+1
Hi,
I’d be interested in getting feedback from people that are already using HPE Apollo’s or other high density servers with local storage as repository server. It seems that a lot of people are currently planning to use them.
+1
Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.
# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting
Run status group 0 (all jobs):
WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec
Disk stats (read/write):
sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
I think, you can sleep without worries 
Do you see average latency during tests?
For the last test: lat (msec): min=4, max=434, avg=79.64, stdev=37.26
Oh, average is quite high.
You may can reduce it by slightly decreasing queue depth. Probably performance will not change that much if reduction is small. I would try --iodepth=16
Similar throughput but avg latency below 50ms then.
+13
Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.
# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting
Run status group 0 (all jobs):
WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec
Disk stats (read/write):
sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
I think, you can sleep without worries
Do you see average latency during tests?
For the last test: lat (msec): min=4, max=434, avg=79.64, stdev=37.26
Oh, average is quite high.
You may can reduce it by slightly decreasing queue depth. Probably performance will not change that much if reduction is small. I would try --iodepth=16
+1
Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.
# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting
Run status group 0 (all jobs):
WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec
Disk stats (read/write):
sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
I think, you can sleep without worries
Do you see average latency during tests?
For the last test: lat (msec): min=4, max=434, avg=79.64, stdev=37.26
+13
Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.
# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting
Run status group 0 (all jobs):
WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec
Disk stats (read/write):
sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
I think, you can sleep without worries
Do you see average latency during tests?
+1
Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.
# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting
Run status group 0 (all jobs):
WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec
Disk stats (read/write):
sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
+13
I looked into io scheduler and wanted to switch to noop, but it’s not available in RHEL 8. I did not touch io-queue depth yet.
# cat /sys/block/sdf/queue/nr_requests
1013
Should be FIO parameter:
--iodepth=
+1
I looked into io scheduler and wanted to switch to noop, but it’s not available in RHEL 8. I did not touch io-queue depth yet.
# cat /sys/block/sdf/queue/nr_requests
1013
+13
I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of 3GB/s.
Which screenshot do you mean?
The link that was in my earlier post https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqQarUzZWJ → screenshot 4 → https://community.hpe.com/t5/image/serverpage/image-id/122029i134C52F696347813/image-size/large?v=v2&px=2000
Thanks, now I see your point: Purple and azure lines.
Did you try to rise io-queue depth in tests as well?
+1
I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of 3GB/s.
Which screenshot do you mean?
The link that was in my earlier post https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqQarUzZWJ → screenshot 4 → https://community.hpe.com/t5/image/serverpage/image-id/122029i134C52F696347813/image-size/large?v=v2&px=2000
+13
I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of 3GB/s.
Which screenshot do you mean?
+1
I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of 3GB/s.
+13
I’m not sure, I guess those are good results. Looking at https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqHi7UzZWI both controllers should be able to write with ~3GB/s. So I’m missing some hundred MB/s. But overall performance is quiet good, especially with the low price of such a box.
Yes, but you have half the amount of disks! This beast operates 58 disks in 2 Raid60 arrays.
+1
I’m not sure, I guess those are good results. Looking at https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqHi7UzZWI both controllers should be able to write with ~3GB/s. So I’m missing some hundred MB/s. But overall performance is quiet good, especially with the low price of such a box.
+13
I would say, quite good performance!
Is this what you expected?
+1
70/30 mix
fio --rw=readwrite --name=test --size=100G --direct=1 --bs=512k --numjobs=20 --rwmixread=70
...Run status group 0 (all jobs):
READ: bw=1562MiB/s (1638MB/s), 77.0MiB/s-352MiB/s (81.8MB/s-369MB/s), io=1400GiB (1503GB), run=203566-917518msec
WRITE: bw=670MiB/s (703MB/s), 33.6MiB/s-151MiB/s (35.2MB/s-158MB/s), io=600GiB (645GB), run=203566-917518msecDisk stats (read/write):
sdb: ios=2866217/1229705, merge=159/59, ticks=14433112/258465, in_queue=13229596, util=100.00%
+13
Thanks!
Very interesting! I got <300MB/sec with Windows and IOmeter with one worker reading.
+1
Did you try a single-job (--numjobs=1) test? For read, I think, this is also interesting.
Here are numbers for write and read with one task.
# fio --rw=read --name=test --size=100G --direct=1 --bs=512k --numjobs=1
test: (g=0): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=2052MiB/s][r=4104 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=110268: Tue May 11 14:34:11 2021
read: IOPS=4493, BW=2247MiB/s (2356MB/s)(100GiB/45572msec)
clat (usec): min=126, max=87993, avg=221.41, stdev=921.48
lat (usec): min=126, max=87994, avg=221.57, stdev=921.49
clat percentiles (usec):
| 1.00th=[ 128], 5.00th=[ 129], 10.00th=[ 130], 20.00th=[ 133],
| 30.00th=[ 143], 40.00th=[ 163], 50.00th=[ 167], 60.00th=[ 169],
| 70.00th=[ 169], 80.00th=[ 172], 90.00th=[ 172], 95.00th=[ 176],
| 99.00th=[ 502], 99.50th=[ 4752], 99.90th=[15008], 99.95th=[18744],
| 99.99th=[26346]
bw ( MiB/s): min= 906, max= 2671, per=100.00%, avg=2256.48, stdev=334.99, samples=90
iops : min= 1812, max= 5342, avg=4512.97, stdev=669.99, samples=90
lat (usec) : 250=98.46%, 500=0.54%, 750=0.03%, 1000=0.03%
lat (msec) : 2=0.10%, 4=0.23%, 10=0.42%, 20=0.14%, 50=0.04%
lat (msec) : 100=0.01%
cpu : usr=0.82%, sys=11.81%, ctx=204802, majf=0, minf=141
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=204800,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2247MiB/s (2356MB/s), 2247MiB/s-2247MiB/s (2356MB/s-2356MB/s), io=100GiB (107GB), run=45572-45572msec
Disk stats (read/write):
sdb: ios=204355/0, merge=12/0, ticks=40212/0, in_queue=12267, util=99.95%
# fio --rw=write --name=test --size=100G --direct=1 --bs=512k --numjobs=1
test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=2330MiB/s][w=4659 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=110600: Tue May 11 14:36:35 2021
write: IOPS=4684, BW=2342MiB/s (2456MB/s)(100GiB/43718msec); 0 zone resets
clat (usec): min=131, max=3000, avg=200.27, stdev=49.15
lat (usec): min=136, max=3016, avg=212.75, stdev=50.33
clat percentiles (usec):
| 1.00th=[ 137], 5.00th=[ 139], 10.00th=[ 153], 20.00th=[ 169],
| 30.00th=[ 174], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 202],
| 70.00th=[ 212], 80.00th=[ 227], 90.00th=[ 253], 95.00th=[ 289],
| 99.00th=[ 375], 99.50th=[ 416], 99.90th=[ 529], 99.95th=[ 570],
| 99.99th=[ 816]
bw ( MiB/s): min= 2268, max= 2598, per=100.00%, avg=2346.06, stdev=80.44, samples=87
iops : min= 4536, max= 5196, avg=4692.13, stdev=160.85, samples=87
lat (usec) : 250=89.58%, 500=10.27%, 750=0.14%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=6.12%, sys=9.37%, ctx=204800, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=2342MiB/s (2456MB/s), 2342MiB/s-2342MiB/s (2456MB/s-2456MB/s), io=100GiB (107GB), run=43718-43718msec
Disk stats (read/write):
sdb: ios=0/204799, merge=0/12, ticks=0/37238, in_queue=8, util=99.94%
+13
Did you try a single-job (--numjobs=1) test? For read, I think, this is also interesting.
+1
For RW and 20 tasks I see ~75MB/s, for Write-Only and Read-Only with 20 tasks ~100MB/s,
Read-Only:
[root@sdeu2000 sdeu2000_veeam01]# fio --rw=read --name=test --size=100G --direct=1 --bs=512k --numjobs=20
test: (g=0): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
...
fio-3.19
Starting 20 processes
Jobs: 1 (f=1): [_(10),R(1),_(9)][100.0%][r=79.9MiB/s][r=159 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=102943: Tue May 11 13:22:36 2021
read: IOPS=193, BW=96.9MiB/s (102MB/s)(100GiB/1057228msec)
clat (usec): min=113, max=123336, avg=5159.42, stdev=8535.84
lat (usec): min=113, max=123336, avg=5159.74, stdev=8535.85
clat percentiles (usec):
| 1.00th=[ 117], 5.00th=[ 161], 10.00th=[ 408], 20.00th=[ 453],
| 30.00th=[ 482], 40.00th=[ 510], 50.00th=[ 570], 60.00th=[ 709],
| 70.00th=[ 5407], 80.00th=[ 9896], 90.00th=[16712], 95.00th=[22676],
| 99.00th=[37487], 99.50th=[44827], 99.90th=[60556], 99.95th=[67634],
| 99.99th=[86508]
bw ( KiB/s): min=44346, max=801792, per=5.58%, avg=98929.95, stdev=52916.18, samples=2110
iops : min= 86, max= 1566, avg=193.12, stdev=103.36, samples=2110
lat (usec) : 250=8.51%, 500=28.60%, 750=24.22%, 1000=2.82%
lat (msec) : 2=0.92%, 4=2.31%, 10=12.73%, 20=13.11%, 50=6.47%
lat (msec) : 100=0.29%, 250=0.01%
cpu : usr=0.09%, sys=0.68%, ctx=204804, majf=0, minf=143
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=204800,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
test: (groupid=0, jobs=1): err= 0: pid=102944: Tue May 11 13:22:36 2021
read: IOPS=194, BW=97.2MiB/s (102MB/s)(100GiB/1053277msec)
clat (usec): min=113, max=122927, avg=5140.17, stdev=8549.27
lat (usec): min=113, max=122927, avg=5140.47, stdev=8549.27
clat percentiles (usec):
| 1.00th=[ 117], 5.00th=[ 153], 10.00th=[ 330], 20.00th=[ 453],
| 30.00th=[ 482], 40.00th=[ 510], 50.00th=[ 570], 60.00th=[ 693],
| 70.00th=[ 5276], 80.00th=[10028], 90.00th=[16909], 95.00th=[22676],
| 99.00th=[37487], 99.50th=[44827], 99.90th=[61080], 99.95th=[68682],
| 99.99th=[82314]
bw ( KiB/s): min=49152, max=758291, per=5.62%, avg=99605.59, stdev=60406.04, samples=2102
iops : min= 96, max= 1481, avg=194.44, stdev=117.99, samples=2102
lat (usec) : 250=9.65%, 500=27.67%, 750=24.49%, 1000=2.77%
lat (msec) : 2=0.92%, 4=2.18%, 10=12.40%, 20=13.16%, 50=6.45%
lat (msec) : 100=0.31%, 250=0.01%
cpu : usr=0.09%, sys=0.68%, ctx=204807, majf=0, minf=143
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=204800,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
…
...
Run status group 0 (all jobs):
READ: bw=1730MiB/s (1814MB/s), 86.5MiB/s-538MiB/s (90.7MB/s-564MB/s), io=2000GiB (2147GB), run=190303-1183552msec
Disk stats (read/write):
sdb: ios=4095969/6, merge=239/0, ticks=16619508/0, in_queue=14989359, util=100.00%
+13
Did I understand correctly: each task wrote with about 100MB/sec?
Did you also do read-tests? Or did I miss them in the amount of text?
+13
fio write only test with 20 tasks
I’m not sure about the block size but I think it’s 512K in Veeam.
[root@sdeu2000 sdeu2000_veeam01]# fio --rw=write --name=test --size=100G --direct=1 --bs=512k --numjobs=20
test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
...
fio-3.19
Starting 20 processes
Jobs: 19 (f=19): [_(1),W(19)][99.9%][w=1934MiB/s][w=3868 IOPS][eta 00m:01s]
test: (groupid=0, jobs=1): err= 0: pid=98126: Tue May 11 12:23:30 2021
write: IOPS=200, BW=100MiB/s (105MB/s)(100GiB/1021525msec); 0 zone resets
clat (usec): min=139, max=143292, avg=4969.42, stdev=2871.79
lat (usec): min=147, max=143310, avg=4984.00, stdev=2871.79
clat percentiles (usec):
| 1.00th=[ 1893], 5.00th=[ 2311], 10.00th=[ 2671], 20.00th=[ 3261],
| 30.00th=[ 3720], 40.00th=[ 4047], 50.00th=[ 4359], 60.00th=[ 4752],
| 70.00th=[ 5276], 80.00th=[ 6325], 90.00th=[ 8029], 95.00th=[ 9503],
| 99.00th=[ 13173], 99.50th=[ 14615], 99.90th=[ 17957], 99.95th=[ 20055],
| 99.99th=[135267]
bw ( KiB/s): min=75776, max=230400, per=5.01%, avg=102812.49, stdev=7472.14, samples=2039
iops : min= 148, max= 450, avg=200.78, stdev=14.60, samples=2039
lat (usec) : 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=1.61%, 4=36.95%, 10=57.32%, 20=4.02%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.02%
cpu : usr=0.40%, sys=0.85%, ctx=204806, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
test: (groupid=0, jobs=1): err= 0: pid=98127: Tue May 11 12:23:30 2021
write: IOPS=200, BW=100MiB/s (105MB/s)(100GiB/1022210msec); 0 zone resets
clat (usec): min=303, max=143285, avg=4969.88, stdev=2876.83
lat (usec): min=316, max=143305, avg=4986.58, stdev=2876.84
clat percentiles (usec):
| 1.00th=[ 1893], 5.00th=[ 2311], 10.00th=[ 2638], 20.00th=[ 3261],
| 30.00th=[ 3720], 40.00th=[ 4047], 50.00th=[ 4359], 60.00th=[ 4752],
| 70.00th=[ 5276], 80.00th=[ 6325], 90.00th=[ 8029], 95.00th=[ 9634],
| 99.00th=[ 13173], 99.50th=[ 14615], 99.90th=[ 18220], 99.95th=[ 20055],
| 99.99th=[133694]
bw ( KiB/s): min=74752, max=205606, per=5.01%, avg=102716.82, stdev=6928.77, samples=2040
iops : min= 146, max= 401, avg=200.59, stdev=13.53, samples=2040
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=1.65%, 4=36.89%, 10=57.32%, 20=4.07%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.02%
cpu : usr=0.43%, sys=0.99%, ctx=204805, majf=0, minf=146
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
….
….
….
Run status group 0 (all jobs):
WRITE: bw=2003MiB/s (2101MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=2000GiB (2147GB), run=1021525-1022299msec
Disk stats (read/write):
sdb: ios=1/4095953, merge=0/239, ticks=0/20170621, in_queue=18120249, util=100.00%
Veeam Block size:
By default Veeam’s block size is set to Local Target, which is 1 MB before compression. Since compression ratio is very often around 2x, with this block size Veeam will write around 512 KB or less to the repository per Veeam block.
+1
fio write only test with 20 tasks
I’m not sure about the block size but I think it’s 512K in Veeam.
[root@sdeu2000 sdeu2000_veeam01]# fio --rw=write --name=test --size=100G --direct=1 --bs=512k --numjobs=20
test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
...
fio-3.19
Starting 20 processes
Jobs: 19 (f=19): [_(1),W(19)][99.9%][w=1934MiB/s][w=3868 IOPS][eta 00m:01s]
test: (groupid=0, jobs=1): err= 0: pid=98126: Tue May 11 12:23:30 2021
write: IOPS=200, BW=100MiB/s (105MB/s)(100GiB/1021525msec); 0 zone resets
clat (usec): min=139, max=143292, avg=4969.42, stdev=2871.79
lat (usec): min=147, max=143310, avg=4984.00, stdev=2871.79
clat percentiles (usec):
| 1.00th=[ 1893], 5.00th=[ 2311], 10.00th=[ 2671], 20.00th=[ 3261],
| 30.00th=[ 3720], 40.00th=[ 4047], 50.00th=[ 4359], 60.00th=[ 4752],
| 70.00th=[ 5276], 80.00th=[ 6325], 90.00th=[ 8029], 95.00th=[ 9503],
| 99.00th=[ 13173], 99.50th=[ 14615], 99.90th=[ 17957], 99.95th=[ 20055],
| 99.99th=[135267]
bw ( KiB/s): min=75776, max=230400, per=5.01%, avg=102812.49, stdev=7472.14, samples=2039
iops : min= 148, max= 450, avg=200.78, stdev=14.60, samples=2039
lat (usec) : 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=1.61%, 4=36.95%, 10=57.32%, 20=4.02%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.02%
cpu : usr=0.40%, sys=0.85%, ctx=204806, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
test: (groupid=0, jobs=1): err= 0: pid=98127: Tue May 11 12:23:30 2021
write: IOPS=200, BW=100MiB/s (105MB/s)(100GiB/1022210msec); 0 zone resets
clat (usec): min=303, max=143285, avg=4969.88, stdev=2876.83
lat (usec): min=316, max=143305, avg=4986.58, stdev=2876.84
clat percentiles (usec):
| 1.00th=[ 1893], 5.00th=[ 2311], 10.00th=[ 2638], 20.00th=[ 3261],
| 30.00th=[ 3720], 40.00th=[ 4047], 50.00th=[ 4359], 60.00th=[ 4752],
| 70.00th=[ 5276], 80.00th=[ 6325], 90.00th=[ 8029], 95.00th=[ 9634],
| 99.00th=[ 13173], 99.50th=[ 14615], 99.90th=[ 18220], 99.95th=[ 20055],
| 99.99th=[133694]
bw ( KiB/s): min=74752, max=205606, per=5.01%, avg=102716.82, stdev=6928.77, samples=2040
iops : min= 146, max= 401, avg=200.59, stdev=13.53, samples=2040
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=1.65%, 4=36.89%, 10=57.32%, 20=4.07%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.02%
cpu : usr=0.43%, sys=0.99%, ctx=204805, majf=0, minf=146
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
….
….
….
Run status group 0 (all jobs):
WRITE: bw=2003MiB/s (2101MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=2000GiB (2147GB), run=1021525-1022299msec
Disk stats (read/write):
sdb: ios=1/4095953, merge=0/239, ticks=0/20170621, in_queue=18120249, util=100.00%
+1
[this is frustrating, I just added a post with some benchmark results and ended up with nothing but a warning that it exceeds the max number of characters….]
Here are some initial numbers for one 28 x 16TB RAID 60 xfs volume.
I added the options su and sw to mkfs, I think they should be right, don’t know why mkfs.xfs is complaining.
[root@sdeu2000 ~]# mkfs.xfs -b size=4096 -m reflink=1,crc=1 -d su=256k,sw=24 /dev/sdb -f
mkfs.xfs: Specified data stripe width 12288 is not the same as the volume stripe width 6144
meta-data=/dev/sdf isize=512 agcount=350, agsize=268435392 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=93755080704, imaxpct=1
= sunit=64 swidth=1536 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
read write test with 20 parallel tasks:
[root@sdeu2000 sdeu2000_veeam01]# fio --rw=readwrite --name=test --size=100G --direct=1 --bs=512k --numjobs=20
test: (g=0): rw=rw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
...
fio-3.19
Starting 20 processes
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [M(1),_(19)][99.7%][r=966MiB/s,w=938MiB/s][r=1932,w=1875 IOPS][eta 00m:02s]
test: (groupid=0, jobs=1): err= 0: pid=94568: Tue May 11 11:36:10 2021
read: IOPS=152, BW=76.4MiB/s (80.2MB/s)(50.0GiB/670315msec)
clat (usec): min=127, max=333675, avg=5907.59, stdev=14202.00
lat (usec): min=128, max=333675, avg=5907.83, stdev=14202.03
clat percentiles (usec):
| 1.00th=[ 133], 5.00th=[ 172], 10.00th=[ 184], 20.00th=[ 249],
| 30.00th=[ 375], 40.00th=[ 537], 50.00th=[ 644], 60.00th=[ 873],
| 70.00th=[ 1680], 80.00th=[ 8160], 90.00th=[ 17171], 95.00th=[ 29754],
| 99.00th=[ 71828], 99.50th=[ 91751], 99.90th=[135267], 99.95th=[154141],
| 99.99th=[208667]
bw ( KiB/s): min= 7168, max=1016832, per=4.97%, avg=77772.72, stdev=115127.85, samples=1337
iops : min= 14, max= 1986, avg=151.77, stdev=224.88, samples=1337
write: IOPS=152, BW=76.3MiB/s (80.0MB/s)(49.0GiB/670315msec); 0 zone resets
clat (usec): min=131, max=249424, avg=604.91, stdev=3200.78
lat (usec): min=135, max=249463, avg=630.75, stdev=3201.17
clat percentiles (usec):
| 1.00th=[ 143], 5.00th=[ 165], 10.00th=[ 178], 20.00th=[ 202],
| 30.00th=[ 229], 40.00th=[ 258], 50.00th=[ 293], 60.00th=[ 338],
| 70.00th=[ 408], 80.00th=[ 529], 90.00th=[ 758], 95.00th=[ 1106],
| 99.00th=[ 3916], 99.50th=[ 10683], 99.90th=[ 44827], 99.95th=[ 65799],
| 99.99th=[129500]
….
….
Run status group 0 (all jobs):
READ: bw=1527MiB/s (1602MB/s), 76.4MiB/s-348MiB/s (80.2MB/s-365MB/s), io=1000GiB (1074GB), run=146941-670315msec
WRITE: bw=1528MiB/s (1602MB/s), 76.3MiB/s-349MiB/s (80.0MB/s-366MB/s), io=1000GiB (1074GB), run=146941-670315msec
Disk stats (read/write):
sdb: ios=2047186/2048082, merge=119/120, ticks=7863530/1349428, in_queue=7637829, util=100.00%
+13
Any wishes/hints for testing the xfs filesystem? Fio benchmark? Which options?
I would be interested in the increase of performance respectively bandwidth with every new worker. I did some IOMeter testing with - Windows based - Apollos. There I saw significant increasement with each worker. A single worker was slower than expected.
+1
Any wishes/hints for testing the xfs filesystem? Fio benchmark? Which options?
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
