Skip to main content
  • EN

Hi,

I’d be interested in getting feedback from people that are already using HPE Apollo’s or other high density servers with local storage as repository server. It seems that a lot of people are currently planning to use them.

  • what is you setup?
  • Number of servers? Number of disks?
  • Which type and size of RAID?
  • Number of VM’s / source data
  • how is your performance?
  • do you use them only as backup target or as copy source as well?
  • which filesystem ReFS/XFS with blockcloning/reflink

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

I think, you can sleep without worries :sunglasses:

Do you see average latency during tests?

 

For the last test:      lat (msec): min=4, max=434, avg=79.64, stdev=37.26

Oh, average is quite high.

You may can reduce it by slightly decreasing queue depth. Probably performance will not change that much if reduction is small. I would try --iodepth=16

 

Similar throughput but avg latency below 50ms then.

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

I think, you can sleep without worries :sunglasses:

Do you see average latency during tests?

 

For the last test:      lat (msec): min=4, max=434, avg=79.64, stdev=37.26

Oh, average is quite high.

You may can reduce it by slightly decreasing queue depth. Probably performance will not change that much if reduction is small. I would try --iodepth=16

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

I think, you can sleep without worries :sunglasses:

Do you see average latency during tests?

 

For the last test:      lat (msec): min=4, max=434, avg=79.64, stdev=37.26

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

I think, you can sleep without worries :sunglasses:

Do you see average latency during tests?

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

I looked into io scheduler and wanted to switch to noop, but it’s not available in RHEL 8. I did not touch io-queue depth yet. 

# cat /sys/block/sdf/queue/nr_requests
1013
 

Should be FIO parameter: 

--iodepth=

I looked into io scheduler and wanted to switch to noop, but it’s not available in RHEL 8. I did not touch io-queue depth yet. 

# cat /sys/block/sdf/queue/nr_requests
1013
 

I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of  3GB/s.

Which screenshot do you mean?

 

The link that was in my earlier post https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqQarUzZWJ → screenshot 4 → https://community.hpe.com/t5/image/serverpage/image-id/122029i134C52F696347813/image-size/large?v=v2&px=2000

Thanks, now I see your point: Purple and azure lines. 

Did you try to rise io-queue depth in tests as well? 

 

I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of  3GB/s.

Which screenshot do you mean?

 

The link that was in my earlier post https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqQarUzZWJ → screenshot 4 → https://community.hpe.com/t5/image/serverpage/image-id/122029i134C52F696347813/image-size/large?v=v2&px=2000

I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of  3GB/s.

Which screenshot do you mean?

I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of  3GB/s.

I’m not sure, I guess those are good results. Looking at https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqHi7UzZWI both controllers should be able to write with ~3GB/s. So I’m missing some hundred MB/s. But overall performance is quiet good, especially with the low price of such a box.

Yes, but you have half the amount of disks! This beast operates 58 disks in 2 Raid60 arrays.

I’m not sure, I guess those are good results. Looking at https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqHi7UzZWI both controllers should be able to write with ~3GB/s. So I’m missing some hundred MB/s. But overall performance is quiet good, especially with the low price of such a box.

I would say, quite good performance!

Is this what you expected?

70/30 mix

 

fio --rw=readwrite --name=test --size=100G --direct=1 --bs=512k --numjobs=20 --rwmixread=70
...

Run status group 0 (all jobs):
   READ: bw=1562MiB/s (1638MB/s), 77.0MiB/s-352MiB/s (81.8MB/s-369MB/s), io=1400GiB (1503GB), run=203566-917518msec
  WRITE: bw=670MiB/s (703MB/s), 33.6MiB/s-151MiB/s (35.2MB/s-158MB/s), io=600GiB (645GB), run=203566-917518msec

Disk stats (read/write):
  sdb: ios=2866217/1229705, merge=159/59, ticks=14433112/258465, in_queue=13229596, util=100.00%
 

 

 

Thanks! 

Very interesting! I got <300MB/sec with Windows and IOmeter with one worker reading.

Did you try a single-job (--numjobs=1) test? For read, I think, this is also interesting. 

 

Here are numbers for write and read with one task.

 

# fio --rw=read --name=test --size=100G --direct=1 --bs=512k --numjobs=1
test: (g=0): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): 1R(1)](100.0%].r=2052MiB/s]Br=4104 IOPS]Oeta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=110268: Tue May 11 14:34:11 2021
  read: IOPS=4493, BW=2247MiB/s (2356MB/s)(100GiB/45572msec)
    clat (usec): min=126, max=87993, avg=221.41, stdev=921.48
     lat (usec): min=126, max=87994, avg=221.57, stdev=921.49
    clat percentiles (usec):
     |  1.00th=  128],  5.00th=  129], 10.00th==  130], 20.00th=h  133],
     | 30.00th=<  143], 40.00th=0  163], 50.00th=4  167], 60.00th=  169],
     | 70.00th=  169], 80.00th=   172], 90.00th=   172], 95.00th=  176],
     | 99.00th=0  502], 99.50th= 4752], 99.90th=015008], 99.95th=518744],
     | 99.99th=,26346]
   bw (  MiB/s): min=  906, max= 2671, per=100.00%, avg=2256.48, stdev=334.99, samples=90
   iops        : min= 1812, max= 5342, avg=4512.97, stdev=669.99, samples=90
  lat (usec)   : 250=98.46%, 500=0.54%, 750=0.03%, 1000=0.03%
  lat (msec)   : 2=0.10%, 4=0.23%, 10=0.42%, 20=0.14%, 50=0.04%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.82%, sys=11.81%, ctx=204802, majf=0, minf=141
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=204800,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2247MiB/s (2356MB/s), 2247MiB/s-2247MiB/s (2356MB/s-2356MB/s), io=100GiB (107GB), run=45572-45572msec

Disk stats (read/write):
  sdb: ios=204355/0, merge=12/0, ticks=40212/0, in_queue=12267, util=99.95%
 

 

 

# fio --rw=write --name=test --size=100G --direct=1 --bs=512k --numjobs=1
test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): KW(1)]o100.0%]pw=2330MiB/s]hw=4659 IOPS]3eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=110600: Tue May 11 14:36:35 2021
  write: IOPS=4684, BW=2342MiB/s (2456MB/s)(100GiB/43718msec); 0 zone resets
    clat (usec): min=131, max=3000, avg=200.27, stdev=49.15
     lat (usec): min=136, max=3016, avg=212.75, stdev=50.33
    clat percentiles (usec):
     |  1.00th=n  137],  5.00th=a  139], 10.00th==  153], 20.00th=  169],
     | 30.00th=/  174], 40.00th=0  180], 50.00th=.  192], 60.00th=0  202],
     | 70.00th=1  212], 80.00th=  227], 90.00th=7  253], 95.00th=1  289],
     | 99.00th=t  375], 99.50th=>  416], 99.90th=h  529], 99.95th=t  570],
     | 99.99th=,  816]
   bw (  MiB/s): min= 2268, max= 2598, per=100.00%, avg=2346.06, stdev=80.44, samples=87
   iops        : min= 4536, max= 5196, avg=4692.13, stdev=160.85, samples=87
  lat (usec)   : 250=89.58%, 500=10.27%, 750=0.14%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=6.12%, sys=9.37%, ctx=204800, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2342MiB/s (2456MB/s), 2342MiB/s-2342MiB/s (2456MB/s-2456MB/s), io=100GiB (107GB), run=43718-43718msec

Disk stats (read/write):
  sdb: ios=0/204799, merge=0/12, ticks=0/37238, in_queue=8, util=99.94%
 

Did you try a single-job (--numjobs=1) test? For read, I think, this is also interesting. 

For RW and 20 tasks I see ~75MB/s, for Write-Only and Read-Only with 20 tasks ~100MB/s,

 

Read-Only:
 

test: (g=0): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
...
fio-3.19
Starting 20 processes
Jobs: 1 (f=1): )_(10),R(1),_(9)]9100.0%]0r=79.9MiB/s]/r=159 IOPS]Peta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=102943: Tue May 11 13:22:36 2021
  read: IOPS=193, BW=96.9MiB/s (102MB/s)(100GiB/1057228msec)
    clat (usec): min=113, max=123336, avg=5159.42, stdev=8535.84
     lat (usec): min=113, max=123336, avg=5159.74, stdev=8535.85
    clat percentiles (usec):
     |  1.00th=   117],  5.00th=  161], 10.00th=[  408], 20.00th==  453],
     | 30.00th=b  482], 40.00th=.  510], 50.00th=0  570], 60.00th=5  709],
     | 70.00th=  5407], 80.00th= 9896], 90.00th=016712], 95.00th=922676],
     | 99.00th==37487], 99.50th= 44827], 99.90th=360556], 99.95th=467634],
     | 99.99th=586508]
   bw (  KiB/s): min=44346, max=801792, per=5.58%, avg=98929.95, stdev=52916.18, samples=2110
   iops        : min=   86, max= 1566, avg=193.12, stdev=103.36, samples=2110
  lat (usec)   : 250=8.51%, 500=28.60%, 750=24.22%, 1000=2.82%
  lat (msec)   : 2=0.92%, 4=2.31%, 10=12.73%, 20=13.11%, 50=6.47%
  lat (msec)   : 100=0.29%, 250=0.01%
  cpu          : usr=0.09%, sys=0.68%, ctx=204804, majf=0, minf=143
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=204800,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
test: (groupid=0, jobs=1): err= 0: pid=102944: Tue May 11 13:22:36 2021
  read: IOPS=194, BW=97.2MiB/s (102MB/s)(100GiB/1053277msec)
    clat (usec): min=113, max=122927, avg=5140.17, stdev=8549.27
     lat (usec): min=113, max=122927, avg=5140.47, stdev=8549.27
    clat percentiles (usec):
     |  1.00th=,  117],  5.00th=d  153], 10.00th=  330], 20.00th=t  453],
     | 30.00th=  482], 40.00th=,  510], 50.00th=3  570], 60.00th=3  693],
     | 70.00th=  5276], 80.00th=[10028], 90.00th==16909], 95.00th=h22676],
     | 99.00th=<37487], 99.50th=044827], 99.90th=061080], 99.95th=068682],
     | 99.99th=682314]
   bw (  KiB/s): min=49152, max=758291, per=5.62%, avg=99605.59, stdev=60406.04, samples=2102
   iops        : min=   96, max= 1481, avg=194.44, stdev=117.99, samples=2102
  lat (usec)   : 250=9.65%, 500=27.67%, 750=24.49%, 1000=2.77%
  lat (msec)   : 2=0.92%, 4=2.18%, 10=12.40%, 20=13.16%, 50=6.45%
  lat (msec)   : 100=0.31%, 250=0.01%
  cpu          : usr=0.09%, sys=0.68%, ctx=204807, majf=0, minf=143
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=204800,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
…

...

Run status group 0 (all jobs):
   READ: bw=1730MiB/s (1814MB/s), 86.5MiB/s-538MiB/s (90.7MB/s-564MB/s), io=2000GiB (2147GB), run=190303-1183552msec

Disk stats (read/write):
  sdb: ios=4095969/6, merge=239/0, ticks=16619508/0, in_queue=14989359, util=100.00%
 

 

Did I understand correctly: each task wrote with about 100MB/sec?

Did you also do read-tests? Or did I miss them in the amount of text?

fio write only test with 20 tasks

 

I’m not sure about the block size but I think it’s 512K in Veeam. 

 

lroot@sdeu2000 sdeu2000_veeam01]# fio --rw=write --name=test --size=100G --direct=1 --bs=512k --numjobs=20
test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
...
fio-3.19
Starting 20 processes
Jobs: 19 (f=19): f_(1),W(19)],99.9%][w=1934MiB/s]4w=3868 IOPS]8eta 00m:01s]
test: (groupid=0, jobs=1): err= 0: pid=98126: Tue May 11 12:23:30 2021
  write: IOPS=200, BW=100MiB/s (105MB/s)(100GiB/1021525msec); 0 zone resets
    clat (usec): min=139, max=143292, avg=4969.42, stdev=2871.79
     lat (usec): min=147, max=143310, avg=4984.00, stdev=2871.79
    clat percentiles (usec):
     |  1.00th=  1893],  5.00th==  2311], 10.00th=t  2671], 20.00th=0  3261],
     | 30.00th=]  3720], 40.00th=  4047], 50.00th=,  4359], 60.00th=]  4752],
     | 70.00th=  5276], 80.00th=  6325], 90.00th=  8029], 95.00th=  9503],
     | 99.00th=. 13173], 99.50th=r 14615], 99.90th=0 17957], 99.95th=5 20055],
     | 99.99th=,135267]
   bw (  KiB/s): min=75776, max=230400, per=5.01%, avg=102812.49, stdev=7472.14, samples=2039
   iops        : min=  148, max=  450, avg=200.78, stdev=14.60, samples=2039
  lat (usec)   : 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=1.61%, 4=36.95%, 10=57.32%, 20=4.02%, 50=0.03%
  lat (msec)   : 100=0.01%, 250=0.02%
  cpu          : usr=0.40%, sys=0.85%, ctx=204806, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
test: (groupid=0, jobs=1): err= 0: pid=98127: Tue May 11 12:23:30 2021
  write: IOPS=200, BW=100MiB/s (105MB/s)(100GiB/1022210msec); 0 zone resets
    clat (usec): min=303, max=143285, avg=4969.88, stdev=2876.83
     lat (usec): min=316, max=143305, avg=4986.58, stdev=2876.84
    clat percentiles (usec):
     |  1.00th==  1893],  5.00th=5  2311], 10.00th=b  2638], 20.00th=e  3261],
     | 30.00th=  3720], 40.00th=8  4047], 50.00th=   4359], 60.00th=  4752],
     | 70.00th=/  5276], 80.00th=t  6325], 90.00th=0  8029], 95.00th=0  9634],
     | 99.00th=2 13173], 99.50th=| 14615], 99.90th=] 18220], 99.95th=5 20055],
     | 99.99th=[133694]
   bw (  KiB/s): min=74752, max=205606, per=5.01%, avg=102716.82, stdev=6928.77, samples=2040
   iops        : min=  146, max=  401, avg=200.59, stdev=13.53, samples=2040
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=1.65%, 4=36.89%, 10=57.32%, 20=4.07%, 50=0.03%
  lat (msec)   : 100=0.01%, 250=0.02%
  cpu          : usr=0.43%, sys=0.99%, ctx=204805, majf=0, minf=146
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
….

….

….


Run status group 0 (all jobs):
  WRITE: bw=2003MiB/s (2101MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=2000GiB (2147GB), run=1021525-1022299msec

Disk stats (read/write):
  sdb: ios=1/4095953, merge=0/239, ticks=0/20170621, in_queue=18120249, util=100.00%
 

 

Veeam Block size: 

By default Veeam’s block size is set to Local Target, which is 1 MB before compression. Since compression ratio is very often around 2x, with this block size Veeam will write around 512 KB or less to the repository per Veeam block.

fio write only test with 20 tasks

 

I’m not sure about the block size but I think it’s 512K in Veeam. 

 

lroot@sdeu2000 sdeu2000_veeam01]# fio --rw=write --name=test --size=100G --direct=1 --bs=512k --numjobs=20
test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
...
fio-3.19
Starting 20 processes
Jobs: 19 (f=19): f_(1),W(19)],99.9%][w=1934MiB/s]4w=3868 IOPS]8eta 00m:01s]
test: (groupid=0, jobs=1): err= 0: pid=98126: Tue May 11 12:23:30 2021
  write: IOPS=200, BW=100MiB/s (105MB/s)(100GiB/1021525msec); 0 zone resets
    clat (usec): min=139, max=143292, avg=4969.42, stdev=2871.79
     lat (usec): min=147, max=143310, avg=4984.00, stdev=2871.79
    clat percentiles (usec):
     |  1.00th=  1893],  5.00th==  2311], 10.00th=t  2671], 20.00th=0  3261],
     | 30.00th=1  3720], 40.00th=  4047], 50.00th=,  4359], 60.00th=]  4752],
     | 70.00th=[  5276], 80.00th=  6325], 90.00th=  8029], 95.00th=  9503],
     | 99.00th=5 13173], 99.50th=b 14615], 99.90th=0 17957], 99.95th=5 20055],
     | 99.99th=]135267]
   bw (  KiB/s): min=75776, max=230400, per=5.01%, avg=102812.49, stdev=7472.14, samples=2039
   iops        : min=  148, max=  450, avg=200.78, stdev=14.60, samples=2039
  lat (usec)   : 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=1.61%, 4=36.95%, 10=57.32%, 20=4.02%, 50=0.03%
  lat (msec)   : 100=0.01%, 250=0.02%
  cpu          : usr=0.40%, sys=0.85%, ctx=204806, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
test: (groupid=0, jobs=1): err= 0: pid=98127: Tue May 11 12:23:30 2021
  write: IOPS=200, BW=100MiB/s (105MB/s)(100GiB/1022210msec); 0 zone resets
    clat (usec): min=303, max=143285, avg=4969.88, stdev=2876.83
     lat (usec): min=316, max=143305, avg=4986.58, stdev=2876.84
    clat percentiles (usec):
     |  1.00th=a  1893],  5.00th=6  2311], 10.00th=4  2638], 20.00th=p  3261],
     | 30.00th=  3720], 40.00th=1  4047], 50.00th=  4359], 60.00th=  4752],
     | 70.00th=r  5276], 80.00th=0  6325], 90.00th=0  8029], 95.00th=.  9634],
     | 99.00th=7 13173], 99.50th=  14615], 99.90th=6 18220], 99.95th=2 20055],
     | 99.99th=h133694]
   bw (  KiB/s): min=74752, max=205606, per=5.01%, avg=102716.82, stdev=6928.77, samples=2040
   iops        : min=  146, max=  401, avg=200.59, stdev=13.53, samples=2040
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=1.65%, 4=36.89%, 10=57.32%, 20=4.07%, 50=0.03%
  lat (msec)   : 100=0.01%, 250=0.02%
  cpu          : usr=0.43%, sys=0.99%, ctx=204805, majf=0, minf=146
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
….

….

….


Run status group 0 (all jobs):
  WRITE: bw=2003MiB/s (2101MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=2000GiB (2147GB), run=1021525-1022299msec

Disk stats (read/write):
  sdb: ios=1/4095953, merge=0/239, ticks=0/20170621, in_queue=18120249, util=100.00%
 

 

[this is frustrating, I just added a post with some benchmark results and ended up with nothing but a warning that it exceeds the max number of characters….]

 

Here are some initial numbers for one 28 x 16TB RAID 60 xfs volume.

I added the options su and sw to mkfs, I think they should be right, don’t know why mkfs.xfs is complaining.

eroot@sdeu2000 ~]# mkfs.xfs -b size=4096 -m reflink=1,crc=1 -d su=256k,sw=24 /dev/sdb -f
mkfs.xfs: Specified data stripe width 12288 is not the same as the volume stripe width 6144
meta-data=/dev/sdf               isize=512    agcount=350, agsize=268435392 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=93755080704, imaxpct=1
         =                       sunit=64     swidth=1536 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
 

 

read write test with 20 parallel tasks:

 root@sdeu2000 sdeu2000_veeam01]# fio --rw=readwrite --name=test --size=100G --direct=1 --bs=512k --numjobs=20
test: (g=0): rw=rw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
...
fio-3.19
Starting 20 processes
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
test: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): lM(1),_(19)] 99.7%]Mr=966MiB/s,w=938MiB/s]or=1932,w=1875 IOPS]/eta 00m:02s]
test: (groupid=0, jobs=1): err= 0: pid=94568: Tue May 11 11:36:10 2021
  read: IOPS=152, BW=76.4MiB/s (80.2MB/s)(50.0GiB/670315msec)
    clat (usec): min=127, max=333675, avg=5907.59, stdev=14202.00
     lat (usec): min=128, max=333675, avg=5907.83, stdev=14202.03
    clat percentiles (usec):
     |  1.00th=5   133],  5.00th=0   172], 10.00th=    184], 20.00th=2   249],
     | 30.00th=t   375], 40.00th=>   537], 50.00th=t   644], 60.00th=>   873],
     | 70.00th=  1680], 80.00th=]  8160], 90.00th=4 17171], 95.00th=4 29754],
     | 99.00th= 71828], 99.50th= 91751], 99.90th=[135267], 99.95th==154141],
     | 99.99th=0208667]
   bw (  KiB/s): min= 7168, max=1016832, per=4.97%, avg=77772.72, stdev=115127.85, samples=1337
   iops        : min=   14, max= 1986, avg=151.77, stdev=224.88, samples=1337
  write: IOPS=152, BW=76.3MiB/s (80.0MB/s)(49.0GiB/670315msec); 0 zone resets
    clat (usec): min=131, max=249424, avg=604.91, stdev=3200.78
     lat (usec): min=135, max=249463, avg=630.75, stdev=3201.17
    clat percentiles (usec):
     |  1.00th=b   143],  5.00th=s   165], 10.00th==   178], 20.00th=1   202],
     | 30.00th=l   229], 40.00th=5   258], 50.00th==   293], 60.00th=1   338],
     | 70.00th=e   408], 80.00th=   529], 90.00th=   758], 95.00th==  1106],
     | 99.00th=  3916], 99.50th=, 10683], 99.90th=0 44827], 99.95th=4 65799],
     | 99.99th= 129500]

….

….
Run status group 0 (all jobs):
   READ: bw=1527MiB/s (1602MB/s), 76.4MiB/s-348MiB/s (80.2MB/s-365MB/s), io=1000GiB (1074GB), run=146941-670315msec
  WRITE: bw=1528MiB/s (1602MB/s), 76.3MiB/s-349MiB/s (80.0MB/s-366MB/s), io=1000GiB (1074GB), run=146941-670315msec

Disk stats (read/write):
  sdb: ios=2047186/2048082, merge=119/120, ticks=7863530/1349428, in_queue=7637829, util=100.00%
 

 

 

Any wishes/hints for testing the xfs filesystem? Fio benchmark? Which options?

I would be interested in the increase of performance respectively bandwidth with every new worker. I did some IOMeter testing with - Windows based - Apollos. There I saw significant increasement with each worker. A single worker was slower than expected.

Any wishes/hints for testing the xfs filesystem? Fio benchmark? Which options?

Comment


Terms & Conditions