I think, you can sleep without worries 

Do you see average latency during tests?

No more tests already migrated most of the jobs. I had some problems with sudo/ssh, some jobs failed a lot of data was left in /opt/veeam (configured in registry) and in the Veeam user home, /var is growing rapidly. At one point Veeam could not perform any task on one server as the Veeam user was not able to connect to his nfs home anymore (>100 defunct processes). After disabling sudo and using the root user this did not happen again, but it’s not so nice needing root for this to work properly.

As we are still on v10 this might be much better in v11 with persistent data mover. All of those are no Apollo issues, it’s Veeam + Linux. Performance in real world is still pretty good, I’m just struggling with the available repository task slots as 52 cores per server are be a bit too little with backup + copy + offload tasks. Those 2 Apollos were initially bought only as copy target, but - as usual - mission has changed and they are now also used for backup. We’ll have 2-4 additional Apollos shortly, then the task slot issue should also be solved.

I looked into io scheduler and wanted to switch to noop, but it’s not available in RHEL 8. I did not touch io-queue depth yet. 

# cat /sys/block/sdf/queue/nr_requests
1013
 

For RW and 20 tasks I see ~75MB/s, for Write-Only and Read-Only with 20 tasks ~100MB/s,

Read-Only:
 

can’t wait to see the results!

@Ralf , I would like to hear what your performance is with single VMDK-VM restore. Did you test this scenario? 

Any wishes/hints for testing the xfs filesystem? Fio benchmark? Which options?

I would be interested in the increase of performance respectively bandwidth with every new worker. I did some IOMeter testing with - Windows based - Apollos. There I saw significant increasement with each worker. A single worker was slower than expected.

fio write only test with 20 tasks

I’m not sure about the block size but I think it’s 512K in Veeam. 

Veeam Block size: 

By default Veeam’s block size is set to Local Target, which is 1 MB before compression. Since compression ratio is very often around 2x, with this block size Veeam will write around 512 KB or less to the repository per Veeam block.

Did I understand correctly: each task wrote with about 100MB/sec?

Did you also do read-tests? Or did I miss them in the amount of text?

Did you try a single-job (--numjobs=1) test? For read, I think, this is also interesting. 

Here are numbers for write and read with one task.

# fio --rw=read --name=test --size=100G --direct=1 --bs=512k --numjobs=1
test: (g=0): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): 1R(1)](100.0%].r=2052MiB/s]Br=4104 IOPS]Oeta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=110268: Tue May 11 14:34:11 2021
  read: IOPS=4493, BW=2247MiB/s (2356MB/s)(100GiB/45572msec)
    clat (usec): min=126, max=87993, avg=221.41, stdev=921.48
     lat (usec): min=126, max=87994, avg=221.57, stdev=921.49
    clat percentiles (usec):
     |  1.00th=�  128],  5.00th=  129], 10.00th==  130], 20.00th=h  133],
     | 30.00th=<  143], 40.00th=0  163], 50.00th=4  167], 60.00th=  169],
     | 70.00th=�  169], 80.00th=�  172], 90.00th=�  172], 95.00th=�  176],
     | 99.00th=0  502], 99.50th= 4752], 99.90th=015008], 99.95th=518744],
     | 99.99th=,26346]
   bw (  MiB/s): min=  906, max= 2671, per=100.00%, avg=2256.48, stdev=334.99, samples=90
   iops        : min= 1812, max= 5342, avg=4512.97, stdev=669.99, samples=90
  lat (usec)   : 250=98.46%, 500=0.54%, 750=0.03%, 1000=0.03%
  lat (msec)   : 2=0.10%, 4=0.23%, 10=0.42%, 20=0.14%, 50=0.04%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.82%, sys=11.81%, ctx=204802, majf=0, minf=141
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=204800,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2247MiB/s (2356MB/s), 2247MiB/s-2247MiB/s (2356MB/s-2356MB/s), io=100GiB (107GB), run=45572-45572msec

Disk stats (read/write):
  sdb: ios=204355/0, merge=12/0, ticks=40212/0, in_queue=12267, util=99.95%
 

# fio --rw=write --name=test --size=100G --direct=1 --bs=512k --numjobs=1
test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): KW(1)]o100.0%]pw=2330MiB/s]hw=4659 IOPS]3eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=110600: Tue May 11 14:36:35 2021
  write: IOPS=4684, BW=2342MiB/s (2456MB/s)(100GiB/43718msec); 0 zone resets
    clat (usec): min=131, max=3000, avg=200.27, stdev=49.15
     lat (usec): min=136, max=3016, avg=212.75, stdev=50.33
    clat percentiles (usec):
     |  1.00th=n  137],  5.00th=a  139], 10.00th==  153], 20.00th=  169],
     | 30.00th=/  174], 40.00th=0  180], 50.00th=.  192], 60.00th=0  202],
     | 70.00th=1  212], 80.00th=�  227], 90.00th=7  253], 95.00th=1  289],
     | 99.00th=t  375], 99.50th=>  416], 99.90th=h  529], 99.95th=t  570],
     | 99.99th=,  816]
   bw (  MiB/s): min= 2268, max= 2598, per=100.00%, avg=2346.06, stdev=80.44, samples=87
   iops        : min= 4536, max= 5196, avg=4692.13, stdev=160.85, samples=87
  lat (usec)   : 250=89.58%, 500=10.27%, 750=0.14%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=6.12%, sys=9.37%, ctx=204800, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2342MiB/s (2456MB/s), 2342MiB/s-2342MiB/s (2456MB/s-2456MB/s), io=100GiB (107GB), run=43718-43718msec

Disk stats (read/write):
  sdb: ios=0/204799, merge=0/12, ticks=0/37238, in_queue=8, util=99.94%
 

Thanks! 

Very interesting! I got <300MB/sec with Windows and IOmeter with one worker reading.

I would say, quite good performance!

Is this what you expected?

Hey @Ralf , when you said var is growing rapidly ? What is about? Logs? Cache?  How many mb/gb per days/jobs?

I’m wonderiing, how did you mount the xfs partition? Are you using lvm?

after all i’m now member of apollo gang . Don’t forget like me to configure cache of the controller, by default it was 100% to read. The difference was HUGE. Interesting fact my processing rate quadruplet compared to my old dedup appliance on same backup scenario. I will continue my test to a specific restoration scenario to compare read performances on backup repo.

fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=16 --refill_buffers --group_reporting

test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=16

...

fio-3.19

Starting 20 processes

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

test: Laying out IO file (1 file / 51200MiB)

Jobs: 11 (f=10): 1f(1),W(1),_(1),W(1),_(1),W(1),_(1),W(2),_(4),W(1),_(2),W(4)]W99.5%]9w=2398MiB/s]iw=4795 IOPS]Ieta 00m:02s]

test: (groupid=0, jobs=20): err= 0: pid=276626: Fri Feb 18 12:28:06 2022

  write: IOPS=5030, BW=2515MiB/s (2637MB/s)(1000GiB/407115msec); 0 zone resets

    slat (usec): min=5, max=204138, avg=2829.68, stdev=8357.51

    clat (usec): min=1612, max=905853, avg=60461.38, stdev=32858.38

     lat (usec): min=1621, max=914383, avg=63291.66, stdev=33526.12

    clat percentiles (msec):

     |  1.00th=>   35],  5.00th=1   39], 10.00th=�   41], 20.00th=9   43],

     | 30.00th==   45], 40.00th=>   47], 50.00th=t   50], 60.00th=0   53],

     | 70.00th=5   57], 80.00th=�   65], 90.00th=�   90], 95.00th=�  155],

     | 99.00th=.  186], 99.50th=9  197], 99.90th=<  215], 99.95th=   224],

     | 99.99th=   241]

   bw (  MiB/s): min= 1528, max= 4374, per=100.00%, avg=2521.54, stdev=17.08, samples=16126

   iops        : min= 3055, max= 8738, avg=5032.83, stdev=34.26, samples=16126

  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.09%, 50=51.84%

  lat (msec)   : 100=39.01%, 250=9.04%, 500=0.01%, 750=0.01%, 1000=0.01%

  cpu          : usr=5.01%, sys=1.05%, ctx=2038785, majf=0, minf=77794

  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%

     issued rwts: total=0,2048000,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=16

 

Run status group 0 (all jobs):

  WRITE: bw=2515MiB/s (2637MB/s), 2515MiB/s-2515MiB/s (2637MB/s-2637MB/s), io=1000GiB (1074GB), run=407115-407115msec

 

Disk stats (read/write):

  sda: ios=0/2046821, merge=0/125, ticks=0/103413090, in_queue=103413090, util=100.00%

For capacity tier offloading we use AWS S3 buckets, we want to look into Wasabi in the next couple of weeks now that they have object lock too.

The servers were deployed from Linux team, they use satellite. No special hardening yet.

For my curiosity what kind of object storage are you using? Example of performance for copy job?

i will deploy on rhel 8 too, we’r using kickstart or satellite provisionning. Do you deploy hardening on you repo? rhel csi?

Just a note: we configured the 4510 servers with the 547FLR-QSFP 40G network adapter. This was not the best idea as the max supported DAC cable lenght is 5m and there is only a MPO transceiver or AOC as alternative. BiDi transceiver like “HPE X140 40G QSFP+ LC BiDi MM” are not supported for this adapter. There is an HPE advisory that certain HPE 40/100 network adapters does not work with BiDi due to a power problem…. As we do not use MPO or AOC in our datacenter, we probably have to replace the adapters now.

Hey Ralf, any news about your test?

Yes, we use LVM by default. But not for the Veeam extents. /var has a size of 25 GB now, 17 GB are used, 12 GB Veeam logs.

xfs mount options (not much tuning):

… type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=6144,noquota)

I would also be interested if you are using LVM. IMHO it is not necessary for repositories.

Yes, I think its session-based land balancing. So with just one stream you will be able to use just one link.

I configured balance-alb which should work for incoming traffic too but I did not try other modes. I see that both interfaces are used, just not with 20Gbit/s. But bonding never gave me full line speed even with multiple senders.