For smaller environments (less than 500 VMs) we use sometimes a DELL PowerEdge R740 XD or XD2.

It is able to handle up to 24 disks with a maximum of - I think - 18TB / disk in the moment.

Up to now we used a ReFS repository with these servers. In the future we will give it a try with a linux installation and an XFS immutable repo…

The server work fine in these environments. No problems up to now.

If you don’t know these links, I would recommend you to read is:

• https://forums.veeam.com/veeam-backup-replication-f2/veeam-v11-numa-awareness-t71923.html#p401059
• https://psnow.ext.hpe.com/doc/a50000150enw?jumpid=in_lit-psnow-red

It’s about a Veeam configuration, Veeam and HPE made missive performance tests with v11. There are a lot of tips and reasons for these.

I know both threads, I’ve already spammed the forum. Point is, I don’t see much real world user experience. HPE claims that those devices are heavily used by Veeam customers. But our Veeam contacts could not provide any (larger) company that uses Apollos.

As I’ll not have much time to test our new setup, the decision SAN + Server vs High Density Server has to be 95% the final setup. If I had more time, I’d just test this in our environment for a couple of weeks.

The benchmarks look very solid, but for us it’s not real world as data will not only be written to the Apollo, its the source for copy and offload jobs. And probably surebackup at some point. 

So… any feedback appreciated! 

i use Dell PowerEdge R740, R730. The performance is Good. 

For how many VM’ s and backup data?

20 VMs

We use a lot of HPE StoreEasy devices for smaller environments including Windows Storage Server : perfect cost-performance balance. For larger environments is a HPE Apollo perfect : more storage available, more performant CPUs availabe, more customizable, more flexible (choose your preferred OS)

We just received our 2 demo Apollo 4510 that hopefully become our new copy target. 130Kg! Let’s see if I can test them with the use cases we have for our primary backup storage too. As we are in a hurry to use them as new copy targets (broken copy chains, we have to completely start over) I’m not so sure about it. Anyhow, I’ll be able to get some first impressions soon.

can’t wait to see the results!

@Ralf , I would like to hear what your performance is with single VMDK-VM restore. Did you test this scenario? 

The Apollos are not deployed, we are currently installing them with RHEL 8.2. It will take a couple of days until I’ll be able to test.

Any wishes/hints for testing the xfs filesystem? Fio benchmark? Which options?

I would be interested in the increase of performance respectively bandwidth with every new worker. I did some IOMeter testing with - Windows based - Apollos. There I saw significant increasement with each worker. A single worker was slower than expected.

[this is frustrating, I just added a post with some benchmark results and ended up with nothing but a warning that it exceeds the max number of characters….]

Here are some initial numbers for one 28 x 16TB RAID 60 xfs volume.

I added the options su and sw to mkfs, I think they should be right, don’t know why mkfs.xfs is complaining.

[root@sdeu2000 ~]# mkfs.xfs -b size=4096 -m reflink=1,crc=1 -d su=256k,sw=24 /dev/sdb -f
mkfs.xfs: Specified data stripe width 12288 is not the same as the volume stripe width 6144
meta-data=/dev/sdf               isize=512    agcount=350, agsize=268435392 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=93755080704, imaxpct=1
         =                       sunit=64     swidth=1536 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
 

read write test with 20 parallel tasks:

….
Run status group 0 (all jobs):
   READ: bw=1527MiB/s (1602MB/s), 76.4MiB/s-348MiB/s (80.2MB/s-365MB/s), io=1000GiB (1074GB), run=146941-670315msec
  WRITE: bw=1528MiB/s (1602MB/s), 76.3MiB/s-349MiB/s (80.0MB/s-366MB/s), io=1000GiB (1074GB), run=146941-670315msec

Disk stats (read/write):
  sdb: ios=2047186/2048082, merge=119/120, ticks=7863530/1349428, in_queue=7637829, util=100.00%
 

fio write only test with 20 tasks

I’m not sure about the block size but I think it’s 512K in Veeam. 

Veeam Block size: 

By default Veeam’s block size is set to Local Target, which is 1 MB before compression. Since compression ratio is very often around 2x, with this block size Veeam will write around 512 KB or less to the repository per Veeam block.

Did I understand correctly: each task wrote with about 100MB/sec?

Did you also do read-tests? Or did I miss them in the amount of text?

For RW and 20 tasks I see ~75MB/s, for Write-Only and Read-Only with 20 tasks ~100MB/s,

Read-Only:
 

Did you try a single-job (--numjobs=1) test? For read, I think, this is also interesting. 

Here are numbers for write and read with one task.

# fio --rw=read --name=test --size=100G --direct=1 --bs=512k --numjobs=1
test: (g=0): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=2052MiB/s][r=4104 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=110268: Tue May 11 14:34:11 2021
  read: IOPS=4493, BW=2247MiB/s (2356MB/s)(100GiB/45572msec)
    clat (usec): min=126, max=87993, avg=221.41, stdev=921.48
     lat (usec): min=126, max=87994, avg=221.57, stdev=921.49
    clat percentiles (usec):
     |  1.00th=[  128],  5.00th=[  129], 10.00th=[  130], 20.00th=[  133],
     | 30.00th=[  143], 40.00th=[  163], 50.00th=[  167], 60.00th=[  169],
     | 70.00th=[  169], 80.00th=[  172], 90.00th=[  172], 95.00th=[  176],
     | 99.00th=[  502], 99.50th=[ 4752], 99.90th=[15008], 99.95th=[18744],
     | 99.99th=[26346]
   bw (  MiB/s): min=  906, max= 2671, per=100.00%, avg=2256.48, stdev=334.99, samples=90
   iops        : min= 1812, max= 5342, avg=4512.97, stdev=669.99, samples=90
  lat (usec)   : 250=98.46%, 500=0.54%, 750=0.03%, 1000=0.03%
  lat (msec)   : 2=0.10%, 4=0.23%, 10=0.42%, 20=0.14%, 50=0.04%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.82%, sys=11.81%, ctx=204802, majf=0, minf=141
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=204800,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2247MiB/s (2356MB/s), 2247MiB/s-2247MiB/s (2356MB/s-2356MB/s), io=100GiB (107GB), run=45572-45572msec

Disk stats (read/write):
  sdb: ios=204355/0, merge=12/0, ticks=40212/0, in_queue=12267, util=99.95%
 

# fio --rw=write --name=test --size=100G --direct=1 --bs=512k --numjobs=1
test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=2330MiB/s][w=4659 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=110600: Tue May 11 14:36:35 2021
  write: IOPS=4684, BW=2342MiB/s (2456MB/s)(100GiB/43718msec); 0 zone resets
    clat (usec): min=131, max=3000, avg=200.27, stdev=49.15
     lat (usec): min=136, max=3016, avg=212.75, stdev=50.33
    clat percentiles (usec):
     |  1.00th=[  137],  5.00th=[  139], 10.00th=[  153], 20.00th=[  169],
     | 30.00th=[  174], 40.00th=[  180], 50.00th=[  192], 60.00th=[  202],
     | 70.00th=[  212], 80.00th=[  227], 90.00th=[  253], 95.00th=[  289],
     | 99.00th=[  375], 99.50th=[  416], 99.90th=[  529], 99.95th=[  570],
     | 99.99th=[  816]
   bw (  MiB/s): min= 2268, max= 2598, per=100.00%, avg=2346.06, stdev=80.44, samples=87
   iops        : min= 4536, max= 5196, avg=4692.13, stdev=160.85, samples=87
  lat (usec)   : 250=89.58%, 500=10.27%, 750=0.14%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=6.12%, sys=9.37%, ctx=204800, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2342MiB/s (2456MB/s), 2342MiB/s-2342MiB/s (2456MB/s-2456MB/s), io=100GiB (107GB), run=43718-43718msec

Disk stats (read/write):
  sdb: ios=0/204799, merge=0/12, ticks=0/37238, in_queue=8, util=99.94%
 

Thanks! 

Very interesting! I got <300MB/sec with Windows and IOmeter with one worker reading.

70/30 mix

fio --rw=readwrite --name=test --size=100G --direct=1 --bs=512k --numjobs=20 --rwmixread=70
...

Run status group 0 (all jobs):
   READ: bw=1562MiB/s (1638MB/s), 77.0MiB/s-352MiB/s (81.8MB/s-369MB/s), io=1400GiB (1503GB), run=203566-917518msec
  WRITE: bw=670MiB/s (703MB/s), 33.6MiB/s-151MiB/s (35.2MB/s-158MB/s), io=600GiB (645GB), run=203566-917518msec

Disk stats (read/write):
  sdb: ios=2866217/1229705, merge=159/59, ticks=14433112/258465, in_queue=13229596, util=100.00%
 

I would say, quite good performance!

Is this what you expected?

I’m not sure, I guess those are good results. Looking at https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqHi7UzZWI both controllers should be able to write with ~3GB/s. So I’m missing some hundred MB/s. But overall performance is quiet good, especially with the low price of such a box.

Yes, but you have half the amount of disks! This beast operates 58 disks in 2 Raid60 arrays.