Experience with HPE Apollo or other High Density server as backup repository?


Userlevel 6
Badge +1
  • Comes here often
  • 71 comments

Hi,

I’d be interested in getting feedback from people that are already using HPE Apollo’s or other high density servers with local storage as repository server. It seems that a lot of people are currently planning to use them.

  • what is you setup?
  • Number of servers? Number of disks?
  • Which type and size of RAID?
  • Number of VM’s / source data
  • how is your performance?
  • do you use them only as backup target or as copy source as well?
  • which filesystem ReFS/XFS with blockcloning/reflink

61 comments

Userlevel 6
Badge +1

I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of  3GB/s.

Userlevel 7
Badge +13

I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of  3GB/s.

Which screenshot do you mean?

Userlevel 6
Badge +1

I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of  3GB/s.

Which screenshot do you mean?

 

The link that was in my earlier post https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqQarUzZWJ → screenshot 4 → https://community.hpe.com/t5/image/serverpage/image-id/122029i134C52F696347813/image-size/large?v=v2&px=2000

Userlevel 7
Badge +13

I’ve also 2 RAID 60 with 28 x 16TB. In the reference Apollo it’s a single SOBR with 2 extents (2 RAID volumes). I’m testing one of them. In the screenshot in the link you see that a single volume is performing at max of  3GB/s.

Which screenshot do you mean?

 

The link that was in my earlier post https://community.hpe.com/t5/Around-the-Storage-Block/The-fastest-ever-all-in-one-backup-solution-from-HPE-Storage-and/ba-p/7126052#.YJqQarUzZWJ → screenshot 4 → https://community.hpe.com/t5/image/serverpage/image-id/122029i134C52F696347813/image-size/large?v=v2&px=2000

Thanks, now I see your point: Purple and azure lines. 

Did you try to rise io-queue depth in tests as well? 

 

Userlevel 6
Badge +1

I looked into io scheduler and wanted to switch to noop, but it’s not available in RHEL 8. I did not touch io-queue depth yet. 

# cat /sys/block/sdf/queue/nr_requests
1013
 

Userlevel 7
Badge +13

I looked into io scheduler and wanted to switch to noop, but it’s not available in RHEL 8. I did not touch io-queue depth yet. 

# cat /sys/block/sdf/queue/nr_requests
1013
 

Should be FIO parameter: 

--iodepth=
Userlevel 6
Badge +1

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

Userlevel 7
Badge +13

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

I think, you can sleep without worries :sunglasses:

Do you see average latency during tests?

Userlevel 6
Badge +1

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

I think, you can sleep without worries :sunglasses:

Do you see average latency during tests?

 

For the last test:      lat (msec): min=4, max=434, avg=79.64, stdev=37.26

Userlevel 7
Badge +13

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

I think, you can sleep without worries :sunglasses:

Do you see average latency during tests?

 

For the last test:      lat (msec): min=4, max=434, avg=79.64, stdev=37.26

Oh, average is quite high.

You may can reduce it by slightly decreasing queue depth. Probably performance will not change that much if reduction is small. I would try --iodepth=16

Userlevel 6
Badge +1

Well, depending on the test parameter I can reach 2,5GB/s. I mean… we probably will not be able to saturate this anyway in the near future. I just want to be sure that my RAID and fs setup correct as this is hard to modify once data is on the disk.

 

# fio --rw=write --name=test --size=50G --direct=1 --bs=512k --numjobs=20 --ioengine=libaio --iodepth=20 --refill_buffers --group_reporting

 


Run status group 0 (all jobs):
  WRITE: bw=2463MiB/s (2582MB/s), 2463MiB/s-2463MiB/s (2582MB/s-2582MB/s), io=1000GiB (1074GB), run=415836-415836msec

Disk stats (read/write):
  sdb: ios=0/2047239, merge=0/119, ticks=0/165897764, in_queue=164873511, util=100.00%
 

I think, you can sleep without worries :sunglasses:

Do you see average latency during tests?

 

For the last test:      lat (msec): min=4, max=434, avg=79.64, stdev=37.26

Oh, average is quite high.

You may can reduce it by slightly decreasing queue depth. Probably performance will not change that much if reduction is small. I would try --iodepth=16

 

Similar throughput but avg latency below 50ms then.

Userlevel 7
Badge +13

Sounds good! 50ms on SATA disks I would call “acceptable”. And this is near saturation; as you already said, you will probably will not see such counters in real life.

Userlevel 7
Badge +8
Reading @vNote42 and @ralf exchanges

Thank you guys for all your answers :sunglasses:

Userlevel 7
Badge +13

Thanks are due to @Ralf ! He did all the testing! Thanks also from me!

Userlevel 7
Badge +20

I’ve been watching this silently for a while but just want to say a huge thank you to both @Ralf and @vNote42 for an awesome technical interaction here.

It’s rare that we as a collective can all see real world numbers outside of specific lab simulations and free from marketing so this has been truly awesome to see. I’m still looking at potentially a Dell PowerEdge XE7100 based system so if I can organise a test I’d love to re-run these benchmarks and get a true Dell vs HPE battle going here!

Userlevel 6
Badge +1

Some iozone results: https://pastebin.com/BC2NaHup 

Userlevel 7
Badge +13

I’ve been watching this silently for a while but just want to say a huge thank you to both @Ralf and @vNote42 for an awesome technical interaction here.

It’s rare that we as a collective can all see real world numbers outside of specific lab simulations and free from marketing so this has been truly awesome to see. I’m still looking at potentially a Dell PowerEdge XE7100 based system so if I can organise a test I’d love to re-run these benchmarks and get a true Dell vs HPE battle going here!

@MicoolPaul , let us know when you get your Dell server! We will test it too :sunglasses:

Userlevel 6
Badge +1

Hm… this was during 5% CPU load.

 

# grep temperature /var/log/messages
May 12 19:07:11 sdeu2000 kernel: CPU46: Core temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU98: Core temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU45: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU97: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU98: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU46: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU26: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU78: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU94: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU27: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU85: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU88: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU42: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU91: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU28: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU80: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU33: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU41: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU36: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU93: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU39: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU44: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU51: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU82: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU32: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU87: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU84: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU79: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU103: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU35: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU48: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU96: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU30: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU43: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU81: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU38: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU29: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU50: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU34: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU100: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU95: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU102: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU40: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU92: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU47: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU99: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU90: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU49: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU89: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU101: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU86: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU31: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU37: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU83: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU99: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU81: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU98: Core temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU95: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU46: Core temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU102: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU94: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU89: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU87: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU35: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU29: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU43: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU42: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU32: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU49: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU38: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU90: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU41: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU93: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU44: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU83: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU31: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU91: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU39: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU82: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU30: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU80: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU28: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU51: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU84: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU100: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU48: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU27: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU79: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU36: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU88: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU47: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU45: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU78: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU97: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU37: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU40: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU92: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU26: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU103: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU101: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU50: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU86: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU85: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU33: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU34: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU98: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU46: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU96: Package temperature/speed normal
 

 

Userlevel 7
Badge +13

Hm… this was during 5% CPU load.

 

# grep temperature /var/log/messages
May 12 19:07:11 sdeu2000 kernel: CPU46: Core temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU98: Core temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU45: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU97: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU98: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU46: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU26: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU78: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU94: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU27: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU85: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU88: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU42: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU91: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU28: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU80: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU33: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU41: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU36: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU93: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU39: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU44: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU51: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU82: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU32: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU87: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU84: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU79: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU103: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU35: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU48: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU96: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU30: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU43: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU81: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU38: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU29: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU50: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU34: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU100: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU95: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU102: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU40: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU92: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU47: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU99: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU90: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU49: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU89: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU101: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU86: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU31: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU37: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU83: Package temperature above threshold, cpu clock throttled (total events = 1)
May 12 19:07:11 sdeu2000 kernel: CPU99: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU81: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU98: Core temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU95: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU46: Core temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU102: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU94: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU89: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU87: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU35: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU29: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU43: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU42: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU32: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU49: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU38: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU90: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU41: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU93: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU44: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU83: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU31: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU91: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU39: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU82: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU30: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU80: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU28: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU51: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU84: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU100: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU48: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU27: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU79: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU36: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU88: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU47: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU45: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU78: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU97: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU37: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU40: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU92: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU26: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU103: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU101: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU50: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU86: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU85: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU33: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU34: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU98: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU46: Package temperature/speed normal
May 12 19:07:11 sdeu2000 kernel: CPU96: Package temperature/speed normal
 

 

What's your Workload-Profile respectively Power-Regulator settings?

Userlevel 6
Badge +1

High Performance. But the error cleared the same second it started, I think this was just a hickup.

Userlevel 6
Badge +1

I wanted to post some Veeam benchmarks too, but this doesn’t make much sense as I can see that jobs are limited by network. We have a 40/50 GbE 547FLR network adapter with QSFP → SFP adapter, so this is currently limited by 2x10 GbE bonded linux interfaces. I see ~1,3GB/s and Veeam is showing network or source as bottleneck. Our network equipment will be replaced in the next months, then I will switch to 40GbE. As we are testing with v10, I also can not use the Apollos as proxy to backup directly from storage snapshot. Our existing proxies are sending the data to the Apollo via network.

One advice: make a large /tmp fs is you use a dedicated partition or lvm volume. Veeam is writing a lot to /tmp and my first jobs failed because it was only 4GB.

 

Userlevel 7
Badge +13

I wanted to post some Veeam benchmarks too, but this doesn’t make much sense as I can see that jobs are limited by network. We have a 40/50 GbE 547FLR network adapter with QSFP → SFP adapter, so this is currently limited by 2x10 GbE bonded linux interfaces. I see ~1,3GB/s and Veeam is showing network or source as bottleneck. Our network equipment will be replaced in the next months, then I will switch to 40GbE. As we are testing with v10, I also can not use the Apollos as proxy to backup directly from storage snapshot. Our existing proxies are sending the data to the Apollo via network.

One advice: make a large /tmp fs is you use a dedicated partition or lvm volume. Veeam is writing a lot to /tmp and my first jobs failed because it was only 4GB.

 

Thanks for your advise!

Did you try different nic-bonding policies? I am not the network-guy, but it should work - with correct policy - to use both uplink simultaneously when multiple backup using multiple proxy server to write into your Apollo server. 

 

Userlevel 6
Badge +1

I configured balance-alb which should work for incoming traffic too but I did not try other modes. I see that both interfaces are used, just not with 20Gbit/s. But bonding never gave me full line speed even with multiple senders.

Userlevel 7
Badge +13

Yes, I think its session-based land balancing. So with just one stream you will be able to use just one link.

Userlevel 3
Badge +1

The Apollos are not deployed, we are currently installing them with RHEL 8.2. It will take a couple of days until I’ll be able to test.

 

just curious, why in your case did you choose RHEL 8.2 , instead of (e.g.) Ubuntu?

Comment