随着越来越多的客户都在使用Linux存储库来存放Veeam备份数据，所以很有必要使用一些测试工具来模拟Veeam的常规操作，以评估Linux存储库本身的磁盘读写性能。
本文将介绍使用Fio测试工具来模拟：Active Full、Synthetic Full、Reverse Incremental、Restore的Veeam常规操作时的预期性能。
[root@CentOS04 ~]# yum install fio #安装fio
1、模拟Active Full / Forward Incremental的测试：
fio --name=full-write-test --filename=/backups/testfile.dat --size=25G --bs=512k --rw=write --ioengine=libaio --direct=1 --time_based --runtime=600s测试命令解析：模拟在600秒内、持续以512k的块大小，在/backups(注意这是Linux存储库的实际目录)下1次写入25GB的testfile.dat文件。
如有需要可自行更改测试参数。测试完成后，不会自动删除testfile.dat文件，需手工删除。(不再赘述)
以下是Active Full测试的实验室输出结果，仅供参考。
[root@CentOS04 ~] # fio --name=full-write-test --filename=/backups/testfile.dat --size=1G --bs=512k --rw=write --ioengine=libaio --direct=1 --time_based --runtime=10s
full-write-test: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1
fio-3.19
Starting 1 process
full-write-test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=188MiB/s][w=376 IOPS][eta 00m:00s]
full-write-test: (groupid=0, jobs=1): err= 0: pid=241320: Thu Nov 23 16:13:43 2023
write: IOPS=458, BW=229MiB/s (241MB/s)(2295MiB/10001msec); 0 zone resets
slat (usec): min=15, max=205, avg=60.00, stdev=28.70
clat (usec): min=980, max=99204, avg=2116.01, stdev=2404.43
lat (usec): min=1019, max=99244, avg=2176.30, stdev=2407.28
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1004], 10.00th=[ 1020], 20.00th=[ 1205],
| 30.00th=[ 1450], 40.00th=[ 1713], 50.00th=[ 1975], 60.00th=[ 2245],
| 70.00th=[ 2540], 80.00th=[ 2835], 90.00th=[ 3032], 95.00th=[ 3195],
| 99.00th=[ 3523], 99.50th=[ 4424], 99.90th=[31065], 99.95th=[61080],
| 99.99th=[99091]
bw ( KiB/s): min=177152, max=303104, per=100.00%, avg=236822.74, stdev=32993.20, samples=19
iops : min= 346, max= 592, avg=462.53, stdev=64.43, samples=19
lat (usec) : 1000=3.75%
lat (msec) : 2=47.49%, 4=48.10%, 10=0.41%, 20=0.04%, 50=0.11%
lat (msec) : 100=0.09%
cpu : usr=0.73%, sys=2.37%, ctx=4596, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4590,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=229MiB/s (241MB/s), 229MiB/s-229MiB/s (241MB/s-241MB/s), io=2295MiB (2406MB), run=10001-10001msec
Disk stats (read/write):
dm-2: ios=0/4563, merge=0/0, ticks=0/9616, in_queue=9616, util=98.46%, aggrios=0/4590, aggrmerge=0/0, aggrticks=0/9769, aggrin_queue=7898, aggrutil=98.86%
sdb: ios=0/4590, merge=0/0, ticks=0/9769, in_queue=7898, util=98.86%
[root@CentOS04 ~] #留意以下测试结果：
Run status group 0 (all jobs): WRITE: bw=229MiB/s (241MB/s), 229MiB/s-229MiB/s (241MB/s-241MB/s), io=2295MiB (2406MB), run=10001-10001msec
2、模拟Synthetic Full / Forever Forward Incremental Merge的测试：
-
此测试适用于：Synthetic Full / Forever Forward Incremental Merge操作中涉及的所有还原点都在同一个Linux存储库上。
-
此测试不适用于：Linux存储库上使用FastClone快速克隆、或使用SOBR的Performance模式的横向扩展备份存储库（即完整还原点和增量还原点保存在不同的存储上）。
fio --name=merge-test --filename=/backups/testfile.dat --size=100G --bs=512k --rw=randrw --rwmixwrite=50 --direct=1 --ioengine=libaio --iodepth=4 --runtime=600 --time_based测试命令解析：模拟在600秒内、持续以512k的块大小，在/backups(注意这是Linux存储库的实际目录)下1次写入、1次读取100GB的testfile.dat文件。
以下是Synthetic Full测试的实验室输出结果，仅供参考。
[root@CentOS04 ~] # fio --name=merge-test --filename=/backups/testfile.dat --size=1G --bs=512k --rw=randrw --rwmixwrite=50 --direct=1 --ioengine=libaio --iodepth=4 --runtime=10 --time_based
merge-test: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=4
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=79.4MiB/s,w=97.4MiB/s][r=158,w=194 IOPS][eta 00m:00s]
merge-test: (groupid=0, jobs=1): err= 0: pid=246148: Thu Nov 23 16:25:26 2023
read: IOPS=205, BW=103MiB/s (108MB/s)(1027MiB/10013msec)
slat (usec): min=11, max=170, avg=51.66, stdev=28.81
clat (usec): min=2666, max=32629, avg=7617.71, stdev=2903.06
lat (usec): min=2680, max=32688, avg=7669.76, stdev=2910.06
clat percentiles (usec):
| 1.00th=[ 2999], 5.00th=[ 3720], 10.00th=[ 4178], 20.00th=[ 5211],
| 30.00th=[ 5866], 40.00th=[ 6587], 50.00th=[ 7373], 60.00th=[ 8029],
| 70.00th=[ 8717], 80.00th=[ 9634], 90.00th=[11338], 95.00th=[12649],
| 99.00th=[15533], 99.50th=[18220], 99.90th=[28443], 99.95th=[29492],
| 99.99th=[32637]
bw ( KiB/s): min=77824, max=182009, per=99.01%, avg=103990.68, stdev=27070.77, samples=19
iops : min= 152, max= 355, avg=203.05, stdev=52.78, samples=19
write: IOPS=216, BW=108MiB/s (113MB/s)(1082MiB/10013msec); 0 zone resets
slat (usec): min=17, max=207, avg=69.75, stdev=33.40
clat (usec): min=1901, max=28883, avg=11145.48, stdev=4363.98
lat (usec): min=1933, max=28929, avg=11215.68, stdev=4373.63
clat percentiles (usec):
| 1.00th=[ 3949], 5.00th=[ 5014], 10.00th=[ 5932], 20.00th=[ 7177],
| 30.00th=[ 8291], 40.00th=[ 9503], 50.00th=[10552], 60.00th=[11731],
| 70.00th=[13173], 80.00th=[15008], 90.00th=[17171], 95.00th=[18744],
| 99.00th=[23200], 99.50th=[24511], 99.90th=[27132], 99.95th=[28705],
| 99.99th=[28967]
bw ( KiB/s): min=83968, max=201001, per=98.83%, avg=109354.47, stdev=29050.83, samples=19
iops : min= 164, max= 392, avg=213.53, stdev=56.62, samples=19
lat (msec) : 2=0.02%, 4=4.53%, 10=58.25%, 20=35.32%, 50=1.87%
cpu : usr=0.62%, sys=2.28%, ctx=4221, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2054,2164,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=1027MiB (1077MB), run=10013-10013msec
WRITE: bw=108MiB/s (113MB/s), 108MiB/s-108MiB/s (113MB/s-113MB/s), io=1082MiB (1135MB), run=10013-10013msec
Disk stats (read/write):
dm-2: ios=2029/2149, merge=0/0, ticks=15500/23950, in_queue=39450, util=99.03%, aggrios=2054/2165, aggrmerge=0/0, aggrticks=15696/24174, aggrin_queue=37728, aggrutil=98.90%
sdb: ios=2054/2165, merge=0/0, ticks=15696/24174, in_queue=37728, util=98.90%
[root@CentOS04 ~] #留意以下测试结果：
Run status group 0 (all jobs): READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=1027MiB (1077MB), run=10013-10013msec WRITE: bw=108MiB/s (113MB/s), 108MiB/s-108MiB/s (113MB/s-113MB/s), io=1082MiB (1135MB), run=10013-10013msec
由于Veeam对于每个处理的块进行两次I/O操作（1次写入、1次读取），所以需要把上述结果(103+108)/2=105.5MB/s。
3、模拟Reverse Incremental的测试：
fio --name=reverse-inc-test --filename=/tmp/testfile.dat --size=100G --bs=512k --rw=randrw --rwmixwrite=67 --direct=1 --ioengine=libaio --iodepth=4 --runtime=600 --time_based测试命令解析：模拟在600秒内、持续以512k的块大小，在/backups(注意这是Linux存储库的实际目录)下1次写入、2次读取100GB的testfile.dat文件。
以下是Reverse Incremental测试的实验室输出结果，仅供参考。
[root@CentOS04 ~] # fio --name=reverse-inc-test --filename=/backups/testfile.dat --size=1G --bs=512k --rw=randrw --rwmixwrite=67 --direct=1 --ioengine=libaio --iodepth=4 --runtime=10 --time_based
reverse-inc-test: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=4
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=5114KiB/s,w=16.5MiB/s][r=9,w=32 IOPS][eta 00m:00s]
reverse-inc-test: (groupid=0, jobs=1): err= 0: pid=247457: Thu Nov 23 16:28:43 2023
read: IOPS=19, BW=9.97MiB/s (10.5MB/s)(101MiB/10080msec)
slat (usec): min=12, max=297, avg=41.15, stdev=27.47
clat (msec): min=2, max=146, avg=40.08, stdev=35.17
lat (msec): min=2, max=146, avg=40.12, stdev=35.17
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 7], 50.00th=[ 43], 60.00th=[ 55],
| 70.00th=[ 65], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 88],
| 99.00th=[ 128], 99.50th=[ 136], 99.90th=[ 148], 99.95th=[ 148],
| 99.99th=[ 148]
bw ( KiB/s): min= 1024, max=65293, per=94.37%, avg=9634.37, stdev=13701.85, samples=19
iops : min= 2, max= 127, avg=18.79, stdev=26.64, samples=19
write: IOPS=46, BW=23.1MiB/s (24.2MB/s)(233MiB/10080msec); 0 zone resets
slat (usec): min=21, max=228, avg=58.95, stdev=25.26
clat (msec): min=3, max=178, avg=69.18, stdev=51.12
lat (msec): min=3, max=178, avg=69.24, stdev=51.12
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6],
| 30.00th=[ 8], 40.00th=[ 55], 50.00th=[ 75], 60.00th=[ 104],
| 70.00th=[ 106], 80.00th=[ 122], 90.00th=[ 129], 95.00th=[ 148],
| 99.00th=[ 171], 99.50th=[ 176], 99.90th=[ 180], 99.95th=[ 180],
| 99.99th=[ 180]
bw ( KiB/s): min=14336, max=123421, per=94.59%, avg=22340.89, stdev=24548.96, samples=19
iops : min= 28, max= 241, avg=43.63, stdev=47.93, samples=19
lat (msec) : 4=9.46%, 10=24.47%, 20=0.30%, 50=6.61%, 100=27.78%
lat (msec) : 250=31.38%
cpu : usr=0.12%, sys=0.30%, ctx=666, majf=0, minf=13
IO depths : 1=0.2%, 2=0.3%, 4=99.5%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=201,465,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=9.97MiB/s (10.5MB/s), 9.97MiB/s-9.97MiB/s (10.5MB/s-10.5MB/s), io=101MiB (105MB), run=10080-10080msec
WRITE: bw=23.1MiB/s (24.2MB/s), 23.1MiB/s-23.1MiB/s (24.2MB/s-24.2MB/s), io=233MiB (244MB), run=10080-10080msec
Disk stats (read/write):
dm-2: ios=201/460, merge=0/0, ticks=7945/31428, in_queue=39373, util=98.72%, aggrios=201/465, aggrmerge=0/0, aggrticks=8058/32117, aggrin_queue=39826, aggrutil=98.96%
sdb: ios=201/465, merge=0/0, ticks=8058/32117, in_queue=39826, util=98.96%
[root@CentOS04 ~] #留意以下测试结果：
Run status group 0 (all jobs): READ: bw=9.97MiB/s (10.5MB/s), 9.97MiB/s-9.97MiB/s (10.5MB/s-10.5MB/s), io=101MiB (105MB), run=10080-10080msec WRITE: bw=23.1MiB/s (24.2MB/s), 23.1MiB/s-23.1MiB/s (24.2MB/s-24.2MB/s), io=233MiB (244MB), run=10080-10080msec
由于Veeam对于每个处理的块进行三次I/O操作（1次写入、2次读取），所以需要把上述结果(9.97+23.1)/3=11MB/s。
4、模拟Restore的测试：
Veeam在恢复期间的读取性能，可能会随着备份数据所在磁盘块的碎片级别而变化。以下模拟最大上限和下限的恢复测试，即实际恢复速度会介于两者之间。
零碎片读取测试 （上限）：以执行顺序读取来模拟恢复操作，即模拟正在读取备份数据所在磁盘块没有碎片。
fio --name=seq-read-test --filename=/backups/JobName/VMname.vbk --bs=512k --rw=read --ioengine=libaio --direct=1 --time_based --runtime=600最大碎片读取测试（下限）：以执行随机读取来模拟恢复操作，即模拟正在读取备份数据所在磁盘块存有大量碎片。
fio --name=frag-read-test --filename=/backups/JobName/VMname.vbk --bs=512k --rw=randread --ioengine=libaio --direct=1 --time_based --runtime=600s注意：需要修改以上测试命令的实际filename，即vbk备份文件所在的真实目录。
以下是零碎片读取测试 （上限）测试的实验室输出结果，仅供参考。
[root@CentOS4 ~] # fio --name=seq-read-test --filename=/backups/'VM Backup Job 4_Oracle'/'VM Backup Job 4_OracleD2023-11-11T165948_C88D.vbk' --bs=512k --rw=read --ioengine=libaio --direct=1 --time_based --runtime=10
seq-read-test: (g=0): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=312MiB/s][r=624 IOPS][eta 00m:00s]
seq-read-test: (groupid=0, jobs=1): err= 0: pid=279450: Thu Nov 23 17:47:00 2023
read: IOPS=636, BW=318MiB/s (334MB/s)(3182MiB/10001msec)
slat (usec): min=11, max=918, avg=33.75, stdev=20.64
clat (usec): min=1040, max=5999, avg=1535.47, stdev=172.97
lat (usec): min=1103, max=6073, avg=1569.45, stdev=173.57
clat percentiles (usec):
| 1.00th=[ 1237], 5.00th=[ 1319], 10.00th=[ 1352], 20.00th=[ 1418],
| 30.00th=[ 1450], 40.00th=[ 1483], 50.00th=[ 1516], 60.00th=[ 1549],
| 70.00th=[ 1598], 80.00th=[ 1647], 90.00th=[ 1729], 95.00th=[ 1811],
| 99.00th=[ 2024], 99.50th=[ 2147], 99.90th=[ 2671], 99.95th=[ 2835],
| 99.99th=[ 5997]
bw ( KiB/s): min=312320, max=349234, per=100.00%, avg=325958.00, stdev=10626.17, samples=19
iops : min= 610, max= 682, avg=636.63, stdev=20.74, samples=19
lat (msec) : 2=98.81%, 4=1.16%, 10=0.03%
cpu : usr=0.30%, sys=2.19%, ctx=6367, majf=0, minf=140
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=6364,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=318MiB/s (334MB/s), 318MiB/s-318MiB/s (334MB/s-334MB/s), io=3182MiB (3337MB), run=10001-10001msec
Disk stats (read/write):
dm-2: ios=6357/1, merge=0/0, ticks=9777/1, in_queue=9778, util=99.11%, aggrios=6412/1, aggrmerge=0/0, aggrticks=9886/1, aggrin_queue=6463, aggrutil=98.92%
sdb: ios=6412/1, merge=0/0, ticks=9886/1, in_queue=6463, util=98.92%
[root@CentOS4 ~] #留意以下测试结果：
Run status group 0 (all jobs): READ: bw=318MiB/s (334MB/s), 318MiB/s-318MiB/s (334MB/s-334MB/s), io=3182MiB (3337MB), run=10001-10001msec
以下是最大碎片读取测试（下限）测试的实验室输出结果，仅供参考。
[root@CentOS4 ~] # fio --name=frag-read-test --filename=/backups/'VM Backup Job 4_Oracle'/'VM Backup Job 4_OracleD2023-11-11T165948_C88D.vbk' --bs=512k --rw=randread --ioengine=libaio --direct=1 --time_based --runtime=10s
frag-read-test: (g=0): rw=randread, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=288MiB/s][r=575 IOPS][eta 00m:00s]
frag-read-test: (groupid=0, jobs=1): err= 0: pid=284652: Thu Nov 23 17:58:53 2023
read: IOPS=561, BW=281MiB/s (294MB/s)(2809MiB/10001msec)
slat (usec): min=11, max=187, avg=31.27, stdev=16.56
clat (usec): min=1082, max=18920, avg=1746.67, stdev=362.10
lat (usec): min=1117, max=18946, avg=1778.17, stdev=362.39
clat percentiles (usec):
| 1.00th=[ 1352], 5.00th=[ 1467], 10.00th=[ 1532], 20.00th=[ 1598],
| 30.00th=[ 1647], 40.00th=[ 1680], 50.00th=[ 1713], 60.00th=[ 1745],
| 70.00th=[ 1778], 80.00th=[ 1844], 90.00th=[ 1926], 95.00th=[ 2114],
| 99.00th=[ 2737], 99.50th=[ 2933], 99.90th=[ 3720], 99.95th=[ 4293],
| 99.99th=[19006]
bw ( KiB/s): min=227555, max=299008, per=100.00%, avg=288672.16, stdev=17426.87, samples=19
iops : min= 444, max= 584, avg=563.79, stdev=34.12, samples=19
lat (msec) : 2=92.86%, 4=7.07%, 10=0.04%, 20=0.04%
cpu : usr=0.29%, sys=1.80%, ctx=5617, majf=0, minf=137
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5617,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=281MiB/s (294MB/s), 281MiB/s-281MiB/s (294MB/s-294MB/s), io=2809MiB (2945MB), run=10001-10001msec
Disk stats (read/write):
dm-2: ios=5680/0, merge=0/0, ticks=9838/0, in_queue=9838, util=99.15%, aggrios=5721/0, aggrmerge=8/0, aggrticks=9966/0, aggrin_queue=6121, aggrutil=98.98%
sdb: ios=5721/0, merge=8/0, ticks=9966/0, in_queue=6121, util=98.98%
[root@CentOS4 ~] #留意以下测试结果：
Run status group 0 (all jobs): READ: bw=281MiB/s (294MB/s), 281MiB/s-281MiB/s (294MB/s-294MB/s), io=2809MiB (2945MB), run=10001-10001msec
最后，实际的恢复速度可能会在：318MB/s(上限值)与281MB/s(下限值)之间。
如对以上过程有技术疑问，请联系本文作者📧：helly.wu@veeam.com
