start test TEST-4K RandomRW: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: Laying out IO file (1 file / 5120MiB) RandomRW: Laying out IO file (1 file / 5120MiB) RandomRW: Laying out IO file (1 file / 5120MiB) RandomRW: Laying out IO file (1 file / 5120MiB) RandomRW: Laying out IO file (1 file / 5120MiB) RandomRW: (groupid=0, jobs=5): err= 0: pid=11710: Tue Jan 21 21:58:49 2020 read: IOPS=3299, BW=12.9MiB/s (13.5MB/s)(775MiB/60141msec) slat (usec): min=5, max=61878, avg=1033.58, stdev=2990.77 clat (msec): min=5, max=1664, avg=72.57, stdev=47.65 lat (msec): min=5, max=1664, avg=73.61, stdev=47.87 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 32], 10.00th=[ 36], 20.00th=[ 44], | 30.00th=[ 50], 40.00th=[ 56], 50.00th=[ 63], 60.00th=[ 70], | 70.00th=[ 81], 80.00th=[ 94], 90.00th=[ 117], 95.00th=[ 142], | 99.00th=[ 226], 99.50th=[ 292], 99.90th=[ 584], 99.95th=[ 735], | 99.99th=[ 1133] bw ( KiB/s): min= 1248, max= 4464, per=20.02%, avg=2641.70, stdev=691.77, samples=600 iops : min= 312, max= 1116, avg=660.40, stdev=172.94, samples=600 write: IOPS=1423, BW=5694KiB/s (5831kB/s)(334MiB/60141msec) slat (usec): min=6, max=52163, avg=1081.00, stdev=3064.55 clat (msec): min=4, max=277, avg=52.85, stdev=26.67 lat (msec): min=4, max=284, avg=53.94, stdev=27.06 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 22], 10.00th=[ 26], 20.00th=[ 32], | 30.00th=[ 37], 40.00th=[ 42], 50.00th=[ 47], 60.00th=[ 54], | 70.00th=[ 61], 80.00th=[ 71], 90.00th=[ 88], 95.00th=[ 105], | 99.00th=[ 142], 99.50th=[ 157], 99.90th=[ 190], 99.95th=[ 205], | 99.99th=[ 249] bw ( KiB/s): min= 456, max= 2008, per=20.02%, avg=1140.21, stdev=305.21, samples=600 iops : min= 114, max= 502, avg=285.02, stdev=76.32, samples=600 lat (msec) : 10=0.06%, 20=1.35%, 50=37.20%, 100=48.07%, 250=12.79% lat (msec) : 500=0.42%, 750=0.08%, 1000=0.02% cpu : usr=0.58%, sys=2.71%, ctx=49351, majf=0, minf=149 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=198406,85613,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=12.9MiB/s (13.5MB/s), 12.9MiB/s-12.9MiB/s (13.5MB/s-13.5MB/s), io=775MiB (813MB), run=60141-60141msec WRITE: bw=5694KiB/s (5831kB/s), 5694KiB/s-5694KiB/s (5831kB/s-5831kB/s), io=334MiB (351MB), run=60141-60141msec Disk stats (read/write): vda: ios=198441/85579, merge=7/1306, ticks=6376854/1132928, in_queue=7516239, util=99.91% TEST-16K RandomRW: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=11723: Tue Jan 21 21:59:51 2020 read: IOPS=2468, BW=38.6MiB/s (40.4MB/s)(2324MiB/60257msec) slat (usec): min=6, max=73062, avg=1383.12, stdev=3974.10 clat (msec): min=2, max=979, avg=96.80, stdev=48.56 lat (msec): min=2, max=979, avg=98.19, stdev=48.88 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 43], 10.00th=[ 51], 20.00th=[ 61], | 30.00th=[ 69], 40.00th=[ 78], 50.00th=[ 87], 60.00th=[ 97], | 70.00th=[ 110], 80.00th=[ 127], 90.00th=[ 155], 95.00th=[ 184], | 99.00th=[ 259], 99.50th=[ 296], 99.90th=[ 443], 99.95th=[ 527], | 99.99th=[ 760] bw ( KiB/s): min= 4192, max=13344, per=20.05%, avg=7918.23, stdev=1831.91, samples=600 iops : min= 262, max= 834, avg=494.85, stdev=114.48, samples=600 write: IOPS=1064, BW=16.6MiB/s (17.4MB/s)(1002MiB/60257msec) slat (usec): min=7, max=92529, avg=1437.24, stdev=4078.56 clat (msec): min=4, max=314, avg=70.87, stdev=34.71 lat (msec): min=4, max=324, avg=72.31, stdev=35.16 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 29], 10.00th=[ 34], 20.00th=[ 43], | 30.00th=[ 50], 40.00th=[ 57], 50.00th=[ 64], 60.00th=[ 72], | 70.00th=[ 83], 80.00th=[ 96], 90.00th=[ 116], 95.00th=[ 138], | 99.00th=[ 186], 99.50th=[ 203], 99.90th=[ 249], 99.95th=[ 266], | 99.99th=[ 292] bw ( KiB/s): min= 1536, max= 5792, per=20.06%, avg=3415.55, stdev=841.90, samples=600 iops : min= 96, max= 362, avg=213.43, stdev=52.63, samples=600 lat (msec) : 4=0.01%, 10=0.02%, 20=0.39%, 50=15.82%, 100=52.34% lat (msec) : 250=30.58%, 500=0.82%, 750=0.03%, 1000=0.01% cpu : usr=0.48%, sys=2.28%, ctx=37722, majf=0, minf=154 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=148732,64129,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=38.6MiB/s (40.4MB/s), 38.6MiB/s-38.6MiB/s (40.4MB/s-40.4MB/s), io=2324MiB (2437MB), run=60257-60257msec WRITE: bw=16.6MiB/s (17.4MB/s), 16.6MiB/s-16.6MiB/s (17.4MB/s-17.4MB/s), io=1002MiB (1051MB), run=60257-60257msec Disk stats (read/write): vda: ios=148726/64162, merge=0/13, ticks=6372766/1156409, in_queue=7532151, util=99.93% TEST-32K RandomRW: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=11731: Tue Jan 21 22:00:52 2020 read: IOPS=2479, BW=77.5MiB/s (81.3MB/s)(4680MiB/60390msec) slat (usec): min=8, max=117043, avg=1383.08, stdev=4019.79 clat (msec): min=7, max=1439, avg=96.42, stdev=61.29 lat (msec): min=7, max=1458, avg=97.81, stdev=61.70 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 39], 10.00th=[ 46], 20.00th=[ 57], | 30.00th=[ 65], 40.00th=[ 74], 50.00th=[ 84], 60.00th=[ 94], | 70.00th=[ 108], 80.00th=[ 126], 90.00th=[ 157], 95.00th=[ 192], | 99.00th=[ 313], 99.50th=[ 405], 99.90th=[ 693], 99.95th=[ 877], | 99.99th=[ 1116] bw ( KiB/s): min= 1472, max=28992, per=20.09%, avg=15945.38, stdev=4234.24, samples=600 iops : min= 46, max= 906, avg=498.25, stdev=132.30, samples=600 write: IOPS=1068, BW=33.4MiB/s (35.0MB/s)(2017MiB/60390msec) slat (usec): min=9, max=89677, avg=1409.10, stdev=4008.67 clat (msec): min=4, max=946, avg=70.54, stdev=39.71 lat (msec): min=4, max=1001, avg=71.95, stdev=40.29 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 28], 10.00th=[ 34], 20.00th=[ 42], | 30.00th=[ 49], 40.00th=[ 56], 50.00th=[ 64], 60.00th=[ 71], | 70.00th=[ 82], 80.00th=[ 94], 90.00th=[ 115], 95.00th=[ 136], | 99.00th=[ 186], 99.50th=[ 215], 99.90th=[ 481], 99.95th=[ 550], | 99.99th=[ 844] bw ( KiB/s): min= 832, max=13888, per=20.10%, avg=6876.94, stdev=1968.37, samples=600 iops : min= 26, max= 434, avg=214.85, stdev=61.52, samples=600 lat (msec) : 10=0.02%, 20=0.44%, 50=18.99%, 100=51.01%, 250=28.06% lat (msec) : 500=1.26%, 750=0.17%, 1000=0.05% cpu : usr=0.56%, sys=2.56%, ctx=38070, majf=0, minf=156 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=149763,64555,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=77.5MiB/s (81.3MB/s), 77.5MiB/s-77.5MiB/s (81.3MB/s-81.3MB/s), io=4680MiB (4907MB), run=60390-60390msec WRITE: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=2017MiB (2115MB), run=60390-60390msec Disk stats (read/write): vda: ios=149793/64574, merge=0/13, ticks=6377078/1144337, in_queue=7547559, util=99.91% TEST-64K RandomRW: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=11743: Tue Jan 21 22:01:53 2020 read: IOPS=2000, BW=125MiB/s (131MB/s)(7546MiB/60355msec) slat (usec): min=11, max=308080, avg=1711.66, stdev=5391.74 clat (msec): min=6, max=1447, avg=119.34, stdev=81.34 lat (msec): min=6, max=1447, avg=121.05, stdev=81.89 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 43], 10.00th=[ 52], 20.00th=[ 65], | 30.00th=[ 77], 40.00th=[ 88], 50.00th=[ 101], 60.00th=[ 115], | 70.00th=[ 133], 80.00th=[ 159], 90.00th=[ 203], 95.00th=[ 255], | 99.00th=[ 443], 99.50th=[ 542], 99.90th=[ 760], 99.95th=[ 953], | 99.99th=[ 1334] bw ( KiB/s): min= 3968, max=48896, per=20.08%, avg=25702.37, stdev=8704.22, samples=600 iops : min= 62, max= 764, avg=401.56, stdev=136.00, samples=600 write: IOPS=864, BW=54.0MiB/s (56.6MB/s)(3259MiB/60355msec) slat (usec): min=13, max=292667, avg=1762.99, stdev=5304.53 clat (msec): min=3, max=721, avg=87.55, stdev=51.97 lat (msec): min=3, max=721, avg=89.32, stdev=52.70 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 35], 10.00th=[ 42], 20.00th=[ 51], | 30.00th=[ 59], 40.00th=[ 68], 50.00th=[ 77], 60.00th=[ 87], | 70.00th=[ 99], 80.00th=[ 115], 90.00th=[ 144], 95.00th=[ 176], | 99.00th=[ 284], 99.50th=[ 359], 99.90th=[ 523], 99.95th=[ 567], | 99.99th=[ 600] bw ( KiB/s): min= 1536, max=20608, per=20.08%, avg=11105.91, stdev=3825.36, samples=600 iops : min= 24, max= 322, avg=173.47, stdev=59.78, samples=600 lat (msec) : 4=0.01%, 10=0.02%, 20=0.17%, 50=12.15%, 100=44.08% lat (msec) : 250=39.42%, 500=3.65%, 750=0.44%, 1000=0.04% cpu : usr=0.54%, sys=2.51%, ctx=31236, majf=0, minf=155 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=120740,52149,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=7546MiB (7913MB), run=60355-60355msec WRITE: bw=54.0MiB/s (56.6MB/s), 54.0MiB/s-54.0MiB/s (56.6MB/s-56.6MB/s), io=3259MiB (3418MB), run=60355-60355msec Disk stats (read/write): vda: ios=120746/52157, merge=2/9, ticks=6358158/1160956, in_queue=7543576, util=100.00% TEST-128K RandomRW: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=11763: Tue Jan 21 22:02:55 2020 read: IOPS=1317, BW=165MiB/s (173MB/s)(9.77GiB/60733msec) slat (usec): min=17, max=198316, avg=2557.64, stdev=8203.00 clat (usec): min=918, max=5748.9k, avg=180712.90, stdev=142437.04 lat (usec): min=1263, max=5758.5k, avg=183271.91, stdev=143328.50 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 52], 10.00th=[ 65], 20.00th=[ 85], | 30.00th=[ 103], 40.00th=[ 123], 50.00th=[ 144], 60.00th=[ 169], | 70.00th=[ 205], 80.00th=[ 251], 90.00th=[ 338], 95.00th=[ 426], | 99.00th=[ 659], 99.50th=[ 785], 99.90th=[ 1552], 99.95th=[ 1770], | 99.99th=[ 2140] bw ( KiB/s): min= 256, max=78848, per=20.15%, avg=33980.11, stdev=15576.69, samples=601 iops : min= 2, max= 616, avg=265.42, stdev=121.71, samples=601 write: IOPS=569, BW=71.2MiB/s (74.6MB/s)(4324MiB/60733msec) slat (usec): min=18, max=189885, avg=2723.78, stdev=8524.64 clat (usec): min=1038, max=3058.9k, avg=132549.50, stdev=87773.27 lat (usec): min=1072, max=3058.9k, avg=135274.67, stdev=89219.63 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 44], 10.00th=[ 54], 20.00th=[ 69], | 30.00th=[ 82], 40.00th=[ 95], 50.00th=[ 110], 60.00th=[ 128], | 70.00th=[ 150], 80.00th=[ 182], 90.00th=[ 241], 95.00th=[ 305], | 99.00th=[ 439], 99.50th=[ 510], 99.90th=[ 743], 99.95th=[ 852], | 99.99th=[ 936] bw ( KiB/s): min= 2560, max=37120, per=20.19%, avg=14720.96, stdev=6963.94, samples=600 iops : min= 20, max= 290, avg=114.93, stdev=54.41, samples=600 lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.07%, 10=0.04%, 20=0.07%, 50=5.21% lat (msec) : 100=27.68%, 250=50.09%, 500=14.66%, 750=1.70%, 1000=0.27% cpu : usr=0.41%, sys=2.27%, ctx=20375, majf=0, minf=156 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=80030,34588,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=9.77GiB (10.5GB), run=60733-60733msec WRITE: bw=71.2MiB/s (74.6MB/s), 71.2MiB/s-71.2MiB/s (74.6MB/s-74.6MB/s), io=4324MiB (4534MB), run=60733-60733msec Disk stats (read/write): vda: ios=80027/34591, merge=0/1, ticks=6370850/1208875, in_queue=7563562, util=100.00% TEST-256K RandomRW: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=11774: Tue Jan 21 22:03:56 2020 read: IOPS=832, BW=208MiB/s (218MB/s)(12.4GiB/60814msec) slat (usec): min=29, max=818741, avg=3994.54, stdev=13536.22 clat (msec): min=4, max=6192, avg=285.00, stdev=238.75 lat (msec): min=4, max=6193, avg=288.99, stdev=240.33 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 67], 10.00th=[ 85], 20.00th=[ 121], | 30.00th=[ 161], 40.00th=[ 199], 50.00th=[ 236], 60.00th=[ 279], | 70.00th=[ 330], 80.00th=[ 393], 90.00th=[ 510], 95.00th=[ 642], | 99.00th=[ 1234], 99.50th=[ 1552], 99.90th=[ 2299], 99.95th=[ 2567], | 99.99th=[ 5000] bw ( KiB/s): min= 413, max=144896, per=20.27%, avg=43208.93, stdev=25103.44, samples=597 iops : min= 1, max= 566, avg=168.74, stdev=98.08, samples=597 write: IOPS=360, BW=90.2MiB/s (94.6MB/s)(5487MiB/60814msec) slat (usec): min=33, max=818275, avg=4424.62, stdev=15723.90 clat (msec): min=4, max=1753, avg=208.47, stdev=146.05 lat (msec): min=5, max=1790, avg=212.89, stdev=148.92 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 58], 10.00th=[ 72], 20.00th=[ 94], | 30.00th=[ 123], 40.00th=[ 153], 50.00th=[ 182], 60.00th=[ 211], | 70.00th=[ 247], 80.00th=[ 292], 90.00th=[ 363], 95.00th=[ 435], | 99.00th=[ 785], 99.50th=[ 1003], 99.90th=[ 1351], 99.95th=[ 1502], | 99.99th=[ 1603] bw ( KiB/s): min= 1532, max=66048, per=20.34%, avg=18794.18, stdev=11266.78, samples=596 iops : min= 5, max= 258, avg=73.36, stdev=44.03, samples=596 lat (msec) : 10=0.08%, 20=0.06%, 50=1.72%, 100=15.23%, 250=41.72% lat (msec) : 500=32.96%, 750=5.51%, 1000=1.36% cpu : usr=0.39%, sys=2.09%, ctx=13069, majf=0, minf=157 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=50639,21948,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=12.4GiB (13.3GB), run=60814-60814msec WRITE: bw=90.2MiB/s (94.6MB/s), 90.2MiB/s-90.2MiB/s (94.6MB/s-94.6MB/s), io=5487MiB (5754MB), run=60814-60814msec Disk stats (read/write): vda: ios=50657/21956, merge=1/0, ticks=6362397/1129932, in_queue=7556099, util=100.00% TEST-512K RandomRW: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=11783: Tue Jan 21 22:04:58 2020 read: IOPS=576, BW=288MiB/s (302MB/s)(17.0GiB/60541msec) slat (usec): min=55, max=1148.3k, avg=5757.86, stdev=22491.41 clat (msec): min=6, max=9037, avg=413.33, stdev=405.89 lat (msec): min=6, max=9074, avg=419.09, stdev=409.51 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 90], 10.00th=[ 112], 20.00th=[ 144], | 30.00th=[ 180], 40.00th=[ 226], 50.00th=[ 288], 60.00th=[ 363], | 70.00th=[ 472], 80.00th=[ 609], 90.00th=[ 827], 95.00th=[ 1099], | 99.00th=[ 2039], 99.50th=[ 2500], 99.90th=[ 3876], 99.95th=[ 4463], | 99.99th=[ 6678] bw ( KiB/s): min= 1024, max=208896, per=20.27%, avg=59864.98, stdev=41878.74, samples=593 iops : min= 2, max= 408, avg=116.84, stdev=81.82, samples=593 write: IOPS=251, BW=126MiB/s (132MB/s)(7616MiB/60541msec) slat (usec): min=66, max=993982, avg=6469.52, stdev=22248.40 clat (msec): min=6, max=2224, avg=303.03, stdev=266.27 lat (msec): min=6, max=2453, avg=309.50, stdev=270.85 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 82], 10.00th=[ 99], 20.00th=[ 125], | 30.00th=[ 148], 40.00th=[ 176], 50.00th=[ 218], 60.00th=[ 271], | 70.00th=[ 342], 80.00th=[ 439], 90.00th=[ 600], 95.00th=[ 768], | 99.00th=[ 1536], 99.50th=[ 1821], 99.90th=[ 2056], 99.95th=[ 2140], | 99.99th=[ 2232] bw ( KiB/s): min= 1021, max=96063, per=20.38%, avg=26251.09, stdev=18765.73, samples=591 iops : min= 1, max= 187, avg=51.18, stdev=36.66, samples=591 lat (msec) : 10=0.02%, 20=0.09%, 50=0.35%, 100=7.81%, 250=39.91% lat (msec) : 500=28.01%, 750=13.29%, 1000=5.34% cpu : usr=0.38%, sys=2.50%, ctx=9103, majf=0, minf=156 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.4% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=34918,15231,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=288MiB/s (302MB/s), 288MiB/s-288MiB/s (302MB/s-302MB/s), io=17.0GiB (18.3GB), run=60541-60541msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=7616MiB (7985MB), run=60541-60541msec Disk stats (read/write): vda: ios=34907/15233, merge=34918/15231, ticks=6333899/1134745, in_queue=7545484, util=100.00% TEST-1024K RandomRW: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=11791: Tue Jan 21 22:06:02 2020 read: IOPS=586, BW=587MiB/s (615MB/s)(36.3GiB/63386msec) slat (usec): min=114, max=1247.9k, avg=5539.65, stdev=28681.04 clat (msec): min=36, max=10259, avg=362.42, stdev=877.95 lat (msec): min=36, max=10259, avg=367.96, stdev=890.59 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 81], 10.00th=[ 89], 20.00th=[ 101], | 30.00th=[ 110], 40.00th=[ 120], 50.00th=[ 132], 60.00th=[ 153], | 70.00th=[ 197], 80.00th=[ 300], 90.00th=[ 567], 95.00th=[ 1070], | 99.00th=[ 5738], 99.50th=[ 6477], 99.90th=[ 7684], 99.95th=[ 8221], | 99.99th=[ 9329] bw ( KiB/s): min= 2043, max=498714, per=21.72%, avg=130489.19, stdev=144520.18, samples=580 iops : min= 1, max= 487, avg=127.33, stdev=141.13, samples=580 write: IOPS=255, BW=255MiB/s (268MB/s)(15.8GiB/63386msec) slat (usec): min=145, max=1447.7k, avg=5786.88, stdev=29528.11 clat (msec): min=47, max=13728, avg=392.15, stdev=1204.25 lat (msec): min=48, max=13759, avg=397.94, stdev=1215.90 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 82], 10.00th=[ 89], 20.00th=[ 101], | 30.00th=[ 110], 40.00th=[ 118], 50.00th=[ 129], 60.00th=[ 144], | 70.00th=[ 174], 80.00th=[ 264], 90.00th=[ 477], 95.00th=[ 902], | 99.00th=[ 7617], 99.50th=[ 9194], 99.90th=[11879], 99.95th=[12550], | 99.99th=[13489] bw ( KiB/s): min= 2043, max=227328, per=23.82%, avg=62280.75, stdev=63591.10, samples=528 iops : min= 1, max= 222, avg=60.68, stdev=62.07, samples=528 lat (msec) : 50=0.05%, 100=20.06%, 250=56.75%, 500=12.37%, 750=3.62% lat (msec) : 1000=2.01% cpu : usr=0.92%, sys=6.99%, ctx=22502, majf=0, minf=157 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.3%, >=64=99.4% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=37186,16184,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=587MiB/s (615MB/s), 587MiB/s-587MiB/s (615MB/s-615MB/s), io=36.3GiB (38.0GB), run=63386-63386msec WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=15.8GiB (16.0GB), run=63386-63386msec Disk stats (read/write): vda: ios=111558/48558, merge=0/0, ticks=4529969/3034036, in_queue=7633574, util=100.00%