start test TEST-4K RandomRW: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: Laying out IO file (1 file / 5120MiB) RandomRW: Laying out IO file (1 file / 5120MiB) RandomRW: Laying out IO file (1 file / 5120MiB) RandomRW: Laying out IO file (1 file / 5120MiB) RandomRW: Laying out IO file (1 file / 5120MiB) fio: terminating on signal 2 RandomRW: (groupid=0, jobs=5): err= 0: pid=3165: Tue Jan 21 15:13:07 2020 read: IOPS=14.5k, BW=56.5MiB/s (59.2MB/s)(3116MiB/55148msec) slat (usec): min=3, max=5107.9k, avg=223.78, stdev=12235.88 clat (usec): min=434, max=5491.9k, avg=15948.82, stdev=95056.36 lat (usec): min=443, max=5491.9k, avg=16173.91, stdev=95925.76 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 25], 95.00th=[ 32], | 99.00th=[ 74], 99.50th=[ 169], 99.90th=[ 709], 99.95th=[ 1620], | 99.99th=[ 5134] bw ( KiB/s): min= 24, max=28432, per=22.67%, avg=13118.61, stdev=7181.60, samples=485 iops : min= 6, max= 7108, avg=3279.62, stdev=1795.39, samples=485 write: IOPS=6214, BW=24.3MiB/s (25.5MB/s)(1339MiB/55148msec) slat (usec): min=4, max=5106.6k, avg=225.69, stdev=10008.86 clat (usec): min=430, max=5458.3k, avg=13590.12, stdev=95037.64 lat (usec): min=438, max=5458.3k, avg=13817.11, stdev=95690.60 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 19], 95.00th=[ 24], | 99.00th=[ 54], 99.50th=[ 122], 99.90th=[ 709], 99.95th=[ 1636], | 99.99th=[ 5134] bw ( KiB/s): min= 63, max=12550, per=22.68%, avg=5636.47, stdev=3101.69, samples=485 iops : min= 15, max= 3137, avg=1409.09, stdev=775.42, samples=485 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.23%, 4=5.44%, 10=52.32%, 20=29.03%, 50=11.49% lat (msec) : 100=0.74%, 250=0.43%, 500=0.12%, 750=0.10%, 1000=0.03% cpu : usr=2.23%, sys=5.13%, ctx=67125, majf=0, minf=162 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=797650,342700,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=56.5MiB/s (59.2MB/s), 56.5MiB/s-56.5MiB/s (59.2MB/s-59.2MB/s), io=3116MiB (3267MB), run=55148-55148msec WRITE: bw=24.3MiB/s (25.5MB/s), 24.3MiB/s-24.3MiB/s (25.5MB/s-25.5MB/s), io=1339MiB (1404MB), run=55148-55148msec Disk stats (read/write): sda: ios=796616/342254, merge=8/95, ticks=6506688/2397594, in_queue=8905136, util=99.91% TEST-8K RandomRW: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=16961: Wed Jan 29 13:17:01 2020 read: IOPS=15.1k, BW=118MiB/s (123MB/s)(7070MiB/60070msec) slat (usec): min=3, max=1640.3k, avg=213.18, stdev=6056.96 clat (usec): min=257, max=1711.5k, avg=15424.89, stdev=50886.22 lat (usec): min=452, max=1711.5k, avg=15639.37, stdev=51263.30 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 26], 95.00th=[ 36], | 99.00th=[ 78], 99.50th=[ 114], 99.90th=[ 911], 99.95th=[ 1070], | 99.99th=[ 1670] bw ( KiB/s): min= 112, max=59120, per=21.10%, avg=25436.57, stdev=12006.21, samples=569 iops : min= 14, max= 7390, avg=3179.53, stdev=1500.77, samples=569 write: IOPS=6476, BW=50.6MiB/s (53.1MB/s)(3039MiB/60070msec) slat (usec): min=4, max=1640.1k, avg=222.13, stdev=6172.36 clat (usec): min=682, max=1685.6k, avg=12762.85, stdev=47764.26 lat (usec): min=696, max=1685.6k, avg=12986.33, stdev=48177.20 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 20], 95.00th=[ 24], | 99.00th=[ 41], 99.50th=[ 78], 99.90th=[ 902], 99.95th=[ 1011], | 99.99th=[ 1670] bw ( KiB/s): min= 16, max=24432, per=21.11%, avg=10934.14, stdev=5167.27, samples=569 iops : min= 2, max= 3054, avg=1366.73, stdev=645.91, samples=569 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.75%, 4=9.23%, 10=46.11%, 20=30.27%, 50=11.64% lat (msec) : 100=1.45%, 250=0.22%, 500=0.12%, 750=0.05%, 1000=0.08% cpu : usr=2.45%, sys=5.35%, ctx=76757, majf=0, minf=167 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=905014,389020,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=118MiB/s (123MB/s), 118MiB/s-118MiB/s (123MB/s-123MB/s), io=7070MiB (7414MB), run=60070-60070msec WRITE: bw=50.6MiB/s (53.1MB/s), 50.6MiB/s-50.6MiB/s (53.1MB/s-53.1MB/s), io=3039MiB (3187MB), run=60070-60070msec Disk stats (read/write): sda: ios=903250/388406, merge=37/136, ticks=7420802/2336485, in_queue=9760168, util=99.93% TEST-16K RandomRW: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=3177: Tue Jan 21 15:14:10 2020 read: IOPS=15.0k, BW=235MiB/s (246MB/s)(13.0GiB/60886msec) slat (usec): min=4, max=2607.7k, avg=208.29, stdev=9275.22 clat (usec): min=493, max=4926.8k, avg=15186.02, stdev=87257.69 lat (usec): min=665, max=4926.8k, avg=15395.73, stdev=87824.29 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 21], 95.00th=[ 32], | 99.00th=[ 91], 99.50th=[ 146], 99.90th=[ 1821], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 606, max=118930, per=24.10%, avg=57929.36, stdev=30577.77, samples=505 iops : min= 37, max= 7433, avg=3620.52, stdev=1911.12, samples=505 write: IOPS=6460, BW=101MiB/s (106MB/s)(6146MiB/60886msec) slat (usec): min=4, max=2607.1k, avg=239.02, stdev=12565.19 clat (usec): min=718, max=4903.5k, avg=13464.06, stdev=81817.85 lat (usec): min=734, max=4903.5k, avg=13704.48, stdev=83019.00 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 18], 95.00th=[ 25], | 99.00th=[ 47], 99.50th=[ 94], 99.90th=[ 1804], 99.95th=[ 2299], | 99.99th=[ 2601] bw ( KiB/s): min= 256, max=50714, per=24.10%, avg=24908.44, stdev=13178.65, samples=505 iops : min= 16, max= 3169, avg=1556.71, stdev=823.68, samples=505 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.44%, 4=8.38%, 10=59.93%, 20=21.46%, 50=7.73% lat (msec) : 100=1.36%, 250=0.36%, 500=0.12%, 750=0.02%, 1000=0.07% cpu : usr=2.56%, sys=6.13%, ctx=70827, majf=0, minf=170 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=914819,393348,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=235MiB/s (246MB/s), 235MiB/s-235MiB/s (246MB/s-246MB/s), io=13.0GiB (14.0GB), run=60886-60886msec WRITE: bw=101MiB/s (106MB/s), 101MiB/s-101MiB/s (106MB/s-106MB/s), io=6146MiB (6445MB), run=60886-60886msec Disk stats (read/write): sda: ios=914624/393312, merge=151/137, ticks=6917414/2565873, in_queue=9901600, util=99.98% TEST-32K RandomRW: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=3191: Tue Jan 21 15:15:11 2020 read: IOPS=14.6k, BW=457MiB/s (479MB/s)(26.8GiB/60005msec) slat (usec): min=4, max=6818.7k, avg=232.76, stdev=23519.03 clat (usec): min=506, max=6853.3k, avg=14715.57, stdev=174580.70 lat (usec): min=522, max=6853.3k, avg=14949.76, stdev=176167.17 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 42], 99.50th=[ 90], 99.90th=[ 3104], 99.95th=[ 5134], | 99.99th=[ 6812] bw ( KiB/s): min= 1472, max=257277, per=32.86%, avg=153778.47, stdev=60139.69, samples=365 iops : min= 46, max= 8039, avg=4805.47, stdev=1879.35, samples=365 write: IOPS=6284, BW=196MiB/s (206MB/s)(11.5GiB/60005msec) slat (usec): min=5, max=6782.0k, avg=206.80, stdev=19020.71 clat (usec): min=944, max=6833.3k, avg=15891.51, stdev=187928.86 lat (usec): min=971, max=6835.4k, avg=16099.79, stdev=188894.71 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 32], 99.50th=[ 79], 99.90th=[ 4010], 99.95th=[ 5134], | 99.99th=[ 6812] bw ( KiB/s): min= 576, max=106347, per=32.86%, avg=66069.91, stdev=25847.44, samples=365 iops : min= 18, max= 3323, avg=2064.61, stdev=807.73, samples=365 lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.32%, 4=6.10%, 10=77.70%, 20=14.00%, 50=1.12% lat (msec) : 100=0.31%, 250=0.17%, 500=0.02%, 750=0.08%, 1000=0.03% cpu : usr=2.58%, sys=6.75%, ctx=57907, majf=0, minf=172 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=877615,377075,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=457MiB/s (479MB/s), 457MiB/s-457MiB/s (479MB/s-479MB/s), io=26.8GiB (28.8GB), run=60005-60005msec WRITE: bw=196MiB/s (206MB/s), 196MiB/s-196MiB/s (206MB/s-206MB/s), io=11.5GiB (12.4GB), run=60005-60005msec Disk stats (read/write): sda: ios=872942/375285, merge=2031/528, ticks=5984131/3937948, in_queue=9923734, util=99.95% TEST-64K RandomRW: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=3264: Tue Jan 21 15:16:12 2020 read: IOPS=12.8k, BW=801MiB/s (839MB/s)(46.9GiB/60008msec) slat (usec): min=5, max=4015.4k, avg=266.45, stdev=19658.15 clat (usec): min=101, max=4034.4k, avg=16795.06, stdev=150037.96 lat (usec): min=231, max=4034.4k, avg=17062.98, stdev=151311.76 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 29], 99.50th=[ 234], 99.90th=[ 3306], 99.95th=[ 3742], | 99.99th=[ 4044] bw ( KiB/s): min= 3584, max=450560, per=32.29%, avg=264751.00, stdev=118260.55, samples=370 iops : min= 56, max= 7040, avg=4136.66, stdev=1847.81, samples=370 write: IOPS=5500, BW=344MiB/s (361MB/s)(20.1GiB/60008msec) slat (usec): min=7, max=4016.5k, avg=248.31, stdev=18437.31 clat (usec): min=237, max=4034.6k, avg=18159.94, stdev=160412.74 lat (usec): min=254, max=4034.7k, avg=18409.74, stdev=161466.30 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 28], 99.50th=[ 326], 99.90th=[ 3306], 99.95th=[ 3775], | 99.99th=[ 4044] bw ( KiB/s): min= 512, max=203264, per=32.30%, avg=113705.34, stdev=50862.33, samples=370 iops : min= 8, max= 3176, avg=1776.56, stdev=794.71, samples=370 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.79%, 4=3.04%, 10=74.42%, 20=20.10%, 50=0.92% lat (msec) : 100=0.16%, 250=0.06%, 500=0.15%, 750=0.03% cpu : usr=2.51%, sys=7.62%, ctx=45907, majf=0, minf=173 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=768671,330096,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=801MiB/s (839MB/s), 801MiB/s-801MiB/s (839MB/s-839MB/s), io=46.9GiB (50.4GB), run=60008-60008msec WRITE: bw=344MiB/s (361MB/s), 344MiB/s-344MiB/s (361MB/s-361MB/s), io=20.1GiB (21.6GB), run=60008-60008msec Disk stats (read/write): sda: ios=762820/328172, merge=2891/762, ticks=6210527/3888323, in_queue=10101122, util=100.00% TEST-128K RandomRW: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=3331: Tue Jan 21 15:17:17 2020 read: IOPS=8143, BW=1018MiB/s (1067MB/s)(63.8GiB/64149msec) slat (usec): min=8, max=5981.4k, avg=398.81, stdev=33889.60 clat (usec): min=1349, max=6005.8k, avg=25776.57, stdev=266182.47 lat (usec): min=1375, max=6005.9k, avg=26176.94, stdev=268317.61 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 16], 95.00th=[ 19], | 99.00th=[ 42], 99.50th=[ 451], 99.90th=[ 5738], 99.95th=[ 5805], | 99.99th=[ 6007] bw ( KiB/s): min= 2816, max=787456, per=40.31%, avg=420156.20, stdev=199941.48, samples=318 iops : min= 22, max= 6150, avg=3282.36, stdev=1562.00, samples=318 write: IOPS=3499, BW=437MiB/s (459MB/s)(27.4GiB/64149msec) slat (usec): min=11, max=5978.3k, avg=436.83, stdev=34876.35 clat (usec): min=1867, max=6472.2k, avg=30037.34, stdev=298024.38 lat (usec): min=1905, max=6472.3k, avg=30475.80, stdev=300045.64 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 17], 95.00th=[ 20], | 99.00th=[ 78], 99.50th=[ 485], 99.90th=[ 5738], 99.95th=[ 6007], | 99.99th=[ 6007] bw ( KiB/s): min= 1532, max=324726, per=40.56%, avg=181711.82, stdev=84885.89, samples=316 iops : min= 11, max= 2536, avg=1419.52, stdev=663.14, samples=316 lat (msec) : 2=0.03%, 4=0.88%, 10=58.79%, 20=36.85%, 50=2.46% lat (msec) : 100=0.07%, 250=0.21%, 500=0.38%, 750=0.01%, 1000=0.04% cpu : usr=2.02%, sys=6.71%, ctx=32451, majf=0, minf=168 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=522391,224503,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=1018MiB/s (1067MB/s), 1018MiB/s-1018MiB/s (1067MB/s-1067MB/s), io=63.8GiB (68.5GB), run=64149-64149msec WRITE: bw=437MiB/s (459MB/s), 437MiB/s-437MiB/s (459MB/s-459MB/s), io=27.4GiB (29.4GB), run=64149-64149msec Disk stats (read/write): sda: ios=518783/223801, merge=3607/831, ticks=5576314/4373939, in_queue=10176336, util=100.00% TEST-256K RandomRW: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=3340: Tue Jan 21 15:18:21 2020 read: IOPS=4777, BW=1194MiB/s (1252MB/s)(73.3GiB/62881msec) slat (usec): min=13, max=8633.9k, avg=672.70, stdev=46513.43 clat (usec): min=587, max=8666.7k, avg=40866.30, stdev=369413.09 lat (usec): min=620, max=8666.8k, avg=41540.63, stdev=372295.00 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 18], 90.00th=[ 22], 95.00th=[ 26], | 99.00th=[ 288], 99.50th=[ 2039], 99.90th=[ 8356], 99.95th=[ 8658], | 99.99th=[ 8658] bw ( KiB/s): min= 3584, max=1015808, per=46.18%, avg=564721.48, stdev=275691.33, samples=272 iops : min= 14, max= 3968, avg=2205.83, stdev=1076.92, samples=272 write: IOPS=2051, BW=513MiB/s (538MB/s)(31.5GiB/62881msec) slat (usec): min=20, max=8640.9k, avg=828.69, stdev=56055.93 clat (msec): min=2, max=9169, avg=57.69, stdev=450.10 lat (msec): min=2, max=9169, avg=58.52, stdev=453.83 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 45], | 99.00th=[ 1636], 99.50th=[ 2500], 99.90th=[ 8356], 99.95th=[ 8658], | 99.99th=[ 8792] bw ( KiB/s): min= 510, max=408064, per=46.33%, avg=243360.93, stdev=117770.60, samples=271 iops : min= 1, max= 1594, avg=950.51, stdev=460.06, samples=271 lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.36%, 10=26.31%, 20=56.58%, 50=14.24% lat (msec) : 100=0.73%, 250=0.48%, 500=0.39%, 750=0.08% cpu : usr=1.66%, sys=6.08%, ctx=18666, majf=0, minf=166 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=300388,129023,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=1194MiB/s (1252MB/s), 1194MiB/s-1194MiB/s (1252MB/s-1252MB/s), io=73.3GiB (78.7GB), run=62881-62881msec WRITE: bw=513MiB/s (538MB/s), 513MiB/s-513MiB/s (538MB/s-538MB/s), io=31.5GiB (33.8GB), run=62881-62881msec Disk stats (read/write): sda: ios=297061/128269, merge=3251/821, ticks=4812490/4824043, in_queue=10284514, util=100.00% TEST-512K RandomRW: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=3351: Tue Jan 21 15:19:23 2020 read: IOPS=3380, BW=1690MiB/s (1772MB/s)(99.6GiB/60370msec) slat (usec): min=25, max=6664.8k, avg=968.32, stdev=53079.39 clat (msec): min=2, max=11329, avg=52.09, stdev=398.17 lat (msec): min=2, max=11330, avg=53.06, stdev=402.42 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 25], 80.00th=[ 28], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 380], 99.50th=[ 3138], 99.90th=[ 6611], 99.95th=[ 6678], | 99.99th=[ 6812] bw ( KiB/s): min= 1024, max=1097728, per=39.01%, avg=675201.37, stdev=301868.54, samples=309 iops : min= 2, max= 2144, avg=1318.63, stdev=589.56, samples=309 write: IOPS=1459, BW=730MiB/s (765MB/s)(43.0GiB/60370msec) slat (usec): min=38, max=6661.0k, avg=1140.41, stdev=58703.52 clat (msec): min=3, max=11821, avg=95.03, stdev=573.13 lat (msec): min=3, max=11821, avg=96.17, stdev=576.19 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 27], | 70.00th=[ 31], 80.00th=[ 36], 90.00th=[ 80], 95.00th=[ 197], | 99.00th=[ 1284], 99.50th=[ 4665], 99.90th=[ 6879], 99.95th=[11342], | 99.99th=[11476] bw ( KiB/s): min= 4096, max=452608, per=38.99%, avg=291271.71, stdev=130277.44, samples=309 iops : min= 8, max= 884, avg=568.76, stdev=254.43, samples=309 lat (msec) : 4=0.02%, 10=7.63%, 20=39.67%, 50=47.49%, 100=1.31% lat (msec) : 250=1.97%, 500=0.78%, 750=0.27%, 1000=0.12% cpu : usr=1.82%, sys=7.10%, ctx=14349, majf=0, minf=167 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=204076,88093,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=1690MiB/s (1772MB/s), 1690MiB/s-1690MiB/s (1772MB/s-1772MB/s), io=99.6GiB (107GB), run=60370-60370msec WRITE: bw=730MiB/s (765MB/s), 730MiB/s-730MiB/s (765MB/s-765MB/s), io=43.0GiB (46.2GB), run=60370-60370msec Disk stats (read/write): sda: ios=203978/88052, merge=204072/88171, ticks=4506422/5649263, in_queue=10200750, util=99.95% TEST-1024K RandomRW: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64 ... fio-3.7 Starting 5 processes RandomRW: (groupid=0, jobs=5): err= 0: pid=3360: Tue Jan 21 15:20:24 2020 read: IOPS=1891, BW=1892MiB/s (1984MB/s)(111GiB/60117msec) slat (usec): min=48, max=5640.1k, avg=1822.64, stdev=58897.46 clat (msec): min=8, max=7627, avg=101.41, stdev=467.78 lat (msec): min=9, max=7627, avg=103.24, stdev=472.66 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 32], 20.00th=[ 35], | 30.00th=[ 39], 40.00th=[ 41], 50.00th=[ 44], 60.00th=[ 46], | 70.00th=[ 50], 80.00th=[ 54], 90.00th=[ 61], 95.00th=[ 82], | 99.00th=[ 2265], 99.50th=[ 3910], 99.90th=[ 7617], 99.95th=[ 7617], | 99.99th=[ 7617] bw ( KiB/s): min= 8192, max=1169408, per=33.30%, avg=645094.38, stdev=309332.25, samples=360 iops : min= 8, max= 1142, avg=629.84, stdev=302.08, samples=360 write: IOPS=816, BW=817MiB/s (856MB/s)(47.9GiB/60117msec) slat (usec): min=75, max=3971.2k, avg=1848.78, stdev=46192.47 clat (msec): min=10, max=7860, avg=150.35, stdev=568.60 lat (msec): min=10, max=7860, avg=152.20, stdev=571.68 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 43], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 53], | 70.00th=[ 58], 80.00th=[ 70], 90.00th=[ 201], 95.00th=[ 355], | 99.00th=[ 2534], 99.50th=[ 4665], 99.90th=[ 7617], 99.95th=[ 7684], | 99.99th=[ 7752] bw ( KiB/s): min= 4079, max=485376, per=33.28%, avg=278258.29, stdev=133082.61, samples=360 iops : min= 3, max= 474, avg=271.61, stdev=129.98, samples=360 lat (msec) : 10=0.02%, 20=0.20%, 50=66.23%, 100=25.23%, 250=3.91% lat (msec) : 500=1.77%, 750=1.00%, 1000=0.17% cpu : usr=1.75%, sys=7.99%, ctx=22136, majf=0, minf=163 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=113725,49089,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=1892MiB/s (1984MB/s), 1892MiB/s-1892MiB/s (1984MB/s-1984MB/s), io=111GiB (119GB), run=60117-60117msec WRITE: bw=817MiB/s (856MB/s), 817MiB/s-817MiB/s (856MB/s-856MB/s), io=47.9GiB (51.5GB), run=60117-60117msec Disk stats (read/write): sda: ios=341189/147270, merge=0/150, ticks=4190913/5651241, in_queue=9850647, util=99.85%