start test

TEST-4K

RandomRW: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=2836: Thu Jan 23 16:40:19 2020
   read: IOPS=1919, BW=7678KiB/s (7862kB/s)(451MiB/60105msec)
    slat (usec): min=3, max=174293, avg=1766.03, stdev=13030.76
    clat (usec): min=995, max=368487, avg=114730.45, stdev=96464.88
     lat (usec): min=1001, max=368496, avg=116496.67, stdev=97255.26
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[   13], 10.00th=[   15], 20.00th=[   18],
     | 30.00th=[   22], 40.00th=[   27], 50.00th=[   48], 60.00th=[  213],
     | 70.00th=[  213], 80.00th=[  213], 90.00th=[  215], 95.00th=[  234],
     | 99.00th=[  239], 99.50th=[  259], 99.90th=[  330], 99.95th=[  368],
     | 99.99th=[  368]
   bw (  KiB/s): min=  488, max=10552, per=19.99%, avg=1534.87, stdev=2204.69, samples=600
   iops        : min=  122, max= 2638, avg=383.68, stdev=551.17, samples=600
  write: IOPS=828, BW=3315KiB/s (3394kB/s)(195MiB/60105msec)
    slat (usec): min=3, max=172529, avg=1889.42, stdev=13491.76
    clat (usec): min=1085, max=368488, avg=114195.22, stdev=96458.37
     lat (usec): min=1095, max=368494, avg=116084.83, stdev=97218.43
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[   12], 10.00th=[   15], 20.00th=[   18],
     | 30.00th=[   22], 40.00th=[   26], 50.00th=[   42], 60.00th=[  197],
     | 70.00th=[  213], 80.00th=[  213], 90.00th=[  215], 95.00th=[  234],
     | 99.00th=[  236], 99.50th=[  255], 99.90th=[  330], 99.95th=[  368],
     | 99.99th=[  368]
   bw (  KiB/s): min=  168, max= 4464, per=20.00%, avg=662.71, stdev=954.52, samples=600
   iops        : min=   42, max= 1116, avg=165.62, stdev=238.64, samples=600
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.05%, 4=0.09%, 10=2.81%, 20=23.53%, 50=23.59%
  lat (msec)   : 100=0.46%, 250=48.61%, 500=0.86%
  cpu          : usr=0.18%, sys=0.57%, ctx=7701, majf=0, minf=157
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=115369,49805,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=7678KiB/s (7862kB/s), 7678KiB/s-7678KiB/s (7862kB/s-7862kB/s), io=451MiB (473MB), run=60105-60105msec
  WRITE: bw=3315KiB/s (3394kB/s), 3315KiB/s-3315KiB/s (3394kB/s-3394kB/s), io=195MiB (204MB), run=60105-60105msec

Disk stats (read/write):
  sda: ios=115154/49732, merge=0/1, ticks=6194728/2669390, in_queue=8875780, util=100.00%

TEST-8K
RandomRW: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=2860: Thu Jan 23 16:41:20 2020
   read: IOPS=1055, BW=8448KiB/s (8650kB/s)(496MiB/60108msec)
    slat (usec): min=3, max=197147, avg=3258.67, stdev=18355.34
    clat (msec): min=2, max=391, avg=208.53, stdev=31.89
     lat (msec): min=2, max=391, avg=211.79, stdev=27.58
    clat percentiles (msec):
     |  1.00th=[   87],  5.00th=[  171], 10.00th=[  192], 20.00th=[  213],
     | 30.00th=[  213], 40.00th=[  213], 50.00th=[  213], 60.00th=[  213],
     | 70.00th=[  215], 80.00th=[  215], 90.00th=[  234], 95.00th=[  234],
     | 99.00th=[  264], 99.50th=[  305], 99.90th=[  376], 99.95th=[  393],
     | 99.99th=[  393]
   bw (  KiB/s): min= 1024, max= 3912, per=19.97%, avg=1686.53, stdev=249.53, samples=600
   iops        : min=  128, max=  489, avg=210.79, stdev=31.19, samples=600
  write: IOPS=455, BW=3642KiB/s (3730kB/s)(214MiB/60108msec)
    slat (usec): min=4, max=195073, avg=3380.54, stdev=18670.77
    clat (usec): min=1508, max=391461, avg=208177.34, stdev=32941.34
     lat (usec): min=1577, max=391468, avg=211558.04, stdev=27974.35
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[  153], 10.00th=[  192], 20.00th=[  209],
     | 30.00th=[  213], 40.00th=[  213], 50.00th=[  213], 60.00th=[  213],
     | 70.00th=[  215], 80.00th=[  215], 90.00th=[  234], 95.00th=[  234],
     | 99.00th=[  279], 99.50th=[  305], 99.90th=[  388], 99.95th=[  393],
     | 99.99th=[  393]
   bw (  KiB/s): min=  383, max= 1808, per=19.96%, avg=726.92, stdev=136.93, samples=600
   iops        : min=   47, max=  226, avg=90.82, stdev=17.11, samples=600
  lat (msec)   : 2=0.01%, 4=0.05%, 10=0.55%, 20=0.23%, 50=0.06%
  lat (msec)   : 100=0.46%, 250=97.00%, 500=1.66%
  cpu          : usr=0.08%, sys=0.29%, ctx=3051, majf=0, minf=160
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=63472,27366,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=8448KiB/s (8650kB/s), 8448KiB/s-8448KiB/s (8650kB/s-8650kB/s), io=496MiB (520MB), run=60108-60108msec
  WRITE: bw=3642KiB/s (3730kB/s), 3642KiB/s-3642KiB/s (3730kB/s-3730kB/s), io=214MiB (224MB), run=60108-60108msec

Disk stats (read/write):
  sda: ios=63393/27274, merge=0/1, ticks=6144102/2645109, in_queue=8798446, util=99.97%

TEST-16K

RandomRW: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=3106: Thu Jan 23 16:59:26 2020
   read: IOPS=1919, BW=29.0MiB/s (31.5MB/s)(1802MiB/60091msec)
    slat (usec): min=4, max=177683, avg=1818.08, stdev=11984.49
    clat (usec): min=510, max=370747, avg=97724.45, stdev=86414.27
     lat (usec): min=518, max=370753, avg=99542.69, stdev=87033.36
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[   24], 10.00th=[   26], 20.00th=[   29],
     | 30.00th=[   32], 40.00th=[   34], 50.00th=[   37], 60.00th=[   61],
     | 70.00th=[  213], 80.00th=[  213], 90.00th=[  213], 95.00th=[  215],
     | 99.00th=[  234], 99.50th=[  236], 99.90th=[  300], 99.95th=[  347],
     | 99.99th=[  372]
   bw (  KiB/s): min= 1792, max=26080, per=19.99%, avg=6139.10, stdev=4083.25, samples=600
   iops        : min=  112, max= 1630, avg=383.64, stdev=255.20, samples=600
  write: IOPS=828, BW=12.9MiB/s (13.6MB/s)(778MiB/60091msec)
    slat (usec): min=4, max=129261, avg=1799.48, stdev=12012.51
    clat (usec): min=751, max=370755, avg=153619.29, stdev=48904.76
     lat (usec): min=764, max=370761, avg=155418.93, stdev=48847.54
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[  100], 10.00th=[  109], 20.00th=[  118],
     | 30.00th=[  124], 40.00th=[  129], 50.00th=[  134], 60.00th=[  144],
     | 70.00th=[  213], 80.00th=[  213], 90.00th=[  213], 95.00th=[  215],
     | 99.00th=[  234], 99.50th=[  236], 99.90th=[  300], 99.95th=[  347],
     | 99.99th=[  372]
   bw (  KiB/s): min=  736, max=10848, per=19.99%, avg=2648.64, stdev=1751.08, samples=600
   iops        : min=   46, max=  678, avg=165.49, stdev=109.46, samples=600
  lat (usec)   : 750=0.02%, 1000=0.05%
  lat (msec)   : 2=0.06%, 4=0.54%, 10=1.09%, 20=0.64%, 50=37.99%
  lat (msec)   : 100=5.74%, 250=53.44%, 500=0.43%
  cpu          : usr=0.18%, sys=0.55%, ctx=5526, majf=0, minf=153
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=115359,49762,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=29.0MiB/s (31.5MB/s), 29.0MiB/s-29.0MiB/s (31.5MB/s-31.5MB/s), io=1802MiB (1890MB), run=60091-60091msec
  WRITE: bw=12.9MiB/s (13.6MB/s), 12.9MiB/s-12.9MiB/s (13.6MB/s-13.6MB/s), io=778MiB (815MB), run=60091-60091msec

Disk stats (read/write):
  sda: ios=115187/49717, merge=0/2, ticks=4138965/4426169, in_queue=8576827, util=99.98%

TEST-32K

RandomRW: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=3401: Thu Jan 23 17:17:45 2020
   read: IOPS=1312, BW=41.0MiB/s (43.0MB/s)(2469MiB/60187msec)
    slat (usec): min=5, max=170053, avg=2686.46, stdev=16291.94
    clat (usec): min=586, max=660821, avg=145979.72, stdev=131702.29
     lat (usec): min=596, max=660878, avg=148666.32, stdev=133230.44
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   51], 10.00th=[   54], 20.00th=[   59],
     | 30.00th=[   62], 40.00th=[   65], 50.00th=[   69], 60.00th=[   75],
     | 70.00th=[  130], 80.00th=[  292], 90.00th=[  414], 95.00th=[  439],
     | 99.00th=[  451], 99.50th=[  460], 99.90th=[  518], 99.95th=[  527],
     | 99.99th=[  575]
   bw (  KiB/s): min= 3584, max=29632, per=19.99%, avg=8396.56, stdev=3628.66, samples=600
   iops        : min=  112, max=  926, avg=262.35, stdev=113.41, samples=600
  write: IOPS=567, BW=17.7MiB/s (18.6MB/s)(1067MiB/60187msec)
    slat (usec): min=7, max=262716, avg=2563.85, stdev=15976.06
    clat (usec): min=551, max=394356, avg=217004.75, stdev=56750.86
     lat (usec): min=564, max=525202, avg=219568.73, stdev=57811.92
    clat percentiles (msec):
     |  1.00th=[   21],  5.00th=[  136], 10.00th=[  142], 20.00th=[  148],
     | 30.00th=[  186], 40.00th=[  232], 50.00th=[  241], 60.00th=[  247],
     | 70.00th=[  255], 80.00th=[  262], 90.00th=[  271], 95.00th=[  279],
     | 99.00th=[  309], 99.50th=[  317], 99.90th=[  338], 99.95th=[  376],
     | 99.99th=[  384]
   bw (  KiB/s): min=  512, max=10944, per=20.03%, avg=3636.65, stdev=1541.85, samples=600
   iops        : min=   16, max=  342, avg=113.58, stdev=48.19, samples=600
  lat (usec)   : 750=0.01%, 1000=0.02%
  lat (msec)   : 2=0.02%, 4=0.13%, 10=0.74%, 20=0.21%, 50=2.54%
  lat (msec)   : 100=41.34%, 250=25.10%, 500=29.81%, 750=0.07%
  cpu          : usr=0.12%, sys=0.42%, ctx=3780, majf=0, minf=157
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=79004,34148,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=41.0MiB/s (43.0MB/s), 41.0MiB/s-41.0MiB/s (43.0MB/s-43.0MB/s), io=2469MiB (2589MB), run=60187-60187msec
  WRITE: bw=17.7MiB/s (18.6MB/s), 17.7MiB/s-17.7MiB/s (18.6MB/s-18.6MB/s), io=1067MiB (1119MB), run=60187-60187msec

Disk stats (read/write):
  sda: ios=78838/34153, merge=0/1, ticks=4005560/4184942, in_queue=8203282, util=99.96%

TEST-64K

RandomRW: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=3636: Thu Jan 23 17:36:16 2020
   read: IOPS=655, BW=40.0MiB/s (42.0MB/s)(2473MiB/60384msec)
   RandomRW: (groupid=0, jobs=5): err= 0: pid=3636: Thu Jan 23 17:36:16 2020
   read: IOPS=655, BW=40.0MiB/s (42.0MB/s)(2473MiB/60384msec)
    slat (usec): min=7, max=338115, avg=5342.61, stdev=32655.95
    clat (usec): min=1055, max=1200.1k, avg=289948.52, stdev=261023.76
     lat (usec): min=1097, max=1200.1k, avg=295291.28, stdev=264156.44
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[  105], 10.00th=[  110], 20.00th=[  121],
     | 30.00th=[  126], 40.00th=[  131], 50.00th=[  136], 60.00th=[  148],
     | 70.00th=[  259], 80.00th=[  584], 90.00th=[  818], 95.00th=[  877],
     | 99.00th=[  911], 99.50th=[  919], 99.90th=[  953], 99.95th=[ 1070],
     | 99.99th=[ 1200]
   bw (  KiB/s): min= 2299, max=30976, per=19.99%, avg=8386.00, stdev=3828.89, samples=600
   iops        : min=   35, max=  484, avg=130.97, stdev=59.85, samples=600
  write: IOPS=285, BW=17.8MiB/s (18.7MB/s)(1076MiB/60384msec)
    slat (usec): min=8, max=337837, avg=5148.59, stdev=31569.54
    clat (msec): min=2, max=694, avg=437.82, stdev=111.42
     lat (msec): min=2, max=743, avg=442.97, stdev=113.61
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[  275], 10.00th=[  284], 20.00th=[  300],
     | 30.00th=[  384], 40.00th=[  472], 50.00th=[  489], 60.00th=[  502],
     | 70.00th=[  510], 80.00th=[  523], 90.00th=[  535], 95.00th=[  550],
     | 99.00th=[  609], 99.50th=[  634], 99.90th=[  667], 99.95th=[  684],
     | 99.99th=[  693]
   bw (  KiB/s): min=  640, max= 9856, per=20.06%, avg=3658.10, stdev=1614.09, samples=600
   iops        : min=   10, max=  154, avg=57.09, stdev=25.24, samples=600
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.29%, 20=0.33%, 50=0.63%
  lat (msec)   : 100=1.97%, 250=44.93%, 500=20.95%, 750=23.75%, 1000=7.06%
  cpu          : usr=0.07%, sys=0.29%, ctx=1936, majf=0, minf=150
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.3%, >=64=99.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=39575,17210,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=40.0MiB/s (42.0MB/s), 40.0MiB/s-40.0MiB/s (42.0MB/s-42.0MB/s), io=2473MiB (2594MB), run=60384-60384msec
  WRITE: bw=17.8MiB/s (18.7MB/s), 17.8MiB/s-17.8MiB/s (18.7MB/s-18.7MB/s), io=1076MiB (1128MB), run=60384-60384msec

Disk stats (read/write):
  sda: ios=39500/17222, merge=0/1, ticks=3926641/4265416, in_queue=8212361, util=99.92%

TEST-128K

RandomRW: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=3883: Thu Jan 23 17:54:48 2020
   read: IOPS=327, BW=40.9MiB/s (42.9MB/s)(2481MiB/60697msec)
    slat (usec): min=13, max=650243, avg=10275.47, stdev=62969.68
    clat (msec): min=6, max=2404, avg=578.47, stdev=521.82
     lat (msec): min=7, max=2404, avg=588.74, stdev=527.05
  clat (msec): min=6, max=2404, avg=578.47, stdev=521.82
     lat (msec): min=7, max=2404, avg=588.74, stdev=527.05
    clat percentiles (msec):
     |  1.00th=[   68],  5.00th=[  199], 10.00th=[  213], 20.00th=[  234],
     | 30.00th=[  251], 40.00th=[  262], 50.00th=[  275], 60.00th=[  305],
     | 70.00th=[  527], 80.00th=[ 1150], 90.00th=[ 1636], 95.00th=[ 1737],
     | 99.00th=[ 1804], 99.50th=[ 1854], 99.90th=[ 2198], 99.95th=[ 2299],
     | 99.99th=[ 2400]
   bw (  KiB/s): min=  256, max=42240, per=21.07%, avg=8820.30, stdev=4172.39, samples=568
   iops        : min=    2, max=  330, avg=68.83, stdev=32.59, samples=568
  write: IOPS=143, BW=17.9MiB/s (18.8MB/s)(1086MiB/60697msec)
    slat (usec): min=18, max=650224, avg=11056.32, stdev=67654.62
    clat (msec): min=3, max=1404, avg=877.65, stdev=235.26
     lat (msec): min=3, max=1633, avg=888.70, stdev=238.48
    clat percentiles (msec):
     |  1.00th=[   73],  5.00th=[  535], 10.00th=[  558], 20.00th=[  592],
     | 30.00th=[  760], 40.00th=[  936], 50.00th=[  969], 60.00th=[ 1003],
     | 70.00th=[ 1028], 80.00th=[ 1062], 90.00th=[ 1099], 95.00th=[ 1167],
     | 99.00th=[ 1267], 99.50th=[ 1301], 99.90th=[ 1351], 99.95th=[ 1351],
     | 99.99th=[ 1401]
   bw (  KiB/s): min= 1024, max=11008, per=21.50%, avg=3937.55, stdev=1632.81, samples=562
   iops        : min=    8, max=   86, avg=30.67, stdev=12.75, samples=562
  lat (msec)   : 4=0.01%, 10=0.07%, 20=0.14%, 50=0.46%, 100=1.04%
  lat (msec)   : 250=20.66%, 500=26.18%, 750=10.50%, 1000=10.37%
  cpu          : usr=0.04%, sys=0.21%, ctx=1052, majf=0, minf=151
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=0.6%, >=64=98.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=19849,8686,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=40.9MiB/s (42.9MB/s), 40.9MiB/s-40.9MiB/s (42.9MB/s-42.9MB/s), io=2481MiB (2602MB), run=60697-60697msec
  WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=1086MiB (1138MB), run=60697-60697msec

Disk stats (read/write):
  sda: ios=19829/8697, merge=0/0, ticks=3955599/4252682, in_queue=8224574, util=99.97%

TEST-256K

RandomRW: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=4169: Thu Jan 23 18:13:21 2020
   read: IOPS=162, BW=40.6MiB/s (42.6MB/s)(2498MiB/61446msec)
    slat (usec): min=19, max=1292.1k, avg=20906.80, stdev=129186.49
    clat (msec): min=3, max=4487, avg=1177.71, stdev=1041.83
     lat (msec): min=4, max=4487, avg=1198.61, stdev=1054.37
    clat percentiles (msec):
     |  1.00th=[  112],  5.00th=[  372], 10.00th=[  430], 20.00th=[  464],
     | 30.00th=[  493], 40.00th=[  518], 50.00th=[  567], 60.00th=[  659],
     | 70.00th=[ 1183], 80.00th=[ 2333], 90.00th=[ 3004], 95.00th=[ 3507],
     | 99.00th=[ 3608], 99.50th=[ 3675], 99.90th=[ 4463], 99.95th=[ 4463],
     | 99.99th=[ 4463]
   bw (  KiB/s): min=  510, max=54784, per=28.29%, avg=11774.57, stdev=4165.36, samples=423
   iops        : min=    1, max=  214, avg=45.94, stdev=16.28, samples=423
  write: IOPS=70, BW=17.6MiB/s (18.5MB/s)(1083MiB/61446msec)
    slat (usec): min=30, max=1281.0k, avg=21121.53, stdev=129222.91
    clat (msec): min=6, max=2760, avg=1733.02, stdev=466.09
     lat (msec): min=7, max=3382, avg=1754.14, stdev=475.82
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[ 1083], 10.00th=[ 1116], 20.00th=[ 1183],
     | 30.00th=[ 1485], 40.00th=[ 1821], 50.00th=[ 1905], 60.00th=[ 1989],
     | 70.00th=[ 2056], 80.00th=[ 2123], 90.00th=[ 2198], 95.00th=[ 2232],
     | 99.00th=[ 2433], 99.50th=[ 2500], 99.90th=[ 2668], 99.95th=[ 2769],
     | 99.99th=[ 2769]
   bw (  KiB/s): min=  512, max=16384, per=29.25%, avg=5279.84, stdev=1906.55, samples=415
   iops        : min=    2, max=   64, avg=20.57, stdev= 7.49, samples=415
  lat (msec)   : 4=0.01%, 10=0.06%, 20=0.03%, 50=0.35%, 100=0.54%
  lat (msec)   : 250=1.36%, 500=20.56%, 750=20.44%, 1000=4.05%
  cpu          : usr=0.04%, sys=0.15%, ctx=605, majf=0, minf=151
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.6%, 32=1.1%, >=64=97.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=9991,4333,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=40.6MiB/s (42.6MB/s), 40.6MiB/s-40.6MiB/s (42.6MB/s-42.6MB/s), io=2498MiB (2619MB), run=61446-61446msec
  WRITE: bw=17.6MiB/s (18.5MB/s), 17.6MiB/s-17.6MiB/s (18.5MB/s-18.5MB/s), io=1083MiB (1136MB), run=61446-61446msec

Disk stats (read/write):
  sda: ios=10072/4355, merge=0/5, ticks=4447297/4274648, in_queue=8733005, util=99.98%

TEST-512K

RandomRW: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=4410: Thu Jan 23 18:31:53 2020
   read: IOPS=81, BW=40.6MiB/s (42.6MB/s)(2503MiB/61588msec)
    slat (usec): min=47, max=1331.8k, avg=41757.20, stdev=178068.20
    clat (msec): min=73, max=7318, avg=2476.33, stdev=1535.10
     lat (msec): min=73, max=7318, avg=2518.09, stdev=1555.77
    clat percentiles (msec):
     |  1.00th=[  284],  5.00th=[ 1116], 10.00th=[ 1334], 20.00th=[ 1435],
     | 30.00th=[ 1519], 40.00th=[ 1586], 50.00th=[ 1670], 60.00th=[ 1787],
     | 70.00th=[ 2299], 80.00th=[ 4463], 90.00th=[ 4732], 95.00th=[ 5671],
     | 99.00th=[ 5873], 99.50th=[ 6007], 99.90th=[ 6611], 99.95th=[ 6611],
     | 99.99th=[ 7349]
   | 99.00th=[ 5873], 99.50th=[ 6007], 99.90th=[ 6611], 99.95th=[ 6611],
     | 99.99th=[ 7349]
   bw (  KiB/s): min= 1024, max=47104, per=28.09%, avg=11691.49, stdev=3855.78, samples=417
   iops        : min=    2, max=   92, avg=22.75, stdev= 7.51, samples=417
  write: IOPS=35, BW=17.9MiB/s (18.8MB/s)(1104MiB/61588msec)
    slat (usec): min=67, max=1300.7k, avg=41739.60, stdev=181458.17
    clat (msec): min=73, max=6055, avg=3097.34, stdev=572.21
     lat (msec): min=73, max=6055, avg=3139.08, stdev=597.41
    clat percentiles (msec):
     |  1.00th=[  279],  5.00th=[ 2400], 10.00th=[ 2567], 20.00th=[ 2769],
     | 30.00th=[ 2937], 40.00th=[ 3037], 50.00th=[ 3138], 60.00th=[ 3272],
     | 70.00th=[ 3339], 80.00th=[ 3473], 90.00th=[ 3574], 95.00th=[ 3842],
     | 99.00th=[ 4396], 99.50th=[ 4463], 99.90th=[ 4933], 99.95th=[ 4933],
     | 99.99th=[ 6074]
   bw (  KiB/s): min= 1021, max=15360, per=29.58%, avg=5426.52, stdev=2236.82, samples=403
   iops        : min=    1, max=   30, avg=10.52, stdev= 4.39, samples=403
  lat (msec)   : 100=0.24%, 250=0.53%, 500=0.58%, 750=0.33%, 1000=1.37%
  cpu          : usr=0.04%, sys=0.15%, ctx=528, majf=0, minf=147
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.6%, 16=1.1%, 32=2.2%, >=64=95.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=5006,2207,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=40.6MiB/s (42.6MB/s), 40.6MiB/s-40.6MiB/s (42.6MB/s-42.6MB/s), io=2503MiB (2625MB), run=61588-61588msec
  WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=1104MiB (1157MB), run=61588-61588msec

Disk stats (read/write):
  sda: ios=10005/4436, merge=0/1, ticks=4396573/4321493, in_queue=8731916, util=99.93%

TEST-1024K

RandomRW: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=4649: Thu Jan 23 18:50:26 2020
   read: IOPS=40, BW=40.7MiB/s (42.7MB/s)(2499MiB/61417msec)
    slat (usec): min=95, max=1445.0k, avg=82302.58, stdev=248240.48
    clat (msec): min=260, max=10875, avg=4985.46, stdev=2476.33
     lat (msec): min=260, max=11743, avg=5067.76, stdev=2517.73
    clat percentiles (msec):
     |  1.00th=[ 1435],  5.00th=[ 2769], 10.00th=[ 2869], 20.00th=[ 3272],
     | 30.00th=[ 3608], 40.00th=[ 3742], 50.00th=[ 3842], 60.00th=[ 4010],
     | 70.00th=[ 4799], 80.00th=[ 8490], 90.00th=[ 9329], 95.00th=[ 9597],
     | 99.00th=[10537], 99.50th=[10537], 99.90th=[10805], 99.95th=[10805],
     | 99.99th=[10939]
   bw (  KiB/s): min= 2048, max=30658, per=28.97%, avg=12068.99, stdev=4264.22, samples=387
   iops        : min=    2, max=   29, avg=11.71, stdev= 4.16, samples=387
  write: IOPS=18, BW=18.3MiB/s (19.2MB/s)(1122MiB/61417msec)
    slat (usec): min=134, max=1347.3k, avg=84360.62, stdev=241312.10
 write: IOPS=18, BW=18.3MiB/s (19.2MB/s)(1122MiB/61417msec)
    slat (usec): min=134, max=1347.3k, avg=84360.62, stdev=241312.10
    clat (msec): min=151, max=9418, avg=5762.86, stdev=1723.25
     lat (msec): min=151, max=9722, avg=5847.22, stdev=1742.55
    clat percentiles (msec):
     |  1.00th=[ 1183],  5.00th=[ 3742], 10.00th=[ 4279], 20.00th=[ 4597],
     | 30.00th=[ 4866], 40.00th=[ 5067], 50.00th=[ 5269], 60.00th=[ 5470],
     | 70.00th=[ 6477], 80.00th=[ 7953], 90.00th=[ 8221], 95.00th=[ 8490],
     | 99.00th=[ 9060], 99.50th=[ 9194], 99.90th=[ 9329], 99.95th=[ 9463],
     | 99.99th=[ 9463]
   bw (  KiB/s): min= 2043, max=16384, per=30.96%, avg=5792.37, stdev=2846.25, samples=362
   iops        : min=    1, max=   16, avg= 5.59, stdev= 2.79, samples=362
  lat (msec)   : 250=0.08%, 500=0.25%, 750=0.25%
  cpu          : usr=0.04%, sys=0.14%, ctx=503, majf=0, minf=146
  IO depths    : 1=0.1%, 2=0.3%, 4=0.6%, 8=1.1%, 16=2.2%, 32=4.4%, >=64=91.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.2%, >=64=0.0%
     issued rwts: total=2499,1122,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=40.7MiB/s (42.7MB/s), 40.7MiB/s-40.7MiB/s (42.7MB/s-42.7MB/s), io=2499MiB (2620MB), run=61417-61417msec
  WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=1122MiB (1177MB), run=61417-61417msec

Disk stats (read/write):
  sda: ios=10004/4513, merge=0/1, ticks=4268656/4481065, in_queue=8757499, util=99.83%