start test

TEST-4K

RandomRW: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=26317: Tue Jan 28 19:14:13 2020
   read: IOPS=122k, BW=475MiB/s (499MB/s)(27.9GiB/60001msec)
    slat (usec): min=2, max=34876, avg=25.70, stdev=124.31
    clat (usec): min=191, max=69726, avg=1868.98, stdev=1070.24
     lat (usec): min=196, max=69735, avg=1894.86, stdev=1078.61
    clat percentiles (usec):
     |  1.00th=[  750],  5.00th=[  898], 10.00th=[ 1004], 20.00th=[ 1188],
     | 30.00th=[ 1352], 40.00th=[ 1516], 50.00th=[ 1663], 60.00th=[ 1827],
     | 70.00th=[ 2040], 80.00th=[ 2343], 90.00th=[ 2868], 95.00th=[ 3458],
     | 99.00th=[ 5211], 99.50th=[ 6652], 99.90th=[12518], 99.95th=[14222],
     | 99.99th=[23725]
   bw (  KiB/s): min=57160, max=169776, per=19.98%, avg=97272.71, stdev=16764.93, samples=600
   iops        : min=14290, max=42444, avg=24318.09, stdev=4191.23, samples=600
  write: IOPS=52.2k, BW=204MiB/s (214MB/s)(11.9GiB/60001msec)
    slat (usec): min=2, max=35314, avg=26.49, stdev=129.60
    clat (usec): min=125, max=62184, avg=1680.96, stdev=1034.39
     lat (usec): min=135, max=62195, avg=1707.63, stdev=1043.70
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[  816], 10.00th=[  906], 20.00th=[ 1057],
     | 30.00th=[ 1221], 40.00th=[ 1369], 50.00th=[ 1516], 60.00th=[ 1663],
     | 70.00th=[ 1844], 80.00th=[ 2089], 90.00th=[ 2507], 95.00th=[ 2999],
     | 99.00th=[ 4686], 99.50th=[ 6063], 99.90th=[12125], 99.95th=[14091],
     | 99.99th=[32637]
   bw (  KiB/s): min=24103, max=71800, per=19.98%, avg=41711.31, stdev=7197.03, samples=600
   iops        : min= 6025, max=17950, avg=10427.73, stdev=1799.27, samples=600
  lat (usec)   : 250=0.01%, 500=0.04%, 750=1.52%, 1000=10.05%
  lat (msec)   : 2=59.49%, 4=26.54%, 10=2.09%, 20=0.24%, 50=0.03%
  lat (msec)   : 100=0.01%
  cpu          : usr=8.49%, sys=50.95%, ctx=250062, majf=0, minf=153
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=7303417,3131764,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=475MiB/s (499MB/s), 475MiB/s-475MiB/s (499MB/s-499MB/s), io=27.9GiB (29.9GB), run=60001-60001msec
  WRITE: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=11.9GiB (12.8GB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=7291028/3126566, merge=0/7, ticks=5535839/1862727, in_queue=7402774, util=100.00%

TEST-8K

RandomRW: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=1457: Tue Jan 28 19:15:14 2020
   read: IOPS=127k, BW=994MiB/s (1043MB/s)(58.3GiB/60002msec)
    slat (usec): min=2, max=80726, avg=22.52, stdev=278.05
    clat (usec): min=113, max=81647, avg=1753.20, stdev=2506.72
     lat (usec): min=127, max=81653, avg=1775.91, stdev=2520.94
    clat percentiles (usec):
     |  1.00th=[  302],  5.00th=[  429], 10.00th=[  523], 20.00th=[  644],
     | 30.00th=[  791], 40.00th=[  988], 50.00th=[ 1221], 60.00th=[ 1549],
     | 70.00th=[ 1893], 80.00th=[ 2278], 90.00th=[ 2802], 95.00th=[ 3490],
     | 99.00th=[12780], 99.50th=[22676], 99.90th=[26084], 99.95th=[31589],
     | 99.99th=[45876]
   bw (  KiB/s): min=44679, max=348576, per=19.94%, avg=203040.81, stdev=36242.36, samples=600
   iops        : min= 5584, max=43572, avg=25379.91, stdev=4530.33, samples=600
  write: IOPS=54.6k, BW=426MiB/s (447MB/s)(24.0GiB/60002msec)
    slat (usec): min=2, max=55036, avg=22.82, stdev=274.02
    clat (usec): min=82, max=81362, avg=1693.54, stdev=2423.89
     lat (usec): min=91, max=81367, avg=1716.53, stdev=2438.45
    clat percentiles (usec):
     |  1.00th=[  265],  5.00th=[  388], 10.00th=[  474], 20.00th=[  586],
     | 30.00th=[  734], 40.00th=[  947], 50.00th=[ 1172], 60.00th=[ 1500],
     | 70.00th=[ 1860], 80.00th=[ 2245], 90.00th=[ 2835], 95.00th=[ 3425],
     | 99.00th=[12256], 99.50th=[22676], 99.90th=[25035], 99.95th=[31327],
     | 99.99th=[45351]
   bw (  KiB/s): min=20142, max=149744, per=19.94%, avg=87064.45, stdev=15525.38, samples=600
   iops        : min= 2517, max=18718, avg=10882.86, stdev=1940.72, samples=600
  lat (usec)   : 100=0.01%, 250=0.43%, 500=9.17%, 750=18.74%, 1000=12.90%
  lat (msec)   : 2=31.94%, 4=23.25%, 10=1.84%, 20=1.01%, 50=0.71%
  lat (msec)   : 100=0.01%
  cpu          : usr=6.35%, sys=29.39%, ctx=212161, majf=0, minf=153
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=7637112,3274816,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=994MiB/s (1043MB/s), 994MiB/s-994MiB/s (1043MB/s-1043MB/s), io=58.3GiB (62.6GB), run=60002-60002msec
  WRITE: bw=426MiB/s (447MB/s), 426MiB/s-426MiB/s (447MB/s-447MB/s), io=24.0GiB (26.8GB), run=60002-60002msec

Disk stats (read/write):
  sda: ios=7607247/3261768, merge=0/8, ticks=4578695/1913196, in_queue=6491280, util=100.00%

TEST-16K

RandomRW: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=1496: Tue Jan 28 19:16:48 2020
   read: IOPS=86.4k, BW=1350MiB/s (1416MB/s)(79.1GiB/60003msec)
    slat (usec): min=3, max=72824, avg=36.12, stdev=242.07
    clat (usec): min=162, max=74151, avg=2551.90, stdev=1911.78
     lat (usec): min=167, max=74159, avg=2588.22, stdev=1925.19
    clat percentiles (usec):
     |  1.00th=[  519],  5.00th=[  865], 10.00th=[ 1156], 20.00th=[ 1401],
     | 30.00th=[ 1713], 40.00th=[ 2089], 50.00th=[ 2343], 60.00th=[ 2606],
     | 70.00th=[ 2933], 80.00th=[ 3359], 90.00th=[ 3851], 95.00th=[ 4359],
     | 99.00th=[ 8455], 99.50th=[12911], 99.90th=[23987], 99.95th=[25560],
     | 99.99th=[44303]
   bw (  KiB/s): min=120382, max=398304, per=19.97%, avg=276066.99, stdev=35902.96, samples=600
   iops        : min= 7523, max=24894, avg=17254.06, stdev=2243.96, samples=600
  write: IOPS=37.1k, BW=579MiB/s (607MB/s)(33.9GiB/60003msec)
    slat (usec): min=3, max=50653, avg=36.65, stdev=237.72
    clat (usec): min=133, max=74004, avg=2559.70, stdev=1917.08
     lat (usec): min=140, max=74013, avg=2596.55, stdev=1929.76
    clat percentiles (usec):
     |  1.00th=[  457],  5.00th=[  783], 10.00th=[ 1106], 20.00th=[ 1369],
     | 30.00th=[ 1631], 40.00th=[ 2024], 50.00th=[ 2311], 60.00th=[ 2573],
     | 70.00th=[ 2999], 80.00th=[ 3490], 90.00th=[ 4015], 95.00th=[ 4621],
     | 99.00th=[ 8160], 99.50th=[12780], 99.90th=[23987], 99.95th=[24511],
     | 99.99th=[44303]
   bw (  KiB/s): min=49752, max=168736, per=19.97%, avg=118424.15, stdev=15386.47, samples=600
   iops        : min= 3109, max=10546, avg=7401.39, stdev=961.68, samples=600
  lat (usec)   : 250=0.01%, 500=1.03%, 750=2.86%, 1000=3.37%
  lat (msec)   : 2=30.79%, 4=53.16%, 10=7.90%, 20=0.57%, 50=0.31%
  lat (msec)   : 100=0.01%
  cpu          : usr=5.45%, sys=25.62%, ctx=209444, majf=0, minf=152
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=5184086,2223782,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1350MiB/s (1416MB/s), 1350MiB/s-1350MiB/s (1416MB/s-1416MB/s), io=79.1GiB (84.9GB), run=60003-60003msec
  WRITE: bw=579MiB/s (607MB/s), 579MiB/s-579MiB/s (607MB/s-607MB/s), io=33.9GiB (36.4GB), run=60003-60003msec

Disk stats (read/write):
  sda: ios=5172690/2218887, merge=0/4, ticks=5635721/2499408, in_queue=8136330, util=100.00%

TEST-32K

RandomRW: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
Jobs: 5 (f=5): [m(5)][100.0%][r=1229MiB/s,w=531MiB/s][r=39.3k,w=17.0k IOPS][eta 00m:00s]
RandomRW: (groupid=0, jobs=5): err= 0: pid=17792: Tue Jan 28 19:18:20 2020
   read: IOPS=37.5k, BW=1171MiB/s (1228MB/s)(68.6GiB/60009msec)
    slat (usec): min=4, max=59200, avg=87.26, stdev=346.52
    clat (usec): min=343, max=138658, avg=5858.06, stdev=2366.94
     lat (usec): min=353, max=138674, avg=5945.65, stdev=2383.70
    clat percentiles (usec):
     |  1.00th=[ 1811],  5.00th=[ 2540], 10.00th=[ 3097], 20.00th=[ 3916],
     | 30.00th=[ 4490], 40.00th=[ 5080], 50.00th=[ 5669], 60.00th=[ 6259],
     | 70.00th=[ 6849], 80.00th=[ 7635], 90.00th=[ 8717], 95.00th=[ 9765],
     | 99.00th=[12256], 99.50th=[13566], 99.90th=[17957], 99.95th=[21365],
     | 99.99th=[33817]
   bw (  KiB/s): min=41708, max=295040, per=19.99%, avg=239775.02, stdev=21710.45, samples=600
   iops        : min= 1303, max= 9220, avg=7492.87, stdev=678.47, samples=600
  write: IOPS=16.1k, BW=503MiB/s (528MB/s)(29.5GiB/60009msec)
    slat (usec): min=4, max=58777, avg=91.29, stdev=362.29
    clat (usec): min=321, max=138596, avg=5878.57, stdev=2731.35
     lat (usec): min=332, max=138614, avg=5970.20, stdev=2752.19
    clat percentiles (usec):
     |  1.00th=[ 1778],  5.00th=[ 2507], 10.00th=[ 3097], 20.00th=[ 3916],
     | 30.00th=[ 4490], 40.00th=[ 5014], 50.00th=[ 5604], 60.00th=[ 6194],
     | 70.00th=[ 6783], 80.00th=[ 7570], 90.00th=[ 8848], 95.00th=[10028],
     | 99.00th=[12911], 99.50th=[14484], 99.90th=[21365], 99.95th=[26608],
     | 99.99th=[92799]
   bw (  KiB/s): min=16351, max=129280, per=19.99%, avg=103008.60, stdev=9554.69, samples=600
   iops        : min=  510, max= 4040, avg=3218.92, stdev=298.61, samples=600
  lat (usec)   : 500=0.01%, 750=0.03%, 1000=0.04%
  lat (msec)   : 2=1.66%, 4=19.62%, 10=74.15%, 20=4.42%, 50=0.07%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=5.37%, sys=26.12%, ctx=151299, majf=0, minf=153
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=2249009,966213,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1171MiB/s (1228MB/s), 1171MiB/s-1171MiB/s (1228MB/s-1228MB/s), io=68.6GiB (73.7GB), run=60009-60009msec
  WRITE: bw=503MiB/s (528MB/s), 503MiB/s-503MiB/s (528MB/s-528MB/s), io=29.5GiB (31.7GB), run=60009-60009msec

Disk stats (read/write):
  sda: ios=2245904/964864, merge=0/4, ticks=6341771/2708597, in_queue=9054601, util=100.00%

TEST-64K

RandomRW: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=1516: Tue Jan 28 19:19:52 2020
   read: IOPS=30.1k, BW=1883MiB/s (1975MB/s)(110GiB/60004msec)
    slat (usec): min=6, max=21103, avg=111.17, stdev=485.19
    clat (usec): min=522, max=30076, avg=7367.38, stdev=2418.16
     lat (usec): min=531, max=30091, avg=7478.77, stdev=2451.25
    clat percentiles (usec):
     |  1.00th=[ 2704],  5.00th=[ 3490], 10.00th=[ 4555], 20.00th=[ 5407],
     | 30.00th=[ 5997], 40.00th=[ 6587], 50.00th=[ 7177], 60.00th=[ 7767],
     | 70.00th=[ 8455], 80.00th=[ 9241], 90.00th=[10552], 95.00th=[11600],
     | 99.00th=[14091], 99.50th=[15139], 99.90th=[17433], 99.95th=[18482],
     | 99.99th=[20841]
   bw (  KiB/s): min=345856, max=445056, per=19.99%, avg=385490.81, stdev=12920.04, samples=599
   iops        : min= 5404, max= 6954, avg=6023.17, stdev=201.89, samples=599
  write: IOPS=12.0k, BW=810MiB/s (849MB/s)(47.5GiB/60004msec)
    slat (usec): min=7, max=14273, avg=115.61, stdev=489.21
    clat (usec): min=481, max=27481, avg=7179.65, stdev=2446.54
     lat (usec): min=499, max=27493, avg=7295.50, stdev=2480.40
    clat percentiles (usec):
     |  1.00th=[ 2573],  5.00th=[ 3294], 10.00th=[ 4146], 20.00th=[ 5211],
     | 30.00th=[ 5800], 40.00th=[ 6390], 50.00th=[ 7046], 60.00th=[ 7635],
     | 70.00th=[ 8225], 80.00th=[ 9110], 90.00th=[10421], 95.00th=[11469],
     | 99.00th=[13960], 99.50th=[15008], 99.90th=[17171], 99.95th=[18220],
     | 99.99th=[21103]
   bw (  KiB/s): min=143360, max=181524, per=19.99%, avg=165765.31, stdev=6360.61, samples=599
   iops        : min= 2240, max= 2836, avg=2589.94, stdev=99.38, samples=599
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.14%, 4=7.71%, 10=79.03%, 20=13.09%, 50=0.02%
  cpu          : usr=3.22%, sys=16.22%, ctx=111813, majf=0, minf=149
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=1808214,777603,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1883MiB/s (1975MB/s), 1883MiB/s-1883MiB/s (1975MB/s-1975MB/s), io=110GiB (119GB), run=60004-60004msec
  WRITE: bw=810MiB/s (849MB/s), 810MiB/s-810MiB/s (849MB/s-849MB/s), io=47.5GiB (50.0GB), run=60004-60004msec

Disk stats (read/write):
  sda: ios=1803133/775401, merge=10/4, ticks=6546480/2721042, in_queue=9268379, util=100.00%

TEST-128K

RandomRW: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=1528: Tue Jan 28 19:21:23 2020
   read: IOPS=15.5k, BW=1932MiB/s (2025MB/s)(113GiB/60011msec)
    slat (usec): min=16, max=14934, avg=216.42, stdev=949.61
    clat (usec): min=1926, max=47127, avg=14295.98, stdev=3769.53
     lat (usec): min=1952, max=47160, avg=14512.67, stdev=3795.95
    clat percentiles (usec):
     |  1.00th=[ 7046],  5.00th=[ 9503], 10.00th=[10159], 20.00th=[11207],
     | 30.00th=[11863], 40.00th=[12780], 50.00th=[13829], 60.00th=[14746],
     | 70.00th=[15664], 80.00th=[16909], 90.00th=[19530], 95.00th=[21365],
     | 99.00th=[25560], 99.50th=[27132], 99.90th=[31327], 99.95th=[33162],
     | 99.99th=[37487]
   bw (  KiB/s): min=340566, max=437760, per=19.99%, avg=395346.78, stdev=15772.43, samples=600
   iops        : min= 2660, max= 3420, avg=3088.49, stdev=123.21, samples=600
  write: IOPS=6645, BW=831MiB/s (871MB/s)(48.7GiB/60011msec)
    slat (usec): min=17, max=14853, avg=230.44, stdev=959.82
    clat (usec): min=1991, max=44927, avg=14156.24, stdev=3835.54
     lat (usec): min=2034, max=44967, avg=14386.99, stdev=3829.16
    clat percentiles (usec):
     |  1.00th=[ 6652],  5.00th=[ 9241], 10.00th=[10028], 20.00th=[11076],
     | 30.00th=[11731], 40.00th=[12649], 50.00th=[13698], 60.00th=[14746],
     | 70.00th=[15664], 80.00th=[16909], 90.00th=[19268], 95.00th=[21103],
     | 99.00th=[25560], 99.50th=[27132], 99.90th=[31065], 99.95th=[32900],
     | 99.99th=[36963]
   bw (  KiB/s): min=145920, max=188672, per=19.99%, avg=170013.45, stdev=7500.47, samples=600
   iops        : min= 1140, max= 1474, avg=1328.13, stdev=58.64, samples=600
  lat (msec)   : 2=0.01%, 4=0.03%, 10=8.66%, 20=83.28%, 50=8.04%
  cpu          : usr=2.77%, sys=16.43%, ctx=57706, majf=0, minf=152
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=927349,398804,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1932MiB/s (2025MB/s), 1932MiB/s-1932MiB/s (2025MB/s-2025MB/s), io=113GiB (122GB), run=60011-60011msec
  WRITE: bw=831MiB/s (871MB/s), 831MiB/s-831MiB/s (871MB/s-871MB/s), io=48.7GiB (52.3GB), run=60011-60011msec

Disk stats (read/write):
  sda: ios=924699/397654, merge=0/5, ticks=6489955/2773996, in_queue=9263935, util=100.00%

TEST-256K

RandomRW: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=1538: Tue Jan 28 19:22:57 2020
   read: IOPS=7804, BW=1951MiB/s (2046MB/s)(114GiB/60015msec)
    slat (usec): min=12, max=29935, avg=428.44, stdev=1873.54
    clat (usec): min=2201, max=91155, avg=28249.28, stdev=7856.33
     lat (usec): min=2224, max=91190, avg=28678.03, stdev=7861.55
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[17957], 10.00th=[19268], 20.00th=[21103],
     | 30.00th=[23200], 40.00th=[26084], 50.00th=[27657], 60.00th=[29230],
     | 70.00th=[31065], 80.00th=[34341], 90.00th=[38536], 95.00th=[42730],
     | 99.00th=[51119], 99.50th=[55313], 99.90th=[62129], 99.95th=[64750],
     | 99.99th=[70779]
   bw (  KiB/s): min=297984, max=462848, per=19.98%, avg=399183.67, stdev=22130.17, samples=600
   iops        : min= 1164, max= 1808, avg=1559.11, stdev=86.42, samples=600
  write: IOPS=3356, BW=839MiB/s (880MB/s)(49.2GiB/60015msec)
    slat (usec): min=16, max=32373, avg=462.83, stdev=1898.48
    clat (usec): min=2086, max=90166, avg=28155.49, stdev=7968.49
     lat (usec): min=2126, max=90499, avg=28618.70, stdev=7933.00
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[17957], 10.00th=[19006], 20.00th=[20841],
     | 30.00th=[22938], 40.00th=[25822], 50.00th=[27657], 60.00th=[28967],
     | 70.00th=[31065], 80.00th=[34341], 90.00th=[38536], 95.00th=[42730],
     | 99.00th=[51119], 99.50th=[55313], 99.90th=[62653], 99.95th=[65274],
     | 99.99th=[72877]
   bw (  KiB/s): min=140749, max=207553, per=19.98%, avg=171670.90, stdev=10869.24, samples=600
   iops        : min=  549, max=  810, avg=670.41, stdev=42.48, samples=600
  lat (msec)   : 4=0.01%, 10=0.02%, 20=14.32%, 50=84.39%, 100=1.27%
  cpu          : usr=2.08%, sys=14.13%, ctx=32246, majf=0, minf=148
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=468414,201447,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1951MiB/s (2046MB/s), 1951MiB/s-1951MiB/s (2046MB/s-2046MB/s), io=114GiB (123GB), run=60015-60015msec
  WRITE: bw=839MiB/s (880MB/s), 839MiB/s-839MiB/s (880MB/s-880MB/s), io=49.2GiB (52.8GB), run=60015-60015msec

Disk stats (read/write):
  sda: ios=467424/200996, merge=0/4, ticks=6540020/2822130, in_queue=9363274, util=100.00%

TEST-512K

RandomRW: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=1688: Tue Jan 28 19:24:26 2020
   read: IOPS=3911, BW=1956MiB/s (2051MB/s)(115GiB/60032msec)
    slat (usec): min=46, max=101206, avg=854.75, stdev=3764.69
    clat (msec): min=3, max=152, avg=56.24, stdev=15.26
     lat (msec): min=3, max=152, avg=57.10, stdev=15.27
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   37], 10.00th=[   40], 20.00th=[   43],
     | 30.00th=[   50], 40.00th=[   53], 50.00th=[   55], 60.00th=[   57],
     | 70.00th=[   60], 80.00th=[   69], 90.00th=[   77], 95.00th=[   87],
     | 99.00th=[  102], 99.50th=[  110], 99.90th=[  123], 99.95th=[  129],
     | 99.99th=[  146]
   bw (  KiB/s): min=295776, max=486451, per=19.96%, avg=399750.44, stdev=34069.80, samples=600
   iops        : min=  577, max=  950, avg=780.60, stdev=66.55, samples=600
  write: IOPS=1685, BW=843MiB/s (883MB/s)(49.4GiB/60032msec)
    slat (usec): min=58, max=56871, avg=947.55, stdev=3841.20
    clat (msec): min=3, max=154, avg=56.34, stdev=15.47
     lat (msec): min=3, max=154, avg=57.29, stdev=15.42
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   37], 10.00th=[   40], 20.00th=[   42],
     | 30.00th=[   50], 40.00th=[   53], 50.00th=[   55], 60.00th=[   57],
     | 70.00th=[   60], 80.00th=[   69], 90.00th=[   78], 95.00th=[   88],
     | 99.00th=[  102], 99.50th=[  110], 99.90th=[  124], 99.95th=[  129],
     | 99.99th=[  146]
   bw (  KiB/s): min=120832, max=219720, per=19.96%, avg=172192.95, stdev=16717.87, samples=600
   iops        : min=  236, max=  429, avg=336.15, stdev=32.65, samples=600
  lat (msec)   : 4=0.01%, 10=0.02%, 20=0.02%, 50=31.35%, 100=67.50%
  lat (msec)   : 250=1.11%
  cpu          : usr=1.74%, sys=13.49%, ctx=16658, majf=0, minf=146
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=234795,101154,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1956MiB/s (2051MB/s), 1956MiB/s-1956MiB/s (2051MB/s-2051MB/s), io=115GiB (123GB), run=60032-60032msec
  WRITE: bw=843MiB/s (883MB/s), 843MiB/s-843MiB/s (883MB/s-883MB/s), io=49.4GiB (53.0GB), run=60032-60032msec

Disk stats (read/write):
  sda: ios=234374/100967, merge=0/4, ticks=6527317/2840359, in_queue=9369001, util=100.00%

TEST-1024K

RandomRW: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=1699: Tue Jan 28 19:25:58 2020
   read: IOPS=1940, BW=1940MiB/s (2034MB/s)(114GiB/60030msec)
    slat (usec): min=197, max=84513, avg=1751.53, stdev=5352.64
    clat (msec): min=18, max=239, avg=113.18, stdev=19.93
     lat (msec): min=18, max=240, avg=114.93, stdev=19.99
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   88], 10.00th=[   92], 20.00th=[   96],
     | 30.00th=[  102], 40.00th=[  107], 50.00th=[  111], 60.00th=[  115],
     | 70.00th=[  123], 80.00th=[  130], 90.00th=[  142], 95.00th=[  150],
     | 99.00th=[  169], 99.50th=[  178], 99.90th=[  192], 99.95th=[  197],
     | 99.99th=[  226]
   bw (  KiB/s): min=282624, max=475428, per=19.94%, avg=396223.87, stdev=34835.81, samples=600
   iops        : min=  276, max=  464, avg=386.79, stdev=34.05, samples=600
  write: IOPS=837, BW=838MiB/s (878MB/s)(49.1GiB/60030msec)
    slat (usec): min=233, max=113782, avg=1890.78, stdev=5400.16
    clat (msec): min=18, max=239, avg=113.66, stdev=20.05
     lat (msec): min=18, max=239, avg=115.55, stdev=20.12
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   88], 10.00th=[   92], 20.00th=[   96],
     | 30.00th=[  102], 40.00th=[  108], 50.00th=[  111], 60.00th=[  115],
     | 70.00th=[  124], 80.00th=[  130], 90.00th=[  142], 95.00th=[  150],
     | 99.00th=[  169], 99.50th=[  178], 99.90th=[  194], 99.95th=[  201],
     | 99.99th=[  224]
   bw (  KiB/s): min=106283, max=231424, per=19.94%, avg=171058.52, stdev=18673.45, samples=600
   iops        : min=  103, max=  226, avg=166.84, stdev=18.24, samples=600
  lat (msec)   : 20=0.02%, 50=0.09%, 100=28.11%, 250=71.78%
  cpu          : usr=1.58%, sys=13.09%, ctx=16244, majf=0, minf=146
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=116469,50288,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1940MiB/s (2034MB/s), 1940MiB/s-1940MiB/s (2034MB/s-2034MB/s), io=114GiB (122GB), run=60030-60030msec
  WRITE: bw=838MiB/s (878MB/s), 838MiB/s-838MiB/s (878MB/s-878MB/s), io=49.1GiB (52.7GB), run=60030-60030msec

Disk stats (read/write):
  sda: ios=233163/100460, merge=9/4, ticks=6529490/2850373, in_queue=9379976, util=99.91%