TEST-4K

RandomRW: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=19513: Wed Jan 22 12:49:57 2020
   read: IOPS=449, BW=1797KiB/s (1840kB/s)(106MiB/60110msec)
    slat (usec): min=2, max=908870, avg=6594.47, stdev=38638.85
    clat (usec): min=775, max=2006.2k, avg=477166.55, stdev=212949.23
     lat (msec): min=82, max=2054, avg=483.76, stdev=213.28
    clat percentiles (msec):
     |  1.00th=[  146],  5.00th=[  199], 10.00th=[  241], 20.00th=[  309],
     | 30.00th=[  355], 40.00th=[  397], 50.00th=[  439], 60.00th=[  498],
     | 70.00th=[  558], 80.00th=[  625], 90.00th=[  751], 95.00th=[  860],
     | 99.00th=[ 1133], 99.50th=[ 1318], 99.90th=[ 1737], 99.95th=[ 2005],
     | 99.99th=[ 2005]
   bw (  KiB/s): min=    7, max=  912, per=20.59%, avg=369.98, stdev=156.88, samples=579
   iops        : min=    1, max=  228, avg=92.44, stdev=39.23, samples=579
  write: IOPS=196, BW=785KiB/s (804kB/s)(46.1MiB/60110msec)
    slat (usec): min=3, max=751211, avg=6352.81, stdev=37598.05
    clat (msec): min=10, max=2006, avg=498.95, stdev=211.36
     lat (msec): min=10, max=2006, avg=505.30, stdev=211.26
    clat percentiles (msec):
     |  1.00th=[  163],  5.00th=[  211], 10.00th=[  255], 20.00th=[  330],
     | 30.00th=[  380], 40.00th=[  414], 50.00th=[  468], 60.00th=[  527],
     | 70.00th=[  575], 80.00th=[  642], 90.00th=[  768], 95.00th=[  877],
     | 99.00th=[ 1133], 99.50th=[ 1301], 99.90th=[ 1737], 99.95th=[ 2005],
     | 99.99th=[ 2005]
   bw (  KiB/s): min=    7, max=  360, per=20.93%, avg=164.09, stdev=69.28, samples=570
   iops        : min=    1, max=   90, avg=40.96, stdev=17.32, samples=570
  lat (usec)   : 1000=0.03%
  lat (msec)   : 2=0.02%, 20=0.01%, 100=0.03%, 250=10.80%, 500=49.15%
  lat (msec)   : 750=29.23%, 1000=8.58%
  cpu          : usr=0.09%, sys=0.16%, ctx=2448, majf=0, minf=151
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=27008,11794,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1797KiB/s (1840kB/s), 1797KiB/s-1797KiB/s (1840kB/s-1840kB/s), io=106MiB (111MB), run=60110-60110msec
  WRITE: bw=785KiB/s (804kB/s), 785KiB/s-785KiB/s (804kB/s-804kB/s), io=46.1MiB (48.3MB), run=60110-60110msec

Disk stats (read/write):
  sda: ios=27011/11824, merge=4/80, ticks=6686698/3129618, in_queue=9834334, util=99.87%

TEST-8K

RandomRW: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=19524: Wed Jan 22 12:50:58 2020
   read: IOPS=451, BW=3614KiB/s (3700kB/s)(212MiB/60136msec)
    slat (usec): min=2, max=638175, avg=6614.52, stdev=37336.76
    clat (usec): min=678, max=1555.8k, avg=477141.20, stdev=201151.57
     lat (usec): min=1124, max=1555.8k, avg=483756.98, stdev=201136.43
    clat percentiles (msec):
     |  1.00th=[  146],  5.00th=[  194], 10.00th=[  234], 20.00th=[  317],
     | 30.00th=[  368], 40.00th=[  405], 50.00th=[  443], 60.00th=[  493],
     | 70.00th=[  550], 80.00th=[  625], 90.00th=[  735], 95.00th=[  860],
     | 99.00th=[ 1083], 99.50th=[ 1183], 99.90th=[ 1368], 99.95th=[ 1418],
     | 99.99th=[ 1552]
   bw (  KiB/s): min=   15, max= 1744, per=20.25%, avg=731.69, stdev=305.05, samples=589
   iops        : min=    1, max=  218, avg=91.39, stdev=38.15, samples=589
  write: IOPS=197, BW=1578KiB/s (1616kB/s)(92.7MiB/60136msec)
    slat (usec): min=3, max=612189, avg=6613.92, stdev=37831.04
    clat (msec): min=41, max=1555, avg=502.06, stdev=209.33
     lat (msec): min=41, max=1591, avg=508.68, stdev=209.63
    clat percentiles (msec):
     |  1.00th=[  159],  5.00th=[  205], 10.00th=[  266], 20.00th=[  334],
     | 30.00th=[  384], 40.00th=[  422], 50.00th=[  472], 60.00th=[  518],
     | 70.00th=[  575], 80.00th=[  659], 90.00th=[  785], 95.00th=[  894],
     | 99.00th=[ 1133], 99.50th=[ 1301], 99.90th=[ 1368], 99.95th=[ 1552],
     | 99.99th=[ 1552]
   bw (  KiB/s): min=   15, max=  832, per=20.39%, avg=321.52, stdev=140.16, samples=585
   iops        : min=    1, max=  104, avg=40.12, stdev=17.53, samples=585
  lat (usec)   : 750=0.01%, 1000=0.03%
  lat (msec)   : 2=0.02%, 20=0.01%, 50=0.01%, 100=0.04%, 250=10.59%
  lat (msec)   : 500=49.29%, 750=30.08%, 1000=8.29%
  cpu          : usr=0.09%, sys=0.17%, ctx=2495, majf=0, minf=161
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=27164,11860,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=3614KiB/s (3700kB/s), 3614KiB/s-3614KiB/s (3700kB/s-3700kB/s), io=212MiB (223MB), run=60136-60136msec
  WRITE: bw=1578KiB/s (1616kB/s), 1578KiB/s-1578KiB/s (1616kB/s-1616kB/s), io=92.7MiB (97.2MB), run=60136-60136msec

Disk stats (read/write):
  sda: ios=26994/11818, merge=0/33, ticks=6612528/3113492, in_queue=9741592, util=99.95%

TEST-16K

RandomRW: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=19546: Wed Jan 22 12:56:47 2020
   read: IOPS=417, BW=6685KiB/s (6845kB/s)(393MiB/60199msec)
    slat (usec): min=3, max=1393.8k, avg=6941.57, stdev=40534.65
    clat (usec): min=768, max=1744.3k, avg=517365.23, stdev=232362.86
     lat (msec): min=34, max=1744, avg=524.31, stdev=232.31
    clat percentiles (msec):
     |  1.00th=[  171],  5.00th=[  218], 10.00th=[  257], 20.00th=[  338],
     | 30.00th=[  384], 40.00th=[  426], 50.00th=[  468], 60.00th=[  531],
     | 70.00th=[  600], 80.00th=[  676], 90.00th=[  827], 95.00th=[  936],
     | 99.00th=[ 1284], 99.50th=[ 1401], 99.90th=[ 1720], 99.95th=[ 1737],
     | 99.99th=[ 1737]
   bw (  KiB/s): min=   31, max= 3658, per=20.49%, avg=1369.34, stdev=607.02, samples=582
   iops        : min=    1, max=  228, avg=85.51, stdev=37.96, samples=582
  write: IOPS=181, BW=2911KiB/s (2981kB/s)(171MiB/60199msec)
    slat (usec): min=3, max=1075.3k, avg=7036.56, stdev=42601.59
    clat (msec): min=78, max=1744, avg=542.34, stdev=237.48
     lat (msec): min=108, max=1943, avg=549.38, stdev=238.24
    clat percentiles (msec):
     |  1.00th=[  180],  5.00th=[  224], 10.00th=[  275], 20.00th=[  363],
     | 30.00th=[  401], 40.00th=[  447], 50.00th=[  506], 60.00th=[  558],
     | 70.00th=[  625], 80.00th=[  701], 90.00th=[  844], 95.00th=[  978],
     | 99.00th=[ 1334], 99.50th=[ 1485], 99.90th=[ 1720], 99.95th=[ 1737],
     | 99.99th=[ 1737]
   bw (  KiB/s): min=   31, max= 1568, per=21.12%, avg=614.50, stdev=272.80, samples=566
   iops        : min=    1, max=   98, avg=38.32, stdev=17.06, samples=566
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.03%, 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (msec)   : 250=8.12%, 500=44.95%, 750=32.26%, 1000=10.59%
  cpu          : usr=0.09%, sys=0.17%, ctx=2219, majf=0, minf=150
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=25151,10952,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=6685KiB/s (6845kB/s), 6685KiB/s-6685KiB/s (6845kB/s-6845kB/s), io=393MiB (412MB), run=60199-60199msec
  WRITE: bw=2911KiB/s (2981kB/s), 2911KiB/s-2911KiB/s (2981kB/s-2981kB/s), io=171MiB (179MB), run=60199-60199msec

Disk stats (read/write):
  sda: ios=25114/10950, merge=0/70, ticks=6773795/3124289, in_queue=9920477, util=99.96%

TEST-32K

RandomRW: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=19587: Wed Jan 22 13:02:46 2020
   read: IOPS=280, BW=8983KiB/s (9198kB/s)(530MiB/60440msec)
    slat (usec): min=3, max=1212.4k, avg=10411.97, stdev=60066.42
    clat (usec): min=879, max=2358.0k, avg=769624.82, stdev=325524.63
     lat (usec): min=1041, max=2358.0k, avg=780038.10, stdev=325497.66
    clat percentiles (msec):
     |  1.00th=[  245],  5.00th=[  300], 10.00th=[  368], 20.00th=[  498],
     | 30.00th=[  575], 40.00th=[  651], 50.00th=[  726], 60.00th=[  818],
     | 70.00th=[  936], 80.00th=[ 1028], 90.00th=[ 1200], 95.00th=[ 1368],
     | 99.00th=[ 1720], 99.50th=[ 1888], 99.90th=[ 2366], 99.95th=[ 2366],
     | 99.99th=[ 2366]
   bw (  KiB/s): min=   63, max= 5504, per=21.40%, avg=1922.09, stdev=1088.46, samples=557
   iops        : min=    1, max=  172, avg=59.99, stdev=34.01, samples=557
  write: IOPS=122, BW=3931KiB/s (4026kB/s)(232MiB/60440msec)
    slat (usec): min=5, max=1029.8k, avg=10747.75, stdev=61183.66
    clat (msec): min=78, max=2358, avg=797.52, stdev=333.00
     lat (msec): min=78, max=2358, avg=808.27, stdev=333.84
    clat percentiles (msec):
     |  1.00th=[  255],  5.00th=[  309], 10.00th=[  380], 20.00th=[  523],
     | 30.00th=[  600], 40.00th=[  667], 50.00th=[  760], 60.00th=[  852],
     | 70.00th=[  953], 80.00th=[ 1036], 90.00th=[ 1250], 95.00th=[ 1401],
     | 99.00th=[ 1787], 99.50th=[ 1955], 99.90th=[ 2366], 99.95th=[ 2366],
     | 99.99th=[ 2366]
   bw (  KiB/s): min=   63, max= 2816, per=22.80%, avg=896.13, stdev=497.04, samples=524
   iops        : min=    1, max=   88, avg=27.91, stdev=15.55, samples=524
  lat (usec)   : 1000=0.03%
  lat (msec)   : 2=0.03%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.98%
  lat (msec)   : 500=19.14%, 750=31.87%, 1000=24.58%
  cpu          : usr=0.06%, sys=0.13%, ctx=1534, majf=0, minf=151
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.7%, >=64=98.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=16966,7425,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=8983KiB/s (9198kB/s), 8983KiB/s-8983KiB/s (9198kB/s-9198kB/s), io=530MiB (556MB), run=60440-60440msec
  WRITE: bw=3931KiB/s (4026kB/s), 3931KiB/s-3931KiB/s (4026kB/s-4026kB/s), io=232MiB (243MB), run=60440-60440msec

Disk stats (read/write):
  sda: ios=16875/7429, merge=1/68, ticks=6721867/3138635, in_queue=9912692, util=99.94%

TEST-64K

RandomRW: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=19611: Wed Jan 22 13:08:39 2020
   read: IOPS=228, BW=14.3MiB/s (14.0MB/s)(859MiB/60145msec)
    slat (usec): min=4, max=2729.5k, avg=12057.03, stdev=78466.45
    clat (usec): min=13, max=5217.2k, avg=948311.42, stdev=565267.73
     lat (msec): min=96, max=5217, avg=960.37, stdev=567.59
    clat percentiles (msec):
     |  1.00th=[  262],  5.00th=[  326], 10.00th=[  393], 20.00th=[  558],
     | 30.00th=[  642], 40.00th=[  718], 50.00th=[  860], 60.00th=[  944],
     | 70.00th=[ 1062], 80.00th=[ 1234], 90.00th=[ 1586], 95.00th=[ 1938],
     | 99.00th=[ 3306], 99.50th=[ 3675], 99.90th=[ 5201], 99.95th=[ 5201],
     | 99.99th=[ 5201]
   bw (  KiB/s): min=  127, max=11648, per=23.67%, avg=3460.71, stdev=2148.36, samples=500
   iops        : min=    1, max=  182, avg=53.97, stdev=33.58, samples=500
  write: IOPS=99, BW=6384KiB/s (6537kB/s)(375MiB/60145msec)
    slat (usec): min=7, max=1887.2k, avg=12454.79, stdev=76947.64
    clat (msec): min=6, max=5217, avg=981.57, stdev=585.66
     lat (msec): min=173, max=5217, avg=994.02, stdev=587.56
    clat percentiles (msec):
     |  1.00th=[  284],  5.00th=[  347], 10.00th=[  447], 20.00th=[  584],
     | 30.00th=[  651], 40.00th=[  743], 50.00th=[  885], 60.00th=[  978],
     | 70.00th=[ 1099], 80.00th=[ 1267], 90.00th=[ 1603], 95.00th=[ 1938],
     | 99.00th=[ 3306], 99.50th=[ 4463], 99.90th=[ 5201], 99.95th=[ 5201],
     | 99.99th=[ 5201]
   bw (  KiB/s): min=  127, max= 4992, per=26.21%, avg=1673.15, stdev=930.46, samples=451
   iops        : min=    1, max=   78, avg=26.05, stdev=14.52, samples=451
  lat (usec)   : 20=0.01%, 250=0.01%
  lat (msec)   : 2=0.05%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (msec)   : 250=0.54%, 500=13.81%, 750=28.19%, 1000=22.18%
  cpu          : usr=0.06%, sys=0.14%, ctx=1140, majf=0, minf=157
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=0.8%, >=64=98.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=13740,5999,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=14.3MiB/s (14.0MB/s), 14.3MiB/s-14.3MiB/s (14.0MB/s-14.0MB/s), io=859MiB (900MB), run=60145-60145msec
  WRITE: bw=6384KiB/s (6537kB/s), 6384KiB/s-6384KiB/s (6537kB/s-6537kB/s), io=375MiB (393MB), run=60145-60145msec

Disk stats (read/write):
  sda: ios=13691/6015, merge=1/66, ticks=6917215/3142085, in_queue=10111053, util=99.93%

TEST-128K

RandomRW: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=19635: Wed Jan 22 13:14:31 2020
   read: IOPS=176, BW=22.1MiB/s (23.1MB/s)(1333MiB/60384msec)
    slat (usec): min=8, max=1719.4k, avg=17069.51, stdev=96755.57
    clat (usec): min=1381, max=4744.2k, avg=1213496.72, stdev=624788.47
     lat (msec): min=217, max=4744, avg=1230.57, stdev=627.55
    clat percentiles (msec):
     |  1.00th=[  338],  5.00th=[  481], 10.00th=[  584], 20.00th=[  726],
     | 30.00th=[  844], 40.00th=[  978], 50.00th=[ 1099], 60.00th=[ 1217],
     | 70.00th=[ 1368], 80.00th=[ 1586], 90.00th=[ 2005], 95.00th=[ 2433],
     | 99.00th=[ 3406], 99.50th=[ 3943], 99.90th=[ 4732], 99.95th=[ 4732],
     | 99.99th=[ 4732]
   bw (  KiB/s): min=  255, max=16896, per=24.83%, avg=5613.63, stdev=3621.35, samples=476
   iops        : min=    1, max=  132, avg=43.74, stdev=28.31, samples=476
  write: IOPS=76, BW=9823KiB/s (10.1MB/s)(579MiB/60384msec)
    slat (usec): min=13, max=1551.5k, avg=18518.81, stdev=102342.05
    clat (msec): min=235, max=4744, avg=1283.20, stdev=654.13
     lat (msec): min=288, max=4782, avg=1301.72, stdev=656.34
    clat percentiles (msec):
     |  1.00th=[  342],  5.00th=[  518], 10.00th=[  634], 20.00th=[  768],
     | 30.00th=[  885], 40.00th=[ 1011], 50.00th=[ 1167], 60.00th=[ 1267],
     | 70.00th=[ 1469], 80.00th=[ 1670], 90.00th=[ 2123], 95.00th=[ 2500],
     | 99.00th=[ 3675], 99.50th=[ 4178], 99.90th=[ 4732], 99.95th=[ 4732],
     | 99.99th=[ 4732]
   bw (  KiB/s): min=  255, max= 7680, per=28.28%, avg=2777.79, stdev=1665.69, samples=418
   iops        : min=    1, max=   60, avg=21.58, stdev=13.04, samples=418
  lat (msec)   : 2=0.06%, 4=0.06%, 10=0.01%, 250=0.03%, 500=5.27%
  lat (msec)   : 750=15.82%, 1000=20.37%
  cpu          : usr=0.06%, sys=0.11%, ctx=1087, majf=0, minf=152
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=1.0%, >=64=97.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=10664,4634,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=22.1MiB/s (23.1MB/s), 22.1MiB/s-22.1MiB/s (23.1MB/s-23.1MB/s), io=1333MiB (1398MB), run=60384-60384msec
  WRITE: bw=9823KiB/s (10.1MB/s), 9823KiB/s-9823KiB/s (10.1MB/s-10.1MB/s), io=579MiB (607MB), run=60384-60384msec

Disk stats (read/write):
  sda: ios=10647/4651, merge=3/59, ticks=6213746/2900566, in_queue=9144148, util=99.91%

TEST-256K

RandomRW: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=19659: Wed Jan 22 13:20:27 2020
   read: IOPS=182, BW=45.5MiB/s (47.7MB/s)(2739MiB/60140msec)
    slat (usec): min=9, max=1521.1k, avg=17565.61, stdev=95848.61
    clat (usec): min=12, max=3376.2k, avg=1183428.74, stdev=490665.20
     lat (msec): min=134, max=3657, avg=1201.00, stdev=492.08
    clat percentiles (msec):
     |  1.00th=[  397],  5.00th=[  523], 10.00th=[  625], 20.00th=[  785],
     | 30.00th=[  894], 40.00th=[  986], 50.00th=[ 1099], 60.00th=[ 1200],
     | 70.00th=[ 1385], 80.00th=[ 1552], 90.00th=[ 1821], 95.00th=[ 2140],
     | 99.00th=[ 2668], 99.50th=[ 2869], 99.90th=[ 3104], 99.95th=[ 3373],
     | 99.99th=[ 3373]
   bw (  KiB/s): min=  510, max=33724, per=23.66%, avg=11033.83, stdev=6723.85, samples=498
   iops        : min=    1, max=  131, avg=42.97, stdev=26.28, samples=498
  write: IOPS=79, BW=19.8MiB/s (20.7MB/s)(1189MiB/60140msec)
    slat (usec): min=17, max=1382.7k, avg=18346.45, stdev=95889.78
    clat (msec): min=23, max=3376, avg=1230.18, stdev=502.88
     lat (msec): min=248, max=3376, avg=1248.53, stdev=504.90
    clat percentiles (msec):
     |  1.00th=[  405],  5.00th=[  558], 10.00th=[  693], 20.00th=[  827],
     | 30.00th=[  919], 40.00th=[ 1011], 50.00th=[ 1133], 60.00th=[ 1267],
     | 70.00th=[ 1435], 80.00th=[ 1603], 90.00th=[ 1921], 95.00th=[ 2198],
     | 99.00th=[ 2869], 99.50th=[ 2937], 99.90th=[ 3239], 99.95th=[ 3373],
     | 99.99th=[ 3373]
   bw (  KiB/s): min=  506, max=14818, per=26.80%, avg=5425.89, stdev=2843.68, samples=439
   iops        : min=    1, max=   57, avg=21.05, stdev=11.13, samples=439
  lat (usec)   : 20=0.01%
  lat (msec)   : 2=0.02%, 4=0.03%, 10=0.01%, 50=0.03%, 250=0.02%
  lat (msec)   : 500=3.28%, 750=12.27%, 1000=24.81%
  cpu          : usr=0.07%, sys=0.17%, ctx=971, majf=0, minf=152
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=1.0%, >=64=98.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=10954,4756,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=2739MiB (2872MB), run=60140-60140msec
  WRITE: bw=19.8MiB/s (20.7MB/s), 19.8MiB/s-19.8MiB/s (20.7MB/s-20.7MB/s), io=1189MiB (1247MB), run=60140-60140msec

Disk stats (read/write):
  sda: ios=10910/4766, merge=11/70, ticks=6201806/2901136, in_queue=9149819, util=99.88%

TEST-512K

RandomRW: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=19687: Wed Jan 22 13:26:28 2020
   read: IOPS=131, BW=65.6MiB/s (68.8MB/s)(3959MiB/60320msec)
    slat (usec): min=20, max=1712.4k, avg=24932.94, stdev=133250.21
    clat (msec): min=2, max=4393, avg=1624.41, stdev=729.41
     lat (msec): min=60, max=4788, avg=1649.34, stdev=735.75
    clat percentiles (msec):
     |  1.00th=[  550],  5.00th=[  684], 10.00th=[  776], 20.00th=[ 1003],
     | 30.00th=[ 1167], 40.00th=[ 1334], 50.00th=[ 1485], 60.00th=[ 1670],
     | 70.00th=[ 1888], 80.00th=[ 2198], 90.00th=[ 2735], 95.00th=[ 3037],
     | 99.00th=[ 3675], 99.50th=[ 3708], 99.90th=[ 4144], 99.95th=[ 4144],
     | 99.99th=[ 4396]
   bw (  KiB/s): min= 1021, max=53248, per=28.45%, avg=19121.59, stdev=12651.28, samples=411
   iops        : min=    1, max=  104, avg=37.26, stdev=24.70, samples=411
  write: IOPS=56, BW=28.3MiB/s (29.7MB/s)(1706MiB/60320msec)
    slat (usec): min=37, max=1325.1k, avg=27028.99, stdev=136095.20
    clat (msec): min=269, max=4522, avg=1733.42, stdev=760.48
     lat (msec): min=309, max=4522, avg=1760.45, stdev=767.35
    clat percentiles (msec):
     |  1.00th=[  567],  5.00th=[  718], 10.00th=[  827], 20.00th=[ 1083],
     | 30.00th=[ 1267], 40.00th=[ 1401], 50.00th=[ 1569], 60.00th=[ 1821],
     | 70.00th=[ 2005], 80.00th=[ 2333], 90.00th=[ 2903], 95.00th=[ 3205],
     | 99.00th=[ 3742], 99.50th=[ 4144], 99.90th=[ 4396], 99.95th=[ 4530],
     | 99.99th=[ 4530]
   bw (  KiB/s): min= 1021, max=27648, per=32.99%, avg=9554.46, stdev=5727.36, samples=357
   iops        : min=    1, max=   54, avg=18.57, stdev=11.14, samples=357
  lat (msec)   : 4=0.07%, 10=0.02%, 50=0.01%, 100=0.01%, 500=0.34%
  lat (msec)   : 750=6.36%, 1000=11.63%
  cpu          : usr=0.07%, sys=0.18%, ctx=709, majf=0, minf=152
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.7%, 32=1.4%, >=64=97.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=7917,3412,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=65.6MiB/s (68.8MB/s), 65.6MiB/s-65.6MiB/s (68.8MB/s-68.8MB/s), io=3959MiB (4151MB), run=60320-60320msec
  WRITE: bw=28.3MiB/s (29.7MB/s), 28.3MiB/s-28.3MiB/s (29.7MB/s-29.7MB/s), io=1706MiB (1789MB), run=60320-60320msec

Disk stats (read/write):
  sda: ios=8055/3504, merge=7732/3413, ticks=6166858/2894175, in_queue=9088467, util=99.80%

TEST-1024K

RandomRW: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=19719: Wed Jan 22 13:32:26 2020
   read: IOPS=57, BW=57.9MiB/s (60.7MB/s)(3510MiB/60661msec)
    slat (usec): min=48, max=1997.2k, avg=59810.00, stdev=202101.84
    clat (msec): min=172, max=8239, avg=3641.43, stdev=1251.37
     lat (msec): min=287, max=8563, avg=3701.24, stdev=1259.03
    clat percentiles (msec):
     |  1.00th=[  995],  5.00th=[ 1737], 10.00th=[ 2022], 20.00th=[ 2567],
     | 30.00th=[ 2970], 40.00th=[ 3272], 50.00th=[ 3608], 60.00th=[ 3910],
     | 70.00th=[ 4245], 80.00th=[ 4665], 90.00th=[ 5269], 95.00th=[ 5805],
     | 99.00th=[ 7080], 99.50th=[ 7416], 99.90th=[ 7886], 99.95th=[ 8221],
     | 99.99th=[ 8221]
   bw (  KiB/s): min= 2035, max=90112, per=27.88%, avg=16517.00, stdev=12298.65, samples=408
   iops        : min=    1, max=   88, avg=16.07, stdev=12.01, samples=408
  write: IOPS=25, BW=25.6MiB/s (26.9MB/s)(1555MiB/60661msec)
    slat (usec): min=86, max=1993.8k, avg=58651.53, stdev=195074.96
    clat (msec): min=378, max=8239, avg=3739.21, stdev=1282.77
     lat (msec): min=700, max=8467, avg=3797.86, stdev=1296.81
    clat percentiles (msec):
     |  1.00th=[  995],  5.00th=[ 1770], 10.00th=[ 2072], 20.00th=[ 2635],
     | 30.00th=[ 3071], 40.00th=[ 3306], 50.00th=[ 3675], 60.00th=[ 4010],
     | 70.00th=[ 4396], 80.00th=[ 4799], 90.00th=[ 5336], 95.00th=[ 5805],
     | 99.00th=[ 7148], 99.50th=[ 7617], 99.90th=[ 8221], 99.95th=[ 8221],
     | 99.99th=[ 8221]
   bw (  KiB/s): min= 2027, max=47104, per=31.88%, avg=8368.67, stdev=6653.35, samples=357
   iops        : min=    1, max=   46, avg= 8.10, stdev= 6.51, samples=357
  lat (msec)   : 250=0.02%, 500=0.20%, 750=0.39%, 1000=0.77%
  cpu          : usr=0.06%, sys=0.14%, ctx=781, majf=0, minf=147
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.2%, >=64=93.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=3510,1555,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=57.9MiB/s (60.7MB/s), 57.9MiB/s-57.9MiB/s (60.7MB/s-60.7MB/s), io=3510MiB (3681MB), run=60661-60661msec
  WRITE: bw=25.6MiB/s (26.9MB/s), 25.6MiB/s-25.6MiB/s (26.9MB/s-26.9MB/s), io=1555MiB (1631MB), run=60661-60661msec

Disk stats (read/write):
  sda: ios=10514/4689, merge=0/80, ticks=6230183/3064867, in_queue=9329265, util=99.65%


*****************************************************************************