start test

TEST-4K

RandomRW: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
...
fio-2.14
Starting 5 processes
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31694: Fri Jan 24 14:18:13 2020
  read : io=170440KB, bw=2837.7KB/s, iops=709, runt= 60064msec
    slat (usec): min=2, max=135209, avg=4911.14, stdev=11594.22
    clat (msec): min=2, max=677, avg=310.23, stdev=89.04
     lat (msec): min=2, max=682, avg=315.14, stdev=89.86
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[  178], 10.00th=[  208], 20.00th=[  241],
     | 30.00th=[  265], 40.00th=[  285], 50.00th=[  310], 60.00th=[  330],
     | 70.00th=[  355], 80.00th=[  383], 90.00th=[  424], 95.00th=[  457],
     | 99.00th=[  519], 99.50th=[  545], 99.90th=[  611], 99.95th=[  635],
     | 99.99th=[  652]
  write: io=73388KB, bw=1221.9KB/s, iops=305, runt= 60064msec
    slat (usec): min=2, max=164937, avg=4944.12, stdev=11650.18
    clat (msec): min=3, max=677, avg=310.66, stdev=88.37
     lat (msec): min=3, max=732, avg=315.61, stdev=89.12
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[  178], 10.00th=[  208], 20.00th=[  243],
     | 30.00th=[  265], 40.00th=[  289], 50.00th=[  310], 60.00th=[  330],
     | 70.00th=[  355], 80.00th=[  383], 90.00th=[  424], 95.00th=[  457],
     | 99.00th=[  523], 99.50th=[  553], 99.90th=[  619], 99.95th=[  644],
     | 99.99th=[  652]
    lat (msec) : 4=0.08%, 10=1.09%, 20=0.06%, 50=0.05%, 100=0.16%
    lat (msec) : 250=22.10%, 500=74.78%, 750=1.67%
  cpu          : usr=0.04%, sys=0.17%, ctx=19187, majf=0, minf=57
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.3%, >=64=99.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=42610/w=18347/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=170440KB, aggrb=2837KB/s, minb=2837KB/s, maxb=2837KB/s, mint=60064msec, maxt=60064msec
  WRITE: io=73388KB, aggrb=1221KB/s, minb=1221KB/s, maxb=1221KB/s, mint=60064msec, maxt=60064msec

Disk stats (read/write):
  nvme0n1: ios=42681/18350, merge=0/4, ticks=2407464/1036036, in_queue=3359816, util=99.05%

TEST-8K

RandomRW: (g=0): rw=randrw, bs=8K-8K/8K-8K/8K-8K, ioengine=libaio, iodepth=64
...
fio-2.14
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=31705: Fri Jan 24 14:19:13 2020
  read : io=337720KB, bw=5622.8KB/s, iops=702, runt= 60063msec
    slat (usec): min=2, max=139905, avg=4991.86, stdev=11612.33
    clat (msec): min=1, max=689, avg=313.45, stdev=87.18
     lat (msec): min=1, max=705, avg=318.44, stdev=87.87
    clat percentiles (msec):
     |  1.00th=[  135],  5.00th=[  182], 10.00th=[  208], 20.00th=[  239],
     | 30.00th=[  265], 40.00th=[  285], 50.00th=[  310], 60.00th=[  330],
     | 70.00th=[  355], 80.00th=[  388], 90.00th=[  429], 95.00th=[  465],
     | 99.00th=[  537], 99.50th=[  562], 99.90th=[  611], 99.95th=[  635],
     | 99.99th=[  685]
  write: io=145176KB, bw=2417.7KB/s, iops=302, runt= 60063msec
    slat (usec): min=2, max=120934, avg=4911.66, stdev=11549.54
    clat (msec): min=2, max=696, avg=313.12, stdev=88.06
     lat (msec): min=2, max=701, avg=318.03, stdev=88.84
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  182], 10.00th=[  208], 20.00th=[  239],
     | 30.00th=[  265], 40.00th=[  289], 50.00th=[  310], 60.00th=[  330],
     | 70.00th=[  355], 80.00th=[  383], 90.00th=[  429], 95.00th=[  465],
     | 99.00th=[  545], 99.50th=[  578], 99.90th=[  627], 99.95th=[  635],
     | 99.99th=[  685]
    lat (msec) : 2=0.01%, 4=0.12%, 10=0.08%, 20=0.02%, 50=0.06%
    lat (msec) : 100=0.16%, 250=23.77%, 500=73.48%, 750=2.30%
  cpu          : usr=0.03%, sys=0.21%, ctx=19027, majf=0, minf=57
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.3%, >=64=99.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=42215/w=18147/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
 latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=337720KB, aggrb=5622KB/s, minb=5622KB/s, maxb=5622KB/s, mint=60063msec, maxt=60063msec
  WRITE: io=145176KB, aggrb=2417KB/s, minb=2417KB/s, maxb=2417KB/s, mint=60063msec, maxt=60063msec

Disk stats (read/write):
  nvme0n1: ios=42116/18125, merge=0/3, ticks=2406036/1033128, in_queue=3378972, util=99.91%

TEST-16K

RandomRW: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=64
...
fio-2.14
Starting 5 processes
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31760: Fri Jan 24 14:22:33 2020
  read : io=682336KB, bw=11364KB/s, iops=710, runt= 60044msec
    slat (usec): min=2, max=142848, avg=4912.45, stdev=11520.04
    clat (msec): min=4, max=656, avg=310.09, stdev=90.88
     lat (msec): min=4, max=677, avg=315.00, stdev=91.73
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[  174], 10.00th=[  204], 20.00th=[  239],
     | 30.00th=[  265], 40.00th=[  285], 50.00th=[  310], 60.00th=[  330],
     | 70.00th=[  355], 80.00th=[  388], 90.00th=[  424], 95.00th=[  457],
     | 99.00th=[  529], 99.50th=[  553], 99.90th=[  619], 99.95th=[  627],
     | 99.99th=[  644]
  write: io=293648KB, bw=4890.6KB/s, iops=305, runt= 60044msec
    slat (usec): min=2, max=124292, avg=4926.51, stdev=11602.97
    clat (msec): min=5, max=656, avg=309.89, stdev=90.18
     lat (msec): min=5, max=666, avg=314.82, stdev=90.94
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[  174], 10.00th=[  206], 20.00th=[  239],
     | 30.00th=[  265], 40.00th=[  285], 50.00th=[  310], 60.00th=[  330],
     | 70.00th=[  355], 80.00th=[  388], 90.00th=[  424], 95.00th=[  457],
     | 99.00th=[  523], 99.50th=[  553], 99.90th=[  603], 99.95th=[  619],
     | 99.99th=[  644]
    lat (msec) : 10=0.77%, 20=0.51%, 50=0.06%, 100=0.18%, 250=22.85%
    lat (msec) : 500=73.80%, 750=1.82%
  cpu          : usr=0.03%, sys=0.20%, ctx=19108, majf=0, minf=57
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.3%, >=64=99.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=42646/w=18353/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=682336KB, aggrb=11363KB/s, minb=11363KB/s, maxb=11363KB/s, mint=60044msec, maxt=60044msec
  WRITE: io=293648KB, aggrb=4890KB/s, minb=4890KB/s, maxb=4890KB/s, mint=60044msec, maxt=60044msec

Disk stats (read/write):
  nvme0n1: ios=42563/18322, merge=0/5, ticks=2398408/1030216, in_queue=3369956, util=99.82%

TEST-32K

RandomRW: (g=0): rw=randrw, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=64
...
fio-2.14
Starting 5 processes
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31770: Fri Jan 24 14:25:53 2020
  read : io=1333.1MB, bw=22742KB/s, iops=710, runt= 60065msec
    slat (usec): min=3, max=116041, avg=4863.76, stdev=11443.43
    clat (msec): min=2, max=735, avg=309.82, stdev=92.08
     lat (msec): min=2, max=735, avg=314.69, stdev=92.94
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[  172], 10.00th=[  202], 20.00th=[  237],
     | 30.00th=[  265], 40.00th=[  285], 50.00th=[  310], 60.00th=[  330],
     | 70.00th=[  355], 80.00th=[  383], 90.00th=[  424], 95.00th=[  461],
     | 99.00th=[  529], 99.50th=[  553], 99.90th=[  611], 99.95th=[  644],
     | 99.99th=[  701]
  write: io=587808KB, bw=9786.3KB/s, iops=305, runt= 60065msec
    slat (usec): min=3, max=165911, avg=5020.67, stdev=11794.76
    clat (msec): min=4, max=704, avg=309.87, stdev=91.32
     lat (msec): min=4, max=704, avg=314.89, stdev=92.22
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[  174], 10.00th=[  202], 20.00th=[  237],
     | 30.00th=[  265], 40.00th=[  285], 50.00th=[  310], 60.00th=[  334],
     | 70.00th=[  355], 80.00th=[  383], 90.00th=[  424], 95.00th=[  457],
     | 99.00th=[  529], 99.50th=[  553], 99.90th=[  611], 99.95th=[  627],
     | 99.99th=[  701]
    lat (msec) : 4=0.02%, 10=0.10%, 20=1.18%, 50=0.06%, 100=0.24%
    lat (msec) : 250=23.15%, 500=73.21%, 750=2.05%
  cpu          : usr=0.07%, sys=0.21%, ctx=19153, majf=0, minf=56
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.3%, >=64=99.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=42687/w=18369/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=1333.1MB, aggrb=22741KB/s, minb=22741KB/s, maxb=22741KB/s, mint=60065msec, maxt=60065msec
  WRITE: io=587808KB, aggrb=9786KB/s, minb=9786KB/s, maxb=9786KB/s, mint=60065msec, maxt=60065msec

Disk stats (read/write):
  nvme0n1: ios=42604/18351, merge=0/6, ticks=2394732/1028192, in_queue=3363476, util=99.84%

TEST-64K

RandomRW: (g=0): rw=randrw, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=64
...
fio-2.14
Starting 5 processes
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31781: Fri Jan 24 14:29:12 2020
  read : io=2667.6MB, bw=45478KB/s, iops=710, runt= 60063msec
    slat (usec): min=3, max=138093, avg=4912.27, stdev=11463.45
    clat (msec): min=10, max=638, avg=310.15, stdev=88.96
     lat (msec): min=10, max=646, avg=315.06, stdev=89.79
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[  180], 10.00th=[  210], 20.00th=[  243],
     | 30.00th=[  265], 40.00th=[  285], 50.00th=[  306], 60.00th=[  330],
     | 70.00th=[  351], 80.00th=[  379], 90.00th=[  420], 95.00th=[  461],
     | 99.00th=[  529], 99.50th=[  553], 99.90th=[  603], 99.95th=[  611],
     | 99.99th=[  627]
  write: io=1147.2MB, bw=19558KB/s, iops=305, runt= 60063msec
    slat (usec): min=4, max=154503, avg=4916.18, stdev=11626.25
    clat (msec): min=11, max=647, avg=309.31, stdev=88.96
     lat (msec): min=11, max=647, avg=314.23, stdev=89.78
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[  180], 10.00th=[  210], 20.00th=[  241],
     | 30.00th=[  265], 40.00th=[  285], 50.00th=[  306], 60.00th=[  326],
     | 70.00th=[  351], 80.00th=[  379], 90.00th=[  420], 95.00th=[  461],
     | 99.00th=[  537], 99.50th=[  562], 99.90th=[  611], 99.95th=[  619],
     | 99.99th=[  644]
    lat (msec) : 20=0.51%, 50=0.90%, 100=0.20%, 250=21.66%, 500=74.52%
    lat (msec) : 750=2.21%
  cpu          : usr=0.06%, sys=0.23%, ctx=19161, majf=0, minf=57
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.3%, >=64=99.5%
  submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=42680/w=18355/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=2667.6MB, aggrb=45477KB/s, minb=45477KB/s, maxb=45477KB/s, mint=60063msec, maxt=60063msec
  WRITE: io=1147.2MB, aggrb=19558KB/s, minb=19558KB/s, maxb=19558KB/s, mint=60063msec, maxt=60063msec

Disk stats (read/write):
  nvme0n1: ios=42600/18334, merge=0/8, ticks=2394348/1034524, in_queue=3369204, util=99.89%

TEST-128K

RandomRW: (g=0): rw=randrw, bs=128K-128K/128K-128K/128K-128K, ioengine=libaio, iodepth=64
...
fio-2.14
Starting 5 processes
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31798: Fri Jan 24 14:32:32 2020
  read : io=5337.7MB, bw=90994KB/s, iops=710, runt= 60067msec
    slat (usec): min=4, max=132991, avg=4893.31, stdev=11463.55
    clat (msec): min=8, max=766, avg=309.26, stdev=89.43
     lat (msec): min=8, max=809, avg=314.16, stdev=90.28
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[  176], 10.00th=[  204], 20.00th=[  237],
     | 30.00th=[  262], 40.00th=[  285], 50.00th=[  306], 60.00th=[  326],
     | 70.00th=[  351], 80.00th=[  383], 90.00th=[  429], 95.00th=[  461],
     | 99.00th=[  529], 99.50th=[  553], 99.90th=[  611], 99.95th=[  644],
     | 99.99th=[  766]
  write: io=2293.3MB, bw=39094KB/s, iops=305, runt= 60067msec
    slat (usec): min=6, max=153629, avg=4952.36, stdev=11503.49
    clat (msec): min=12, max=809, avg=311.16, stdev=90.19
     lat (msec): min=12, max=842, avg=316.11, stdev=91.03
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[  178], 10.00th=[  204], 20.00th=[  239],
     | 30.00th=[  262], 40.00th=[  285], 50.00th=[  306], 60.00th=[  330],
     | 70.00th=[  355], 80.00th=[  383], 90.00th=[  433], 95.00th=[  465],
     | 99.00th=[  529], 99.50th=[  553], 99.90th=[  611], 99.95th=[  734],
     | 99.99th=[  799]
    lat (msec) : 10=0.01%, 20=0.03%, 50=0.93%, 100=0.31%, 250=23.72%
    lat (msec) : 500=72.95%, 750=2.05%, 1000=0.01%
  cpu          : usr=0.08%, sys=0.26%, ctx=19057, majf=0, minf=57
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.3%, >=64=99.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=42701/w=18346/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=5337.7MB, aggrb=90993KB/s, minb=90993KB/s, maxb=90993KB/s, mint=60067msec, maxt=60067msec
  WRITE: io=2293.3MB, aggrb=39094KB/s, minb=39094KB/s, maxb=39094KB/s, mint=60067msec, maxt=60067msec

TEST-256K

RandomRW: (g=0): rw=randrw, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=64
...
fio-2.14
Starting 5 processes
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31810: Fri Jan 24 14:35:52 2020
  read : io=7764.8MB, bw=132309KB/s, iops=516, runt= 60095msec
    slat (usec): min=7, max=167710, avg=6657.67, stdev=15825.79
    clat (msec): min=40, max=932, avg=425.87, stdev=120.44
     lat (msec): min=40, max=932, avg=432.53, stdev=121.44
    clat percentiles (msec):
     |  1.00th=[  167],  5.00th=[  245], 10.00th=[  281], 20.00th=[  326],
     | 30.00th=[  359], 40.00th=[  392], 50.00th=[  416], 60.00th=[  449],
     | 70.00th=[  482], 80.00th=[  523], 90.00th=[  586], 95.00th=[  635],
     | 99.00th=[  742], 99.50th=[  783], 99.90th=[  873], 99.95th=[  898],
     | 99.99th=[  922]
  write: io=3344.0MB, bw=56981KB/s, iops=222, runt= 60095msec
    slat (usec): min=8, max=148306, avg=6962.45, stdev=16205.72
    clat (msec): min=47, max=932, avg=425.34, stdev=120.22
     lat (msec): min=47, max=932, avg=432.31, stdev=121.28
    clat percentiles (msec):
     |  1.00th=[  167],  5.00th=[  243], 10.00th=[  281], 20.00th=[  326],
     | 30.00th=[  359], 40.00th=[  388], 50.00th=[  416], 60.00th=[  449],
     | 70.00th=[  482], 80.00th=[  519], 90.00th=[  586], 95.00th=[  635],
     | 99.00th=[  742], 99.50th=[  783], 99.90th=[  865], 99.95th=[  889],
     | 99.99th=[  914]
    lat (msec) : 50=0.03%, 100=0.42%, 250=5.25%, 500=69.45%, 750=23.99%
    lat (msec) : 1000=0.86%
  cpu          : usr=0.08%, sys=0.23%, ctx=13848, majf=0, minf=56
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=31059/w=13376/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=7764.8MB, aggrb=132308KB/s, minb=132308KB/s, maxb=132308KB/s, mint=60095msec, maxt=60095msec
  WRITE: io=3344.0MB, aggrb=56980KB/s, minb=56980KB/s, maxb=56980KB/s, mint=60095msec, maxt=60095msec

Disk stats (read/write):
  nvme0n1: ios=30973/13372, merge=0/5, ticks=2393848/1033100, in_queue=3367540, util=99.82%

TEST-512K

RandomRW: (g=0): rw=randrw, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=64
...
fio-2.14
Starting 5 processes
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31821: Fri Jan 24 14:39:12 2020
  read : io=7802.0MB, bw=132935KB/s, iops=259, runt= 60099msec
    slat (usec): min=12, max=223356, avg=13501.10, stdev=19636.40
    clat (msec): min=72, max=1437, avg=849.87, stdev=174.31
     lat (msec): min=80, max=1453, avg=863.37, stdev=175.68
    clat percentiles (msec):
     |  1.00th=[  420],  5.00th=[  586], 10.00th=[  644], 20.00th=[  709],
     | 30.00th=[  758], 40.00th=[  799], 50.00th=[  848], 60.00th=[  889],
     | 70.00th=[  938], 80.00th=[  996], 90.00th=[ 1074], 95.00th=[ 1139],
     | 99.00th=[ 1254], 99.50th=[ 1287], 99.90th=[ 1369], 99.95th=[ 1385],
     | 99.99th=[ 1434]
  write: io=3309.6MB, bw=56389KB/s, iops=110, runt= 60099msec
    slat (usec): min=19, max=143248, avg=13486.99, stdev=19255.86
    clat (msec): min=98, max=1429, avg=851.54, stdev=171.11
     lat (msec): min=98, max=1440, avg=865.03, stdev=172.72
    clat percentiles (msec):
     |  1.00th=[  474],  5.00th=[  594], 10.00th=[  644], 20.00th=[  709],
     | 30.00th=[  758], 40.00th=[  799], 50.00th=[  848], 60.00th=[  889],
     | 70.00th=[  938], 80.00th=[  996], 90.00th=[ 1074], 95.00th=[ 1139],
     | 99.00th=[ 1254], 99.50th=[ 1303], 99.90th=[ 1369], 99.95th=[ 1385],
     | 99.99th=[ 1434]
    lat (msec) : 100=0.09%, 250=0.31%, 500=1.37%, 750=26.68%, 1000=51.74%
    lat (msec) : 2000=19.82%
  cpu          : usr=0.07%, sys=0.20%, ctx=14543, majf=0, minf=56
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=0.7%, >=64=98.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=15604/w=6619/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=7802.0MB, aggrb=132934KB/s, minb=132934KB/s, maxb=132934KB/s, mint=60099msec, maxt=60099msec
  WRITE: io=3309.6MB, aggrb=56389KB/s, minb=56389KB/s, maxb=56389KB/s, mint=60099msec, maxt=60099msec

Disk stats (read/write):
  nvme0n1: ios=31134/13231, merge=0/6, ticks=2449272/1040980, in_queue=3409152, util=99.18%

TEST-1024K

RandomRW: (g=0): rw=randrw, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=64
...
fio-2.14
Starting 5 processes
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)
RandomRW: Laying out IO file(s) (1 file(s) / 5120MB)


RandomRW: (groupid=0, jobs=5): err= 0: pid=31838: Fri Jan 24 14:42:32 2020
  read : io=7788.0MB, bw=132685KB/s, iops=129, runt= 60104msec
    slat (usec): min=24, max=186618, avg=27027.94, stdev=26198.55
    clat (msec): min=62, max=2728, avg=1691.69, stdev=350.53
     lat (msec): min=89, max=2769, avg=1718.72, stdev=353.42
    clat percentiles (msec):
     |  1.00th=[  668],  5.00th=[ 1106], 10.00th=[ 1254], 20.00th=[ 1418],
     | 30.00th=[ 1532], 40.00th=[ 1631], 50.00th=[ 1713], 60.00th=[ 1795],
     | 70.00th=[ 1893], 80.00th=[ 1991], 90.00th=[ 2114], 95.00th=[ 2180],
     | 99.00th=[ 2474], 99.50th=[ 2606], 99.90th=[ 2704], 99.95th=[ 2704],
     | 99.99th=[ 2737]
  write: io=3327.0MB, bw=56683KB/s, iops=55, runt= 60104msec
    slat (usec): min=33, max=160126, avg=26900.70, stdev=26001.21
    clat (msec): min=67, max=2737, avg=1691.36, stdev=353.77
     lat (msec): min=111, max=2764, avg=1718.26, stdev=357.08
    clat percentiles (msec):
     |  1.00th=[  701],  5.00th=[ 1106], 10.00th=[ 1254], 20.00th=[ 1418],
     | 30.00th=[ 1532], 40.00th=[ 1631], 50.00th=[ 1713], 60.00th=[ 1795],
     | 70.00th=[ 1893], 80.00th=[ 1975], 90.00th=[ 2114], 95.00th=[ 2180],
     | 99.00th=[ 2507], 99.50th=[ 2638], 99.90th=[ 2704], 99.95th=[ 2737],
     | 99.99th=[ 2737]
    lat (msec) : 100=0.05%, 250=0.24%, 500=0.42%, 750=0.41%, 1000=1.66%
    lat (msec) : 2000=78.29%, >=2000=18.92%
  cpu          : usr=0.04%, sys=0.20%, ctx=14734, majf=0, minf=56
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.7%, 32=1.4%, >=64=97.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=7788/w=3327/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=7788.0MB, aggrb=132685KB/s, minb=132685KB/s, maxb=132685KB/s, mint=60104msec, maxt=60104msec
  WRITE: io=3327.0MB, aggrb=56682KB/s, minb=56682KB/s, maxb=56682KB/s, mint=60104msec, maxt=60104msec

Disk stats (read/write):
  nvme0n1: ios=31075/13272, merge=0/5, ticks=2479424/1050812, in_queue=3465652, util=99.59%