start test

TEST-4K

RandomRW: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes
RandomRW: Laying out IO file (1 file / 3072MiB)
RandomRW: Laying out IO file (1 file / 3072MiB)
RandomRW: Laying out IO file (1 file / 3072MiB)
RandomRW: Laying out IO file (1 file / 3072MiB)
RandomRW: Laying out IO file (1 file / 3072MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=29604: Tue Jan 21 12:55:04 2020
   read: IOPS=19.2k, BW=74.8MiB/s (78.5MB/s)(4490MiB/60006msec)
    slat (usec): min=3, max=57016, avg=163.30, stdev=827.07
    clat (usec): min=600, max=78118, avg=11717.04, stdev=5932.70
     lat (usec): min=609, max=78125, avg=11881.68, stdev=5989.76
    clat percentiles (usec):
     |  1.00th=[ 2638],  5.00th=[ 3752], 10.00th=[ 5473], 20.00th=[ 6718],
     | 30.00th=[ 8356], 40.00th=[ 9503], 50.00th=[10945], 60.00th=[12256],
     | 70.00th=[13829], 80.00th=[15795], 90.00th=[19006], 95.00th=[22152],
     | 99.00th=[30802], 99.50th=[35390], 99.90th=[48497], 99.95th=[57934],
     | 99.99th=[68682]
   bw (  KiB/s): min=11096, max=18128, per=20.00%, avg=15320.97, stdev=1192.58, samples=600
   iops        : min= 2774, max= 4532, avg=3830.21, stdev=298.15, samples=600
  write: IOPS=8244, BW=32.2MiB/s (33.8MB/s)(1932MiB/60006msec)
    slat (usec): min=4, max=57096, avg=167.51, stdev=835.72
    clat (usec): min=628, max=79577, avg=11017.33, stdev=6056.77
     lat (usec): min=634, max=79584, avg=11186.19, stdev=6116.02
    clat percentiles (usec):
     |  1.00th=[ 2343],  5.00th=[ 3195], 10.00th=[ 4228], 20.00th=[ 5997],
     | 30.00th=[ 7373], 40.00th=[ 8848], 50.00th=[10028], 60.00th=[11600],
     | 70.00th=[13173], 80.00th=[15139], 90.00th=[18482], 95.00th=[21627],
     | 99.00th=[30540], 99.50th=[35390], 99.90th=[47973], 99.95th=[56886],
     | 99.99th=[67634]
   bw (  KiB/s): min= 4744, max= 7824, per=20.00%, avg=6593.59, stdev=537.83, samples=600
   iops        : min= 1186, max= 1956, avg=1648.36, stdev=134.46, samples=600
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.38%, 4=6.29%, 10=38.83%, 20=46.84%, 50=7.57%
  lat (msec)   : 100=0.08%
  cpu          : usr=3.25%, sys=6.06%, ctx=104999, majf=0, minf=151
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=1149480,494699,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=74.8MiB/s (78.5MB/s), 74.8MiB/s-74.8MiB/s (78.5MB/s-78.5MB/s), io=4490MiB (4708MB), run=60006-60006msec
  WRITE: bw=32.2MiB/s (33.8MB/s), 32.2MiB/s-32.2MiB/s (33.8MB/s-33.8MB/s), io=1932MiB (2026MB), run=60006-60006msec

Disk stats (read/write):
  xvda: ios=1146803/493575, merge=123/220, ticks=6818051/2642048, in_queue=9460197, util=99.95%

TEST-16K

RandomRW: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=29624: Tue Jan 21 12:56:05 2020
   read: IOPS=14.5k, BW=226MiB/s (237MB/s)(13.3GiB/60009msec)
    slat (usec): min=4, max=56637, avg=217.75, stdev=1154.81
    clat (usec): min=902, max=113808, avg=15519.81, stdev=8220.12
     lat (usec): min=910, max=113816, avg=15738.89, stdev=8299.11
    clat percentiles (usec):
     |  1.00th=[ 3294],  5.00th=[ 4948], 10.00th=[ 6915], 20.00th=[ 8848],
     | 30.00th=[10814], 40.00th=[12518], 50.00th=[14353], 60.00th=[16057],
     | 70.00th=[18220], 80.00th=[20841], 90.00th=[25297], 95.00th=[29492],
     | 99.00th=[44303], 99.50th=[52691], 99.90th=[66847], 99.95th=[71828],
     | 99.99th=[82314]
   bw (  KiB/s): min=33952, max=55840, per=20.00%, avg=46369.45, stdev=3532.90, samples=600
   iops        : min= 2122, max= 3490, avg=2898.06, stdev=220.81, samples=600
  write: IOPS=6227, BW=97.3MiB/s (102MB/s)(5839MiB/60009msec)
    slat (usec): min=4, max=55061, avg=223.36, stdev=1146.79
    clat (usec): min=995, max=95553, avg=14505.22, stdev=8297.28
     lat (usec): min=1003, max=95563, avg=14729.92, stdev=8380.89
    clat percentiles (usec):
     |  1.00th=[ 2868],  5.00th=[ 3982], 10.00th=[ 5538], 20.00th=[ 7701],
     | 30.00th=[ 9634], 40.00th=[11338], 50.00th=[13304], 60.00th=[15008],
     | 70.00th=[17171], 80.00th=[20055], 90.00th=[24511], 95.00th=[28967],
     | 99.00th=[42730], 99.50th=[51643], 99.90th=[66847], 99.95th=[70779],
     | 99.99th=[82314]
   bw (  KiB/s): min=14688, max=24864, per=19.99%, avg=19922.08, stdev=1614.10, samples=600
   iops        : min=  918, max= 1554, avg=1245.09, stdev=100.90, samples=600
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.19%, 4=3.33%, 10=23.43%, 20=51.08%, 50=21.37%
  lat (msec)   : 100=0.61%, 250=0.01%
  cpu          : usr=2.61%, sys=4.92%, ctx=82876, majf=0, minf=160
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=869768,373698,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=13.3GiB (14.2GB), run=60009-60009msec
  WRITE: bw=97.3MiB/s (102MB/s), 97.3MiB/s-97.3MiB/s (102MB/s-102MB/s), io=5839MiB (6123MB), run=60009-60009msec

Disk stats (read/write):
  xvda: ios=862727/371890, merge=5296/1027, ticks=6838323/2623696, in_queue=9463122, util=99.95%

TEST-32K

RandomRW: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=29643: Tue Jan 21 12:57:05 2020
   read: IOPS=10.8k, BW=336MiB/s (352MB/s)(19.7GiB/60047msec)
    slat (usec): min=5, max=69520, avg=293.52, stdev=1604.06
    clat (usec): min=1223, max=165008, avg=20835.84, stdev=11531.34
     lat (usec): min=1232, max=165016, avg=21130.69, stdev=11646.24
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    9], 20.00th=[   12],
     | 30.00th=[   15], 40.00th=[   17], 50.00th=[   20], 60.00th=[   22],
     | 70.00th=[   25], 80.00th=[   28], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   64], 99.50th=[   72], 99.90th=[   93], 99.95th=[  103],
     | 99.99th=[  129]
   bw (  KiB/s): min=42880, max=88704, per=20.00%, avg=68837.18, stdev=8011.65, samples=600
   iops        : min= 1340, max= 2772, avg=2151.12, stdev=250.37, samples=600
  write: IOPS=4618, BW=144MiB/s (151MB/s)(8667MiB/60047msec)
    slat (usec): min=5, max=69504, avg=312.89, stdev=1647.23
    clat (usec): min=1382, max=133761, avg=19740.22, stdev=11715.56
     lat (usec): min=1393, max=142035, avg=20054.44, stdev=11838.74
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    8], 20.00th=[   11],
     | 30.00th=[   14], 40.00th=[   16], 50.00th=[   18], 60.00th=[   21],
     | 70.00th=[   24], 80.00th=[   28], 90.00th=[   34], 95.00th=[   41],
     | 99.00th=[   63], 99.50th=[   72], 99.90th=[   92], 99.95th=[  101],
     | 99.99th=[  124]
   bw (  KiB/s): min=18048, max=39424, per=20.01%, avg=29569.42, stdev=3454.08, samples=600
   iops        : min=  564, max= 1232, avg=924.01, stdev=107.93, samples=600
  lat (msec)   : 2=0.04%, 4=0.95%, 10=14.77%, 20=39.94%, 50=41.93%
  lat (msec)   : 100=2.31%, 250=0.06%
  cpu          : usr=2.08%, sys=3.99%, ctx=62415, majf=0, minf=161
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=645728,277345,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=336MiB/s (352MB/s), 336MiB/s-336MiB/s (352MB/s-352MB/s), io=19.7GiB (21.2GB), run=60047-60047msec
  WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=8667MiB (9088MB), run=60047-60047msec

Disk stats (read/write):
  xvda: ios=644277/276812, merge=0/15, ticks=6804348/2650794, in_queue=9456145, util=99.95%

TEST-64K

RandomRW: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=29662: Tue Jan 21 12:58:06 2020
   read: IOPS=5563, BW=348MiB/s (365MB/s)(20.4GiB/60032msec)
    slat (usec): min=7, max=74355, avg=606.01, stdev=2238.86
    clat (msec): min=2, max=170, avg=39.93, stdev=13.08
     lat (msec): min=2, max=170, avg=40.54, stdev=13.25
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   23], 10.00th=[   26], 20.00th=[   30],
     | 30.00th=[   33], 40.00th=[   36], 50.00th=[   39], 60.00th=[   42],
     | 70.00th=[   45], 80.00th=[   49], 90.00th=[   56], 95.00th=[   63],
     | 99.00th=[   86], 99.50th=[   95], 99.90th=[  116], 99.95th=[  126],
     | 99.99th=[  148]
   bw (  KiB/s): min=51864, max=87424, per=19.99%, avg=71196.07, stdev=5998.00, samples=600
   iops        : min=  810, max= 1366, avg=1112.38, stdev=93.71, samples=600
  write: IOPS=2392, BW=150MiB/s (157MB/s)(8978MiB/60032msec)
    slat (usec): min=9, max=78005, avg=650.62, stdev=2329.30
    clat (msec): min=3, max=166, avg=38.73, stdev=13.04
     lat (msec): min=3, max=174, avg=39.38, stdev=13.24
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   22], 10.00th=[   25], 20.00th=[   29],
     | 30.00th=[   32], 40.00th=[   35], 50.00th=[   37], 60.00th=[   41],
     | 70.00th=[   44], 80.00th=[   48], 90.00th=[   55], 95.00th=[   62],
     | 99.00th=[   83], 99.50th=[   92], 99.90th=[  116], 99.95th=[  127],
     | 99.99th=[  146]
   bw (  KiB/s): min=22144, max=39040, per=20.00%, avg=30620.64, stdev=3017.61, samples=600
   iops        : min=  346, max=  610, avg=478.42, stdev=47.13, samples=600
  lat (msec)   : 4=0.01%, 10=0.04%, 20=2.40%, 50=80.80%, 100=16.44%
  lat (msec)   : 250=0.32%
  cpu          : usr=1.25%, sys=3.15%, ctx=53378, majf=0, minf=162
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=334018,143647,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=348MiB/s (365MB/s), 348MiB/s-348MiB/s (365MB/s-365MB/s), io=20.4GiB (21.9GB), run=60032-60032msec
  WRITE: bw=150MiB/s (157MB/s), 150MiB/s-150MiB/s (157MB/s-157MB/s), io=8978MiB (9414MB), run=60032-60032msec

Disk stats (read/write):
  xvda: ios=667042/286880, merge=0/16, ticks=6848728/2593639, in_queue=9443213, util=99.93%

TEST-128K

RandomRW: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=29681: Tue Jan 21 12:59:07 2020
   read: IOPS=3201, BW=400MiB/s (420MB/s)(23.5GiB/60024msec)
    slat (usec): min=12, max=76111, avg=1053.60, stdev=3148.65
    clat (msec): min=7, max=200, avg=69.03, stdev=16.85
     lat (msec): min=10, max=206, avg=70.08, stdev=17.11
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   47], 10.00th=[   51], 20.00th=[   56],
     | 30.00th=[   60], 40.00th=[   64], 50.00th=[   67], 60.00th=[   71],
     | 70.00th=[   75], 80.00th=[   81], 90.00th=[   91], 95.00th=[  101],
     | 99.00th=[  124], 99.50th=[  133], 99.90th=[  150], 99.95th=[  157],
     | 99.99th=[  180]
   bw (  KiB/s): min=51968, max=101376, per=19.98%, avg=81871.10, stdev=7971.69, samples=600
   iops        : min=  406, max=  792, avg=639.57, stdev=62.28, samples=600
  write: IOPS=1381, BW=173MiB/s (181MB/s)(10.1GiB/60024msec)
    slat (usec): min=15, max=70975, avg=1146.26, stdev=3307.41
    clat (msec): min=10, max=206, avg=67.94, stdev=17.28
     lat (msec): min=10, max=206, avg=69.09, stdev=17.54
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   45], 10.00th=[   50], 20.00th=[   54],
     | 30.00th=[   58], 40.00th=[   62], 50.00th=[   66], 60.00th=[   70],
     | 70.00th=[   74], 80.00th=[   81], 90.00th=[   90], 95.00th=[  101],
     | 99.00th=[  124], 99.50th=[  133], 99.90th=[  153], 99.95th=[  161],
     | 99.99th=[  180]
   bw (  KiB/s): min=24576, max=47872, per=19.98%, avg=35337.70, stdev=3651.32, samples=600
   iops        : min=  192, max=  374, avg=276.04, stdev=28.53, samples=600
  lat (msec)   : 10=0.01%, 20=0.02%, 50=10.13%, 100=84.82%, 250=5.03%
  cpu          : usr=0.92%, sys=2.88%, ctx=45780, majf=0, minf=156
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=192137,82935,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=400MiB/s (420MB/s), 400MiB/s-400MiB/s (420MB/s-420MB/s), io=23.5GiB (25.2GB), run=60024-60024msec
  WRITE: bw=173MiB/s (181MB/s), 173MiB/s-173MiB/s (181MB/s-181MB/s), io=10.1GiB (10.9GB), run=60024-60024msec

Disk stats (read/write):
  xvda: ios=575050/248301, merge=0/13, ticks=6820275/2625659, in_queue=9447284, util=99.93%

TEST-256K

RandomRW: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=29700: Tue Jan 21 13:00:07 2020
   read: IOPS=1586, BW=397MiB/s (416MB/s)(23.2GiB/60020msec)
    slat (usec): min=21, max=79572, avg=2125.67, stdev=4464.84
    clat (msec): min=8, max=311, avg=138.40, stdev=25.57
     lat (msec): min=11, max=313, avg=140.53, stdev=25.98
    clat percentiles (msec):
     |  1.00th=[   96],  5.00th=[  107], 10.00th=[  112], 20.00th=[  118],
     | 30.00th=[  125], 40.00th=[  130], 50.00th=[  136], 60.00th=[  140],
     | 70.00th=[  146], 80.00th=[  155], 90.00th=[  171], 95.00th=[  188],
     | 99.00th=[  222], 99.50th=[  236], 99.90th=[  275], 99.95th=[  288],
     | 99.99th=[  305]
   bw (  KiB/s): min=46592, max=105984, per=19.96%, avg=81046.35, stdev=9985.40, samples=600
   iops        : min=  182, max=  414, avg=316.55, stdev=39.01, samples=600
  write: IOPS=685, BW=171MiB/s (180MB/s)(10.0GiB/60020msec)
    slat (usec): min=27, max=97617, avg=2341.58, stdev=4663.46
    clat (msec): min=11, max=317, avg=138.81, stdev=25.95
     lat (msec): min=11, max=336, avg=141.16, stdev=26.36
    clat percentiles (msec):
     |  1.00th=[   96],  5.00th=[  106], 10.00th=[  112], 20.00th=[  120],
     | 30.00th=[  125], 40.00th=[  130], 50.00th=[  136], 60.00th=[  140],
     | 70.00th=[  146], 80.00th=[  157], 90.00th=[  171], 95.00th=[  188],
     | 99.00th=[  228], 99.50th=[  239], 99.90th=[  271], 99.95th=[  288],
     | 99.99th=[  305]
   bw (  KiB/s): min=19456, max=50688, per=19.95%, avg=34989.38, stdev=5061.38, samples=600
   iops        : min=   76, max=  198, avg=136.60, stdev=19.76, samples=600
  lat (msec)   : 10=0.01%, 20=0.02%, 50=0.05%, 100=1.90%, 250=97.72%
  lat (msec)   : 500=0.30%
  cpu          : usr=0.65%, sys=2.54%, ctx=45509, majf=0, minf=157
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=95211,41117,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=397MiB/s (416MB/s), 397MiB/s-397MiB/s (416MB/s-416MB/s), io=23.2GiB (24.0GB), run=60020-60020msec
  WRITE: bw=171MiB/s (180MB/s), 171MiB/s-171MiB/s (180MB/s-180MB/s), io=10.0GiB (10.8GB), run=60020-60020msec

Disk stats (read/write):
  xvda: ios=569976/246066, merge=0/15, ticks=6665701/2788074, in_queue=9454737, util=99.94%

TEST-512K

RandomRW: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=29719: Tue Jan 21 13:01:08 2020
   read: IOPS=758, BW=379MiB/s (398MB/s)(22.2GiB/60016msec)
    slat (usec): min=39, max=105945, avg=4447.29, stdev=6790.04
    clat (msec): min=5, max=677, avg=287.99, stdev=52.31
     lat (msec): min=10, max=717, avg=292.44, stdev=52.97
    clat percentiles (msec):
     |  1.00th=[  201],  5.00th=[  226], 10.00th=[  236], 20.00th=[  251],
     | 30.00th=[  262], 40.00th=[  271], 50.00th=[  279], 60.00th=[  292],
     | 70.00th=[  305], 80.00th=[  321], 90.00th=[  351], 95.00th=[  380],
     | 99.00th=[  481], 99.50th=[  518], 99.90th=[  575], 99.95th=[  600],
     | 99.99th=[  642]
   bw (  KiB/s): min=18432, max=118784, per=19.97%, avg=77573.38, stdev=13187.47, samples=596
   iops        : min=   36, max=  232, avg=151.44, stdev=25.77, samples=596
  write: IOPS=328, BW=164MiB/s (172MB/s)(9861MiB/60016msec)
    slat (usec): min=54, max=104431, avg=4904.40, stdev=7201.14
    clat (msec): min=10, max=654, avg=289.98, stdev=51.88
     lat (msec): min=11, max=677, avg=294.88, stdev=52.61
    clat percentiles (msec):
     |  1.00th=[  203],  5.00th=[  228], 10.00th=[  239], 20.00th=[  253],
     | 30.00th=[  262], 40.00th=[  271], 50.00th=[  284], 60.00th=[  292],
     | 70.00th=[  305], 80.00th=[  321], 90.00th=[  351], 95.00th=[  380],
     | 99.00th=[  477], 99.50th=[  523], 99.90th=[  567], 99.95th=[  592],
     | 99.99th=[  642]
   bw (  KiB/s): min= 8175, max=54272, per=19.97%, avg=33597.81, stdev=6792.62, samples=596
   iops        : min=   15, max=  106, avg=65.53, stdev=13.26, samples=596
  lat (msec)   : 10=0.01%, 20=0.04%, 50=0.07%, 100=0.09%, 250=19.28%
  lat (msec)   : 500=79.75%, 750=0.77%
  cpu          : usr=0.53%, sys=1.97%, ctx=43413, majf=0, minf=159
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=45539,19721,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=379MiB/s (398MB/s), 379MiB/s-379MiB/s (398MB/s-398MB/s), io=22.2GiB (23.9GB), run=60016-60016msec
  WRITE: bw=164MiB/s (172MB/s), 164MiB/s-164MiB/s (172MB/s-172MB/s), io=9861MiB (10.3GB), run=60016-60016msec

Disk stats (read/write):
  xvda: ios=544635/235907, merge=1/21, ticks=6438766/2999860, in_queue=9445264, util=99.88%

TEST-1024K

RandomRW: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.7
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=29752: Tue Jan 21 13:02:09 2020
   read: IOPS=389, BW=389MiB/s (408MB/s)(22.8GiB/60014msec)
    slat (usec): min=75, max=80473, avg=8403.46, stdev=7657.50
    clat (msec): min=2, max=798, avg=559.71, stdev=68.65
     lat (msec): min=10, max=808, avg=568.11, stdev=69.31
    clat percentiles (msec):
     |  1.00th=[  430],  5.00th=[  472], 10.00th=[  489], 20.00th=[  510],
     | 30.00th=[  527], 40.00th=[  542], 50.00th=[  550], 60.00th=[  567],
     | 70.00th=[  592], 80.00th=[  609], 90.00th=[  651], 95.00th=[  676],
     | 99.00th=[  726], 99.50th=[  735], 99.90th=[  768], 99.95th=[  776],
     | 99.99th=[  793]
   bw (  KiB/s): min=12288, max=114688, per=19.91%, avg=79368.42, stdev=11822.37, samples=594
   iops        : min=   12, max=  112, avg=77.45, stdev=11.56, samples=594
  write: IOPS=170, BW=170MiB/s (179MB/s)(9.99GiB/60014msec)
    slat (usec): min=110, max=95076, avg=10090.65, stdev=8286.75
    clat (msec): min=9, max=772, avg=561.42, stdev=69.02
     lat (msec): min=11, max=791, avg=571.51, stdev=69.72
    clat percentiles (msec):
     |  1.00th=[  426],  5.00th=[  477], 10.00th=[  493], 20.00th=[  514],
     | 30.00th=[  527], 40.00th=[  542], 50.00th=[  558], 60.00th=[  575],
     | 70.00th=[  592], 80.00th=[  617], 90.00th=[  651], 95.00th=[  676],
     | 99.00th=[  718], 99.50th=[  735], 99.90th=[  751], 99.95th=[  760],
     | 99.99th=[  768]
   bw (  KiB/s): min= 2048, max=63488, per=19.84%, avg=34642.58, stdev=8552.87, samples=594
   iops        : min=    2, max=   62, avg=33.77, stdev= 8.37, samples=594
  lat (msec)   : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.07%, 100=0.07%
  lat (msec)   : 250=0.26%, 500=13.29%, 750=86.09%, 1000=0.18%
  cpu          : usr=0.56%, sys=1.69%, ctx=44871, majf=0, minf=160
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.5%, >=64=99.1%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=23360,10231,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=389MiB/s (408MB/s), 389MiB/s-389MiB/s (408MB/s-408MB/s), io=22.8GiB (24.5GB), run=60014-60014msec
  WRITE: bw=170MiB/s (179MB/s), 170MiB/s-170MiB/s (179MB/s-179MB/s), io=9.99GiB (10.7GB), run=60014-60014msec

Disk stats (read/write):
  xvda: ios=560489/245371, merge=0/14, ticks=6106984/3371245, in_queue=9479229, util=99.78%