start test

TEST-4K

RandomRW: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.1
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31120: Tue Jan 28 17:01:05 2020
   read: IOPS=36.3k, BW=142MiB/s (149MB/s)(8506MiB/60002msec)
    slat (usec): min=2, max=34466, avg=88.60, stdev=366.11
    clat (usec): min=160, max=42862, avg=4856.37, stdev=2139.68
     lat (usec): min=168, max=46101, avg=4945.87, stdev=2172.89
    clat percentiles (usec):
     |  1.00th=[ 1450],  5.00th=[ 2245], 10.00th=[ 2638], 20.00th=[ 3163],
     | 30.00th=[ 3589], 40.00th=[ 4015], 50.00th=[ 4424], 60.00th=[ 4883],
     | 70.00th=[ 5538], 80.00th=[ 6325], 90.00th=[ 7701], 95.00th=[ 8979],
     | 99.00th=[11600], 99.50th=[12649], 99.90th=[15270], 99.95th=[16909],
     | 99.99th=[33424]
   bw (  KiB/s): min=23000, max=34720, per=20.02%, avg=29055.03, stdev=1851.83, samples=600
   iops        : min= 5750, max= 8680, avg=7263.60, stdev=462.95, samples=600
  write: IOPS=15.6k, BW=60.9MiB/s (63.9MB/s)(3654MiB/60002msec)
    slat (usec): min=3, max=33464, avg=94.52, stdev=375.84
    clat (usec): min=535, max=52185, avg=8905.25, stdev=2936.35
     lat (usec): min=1441, max=52192, avg=9000.67, stdev=2950.20
    clat percentiles (usec):
     |  1.00th=[ 3785],  5.00th=[ 4817], 10.00th=[ 5473], 20.00th=[ 6456],
     | 30.00th=[ 7177], 40.00th=[ 7898], 50.00th=[ 8586], 60.00th=[ 9241],
     | 70.00th=[10028], 80.00th=[11076], 90.00th=[12649], 95.00th=[14222],
     | 99.00th=[17433], 99.50th=[18744], 99.90th=[21890], 99.95th=[24511],
     | 99.99th=[41157]
   bw (  KiB/s): min= 9824, max=14688, per=20.01%, avg=12481.55, stdev=821.23, samples=600
   iops        : min= 2456, max= 3672, avg=3120.26, stdev=205.31, samples=600
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.13%
  lat (msec)   : 2=2.09%, 4=26.16%, 10=60.40%, 20=11.09%, 50=0.09%
  lat (msec)   : 100=0.01%
  cpu          : usr=5.42%, sys=10.91%, ctx=289961, majf=0, minf=52
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=2177466,935462,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=8506MiB (8919MB), run=60002-60002msec
  WRITE: bw=60.9MiB/s (63.9MB/s), 60.9MiB/s-60.9MiB/s (63.9MB/s-63.9MB/s), io=3654MiB (3832MB), run=60002-60002msec

Disk stats (read/write):
  vda: ios=2171739/932956, merge=0/39, ticks=2328252/4847852, in_queue=7079696, util=100.00%

TEST-8K

RandomRW: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=64
...
fio-3.1
Starting 5 processes

RandomRW: (groupid=0, jobs=5): err= 0: pid=31127: Tue Jan 28 17:02:06 2020
   read: IOPS=30.3k, BW=237MiB/s (248MB/s)(13.9GiB/60004msec)
    slat (usec): min=2, max=21633, avg=106.98, stdev=434.69
    clat (usec): min=433, max=32011, avg=5797.43, stdev=2525.35
     lat (usec): min=441, max=32385, avg=5905.32, stdev=2566.77
    clat percentiles (usec):
     |  1.00th=[ 1762],  5.00th=[ 2638], 10.00th=[ 3130], 20.00th=[ 3752],
     | 30.00th=[ 4293], 40.00th=[ 4752], 50.00th=[ 5276], 60.00th=[ 5866],
     | 70.00th=[ 6652], 80.00th=[ 7635], 90.00th=[ 9110], 95.00th=[10552],
     | 99.00th=[13829], 99.50th=[15139], 99.90th=[18482], 99.95th=[20055],
     | 99.99th=[23725]
   bw (  KiB/s): min=36240, max=60809, per=20.03%, avg=48587.61, stdev=4327.73, samples=600
   iops        : min= 4530, max= 7601, avg=6073.20, stdev=540.98, samples=600
  write: IOPS=13.0k, BW=102MiB/s (107MB/s)(6114MiB/60004msec)
    slat (usec): min=3, max=16280, avg=114.30, stdev=443.96
    clat (usec): min=1237, max=44613, avg=10675.16, stdev=3456.56
     lat (usec): min=1279, max=44624, avg=10790.37, stdev=3472.10
    clat percentiles (usec):
     |  1.00th=[ 4490],  5.00th=[ 5800], 10.00th=[ 6587], 20.00th=[ 7767],
     | 30.00th=[ 8717], 40.00th=[ 9503], 50.00th=[10290], 60.00th=[11076],
     | 70.00th=[11994], 80.00th=[13304], 90.00th=[15139], 95.00th=[16909],
     | 99.00th=[20841], 99.50th=[22414], 99.90th=[26084], 99.95th=[27657],
     | 99.99th=[31065]
   bw (  KiB/s): min=15663, max=26608, per=20.03%, avg=20894.32, stdev=1907.88, samples=600
   iops        : min= 1957, max= 3326, avg=2611.51, stdev=238.51, samples=600
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.07%
  lat (msec)   : 2=1.06%, 4=16.27%, 10=61.75%, 20=20.37%, 50=0.47%
  cpu          : usr=4.63%, sys=9.76%, ctx=257188, majf=0, minf=57
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=1819658,782570,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=237MiB/s (248MB/s), 237MiB/s-237MiB/s (248MB/s-248MB/s), io=13.9GiB (14.9GB), run=60004-60004msec
  WRITE: bw=102MiB/s (107MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=6114MiB (6411MB), run=60004-60004msec

Disk stats (read/write):
  vda: ios=1816345/781217, merge=0/53, ticks=2225276/5015124, in_queue=7162560, util=100.00%

TEST-16K

RandomRW: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
...
fio-3.1
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31135: Tue Jan 28 17:03:47 2020
   read: IOPS=24.4k, BW=382MiB/s (401MB/s)(22.4GiB/60004msec)
    slat (usec): min=3, max=86616, avg=133.65, stdev=531.79
    clat (usec): min=459, max=116246, avg=7200.48, stdev=3231.36
     lat (usec): min=498, max=116352, avg=7335.09, stdev=3276.51
    clat percentiles (usec):
     |  1.00th=[ 2057],  5.00th=[ 3163], 10.00th=[ 3785], 20.00th=[ 4621],
     | 30.00th=[ 5276], 40.00th=[ 5997], 50.00th=[ 6652], 60.00th=[ 7439],
     | 70.00th=[ 8356], 80.00th=[ 9503], 90.00th=[11338], 95.00th=[12911],
     | 99.00th=[16319], 99.50th=[17695], 99.90th=[21365], 99.95th=[24511],
     | 99.99th=[94897]
   bw (  KiB/s): min=51168, max=97767, per=20.03%, avg=78367.57, stdev=7766.49, samples=600
   iops        : min= 3198, max= 6110, avg=4897.75, stdev=485.40, samples=600
  write: IOPS=10.5k, BW=164MiB/s (172MB/s)(9865MiB/60004msec)
    slat (usec): min=3, max=32758, avg=143.74, stdev=527.71
    clat (usec): min=1097, max=133831, avg=13210.04, stdev=4414.59
     lat (usec): min=1292, max=133843, avg=13354.75, stdev=4426.39
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   15], 80.00th=[   17], 90.00th=[   19], 95.00th=[   21],
     | 99.00th=[   26], 99.50th=[   28], 99.90th=[   33], 99.95th=[   39],
     | 99.99th=[  118]
   bw (  KiB/s): min=21333, max=42400, per=20.03%, avg=33721.53, stdev=3359.29, samples=600
   iops        : min= 1333, max= 2650, avg=2107.36, stdev=209.96, samples=600
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.05%
  lat (msec)   : 2=0.57%, 4=7.97%, 10=56.12%, 20=33.26%, 50=2.00%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=4.00%, sys=8.42%, ctx=230837, majf=0, minf=56
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=1467068,631342,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=382MiB/s (401MB/s), 382MiB/s-382MiB/s (401MB/s-401MB/s), io=22.4GiB (24.0GB), run=60004-60004msec
  WRITE: bw=164MiB/s (172MB/s), 164MiB/s-164MiB/s (172MB/s-172MB/s), io=9865MiB (10.3GB), run=60004-60004msec

Disk stats (read/write):
  vda: ios=1465567/630654, merge=0/96, ticks=2148204/5158292, in_queue=7223148, util=100.00%

TEST-32K

RandomRW: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64
...
fio-3.1
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31144: Tue Jan 28 17:05:29 2020
   read: IOPS=17.3k, BW=540MiB/s (566MB/s)(31.7GiB/60007msec)
    slat (usec): min=3, max=161403, avg=189.77, stdev=891.78
    clat (usec): min=245, max=313435, avg=10202.22, stdev=6503.69
     lat (usec): min=250, max=371817, avg=10393.00, stdev=6597.86
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    5], 10.00th=[    6], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   12], 80.00th=[   14], 90.00th=[   16], 95.00th=[   18],
     | 99.00th=[   23], 99.50th=[   25], 99.90th=[  105], 99.95th=[  144],
     | 99.99th=[  199]
   bw (  KiB/s): min=13146, max=154176, per=20.02%, avg=110758.23, stdev=15148.08, samples=600
   iops        : min=  410, max= 4818, avg=3461.01, stdev=473.38, samples=600
  write: IOPS=7434, BW=232MiB/s (244MB/s)(13.6GiB/60007msec)
    slat (usec): min=4, max=126242, avg=209.22, stdev=913.73
    clat (msec): min=2, max=363, avg=18.65, stdev= 9.94
     lat (msec): min=2, max=401, avg=18.86, stdev=10.02
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[   10], 10.00th=[   12], 20.00th=[   14],
     | 30.00th=[   16], 40.00th=[   17], 50.00th=[   18], 60.00th=[   20],
     | 70.00th=[   21], 80.00th=[   23], 90.00th=[   26], 95.00th=[   29],
     | 99.00th=[   35], 99.50th=[   40], 99.90th=[  184], 99.95th=[  213],
     | 99.99th=[  309]
   bw (  KiB/s): min= 4232, max=65088, per=20.02%, avg=47633.06, stdev=6588.97, samples=600
   iops        : min=  132, max= 2034, avg=1488.36, stdev=205.92, samples=600
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.24%, 4=2.47%, 10=37.15%, 20=48.55%, 50=11.37%
  lat (msec)   : 100=0.05%, 250=0.15%, 500=0.01%
  cpu          : usr=3.15%, sys=6.82%, ctx=173037, majf=0, minf=53
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=1037207,446123,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=540MiB/s (566MB/s), 540MiB/s-540MiB/s (566MB/s-566MB/s), io=31.7GiB (33.0GB), run=60007-60007msec
  WRITE: bw=232MiB/s (244MB/s), 232MiB/s-232MiB/s (244MB/s-244MB/s), io=13.6GiB (14.6GB), run=60007-60007msec

Disk stats (read/write):
  vda: ios=1035092/445176, merge=0/116, ticks=2103020/5248168, in_queue=7276936, util=100.00%

TEST-64K

RandomRW: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.1
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31238: Tue Jan 28 17:07:11 2020
   read: IOPS=10.4k, BW=652MiB/s (683MB/s)(38.2GiB/60009msec)
    slat (usec): min=4, max=65194, avg=315.23, stdev=1230.97
    clat (usec): min=563, max=98551, avg=17089.69, stdev=7321.09
     lat (usec): min=998, max=122691, avg=17405.98, stdev=7392.92
    clat percentiles (usec):
     |  1.00th=[ 4555],  5.00th=[ 6849], 10.00th=[ 8291], 20.00th=[10552],
     | 30.00th=[12518], 40.00th=[14484], 50.00th=[16581], 60.00th=[18482],
     | 70.00th=[20579], 80.00th=[22938], 90.00th=[26346], 95.00th=[29492],
     | 99.00th=[36439], 99.50th=[40109], 99.90th=[56886], 99.95th=[68682],
     | 99.99th=[85459]
   bw (  KiB/s): min=92258, max=173585, per=20.06%, avg=133859.74, stdev=12872.24, samples=600
   iops        : min= 1441, max= 2712, avg=2091.27, stdev=201.13, samples=600
  write: IOPS=4479, BW=280MiB/s (294MB/s)(16.4GiB/60009msec)
    slat (usec): min=6, max=55354, avg=360.14, stdev=1308.33
    clat (msec): min=2, max=144, avg=30.55, stdev= 9.66
     lat (msec): min=5, max=146, avg=30.91, stdev= 9.66
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   17], 10.00th=[   20], 20.00th=[   24],
     | 30.00th=[   26], 40.00th=[   28], 50.00th=[   30], 60.00th=[   32],
     | 70.00th=[   34], 80.00th=[   37], 90.00th=[   43], 95.00th=[   48],
     | 99.00th=[   59], 99.50th=[   64], 99.90th=[   85], 99.95th=[   95],
     | 99.99th=[  115]
   bw (  KiB/s): min=37066, max=80482, per=20.06%, avg=57507.91, stdev=5757.54, samples=600
   iops        : min=  579, max= 1257, avg=898.25, stdev=89.96, samples=600
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.06%, 4=0.40%, 10=11.97%, 20=38.17%, 50=48.19%
  lat (msec)   : 100=1.20%, 250=0.01%
  cpu          : usr=2.12%, sys=4.84%, ctx=109625, majf=0, minf=53
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=625542,268781,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=652MiB/s (683MB/s), 652MiB/s-652MiB/s (683MB/s-683MB/s), io=38.2GiB (40.0GB), run=60009-60009msec
  WRITE: bw=280MiB/s (294MB/s), 280MiB/s-280MiB/s (294MB/s-294MB/s), io=16.4GiB (17.6GB), run=60009-60009msec

Disk stats (read/write):
  vda: ios=624928/268609, merge=0/50, ticks=1973664/5424992, in_queue=7337404, util=100.00%

TEST-128K

RandomRW: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
...
fio-3.1
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31246: Tue Jan 28 17:08:51 2020
   read: IOPS=6654, BW=832MiB/s (872MB/s)(48.8GiB/60024msec)
    slat (usec): min=6, max=45798, avg=491.94, stdev=1848.96
    clat (usec): min=477, max=86027, avg=26747.56, stdev=10876.28
     lat (usec): min=500, max=95385, avg=27240.70, stdev=10944.20
    clat percentiles (usec):
     |  1.00th=[ 6521],  5.00th=[10552], 10.00th=[13042], 20.00th=[16581],
     | 30.00th=[19792], 40.00th=[23200], 50.00th=[26346], 60.00th=[29492],
     | 70.00th=[32375], 80.00th=[35914], 90.00th=[41157], 95.00th=[45351],
     | 99.00th=[54264], 99.50th=[57934], 99.90th=[66323], 99.95th=[69731],
     | 99.99th=[76022]
   bw (  KiB/s): min=122345, max=226081, per=20.12%, avg=171377.71, stdev=16341.89, samples=600
   iops        : min=  955, max= 1766, avg=1338.47, stdev=127.66, samples=600
  write: IOPS=2862, BW=358MiB/s (375MB/s)(20.0GiB/60024msec)
    slat (usec): min=9, max=38797, avg=577.66, stdev=1979.83
    clat (msec): min=5, max=130, avg=47.85, stdev=14.26
     lat (msec): min=7, max=130, avg=48.43, stdev=14.21
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   26], 10.00th=[   31], 20.00th=[   37],
     | 30.00th=[   42], 40.00th=[   44], 50.00th=[   47], 60.00th=[   50],
     | 70.00th=[   54], 80.00th=[   58], 90.00th=[   67], 95.00th=[   74],
     | 99.00th=[   89], 99.50th=[   94], 99.90th=[  106], 99.95th=[  110],
     | 99.99th=[  121]
   bw (  KiB/s): min=48064, max=93370, per=20.11%, avg=73684.74, stdev=7567.81, samples=600
   iops        : min=  375, max=  729, avg=575.32, stdev=59.13, samples=600
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.18%, 10=2.66%, 20=18.93%, 50=64.88%
  lat (msec)   : 100=13.26%, 250=0.06%
  cpu          : usr=1.91%, sys=3.95%, ctx=70361, majf=0, minf=52
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=399403,171831,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=832MiB/s (872MB/s), 832MiB/s-832MiB/s (872MB/s-872MB/s), io=48.8GiB (52.3GB), run=60024-60024msec
  WRITE: bw=358MiB/s (375MB/s), 358MiB/s-358MiB/s (375MB/s-375MB/s), io=20.0GiB (22.5GB), run=60024-60024msec

Disk stats (read/write):
  vda: ios=397987/171179, merge=0/51, ticks=1921972/5450292, in_queue=7298668, util=100.00%

TEST-256K

RandomRW: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.1
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31324: Tue Jan 28 17:10:32 2020
   read: IOPS=3523, BW=881MiB/s (924MB/s)(51.6GiB/60026msec)
    slat (usec): min=10, max=89073, avg=923.85, stdev=3888.96
    clat (usec): min=912, max=177629, avg=50408.02, stdev=22101.77
     lat (usec): min=940, max=177872, avg=51333.10, stdev=22268.74
    clat percentiles (msec):
     |  1.00th=[   11],  5.00th=[   18], 10.00th=[   22], 20.00th=[   29],
     | 30.00th=[   36], 40.00th=[   44], 50.00th=[   51], 60.00th=[   57],
     | 70.00th=[   63], 80.00th=[   70], 90.00th=[   80], 95.00th=[   88],
     | 99.00th=[  105], 99.50th=[  112], 99.90th=[  134], 99.95th=[  144],
     | 99.99th=[  157]
   bw (  KiB/s): min=107008, max=249344, per=20.04%, avg=180734.09, stdev=24549.85, samples=600
   iops        : min=  418, max=  974, avg=705.67, stdev=95.91, samples=600
  write: IOPS=1520, BW=380MiB/s (399MB/s)(22.3GiB/60026msec)
    slat (usec): min=17, max=100100, avg=1119.58, stdev=4225.12
    clat (msec): min=7, max=225, avg=90.27, stdev=26.70
     lat (msec): min=7, max=225, avg=91.39, stdev=26.77
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   50], 10.00th=[   59], 20.00th=[   70],
     | 30.00th=[   78], 40.00th=[   83], 50.00th=[   88], 60.00th=[   94],
     | 70.00th=[  102], 80.00th=[  110], 90.00th=[  126], 95.00th=[  140],
     | 99.00th=[  165], 99.50th=[  176], 99.90th=[  199], 99.95th=[  205],
     | 99.99th=[  220]
   bw (  KiB/s): min=32256, max=111550, per=20.03%, avg=77953.54, stdev=11205.39, samples=600
   iops        : min=  126, max=  435, avg=304.23, stdev=43.77, samples=600
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.06%, 10=0.46%, 20=4.73%, 50=31.53%
  lat (msec)   : 100=52.88%, 250=10.34%
  cpu          : usr=1.28%, sys=2.93%, ctx=37869, majf=0, minf=44
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=211501,91256,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=881MiB/s (924MB/s), 881MiB/s-881MiB/s (924MB/s-924MB/s), io=51.6GiB (55.4GB), run=60026-60026msec
  WRITE: bw=380MiB/s (399MB/s), 380MiB/s-380MiB/s (399MB/s-399MB/s), io=22.3GiB (23.9GB), run=60026-60026msec

Disk stats (read/write):
  vda: ios=211147/91119, merge=0/75, ticks=1710392/5673716, in_queue=7241908, util=99.76%

TEST-512K

RandomRW: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=64
...
fio-3.1
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31346: Tue Jan 28 17:12:11 2020
   read: IOPS=1650, BW=825MiB/s (865MB/s)(48.4GiB/60069msec)
    slat (usec): min=17, max=194578, avg=1935.43, stdev=8250.89
    clat (msec): min=2, max=429, avg=107.24, stdev=46.91
     lat (msec): min=3, max=490, avg=109.18, stdev=47.19
    clat percentiles (msec):
     |  1.00th=[   22],  5.00th=[   35], 10.00th=[   45], 20.00th=[   65],
     | 30.00th=[   82], 40.00th=[   94], 50.00th=[  107], 60.00th=[  120],
     | 70.00th=[  132], 80.00th=[  146], 90.00th=[  165], 95.00th=[  184],
     | 99.00th=[  230], 99.50th=[  259], 99.90th=[  313], 99.95th=[  342],
     | 99.99th=[  388]
   bw (  KiB/s): min=50377, max=273955, per=20.04%, avg=169377.10, stdev=34807.86, samples=600
   iops        : min=   98, max=  535, avg=330.51, stdev=67.96, samples=600
  write: IOPS=711, BW=356MiB/s (373MB/s)(20.9GiB/60069msec)
    slat (usec): min=31, max=143739, avg=2500.09, stdev=9079.88
    clat (msec): min=18, max=683, avg=193.50, stdev=65.37
     lat (msec): min=18, max=683, avg=196.00, stdev=65.46
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   94], 10.00th=[  117], 20.00th=[  146],
     | 30.00th=[  163], 40.00th=[  176], 50.00th=[  186], 60.00th=[  201],
     | 70.00th=[  218], 80.00th=[  241], 90.00th=[  279], 95.00th=[  309],
     | 99.00th=[  380], 99.50th=[  418], 99.90th=[  550], 99.95th=[  600],
     | 99.99th=[  676]
   bw (  KiB/s): min=11354, max=119261, per=20.02%, avg=72959.74, stdev=16734.22, samples=600
   iops        : min=   22, max=  232, avg=142.19, stdev=32.65, samples=600
  lat (msec)   : 4=0.01%, 10=0.11%, 20=0.42%, 50=8.36%, 100=24.39%
  lat (msec)   : 250=61.27%, 500=5.39%, 750=0.06%
  cpu          : usr=0.92%, sys=2.20%, ctx=17195, majf=0, minf=41
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=99148,42756,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=825MiB/s (865MB/s), 825MiB/s-825MiB/s (865MB/s-865MB/s), io=48.4GiB (51.0GB), run=60069-60069msec
  WRITE: bw=356MiB/s (373MB/s), 356MiB/s-356MiB/s (373MB/s-373MB/s), io=20.9GiB (22.4GB), run=60069-60069msec

Disk stats (read/write):
  vda: ios=98824/42621, merge=0/58, ticks=1916472/5534080, in_queue=7179484, util=97.86%

TEST-1024K

RandomRW: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.1
Starting 5 processes
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)
RandomRW: Laying out IO file (1 file / 5120MiB)

RandomRW: (groupid=0, jobs=5): err= 0: pid=31356: Tue Jan 28 17:13:52 2020
   read: IOPS=865, BW=866MiB/s (908MB/s)(50.8GiB/60116msec)
    slat (usec): min=33, max=403563, avg=3601.53, stdev=14803.64
    clat (usec): min=1323, max=895006, avg=201908.33, stdev=88070.70
     lat (usec): min=1442, max=1054.4k, avg=205511.28, stdev=88811.08
    clat percentiles (msec):
     |  1.00th=[   34],  5.00th=[   75], 10.00th=[  101], 20.00th=[  131],
     | 30.00th=[  153], 40.00th=[  174], 50.00th=[  194], 60.00th=[  215],
     | 70.00th=[  239], 80.00th=[  268], 90.00th=[  313], 95.00th=[  351],
     | 99.00th=[  456], 99.50th=[  518], 99.90th=[  651], 99.95th=[  751],
     | 99.99th=[  877]
   bw (  KiB/s): min=24874, max=436224, per=20.09%, avg=178098.15, stdev=51512.13, samples=600
   iops        : min=   24, max=  426, avg=173.58, stdev=50.29, samples=600
  write: IOPS=375, BW=376MiB/s (394MB/s)(22.1GiB/60116msec)
    slat (usec): min=60, max=214796, avg=4944.74, stdev=16542.01
    clat (msec): min=36, max=1133, avg=371.39, stdev=119.56
     lat (msec): min=36, max=1175, avg=376.34, stdev=119.60
    clat percentiles (msec):
     |  1.00th=[  132],  5.00th=[  192], 10.00th=[  226], 20.00th=[  275],
     | 30.00th=[  309], 40.00th=[  334], 50.00th=[  359], 60.00th=[  388],
     | 70.00th=[  422], 80.00th=[  460], 90.00th=[  527], 95.00th=[  584],
     | 99.00th=[  718], 99.50th=[  768], 99.90th=[  927], 99.95th=[ 1099],
     | 99.99th=[ 1133]
   bw (  KiB/s): min= 4096, max=153600, per=20.10%, avg=77378.56, stdev=25678.74, samples=597
   iops        : min=    4, max=  150, avg=75.28, stdev=25.09, samples=597
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.04%, 20=0.29%, 50=1.21%
  lat (msec)   : 100=5.52%, 250=48.99%, 500=39.60%, 750=4.08%, 1000=0.22%
  lat (msec)   : 2000=0.03%
  cpu          : usr=0.82%, sys=1.90%, ctx=8673, majf=0, minf=42
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=52039,22596,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=866MiB/s (908MB/s), 866MiB/s-866MiB/s (908MB/s-908MB/s), io=50.8GiB (54.6GB), run=60116-60116msec
  WRITE: bw=376MiB/s (394MB/s), 376MiB/s-376MiB/s (394MB/s-394MB/s), io=22.1GiB (23.7GB), run=60116-60116msec

Disk stats (read/write):
  vda: ios=51896/22456, merge=0/95, ticks=2335856/5095984, in_queue=7027924, util=95.87%