• Welcome to LiuJason's Blog!
> Coding Projects > Linux笔记 > CentOS下使用fio测试挂载了CEPH rbd磁盘的读写性能|4k随机/顺序读写

CentOS下使用fio测试挂载了CEPH rbd磁盘的读写性能|4k随机/顺序读写

Linux笔记 Jason 4 years ago (2020-12-13) 731 Views 0 Comments QR code of this page
文章目录[隐藏]

背景简介

继续上文的ceph性能调整，有兴趣的可以看看：

使用rbd bench/bench-write自带测速工具和fio对ceph image进行性能测试|测速跑分

文章目录[隐藏] 前言 集群配置 开始测试 前言 最近在折腾CEPH集群，测试环境是4台ceph物理节点，每台8~12个SATA/SAS机械硬盘，共43个OSD。性能远低于预期，目前正在调整配置中...... 正好实测了一下性能，这里分享出来大家参考。对CEPH Pool的性能测试可以看前一篇： 集群配置 4 CEPH节点：SH-1001 SH-1003 SH-1004 SH-1005 2Manager：SH-1001 SH-1005 3Monitor：SH-1004 SH-2001 SH-2003 开始测试 内网测速 root@SH-1005:~# iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.1.0.1, port 42784 [ 5] local 10.1.0.5 port 5201 connected to 10.1.0.1 port 42786 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 […]


使用rados自带测速工具对ceph pool进行性能测试|测速跑分

文章目录[隐藏] 前言 集群配置 开始测试 前言 最近在折腾CEPH集群，测试环境是4台ceph物理节点，每台8~12个SATA/SAS机械硬盘，共43个OSD。性能远低于预期，目前正在调整配置中...... 正好实测了一下性能，这里分享出来大家参考。简要信息通过ceph -s即可获取： 集群配置 4 CEPH节点：SH-1001 SH-1003 SH-1004 SH-1005 2Manager：SH-1001 SH-1005 3Monitor：SH-1004 SH-2001 SH-2003 开始测试 内网测速 root@SH-1005:~# iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.1.0.1, port 42784 [ 5] local 10.1.0.5 port 5201 connected to 10.1.0.1 port 42786 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 […]


测试环境的Ceph集群配置同前文，所以就不赘述了。将上述的Ceph pool中新建raw格式的ceph rbd，并挂载在虚拟机的/vz下，qemu cache设置为writeback

[root@hn03 ~]# df -h
Filesystem                            Size  Used Avail Use% Mounted on
devtmpfs                              126G     0  126G   0% /dev
tmpfs                                 126G   12K  126G   1% /dev/shm
tmpfs                                 126G 1008K  126G   1% /run
tmpfs                                 126G     0  126G   0% /sys/fs/cgroup
/dev/sda4                             165G  5.7G  151G   4% /
tmpfs                                 126G   20K  126G   1% /tmp
/dev/sda2                             976M  170M  739M  19% /boot
/dev/mapper/vhs_jelastic--host--3-vz  8.0T   17G  7.6T   1% /vz
/dev/ploop13141p1                     9.8G  1.3G  8.0G  14% /vz/pfcache
10.1.3.2:/home/nfs/backup/hn03         33T   24T  7.3T  77% /vz/backups
/dev/ploop48143p1                     241G  548M  230G   1% /vz/root/2629
tmpfs                                  26G     0   26G   0% /run/user/0

开始测试

4K随机写性能

[root@hn03 ~]# fio -filename=/vz/test_4kwrite -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=100M -numjobs=10 -runtime=60 -group_reporting -name=test4kwrite
test4kwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.14
Starting 10 threads
test4kwrite: Laying out IO file (1 file / 100MiB)
Jobs: 10 (f=10): [w(10)][100.0%][w=4336KiB/s][w=1084 IOPS][eta 00m:00s]
test4kwrite: (groupid=0, jobs=10): err= 0: pid=292932: Sun Dec 13 16:10:38 2020
  write: IOPS=2454, BW=9820KiB/s (10.1MB/s)(577MiB/60196msec)
    clat (usec): min=141, max=1677.1k, avg=4061.46, stdev=52237.91
     lat (usec): min=142, max=1677.1k, avg=4062.74, stdev=52237.96
    clat percentiles (usec):
     |  1.00th=[    157],  5.00th=[    161], 10.00th=[    163],
     | 20.00th=[    167], 30.00th=[    174], 40.00th=[    180],
     | 50.00th=[    184], 60.00th=[    190], 70.00th=[    200],
     | 80.00th=[    215], 90.00th=[    258], 95.00th=[  10945],
     | 99.00th=[  53216], 99.50th=[  68682], 99.90th=[1216349],
     | 99.95th=[1350566], 99.99th=[1619002]
   bw (  KiB/s): min=  159, max=48638, per=100.00%, avg=14139.18, stdev=1074.69, samples=836
   iops        : min=   39, max=12158, avg=3534.29, stdev=268.68, samples=836
  lat (usec)   : 250=89.14%, 500=4.80%, 750=0.06%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=0.10%, 10=0.72%, 20=1.17%, 50=2.83%
  lat (msec)   : 100=0.88%, 250=0.09%, 500=0.01%, 1000=0.02%, 2000=0.13%
  cpu          : usr=0.27%, sys=2.23%, ctx=295619, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,147774,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9820KiB/s (10.1MB/s), 9820KiB/s-9820KiB/s (10.1MB/s-10.1MB/s), io=577MiB (605MB), run=60196-60196msec

Disk stats (read/write):
    dm-0: ios=1/148479, merge=0/0, ticks=74/176011, in_queue=218493, util=96.85%, aggrios=1/147862, aggrmerge=0/636, aggrticks=74/158079, aggrin_queue=159152, aggrutil=96.39%
  sdb: ios=1/147862, merge=0/636, ticks=74/158079, in_queue=159152, util=96.39%

4k随机读

[root@hn03 ~]# fio -filename=/vz/test_4kread -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=4k -size=100M -numjobs=10 -runtime=60 -group_reporting -name=test4kread
test4kread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.14
Starting 10 threads
Jobs: 2 (f=2): [_(1),r(1),_(1),r(1),_(6)][100.0%][r=42.4MiB/s][r=10.8k IOPS][eta 00m:00s]
test4kread: (groupid=0, jobs=10): err= 0: pid=303812: Sun Dec 13 16:17:48 2020
  read: IOPS=7743, BW=30.2MiB/s (31.7MB/s)(1000MiB/33062msec)
    clat (usec): min=6, max=465810, avg=1267.16, stdev=6697.91
     lat (usec): min=7, max=465811, avg=1268.53, stdev=6697.91
    clat percentiles (usec):
     |  1.00th=[   145],  5.00th=[   167], 10.00th=[   186], 20.00th=[   239],
     | 30.00th=[   478], 40.00th=[   709], 50.00th=[   766], 60.00th=[   824],
     | 70.00th=[   881], 80.00th=[   955], 90.00th=[  1090], 95.00th=[  1270],
     | 99.00th=[ 20579], 99.50th=[ 37487], 99.90th=[ 74974], 99.95th=[119014],
     | 99.99th=[283116]
   bw (  KiB/s): min=  552, max=62816, per=100.00%, avg=31085.23, stdev=2034.02, samples=648
   iops        : min=  138, max=15704, avg=7770.71, stdev=508.54, samples=648
  lat (usec)   : 10=0.04%, 20=0.05%, 50=0.05%, 100=0.02%, 250=21.29%
  lat (usec)   : 500=8.64%, 750=16.49%, 1000=37.87%
  lat (msec)   : 2=13.23%, 4=0.26%, 10=0.57%, 20=0.47%, 50=0.79%
  lat (msec)   : 100=0.16%, 250=0.05%, 500=0.01%
  cpu          : usr=1.08%, sys=6.70%, ctx=256844, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=256000,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=30.2MiB/s (31.7MB/s), 30.2MiB/s-30.2MiB/s (31.7MB/s-31.7MB/s), io=1000MiB (1049MB), run=33062-33062msec

Disk stats (read/write):
    dm-0: ios=255350/0, merge=0/0, ticks=305061/0, in_queue=306628, util=99.92%, aggrios=255590/0, aggrmerge=0/0, aggrticks=305374/0, aggrin_queue=305203, aggrutil=99.72%
  sdb: ios=255590/0, merge=0/0, ticks=305374/0, in_queue=305203, util=99.72%

顺序写

[root@hn03 ~]# fio -filename=/vz/test_seqwrite -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1G -size=10G -numjobs=10 -runtime=60 -group_reporting -name=testseqwrite
testseqwrite: (g=0): rw=write, bs=(R) 1024MiB-1024MiB, (W) 1024MiB-1024MiB, (T) 1024MiB-1024MiB, ioengine=psync, iodepth=1
...
fio-3.14
Starting 10 threads
testseqwrite: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(9),W(1)][13.6%][eta 12m:23s]                                                
testseqwrite: (groupid=0, jobs=10): err= 0: pid=307042: Sun Dec 13 16:21:50 2020
  write: IOPS=0, BW=159MiB/s (167MB/s)(18.0GiB/115971msec)
    clat (msec): min=20002, max=64182, avg=50736.31, stdev=14795.11
     lat (msec): min=20103, max=64581, avg=50949.70, stdev=14763.00
    clat percentiles (msec):
     |  1.00th=[17113],  5.00th=[17113], 10.00th=[17113], 20.00th=[17113],
     | 30.00th=[17113], 40.00th=[17113], 50.00th=[17113], 60.00th=[17113],
     | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  MiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=8
   iops        : min=   16, max=   16, avg=16.00, stdev= 0.00, samples=8
  lat (msec)   : >=2000=100.00%
  cpu          : usr=0.42%, sys=0.43%, ctx=380, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,18,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=18.0GiB (19.3GB), run=115971-115971msec

Disk stats (read/write):
    dm-0: ios=0/37098, merge=0/0, ticks=0/17552840, in_queue=17710730, util=99.92%, aggrios=0/36944, aggrmerge=0/179, aggrticks=0/17266133, aggrin_queue=17265866, aggrutil=99.22%
  sdb: ios=0/36944, merge=0/179, ticks=0/17266133, in_queue=17265866, util=99.22%

顺序读

[root@hn03 ~]# fio -filename=/vz/test_seqread -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1G -size=10G -numjobs=10 -runtime=60 -group_reporting -name=testseqread
testseqread: (g=0): rw=read, bs=(R) 1024MiB-1024MiB, (W) 1024MiB-1024MiB, (T) 1024MiB-1024MiB, ioengine=psync, iodepth=1
...
fio-3.14
Starting 10 threads
testseqread: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(6),R(1),_(3)][60.0%][eta 00m:46s]                                      
testseqread: (groupid=0, jobs=10): err= 0: pid=319596: Sun Dec 13 16:30:05 2020
  read: IOPS=1, BW=1043MiB/s (1094MB/s)(70.0GiB/68729msec)
    clat (msec): min=967, max=10553, avg=8866.24, stdev=2072.27
     lat (msec): min=967, max=10553, avg=8866.24, stdev=2072.27
    clat percentiles (msec):
     |  1.00th=[  969],  5.00th=[ 2869], 10.00th=[ 5403], 20.00th=[ 9060],
     | 30.00th=[ 9329], 40.00th=[ 9329], 50.00th=[ 9463], 60.00th=[ 9597],
     | 70.00th=[ 9731], 80.00th=[ 9866], 90.00th=[10134], 95.00th=[10134],
     | 99.00th=[10537], 99.50th=[10537], 99.90th=[10537], 99.95th=[10537],
     | 99.99th=[10537]
   bw (  MiB/s): min=20447, max=20480, per=100.00%, avg=20472.62, stdev= 2.11, samples=60
   iops        : min=   15, max=   20, avg=18.69, stdev= 0.30, samples=60
  lat (msec)   : 1000=1.43%, 2000=1.43%, >=2000=97.14%
  cpu          : usr=0.00%, sys=3.04%, ctx=4613, majf=0, minf=6158
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=70,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1043MiB/s (1094MB/s), 1043MiB/s-1043MiB/s (1094MB/s-1094MB/s), io=70.0GiB (75.2GB), run=68729-68729msec

Disk stats (read/write):
    dm-0: ios=143360/4, merge=0/0, ticks=9562103/248, in_queue=9565573, util=99.98%, aggrios=143360/4, aggrmerge=0/1, aggrticks=9515022/245, aggrin_queue=9514757, aggrutil=99.95%
  sdb: ios=143360/4, merge=0/1, ticks=9515022/245, in_queue=9514757, util=99.95%
This article is under CC BY-NC-SA 4.0 license.
Please quote the original link：https://www.liujason.com/article/1121.html
Like (0)
    No related articles!
  • No related articles！
发表我的评论
取消评论

表情 贴图 加粗 删除线 居中 斜体 签到

Hi，您需要填写昵称和邮箱！

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址