• Welcome to LiuJason's Blog!
> Coding Projects > Linux笔记 > Debian10下使用SSD作为缓存提高LVM磁盘读写性能

Debian10下使用SSD作为缓存提高LVM磁盘读写性能

Linux笔记 Jason 5 years ago (2020-01-30) 1863 Views 2 Comments QR code of this page

首先查看磁盘的ID，防止重启后磁盘顺序出错。

root@PVE-EU-1 / # ls -l /dev/disk/by-path/ /dev/disk/by-id/
/dev/disk/by-id/:
total 0
lrwxrwxrwx 1 root root  9 Jan 30 10:41 wwn-0x60000000000000000e00000000010001 -> ../../sdd
/dev/disk/by-path/:
total 0
lrwxrwxrwx 1 root root  9 Jan 30 10:41 ip-172.17.3.107:3260-iscsi-iqn.2020-01.pve-eu-3:iscsieu1-lun-1 -> ../../sdd

然后将设备添加到lvm的pv列表中，同时获得UUID

root@PVE-EU-1 / # pvcreate /dev/disk/by-path/ip-172.17.3.107:3260-iscsi-iqn.2020-01.pve-eu-3:iscsieu1-lun-1
WARNING: ext4 signature detected on /dev/disk/by-path/ip-172.17.3.107:3260-iscsi-iqn.2020-01.pve-eu-3:iscsieu1-lun-1 at offset 1080. Wipe it? [y/n]:  y
  Wiping ext4 signature on /dev/disk/by-path/ip-172.17.3.107:3260-iscsi-iqn.2020-01.pve-eu-3:iscsieu1-lun-1.
  Physical volume "/dev/disk/by-path/ip-172.17.3.107:3260-iscsi-iqn.2020-01.pve-eu-3:iscsieu1-lun-1" successfully created.
root@PVE-EU-1 / # pvdisplay
  "/dev/sdd" is a new physical volume of "5.00 TiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdd
  VG Name               
  PV Size               5.00 TiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               1IXSXB-88kX-SFLb-JVqt-QMCj-d1h5-bD1bZn

然后使用路径名将磁盘加入vg组

root@PVE-EU-1 / # vgextend vg0 /dev/disk/by-path/ip-172.17.3.107:3260-iscsi-iqn.2020-01.pve-eu-3:iscsieu1-lun-1
  Volume group "vg0" successfully extended

在SSD上新建一个用于cache的lv

root@PVE-EU-1 / # lvcreate --type cache-pool -l 100%FREE -n iscsi-cache vg0 /dev/md1
  Using <5.38 MiB chunk size instead of default 64.00 KiB, so cache pool has less than 1000000 chunks.
  Logical volume "iscsi-cache" created.

然后再新建一个lv作为存储磁盘，查看一下目前状况

root@PVE-EU-1 / # lvcreate -l 100%FREE -n iscsieu1c vg0 /dev/sdd 
 Logical volume "iscsieu1c" created.

root@PVE-EU-1 / # lvdisplay
   
  --- Logical volume ---
  LV Path                /dev/vg0/iscsi-cache
  LV Name                iscsi-cache
  VG Name                vg0
  LV UUID                POcJ4a-sd9I-UzmM-vdKC-hiNK-Ku8y-ZeTMAl
  LV Write Access        read/write
  LV Creation host, time PVE-EU-1, 2020-01-30 11:12:55 +0100
  LV Pool metadata       iscsi-cache_cmeta
  LV Pool data           iscsi-cache_cdata
  LV Status              NOT available
  LV Size                <99.91 GiB
  Current LE             25576
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/vg0/iscsieu1c
  LV Name                iscsieu1c
  VG Name                vg0
  LV UUID                cDMJXq-NAjz-jyfL-hW2f-dT13-2gr2-PJgzeG
  LV Write Access        read/write
  LV Creation host, time PVE-EU-1, 2020-01-30 11:15:18 +0100
  LV Status              available
  # open                 0
  LV Size                <5.00 TiB
  Current LE             1310719
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

然后将缓存与磁盘合并

root@PVE-EU-1 / # lvconvert --type cache --cachepool iscsi-cache vg0/iscsieu1c
Do you want wipe existing metadata of cache pool vg0/iscsi-cache? [y/n]: y
  Logical volume vg0/iscsieu1c is now cached.

查看结果

root@PVE-EU-1 / # lvdisplay

  --- Logical volume ---
  LV Path                /dev/vg0/iscsieu1c
  LV Name                iscsieu1c
  VG Name                vg0
  LV UUID                cDMJXq-NAjz-jyfL-hW2f-dT13-2gr2-PJgzeG
  LV Write Access        read/write
  LV Creation host, time PVE-EU-1, 2020-01-30 11:15:18 +0100
  Unknown feature in status: 8 49/12288 11008 1/19033 1 20 0 0 0 1 0 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw - 
  LV Cache pool name     iscsi-cache
  LV Cache origin name   iscsieu1c_corig
  LV Status              available
  # open                 0
  LV Size                <5.00 TiB
  Cache used blocks      0.01%
  Cache metadata blocks  0.40%
  Cache dirty blocks     0.00%
  Cache read hits/misses 1 / 20
  Cache wrt hits/misses  0 / 0
  Cache demotions        0
  Cache promotions       1
  Current LE             1310719
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:1

最终的效果如下：

root@PVE-EU-1 / # fio -filename=/dev/vg0/iscsieu1c -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=10G -numjobs=30 -runtime=10 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 30 threads
Jobs: 30 (f=30): [w(30)][100.0%][w=2450KiB/s][w=153 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=15367: Thu Jan 30 11:26:49 2020
  write: IOPS=123, BW=1973KiB/s (2021kB/s)(20.7MiB/10719msec); 0 zone resets
    clat (usec): min=227, max=3152.8k, avg=233039.37, stdev=375505.76
     lat (usec): min=227, max=3152.8k, avg=233040.15, stdev=375505.75
    clat percentiles (usec):
     |  1.00th=[   1418],  5.00th=[  23987], 10.00th=[  34866],
     | 20.00th=[  51119], 30.00th=[  68682], 40.00th=[  88605],
     | 50.00th=[ 120062], 60.00th=[ 149947], 70.00th=[ 210764],
     | 80.00th=[ 295699], 90.00th=[ 459277], 95.00th=[ 775947],
     | 99.00th=[2197816], 99.50th=[2600469], 99.90th=[2936013],
     | 99.95th=[3137340], 99.99th=[3137340]
   bw (  KiB/s): min=   31, max=  448, per=4.84%, avg=95.46, stdev=49.15, samples=433
   iops        : min=    1, max=   28, avg= 5.95, stdev= 3.08, samples=433
  lat (usec)   : 250=0.08%, 500=0.68%, 750=0.08%, 1000=0.08%
  lat (msec)   : 2=0.30%, 4=0.23%, 10=1.59%, 20=1.29%, 50=15.28%
  lat (msec)   : 100=24.21%, 250=32.15%, 500=15.05%, 750=3.86%, 1000=1.36%
  cpu          : usr=0.02%, sys=0.00%, ctx=1748, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1322,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1973KiB/s (2021kB/s), 1973KiB/s-1973KiB/s (2021kB/s-2021kB/s), io=20.7MiB (21.7MB), run=10719-10719msec
root@PVE-EU-1 / # fio -filename=/dev/vg0/iscsieu1c -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=16k -size=10G -numjobs=30 -runtime=10 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 30 threads
Jobs: 30 (f=30): [r(30)][100.0%][r=5968KiB/s][r=373 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=15681: Thu Jan 30 11:27:40 2020
  read: IOPS=730, BW=11.4MiB/s (11.0MB/s)(115MiB/10042msec)
    clat (usec): min=112, max=1249.2k, avg=40850.26, stdev=135960.72
     lat (usec): min=112, max=1249.2k, avg=40850.46, stdev=135960.72
    clat percentiles (usec):
     |  1.00th=[    141],  5.00th=[    151], 10.00th=[    163],
     | 20.00th=[    192], 30.00th=[    293], 40.00th=[    545],
     | 50.00th=[   2507], 60.00th=[   3785], 70.00th=[   5145],
     | 80.00th=[   8094], 90.00th=[  82314], 95.00th=[ 263193],
     | 99.00th=[ 767558], 99.50th=[ 943719], 99.90th=[1115685],
     | 99.95th=[1132463], 99.99th=[1249903]
   bw (  KiB/s): min=   31, max= 2688, per=3.97%, avg=463.97, stdev=570.84, samples=503
   iops        : min=    1, max=  168, avg=28.96, stdev=35.68, samples=503
  lat (usec)   : 250=27.34%, 500=11.19%, 750=5.18%, 1000=1.06%
  lat (msec)   : 2=2.90%, 4=14.03%, 10=19.70%, 20=2.64%, 50=3.69%
  lat (msec)   : 100=3.04%, 250=3.96%, 500=2.57%, 750=1.65%, 1000=0.72%
  cpu          : usr=0.02%, sys=0.03%, ctx=7993, majf=0, minf=120
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=7340,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=11.4MiB/s (11.0MB/s), 11.4MiB/s-11.4MiB/s (11.0MB/s-11.0MB/s), io=115MiB (120MB), run=10042-10042msec
root@PVE-EU-1 / # fio -filename=/dev/vg0/iscsieu1c -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=10G -numjobs=30 -runtime=10 -group_reporting -name=mytest
mytest: (g=0): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 30 threads
Jobs: 30 (f=30): [R(30)][100.0%][r=922MiB/s][r=59.0k IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=15940: Thu Jan 30 11:28:22 2020
  read: IOPS=58.4k, BW=912MiB/s (956MB/s)(9120MiB/10002msec)
    clat (usec): min=47, max=47185, avg=511.73, stdev=284.84
     lat (usec): min=48, max=47185, avg=511.84, stdev=284.85
    clat percentiles (usec):
     |  1.00th=[  149],  5.00th=[  215], 10.00th=[  243], 20.00th=[  289],
     | 30.00th=[  351], 40.00th=[  412], 50.00th=[  478], 60.00th=[  562],
     | 70.00th=[  635], 80.00th=[  701], 90.00th=[  791], 95.00th=[  857],
     | 99.00th=[ 1106], 99.50th=[ 1467], 99.90th=[ 3490], 99.95th=[ 4178],
     | 99.99th=[ 5407]
   bw (  KiB/s): min=24288, max=33184, per=3.33%, avg=31132.71, stdev=1579.07, samples=587
   iops        : min= 1518, max= 2074, avg=1945.78, stdev=98.69, samples=587
  lat (usec)   : 50=0.04%, 100=0.61%, 250=10.56%, 500=40.15%, 750=34.34%
  lat (usec)   : 1000=12.13%
  lat (msec)   : 2=1.95%, 4=0.17%, 10=0.06%, 20=0.01%, 50=0.01%
  cpu          : usr=0.34%, sys=1.50%, ctx=585439, majf=0, minf=120
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=583681,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=912MiB/s (956MB/s), 912MiB/s-912MiB/s (956MB/s-956MB/s), io=9120MiB (9563MB), run=10002-10002msec
root@PVE-EU-1 / # fio -filename=/dev/vg0/iscsieu1c -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=10G -numjobs=30 -runtime=10 -group_reporting -name=mytest
mytest: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
...
fio-3.12
Starting 30 threads
Jobs: 30 (f=30): [W(30)][100.0%][w=448MiB/s][w=28.7k IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=16216: Thu Jan 30 11:29:06 2020
  write: IOPS=27.9k, BW=435MiB/s (457MB/s)(4355MiB/10002msec); 0 zone resets
    clat (usec): min=230, max=51764, avg=1074.34, stdev=867.85
     lat (usec): min=230, max=51765, avg=1074.62, stdev=867.85
    clat percentiles (usec):
     |  1.00th=[  725],  5.00th=[  816], 10.00th=[  865], 20.00th=[  906],
     | 30.00th=[  947], 40.00th=[  971], 50.00th=[ 1004], 60.00th=[ 1020],
     | 70.00th=[ 1057], 80.00th=[ 1090], 90.00th=[ 1188], 95.00th=[ 1450],
     | 99.00th=[ 2278], 99.50th=[ 3228], 99.90th=[13173], 99.95th=[17171],
     | 99.99th=[46400]
   bw (  KiB/s): min=12320, max=16256, per=3.32%, avg=14825.17, stdev=1071.87, samples=575
   iops        : min=  770, max= 1016, avg=926.57, stdev=67.00, samples=575
  lat (usec)   : 250=0.01%, 500=0.08%, 750=1.34%, 1000=48.62%
  lat (msec)   : 2=48.65%, 4=0.90%, 10=0.23%, 20=0.14%, 50=0.03%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.27%, sys=1.55%, ctx=296017, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,278738,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=435MiB/s (457MB/s), 435MiB/s-435MiB/s (457MB/s-457MB/s), io=4355MiB (4567MB), run=10002-10002msec
This article is under CC BY-NC-SA 4.0 license.
Please quote the original link：https://www.liujason.com/article/507.html
Like (0)
    No related articles!
  • No related articles！
发表我的评论
取消评论

表情 贴图 加粗 删除线 居中 斜体 签到

Hi，您需要填写昵称和邮箱！

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
(2)个小伙伴在吐槽
  1. Unknown feature in status 有没有影响我也出现了
    sunny2020-12-10 12:58 Reply Windows 10 | Chrome 87.0.4280.88
    • Jason
      没什么影响，目前性能稳定。正常用了一个月了，持续观察中
      Jason2021-01-11 04:06 Reply Windows 10 | Chrome 87.0.4280.88