Raid 配置的规划

安装 TiDB 集群的物理机配置如下：

组件配置描述CPU2 * Intel Xeon Gold 5218R(2.1GHz, 20Core)内存384GB系统盘2 * 480GB SATA *** RAID1数据盘14 * 1.92TB SATA *** RAID10

用 14 * 1.92TB SATA *** 组成 RAID10 后，减去 RAID 的元数据和 1024 的转换损失，实际共 12819716 MB 可用。因为所有的节点都配置了 *** 磁盘，且数据规模较大，所以需要配置成 TiDB 和 TiKV 的混合部署模式。

目前我们共配置 80 VCPU 的主机，按 16 个 VCPU 一个组件，大概可以配置 5 个组件，每台物理主机共部署 4 个 TiKV 和 1 个 TiDB。按一个 TiKV 约 2 TB 的容量规划，配置一个 3 TB 的空间，保证空间占用在 80% 以下。所以 12819716 MB 的容量空间分成 4 个 3 TB 的空间，挂接在 /data{0123}，和 1 个 500 GB 的空间，挂接在 /deploy。

混合部署的 CPU 和内存配置，可以参考文档 https://docs.pingcap.com/zh/tidb/dev/hybrid-deployment-topology 。

在操作系统中可以使用 RAID 卡的命令行管理工具，实现在线的 RAID 重配置，免去了重启和控制台访问等问题。

目前 x86 服务器中，主要使用 Microchip 的 SmartROC 系列（收购 adaptec 公司后发布的产品）和 Broadcom 的 Megaraid 系列（收购 avago 公司后发布的产品)，以下以这两款 RAID 卡配置为例。

Microchip Adaptec RAID arcconf 

RAID 卡背景信息

使用操作系统命令查看 RAID 卡型号是 ZTE ***服务器 R5300 G4 的 RAID 卡 SDPSA_L-18i

[host bin]$ sudo dmidecode |grep -i "product name"Product Name: R5300 G4 Product Name: R5300G4[host bin]$ sudo dmidecode |grep -i RAID [host bin]$ sudo lspci |grep -i RAID [host bin]$ sudo dmesg |grep -i RAID [ 6.360527] smartpqi 0000:3c:00.0: added 14:1:0:0 4000000000000000 Direct-Access ZTE LOGICAL VOLUME ***SmartPathCap- En- RAID-1(1+0) [ 6.386016]smartpqi 0000:3c:00.0: added14:1:0:1 4000000100000000 Direct-Access ZTE LOGICAL VOLUME ***SmartPathCap- En- RAID-1(1+0) [ 6.396947] smartpqi 0000:3c:00.0: added 14:2:0:0 0000000000000000 RAID ZTE SDPSA_L-18i 2G [ 6.398251] scsi 14:2:0:0: RAID ZTE SDPSA_L-18i 2G 5.00 PQ: 0 ANSI: 5

通过搜索可以查到 SDPSA_L-18i 实际上是 Microchip 的 SmartROC3100 RAID 卡。

核对产品资料确认使用 arcconf 命令行工具可以配置 RAID。

https://ww1.microchip.com/downloads/en/DeviceDoc/00003022A.pdf

软件下载位置

https://storage.microsemi.com/en-us/speed/raid/storage_manager/arcconf_B25611_zip.php

参考产品手册进行配置

https://download.adaptec.com/pdfs/user_guides/adaptec_cli_smarthba_smartraid_11_22_ug.pdf

查看 RAID 卡和数据盘信息

主机配置有两个 vd，vd1 是 14 块盘配置的 RAID 10，需要删除 vd1。

[root@host /tmp/arcconf/linux_x64]# ./arcconf list 1 Controllers found: 1---------------------------------------------------------------------- Controller information ---------------------------------------------------------------------- Controller ID:Status, Slot, Mode, Name, SerialNumber, WWN ---------------------------------------------------------------------- Controller1: : Optimal, Slot 0, Mixed, ZTE SmartROC3100 SDPSA_L-18i 2G, 746001400375, 50015EBE07AA5A3F ---------------------------------------------------------------------- Array Information ---------------------------------------------------------------------- Array ID: Status (Interface, TotalSize MB, FreeSpace MB) ---------------------------------------------------------------------- Array 0 : Ok (SATA ***, 915725 MB, 0 MB) Array 1 : Ok (SATA ***, 25639884 MB, 1 MB)---------------------------------------------------------------------- Logical device information ---------------------------------------------------------------------- Logical ID: Status (RAID, Interface, Size MB)Name ---------------------------------------------------------------------- Logical0 : Optimal (1, Data, 457830 MB) os Logical 1 : Optimal (10, Data, 12819716 MB)vd ---------------------------------------------------------------------- maxCache information ---------------------------------------------------------------------- Array ID: Status (Interface, TotalSize MB, FreeSpace MB)---------------------------------------------------------------------- No maxCache device created ---------------------------------------------------------------------- Physical Device information ---------------------------------------------------------------------- Physical ID: State (Interface, BlockSize, SizeMB, Vendor, Model, Type) WWN, [Location]---------------------------------------------------------------------- Physical0,0 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A41, [Direct Attached, Slot 0(Connector 0:CN0)] Physical 0,1 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A42, [Direct Attached, Slot1(Connector 0:CN0)] Physical 0,2 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A40, [Direct Attached, Slot 2(Connector 0:CN0)] Physical 0,3 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A3F, [Direct Attached, Slot3(Connector 0:CN0)] Physical 0,4 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A45, [Direct Attached, Slot 4(Connector 1:CN1)] Physical 0,5 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A46, [Direct Attached, Slot5(Connector 1:CN1)] Physical 0,6 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A44, [Direct Attached, Slot 6(Connector 1:CN1)] Physical 0,7 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A43, [Direct Attached, Slot7(Connector 1:CN1)] Physical 0,8 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A49, [Direct Attached, Slot 8(Connector 2:CN2)] Physical 0,9 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A4A, [Direct Attached, Slot9(Connector 2:CN2)] Physical 0,10 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A48, [Direct Attached, Slot 10(Connector 2:CN2)] Physical 0,11 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive)30015EBE07AA5A47,[Direct Attached, Slot 11(Connector 2:CN2)] Physical 0,12 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A4D, [Direct Attached, Slot 12(Connector 3:CN3)] Physical 0,13 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive)30015EBE07AA5A4E,[Direct Attached, Slot 13(Connector 3:CN3)] Physical 0,16 : Online (SATA, 512 Bytes, 457862MB, ATA, SAMSUNG MZ7LH480HAHQ-00005, Solid State Drive) 30015EBE07AA5A4F, [Direct Attached, Slot 16(Connector 4:CN4)] Physical 0,17 : Online (SATA, 512Bytes, 457862MB, ATA, SAMSUNG MZ7LH480HAHQ-00005, Solid State Drive) 30015EBE07AA5A50, [Direct Attached, Slot 17(Connector 4:CN4)] Physical 2,0 : Ready (SES2, Not Applicable, Not Applicable, ZTE , Virtual SGPIO, Enclosure Services Device) 50015EBE07AA5A51, [Not Applicable] Command completed successfully. [root@host /tmp/arcconf/linux_x64]# ./arcconf getconfig 1 ld 1Controllers found:1 -------------------------------------------------------- Logical device information -------------------------------------------------------- Logical Device number 1 Logical Device name : vd Disk Name :/dev/sdb Block Size of member drives: 512 Bytes Array : 1 RAID level : 10 Status of Logical Device : Optimal Size : 12819716 MB Stripe-unit size : 256 KB Full Stripe Size : 1792 KB Interface Type : SATA *** Device Type :Data Boot Type: None Heads : 255 Sectors Per Track : 32 Cylinders : 65535 Caching : Enabled Mount Points : Not Mounted LD Acceleration Method : Controller Cache Logical Device On Boot Connector : No Volume Unique Identifier :600508B1001CD9EDCFD7F2DC2F11B617 -------------------------------------------------------- Logical Device segment information -------------------------------------------------------- Group0, Segment 0 : Present (1831420MB, SATA, ***, Connector:0, Device:0)222638F89768 中间输出略 Group6, Segment 1 : Present (1831420MB, SATA, ***, Connector:3, Device:13) 22253910CA2E Command completed successfully. [root@host /tmp/arcconf/linux_x64]#

删除 VD1

删除 vd1，14 块盘是 ready 状态。

[root@host /tmp/arcconf/linux_x64]# ./arcconf delete 1 logicaldrive 1 Controllers found: 1 WARNING: Deleting this logical device will automatically delete array 1 because it is the only logical device present on that array. All data in logical device 1then ENTER to continue or press ENTER to abort: y Deleting: logical device 1 ("vd") Command completed successfully. [root@host /tmp/arcconf/linux_x64]# ./arcconf list 1 Controllers found: 1---------------------------------------------------------------------- Controller information ---------------------------------------------------------------------- Controller ID: Status, Slot, Mode, Name, SerialNumber, WWN ---------------------------------------------------------------------- Controller 1: : Optimal, Slot 0, Mixed, ZTE SmartROC3100 SDPSA_L-18i 2G,746001400375, 50015EBE07AA5A3F ---------------------------------------------------------------------- Array Information ---------------------------------------------------------------------- Array ID : Status (Interface, TotalSize MB, FreeSpace MB) ---------------------------------------------------------------------- Array 0 : Ok (SATA ***, 915725 MB, 0 MB)---------------------------------------------------------------------- Logical device information ---------------------------------------------------------------------- Logical ID: Status (RAID, Interface, Size MB) Name ---------------------------------------------------------------------- Logical 0 : Optimal (1, Data, 457830 MB)os ---------------------------------------------------------------------- maxCache information ---------------------------------------------------------------------- Array ID: Status (Interface, TotalSize MB, FreeSpace MB)---------------------------------------------------------------------- No maxCache device created ---------------------------------------------------------------------- Physical Device information ---------------------------------------------------------------------- Physical ID: State (Interface, BlockSize, SizeMB, Vendor, Model, Type) WWN, [Location] ---------------------------------------------------------------------- Physical 0,0 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A41, [Direct Attached, Slot 0(Connector0:CN0)] Physical 0,1 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A42, [Direct Attached, Slot 1(Connector 0:CN0)] Physical 0,2 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A40, [Direct Attached, Slot 2(Connector0:CN0)] Physical 0,3 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A3F, [Direct Attached, Slot 3(Connector 0:CN0)] Physical 0,4 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A45, [Direct Attached, Slot 4(Connector1:CN1)] Physical 0,5 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A46, [Direct Attached, Slot 5(Connector 1:CN1)] Physical 0,6 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A44, [Direct Attached, Slot 6(Connector1:CN1)] Physical 0,7 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A43, [Direct Attached, Slot 7(Connector 1:CN1)] Physical 0,8 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A49, [Direct Attached, Slot 8(Connector 2:CN2)] Physical 0,9 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A4A, [Direct Attached, Slot 9(Connector 2:CN2)] Physical 0,10 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A48, [Direct Attached, Slot10(Connector 2:CN2)] Physical 0,11 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A47, [Direct Attached, Slot 11(Connector 2:CN2)] Physical 0,12 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A4D, [Direct Attached, Slot12(Connector 3:CN3)] Physical 0,13 : Ready (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A4E, [Direct Attached, Slot 13(Connector 3:CN3)] Physical 0,16 : Online (SATA, 512 Bytes, 457862MB, ATA, SAMSUNG MZ7LH480HAHQ-00005, Solid State Drive) 30015EBE07AA5A4F, [Direct Attached, Slot16(Connector 4:CN4)] Physical 0,17 : Online (SATA, 512 Bytes, 457862MB, ATA, SAMSUNG MZ7LH480HAHQ-00005, Solid State Drive) 30015EBE07AA5A50, [Direct Attached, Slot 17(Connector 4:CN4)] Physical 2,0 : Ready (SES2, Not Applicable, Not Applicable, ZTE , Virtual SGPIO, Enclosure Services Device)50015EBE07AA5A51,[Not Applicable] Command completed successfully.

创建第一个 VD

使用 14 块创建 RAID 10 ,并划分出第一个 3070000 MB 的 LUN。

命令参数

值

arcconf create 1 logicaldrive

在 控制器 1 上创建 LD

name / VD 名字

data0

LDcache / 缓存启用

LON

VD 容量

3070000

RAID 级别

10

磁盘从 0 到 13

0 0,0 1,0 2,0 3,0 4,0 5,0 6,0 7,0 8,0 9,0 10,0 11,0 12,0 13

[root@host /tmp/arcconf/linux_x64]# ./arcconf create 1 logicaldrive name data0 LDcache LON 3070000 10 0 0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0 12 0 13 Controllers found: 1 Do you want to addthen ENTER to continueor press ENTER to abort: y Creating logical device: data0 Command completed successfully.[root@host /tmp/arcconf/linux_x64]# ./arcconf list 1 Controllers found: 1---------------------------------------------------------------------- Controller information ---------------------------------------------------------------------- Controller ID: Status, Slot, Mode, Name, SerialNumber, WWN ---------------------------------------------------------------------- Controller 1: : Optimal, Slot 0, Mixed, ZTE SmartROC3100 SDPSA_L-18i 2G, 746001400375, 50015EBE07AA5A3F ---------------------------------------------------------------------- Array Information ---------------------------------------------------------------------- Array ID : Status (Interface, TotalSize MB, FreeSpace MB) ---------------------------------------------------------------------- Array 0 : Ok (SATA ***, 915725 MB, 0 MB) Array 1 : Ok (SATA ***, 25639884 MB, 19499435 MB)---------------------------------------------------------------------- Logical device information ---------------------------------------------------------------------- Logical ID: Status (RAID, Interface, Size MB) Name ---------------------------------------------------------------------- Logical 0 : Optimal (1, Data, 457830 MB) os Logical 1 :Optimal(10, Data, 3070000 MB) data0 ---------------------------------------------------------------------- maxCache information ---------------------------------------------------------------------- Array ID : Status (Interface, TotalSize MB, FreeSpace MB)---------------------------------------------------------------------- No maxCache device created ---------------------------------------------------------------------- Physical Device information ---------------------------------------------------------------------- Physical ID: State (Interface, BlockSize, SizeMB, Vendor, Model, Type) WWN, [Location] ---------------------------------------------------------------------- Physical 0,0 : Online (SATA, 512Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A41, [Direct Attached, Slot 0(Connector 0:CN0)] Physical 0,1 : Online (SATA, 512 Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A42, [Direct Attached, Slot 1(Connector 0:CN0)] Physical 0,2 : Online (SATA, 512Bytes, 1831420MB, ATA, Micron_5300_MTFDDAK1T9TDS, Solid State Drive) 30015EBE07AA5A40, [Direct Attached, Slot 2(Conn