ceph集群搭建-拓冰建站

一、环境准备

虚拟机（个人最小可用）

192.168.234.129 1c2g
192.168.234.130 1c2g
192.168.234.131 1c2g

系统盘：20G+
独立空白裸盘：每节点至少 1 块（10G+，无分区、无格式化）

二、统一基础配置（所有节点执行）

1. 设置主机名 & hosts

# 节点1
hostnamectl set-hostname ceph-node1
# 节点2
hostnamectl set-hostname ceph-node2
# 节点3
hostnamectl set-hostname ceph-node3

所有节点/etc/hosts加上：

192.168.234.129 ceph-node1
192.168.234.130 ceph-node2
192.168.234.131 ceph-node3

2. 关闭防火墙、SELinux

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

3. 时间同步

yum install -y chrony
systemctl enable --now chronyd

4. 无密码 SSH（admin 节点到其他节点）

在 ceph-node1：

ssh-keygen
ssh-copy-id ceph-node1
ssh-copy-id ceph-node2
ssh-copy-id ceph-node3

三、使用 cephadm 部署

1. 安装 cephadm

只在 ceph-node1（admin 节点）执行：

cat > /etc/yum.repos.d/ceph-quincy.repo <<'EOF'
[ceph-quincy]
name=Ceph Quincy x86_64
baseurl=https://mirrors.aliyun.com/ceph/rpm-quincy/el9/x86_64/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
[ceph-quincy-noarch]
name=Ceph Quincy noarch
baseurl=https://mirrors.aliyun.com/ceph/rpm-quincy/el9/noarch/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
EOF
dnf clean all
dnf makecache
dnf install -y cephadm

2. 引导集群

cephadm bootstrap --mon-ip 192.168.234.129

dashboard 地址、用户名 admin、密码保存好！

3. 拷贝ceph.pub 集群密钥

ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node3

4. 加入其他节点

cephadm shell -- ceph orch host add ceph-node2
cephadm shell -- ceph orch host add ceph-node3

查看节点：

[root@localhost ~]# cephadm shell -- ceph orch host ls
Inferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cd
Inferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/config
Using ceph image with id '259b35566514' and tag 'v17' created on 2024-11-26 00:45:38 +0000 UTC
quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f
HOST ADDR LABELS STATUS
ceph-node1 192.168.234.129 _admin
ceph-node2 192.168.234.130
ceph-node3 192.168.234.131

5. 给所有节点打标签

ceph orch host label add ceph-node1 _admin
ceph orch host label add ceph-node1 mon
ceph orch host label add ceph-node2 mon
ceph orch host label add ceph-node3 mon
查看节点：
[root@localhost ~]# cephadm shell -- ceph orch host ls
Inferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cd
Inferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/config
Using ceph image with id '259b35566514' and tag 'v17' created on 2024-11-26 00:45:38 +0000 UTC
quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f
HOST ADDR LABELS STATUS
ceph-node1 192.168.234.129 _admin,mon
ceph-node2 192.168.234.130 mon
ceph-node3 192.168.234.131 mon

四、添加 OSD（存储盘）

1. 查看可用裸设备

[root@localhost ~]# cephadm shell -- ceph orch device ls
Inferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cd
Inferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/config
Using ceph image with id '259b35566514' and tag 'v17' created on 2024-11-26 00:45:38 +0000 UTC
quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
ceph-node1 /dev/nvme0n2 ssd VMware_Virtual_NVMe_Disk_VMware_NVME_0000 10.0G Yes 27m ago
ceph-node2 /dev/nvme0n2 ssd VMware_Virtual_NVMe_Disk_VMware_NVME_0000 10.0G Yes 8m ago
ceph-node3 /dev/nvme0n2 ssd VMware_Virtual_NVMe_Disk_VMware_NVME_0000 10.0G Yes 3m ago

2. 自动扫描并创建 OSD

ceph orch apply osd --all-available-devices

3. 查看 OSD 状态

[root@localhost ~]# cephadm shell -- ceph osd tree
Inferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cd
Inferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/config
Using ceph image with id '259b35566514' and tag 'v17' created on 2024-11-26 00:45:38 +0000 UTC
quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.02939 root default
-7 0.00980 host ceph-node1
0 ssd 0.00980 osd.0 up 1.00000 1.00000
-5 0.00980 host ceph-node2
1 ssd 0.00980 osd.1 up 1.00000 1.00000
-3 0.00980 host ceph-node3
2 ssd 0.00980 osd.2 up 1.00000 1.00000
[root@localhost ~]# cephadm shell -- ceph -s
Inferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cd
Inferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/config
Using ceph image with id '259b35566514' and tag 'v17' created on 2024-11-26 00:45:38 +0000 UTC
quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f
cluster:
id: 044dbd4a-7c37-11f1-8276-000c295d03cd
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 46m)
mgr: ceph-node1.kvwqoy(active, since 66m), standbys: ceph-node2.jujajl
osd: 3 osds: 3 up (since 43m), 3 in (since 43m)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 885 MiB used, 29 GiB / 30 GiB avail
pgs: 1 active+clean

五、访问 Dashboard（Web 界面）

查看访问地址：
[root@localhost ~]# cephadm shell -- ceph mgr services
Inferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cd
Inferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/config
Using ceph image with id '259b35566514' and tag 'v17' created on 2024-11-26 00:45:38 +0000 UTC
quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f
{
"dashboard": "https://192.168.234.129:8443/",
"prometheus": "http://192.168.234.129:9283/"
}
用前面保存的账号密码登录访问：
http://192.168.234.129:8443/