Ceph集群部署

node节点信息

1
2
3
4
192.168.99.100 k8s-master  deploy、mon*1、osd*3
192.168.99.101 node1  mon*1、 osd*3
192.168.99.102 node2  mon*1、 osd*3

禁用防火墙

1
2
3
4
5
6
sed-i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
setenforce 0
systemctl stop iptables
systemctl stop firewalld
systemctl disable iptables
systemctl disable firewalld

修改yum源

1
2
3
4
5
6
yum clean all
curl http://mirrors.aliyun.com/repo/Centos-7.repo >/etc/yum.repos.d/CentOS-Base.repo
curl http://mirrors.aliyun.com/repo/epel-7.repo >/etc/yum.repos.d/epel.repo 
sed -i '/aliyuncs/d' /etc/yum.repos.d/CentOS-Base.repo
sed -i '/aliyuncs/d' /etc/yum.repos.d/epel.repo
yum makecache
  • 增加ceph源
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    #vim /etc/yum.repos.d/ceph.repo

    [ceph]
    name=ceph
    baseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/x86_64/
    gpgcheck=0
    [ceph-noarch]
    name=cephnoarch
    baseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/noarch/
    gpgcheck=0

时间同步

1
2
3
4
5
6
yum -y install wget ntp vim openssh-server
yum -y install ntp ntp-data ntp-doc
systemctl enable ntpd.service
systemctl start ntpd.service
ntpq -p

  • 修改/etc/ntp.conf，注释掉四行server，添加一行server指向ceph-admin
    1
    2
    3
    4
    5
    6
    7
    server 0.centos.pool.ntp.org iburst
    server 1.centos.pool.ntp.org iburst
    server 2.centos.pool.ntp.org iburst
    server 3.centos.pool.ntp.org iburst

    # 添加一行
    server 192.168.99.100
  • 重启
    1
    2
    3
    systemctl restart ntpd
    ntpq -p

部署ceph-deploy

  • ceph-deploy主机执行：

    1
    2
    3
    yum update && yum -y install ceph-deploy ceph
    mkdir  my-cluster
    cd  my-cluster

  • 版本信息

    1
    2
    3
    4
    [root@k8s-master my-cluster]# ceph-deploy --version
    1.5.39
    [root@k8s-master my-cluster]# ceph -v
    ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)

  • 创建monitor节点

    1
    ceph-deploy new k8s-master node1 node2

  • 执行过程

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    [root@k8s-master my-cluster]# ceph-deploy new k8s-master node1 node2
    [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
    [ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy new k8s-master node1 node2
    [ceph_deploy.cli][INFO  ] ceph-deploy options:
    [ceph_deploy.cli][INFO  ]  username                      : None
    [ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f3700457668>
    [ceph_deploy.cli][INFO  ]  verbose                       : False
    [ceph_deploy.cli][INFO  ]  overwrite_conf                : False
    [ceph_deploy.cli][INFO  ]  quiet                         : False
    [ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f36ffbc7d88>
    [ceph_deploy.cli][INFO  ]  cluster                       : ceph
    [ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
    [ceph_deploy.cli][INFO  ]  mon                           : ['k8s-master', 'node1', 'node2']
    [ceph_deploy.cli][INFO  ]  public_network                : None
    [ceph_deploy.cli][INFO  ]  ceph_conf                     : None
    [ceph_deploy.cli][INFO  ]  cluster_network               : None
    [ceph_deploy.cli][INFO  ]  default_release               : False
    [ceph_deploy.cli][INFO  ]  fsid                          : None
    [ceph_deploy.new][DEBUG ] Creating new cluster named ceph
    [ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
    [k8s-master][DEBUG ] connected to host: k8s-master
    [k8s-master][DEBUG ] detect platform information from remote host
    [k8s-master][DEBUG ] detect machine type
    [k8s-master][DEBUG ] find the location of an executable
    [k8s-master][INFO  ] Running command: /usr/sbin/ip link show
    [k8s-master][INFO  ] Running command: /usr/sbin/ip addr show
    [k8s-master][DEBUG ] IP addresses found: [u'192.168.99.100', u'172.17.0.1', u'10.0.3.15']
    [ceph_deploy.new][DEBUG ] Resolving host k8s-master
    [ceph_deploy.new][DEBUG ] Monitor k8s-master at ::1
    [ceph_deploy.new][INFO  ] Monitors are IPv6, binding Messenger traffic on IPv6
    [ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
    [node1][DEBUG ] connected to host: k8s-master
    [node1][INFO  ] Running command: ssh -CT -o BatchMode=yes node1
    [node1][DEBUG ] connected to host: node1
    [node1][DEBUG ] detect platform information from remote host
    [node1][DEBUG ] detect machine type
    [node1][DEBUG ] find the location of an executable
    [node1][INFO  ] Running command: /usr/sbin/ip link show
    [node1][INFO  ] Running command: /usr/sbin/ip addr show
    [node1][DEBUG ] IP addresses found: [u'192.168.99.101', u'172.17.0.1', u'10.0.3.15']
    [ceph_deploy.new][DEBUG ] Resolving host node1
    [ceph_deploy.new][DEBUG ] Monitor node1 at 192.168.99.101
    [ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
    [node2][DEBUG ] connected to host: k8s-master
    [node2][INFO  ] Running command: ssh -CT -o BatchMode=yes node2
    [ceph_deploy.new][WARNIN] could not connect via SSH
    [ceph_deploy.new][INFO  ] will connect again with password prompt
    The authenticity of host 'node2 (192.168.99.102)' can't be established.
    ECDSA key fingerprint is SHA256:+NaBMN0aj9CV0euaO9gBi7sfDLwrGd9W6I7epxNppIc.
    ECDSA key fingerprint is MD5:55:c7:ec:29:3a:54:eb:6d:17:ed:92:53:80:6d:82:76.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'node2,192.168.99.102' (ECDSA) to the list of known hosts.
    [node2][DEBUG ] connected to host: node2
    [node2][DEBUG ] detect platform information from remote host
    [node2][DEBUG ] detect machine type
    [ceph_deploy.new][INFO  ] adding public keys to authorized_keys
    [node2][DEBUG ] append contents to file
    [node2][DEBUG ] connected to host: node2
    [node2][DEBUG ] detect platform information from remote host
    [node2][DEBUG ] detect machine type
    [node2][DEBUG ] find the location of an executable
    [node2][INFO  ] Running command: /usr/sbin/ip link show
    [node2][INFO  ] Running command: /usr/sbin/ip addr show
    [node2][DEBUG ] IP addresses found: [u'192.168.99.102', u'172.17.0.1', u'10.0.3.15']
    [ceph_deploy.new][DEBUG ] Resolving host node2
    [ceph_deploy.new][DEBUG ] Monitor node2 at 192.168.99.102
    [ceph_deploy.new][DEBUG ] Monitor initial members are ['k8s-master', 'node1', 'node2']
    [ceph_deploy.new][DEBUG ] Monitor addrs are ['[::1]', '192.168.99.101', '192.168.99.102']
    [ceph_deploy.new][DEBUG ] Creating a random mon key...
    [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
    [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

  • 执行完目录会生成key和conf

    1
    2
    [root@k8s-master my-cluster]# ls
    ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring
1
2
3
4
5
6
# public_network 
echo public_network=192.168.99.1/24 >> ceph.conf

# mon之间时差允许范围(默认为0.05s，现改为2s)
echo mon_clock_drift_allowed = 2 >> ceph.conf

部署monitor

1
ceph-deploy mon create-initial