ceph-ansible离线部署ceph-nautilus版本

环境介绍：

软件	版本
CentOS	CentOS Linux release 7.9.2009 (Core)
ceph	nautilus
ansible	2.9.25

部署规划：

主机名	主机地址	CPU	内存	角色
node1.example.com	192.168.174.143	8C	8G	mon,mgr,osd,rgw,mds,ntp-server,ceph-ansible,ansible
node2.example.com	192.168.174.144	8C	8G	mon,mgr,osd,rgw,mds
node3.example.com	192.168.174.145	8C	8G	mon,mgr,osd,rgw,mds
node4.example.com	192.168.174.146	8C	8G	osd,rgw,mds
node5.example.com	192.168.174.147	8C	8G	osd,rgw,mds

安装Ansible

在node1主机解压安装

tar -xvf ansible-2.9.25.tar.gz
cd ansible
rpm -ivh *.rpm --force --nodeps

这里可通过CSDN下载Ansible-2.9.25安装包https://download.csdn.net/download/gswcfl/88040647?spm=1001.2014.3001.5503。

配置普通用户

作用：ansible通过普通用户远程执行命令，配置用户有sudo权限

此步ansible主机使用root帐号对授控主机执行操作。

vim useradd.yml

---
- name: create users
  hosts: all
  tasks:
    - name: create some users
      user:
        name: "alice"
        password: "{
    {'123456'|password_hash('sha512')}}"
        state: present
        comment: "Monitor and Maintenance account"
      notify: add sudo
  handlers:
    - name: add sudo
      copy:
        content: "alice ALL=(ALL) NOPASSWD: ALL"
        dest: /etc/sudoers.d/alice

执行增加用户

ansible-playbook useradd.yml

在下面的ansible远程执行中，将使用alice用户。

配置免密钥登陆

[alice@node1 ~]$ ssh-keygen
[alice@node1 ~]$ for i in 143 144 145 146 147 \
[alice@node1 ~]$ do \
[alice@node1 ~]$ ssh-copy-id [email protected].$i \
[alice@node1 ~]$ ssh-copy-id [email protected].$i \
[alice@node1 ~]$ done

配置hosts文件

创建hosts文件模板

vim hosts.j2

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
{
    %for i in groups.all%}
{
    {
    hostvars[i].ansible_default_ipv4.address}} {
    {
    hostvars[i].ansible_fqdn}} {
    {
    hostvars[i].ansible_hostname}}
{
    %endfor%}

执行hosts剧本

vim deploy-hosts.yml

---
- name: deploy new hosts
  hosts: all
  become: true
  tasks:
    - name: begin deploy new hosts
      template:
         src: hosts.j2
         dest: /etc/hosts

执行剧本：

ansible-playbook deloy-hosts.yml

配置网络时间

安装ansible角色

yum -y install rhel-system-roles
$ cp -rvf /usr/share/ansible/roles/rhel-system-roles.timesync/ roles/

部署chrony剧本

vim timesync.yml

---
- name: install chronyd
  hosts: all
  become: true
  vars:
    timesync_ntp_servers:
      - hostname: 192.168.174.143
        iburst: yes
  roles:
    - rhel-system-roles.timesync
  tasks:
    - name: Set chronyd server
      lineinfile:
        path: /etc/chrony.conf
        line: "{
    {item}}"
      loop:
        - "allow 192.168.174.0/24"
        - "local stratum 8"
      when: (ansible_default_ipv4.address  == "192.168.174.143")
      notify: restart chronyd
  handlers:
    - name: resetart chronyd
      service:
        name: chronyd
        state: restarted

执行剧本

ansible-playbook timesync.yml

关闭Selinux

复制selinux角色文件

$ cp -rvf /usr/share/ansible/roles/rhel-system-roles.selinux/ roles/

部署selinux剧本

vim selinux.yml

---
- name: Configure selinux to disabled
  hosts: all
  vars:
    selinux_policy: targeted
    selinux_state: disabled
  roles:
    - rhel-system-roles.selinux

执行剧本：

ansible-playbook selinux.yml

执行完成后，重启所有主机。

配置Ceph离线YUM仓库

此处省略基于镜像配置

同步网络ceph-ansible仓库到本地

安装ceph-ansible repo文件（能连外网的主机）

yum -y install epel-release centos-release-ceph-nautilus centos-release-openstack-stein

同步最新版本软件包

repotrack ceph-ansible -p /root/ceph-ansible/

制作本地源

cd /root
createrepo -v ceph-ansible/

将ceph-ansible仓库同步到node1主机

cd /root
zip -qvr ceph-ansible.zip ceph-ansible
scp ceph-ansible.zip root@node1:/opt/

同步网络Ceph仓库到本地

配置repo文件（能连外网的主机）

vim /etc/yum.repos.d/ceph.repo

 [x86_64]
name=x86_64
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64
enabled=1
gpgcheck=0

[noarch]
name=noarch
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch
enabled=0
gpgcheck=0

同步最新版本软件包

mkdir -p /root/ceph-nautilus/rpm-nautilus/el7
cd /root/ceph-nautilus/rpm-nautilus/el7
reposync -n -r x86_64
reposync -n -r noarch

制作本地源

分别进行x86_64和noarch目录执行createrepo命令，如果没有此命令，执行yum -y install createrepo安装

createrepo -v .

下载key文件

mkdir /root/ceph-nautilus/keys
cd /root/ceph-nautilus/keys
wget http://mirrors.aliyun.com/ceph/keys/release.asc

将ceph仓库同步到node1主机

cd /root
zip -qvr ceph-nautilus.zip ceph-nautilus
scp ceph-nautilus.zip root@node1:/opt/

配置本地ceph-ansible和Ceph仓库

在node1主机上解压仓库文件

cd /opt
unzip ceph-nautilus.zip
unzip ceph-ansible.zip

通过nginx配置内网环境ceph仓库

[root@node1 ~]# yum -y install nginx
[root@node1 ~]# vim /etc/nginx/conf.d/ceph.conf 
server {
    
        listen 8088;
        server_name localhost;
        root /opt/ceph-ansible;
        location / {
    
                autoindex on;
                autoindex_exact_size off;
                autoindex_localtime on;
                }
}
server {
    
        listen 8089;
        server_name localhost;
        root /opt/ceph-nautilus;
        location / {
    
                autoindex on;
                autoindex_exact_size off;
                autoindex_localtime on;
                }
}
[root@node1 ~]# chown nginx.nginx -R /opt/ceph-ansible
[root@node1 ~]# chown nginx.nginx -R /opt/ceph-nautilus

所有内网ceph主机上配置repo

[root@node1 ~]# vim /etc/yum.repos.d/ceph.repo 
[ceph-ansible]
name=ceph-ansible
baseurl=http://192.168.174.143:8088
enabled=1
gpgcheck=0

[ceph-noarch]
name=ceph-noarch
baseurl=http://192.168.174.143:8089/rpm-nautilus/el7/noarch/
enabled=1
gpgcheck=0
[ceph-x86_64]
name=ceph-x86_64
baseurl=http://192.168.174.143:8089/rpm-nautilus/el7/x86_64/
enabled=1
gpgcheck=0

配置ceph-ansible节点

安装ceph-ansible

在node1还节点安装ceph-ansible

yum -y install ceph-ansible

配置ceph变量信息

配置hosts文件

[root@node1]# cd /usr/share/ceph-ansible/
[root@node1 ceph-ansible]# vim hosts
[mons]
node1
node2
node3

[mgrs]
node1
node2
node3

[osds]
node1
node2
node3
node4
node5

[rgws]
node1
node2
node3
node4
node5

[mdss]
node1
node2
node3
node4
node5

[grafana-server]
node1

配置镜像、网络、登陆信息

[root@node1 group_vars]# cd /usr/share/ceph-ansible/group_vars
[root@node1 group_vars]# cp all.yml.sample all.yml
[root@node1 group_vars]# vim all.yml
---
dummy:
configure_firewall: False
ntp_service_enabled: False
ceph_origin: repository
ceph_repository: community
ceph_mirror: http://192.168.174.143:8089
ceph_stable_key: http://192.168.174.143:8089/keys/release.asc
ceph_stable_release: nautilus
ceph_stable_repo: "{
    { ceph_mirror }}/"
rbd_cache: "true"
rbd_cache_writethrough_until_flush: "False"
rbd_client_directories: False # this will create rbd_client_log_path and rbd_client_admin_socket_path directories with proper permissions
monitor_interface: ens33
journal_size: 1024 # OSD journal size in MB
public_network: 192.168.174.0/24
cluster_network: "{
    { public_network }}"
dashboard_enabled: False
dashboard_port: 8443
dashboard_admin_user: admin
dashboard_admin_password: admin
grafana_admin_user: admin
grafana_admin_password: admin

配置osds

[root@node1 group_vars]# cp osds.yml.sample osds.yml
[root@node1 group_vars]# vim osds.yml
---
dummy:
devices:
  - /dev/sdb
  - /dev/sdc
osd_scenario: "collecated"

配置rgws

[root@node1 group_vars]# cp rgws.yml.sample rgws.yml
[root@node1 group_vars]# vim rgws.yml
---
dummy:
radosgw_interface: ens33

配置site

[root@node1]# cd /usr/share/ceph-ansible
[root@node1 ceph-ansible]# cp site.yml.sample site.yml
[root@node1 ceph-ansible]# vim site.yml
- hosts:
  - mons
  - osds
  - mdss
  - rgws
  - nfss
  - rbdmirrors
  - clients
  - mgrs
  - iscsigws
  - iscsi-gws # for backward compatibility only!
  - grafana-server
  - rgwloadbalancers

说明：此处的主机组名可以都不注释，根据hosts文件的组名进行安装，未在hosts文件中设置的不进行安装。

开始安装

[root@node1]# cd /usr/share/ceph-ansible
[root@node1 ceph-ansible]# ansible-playbook -i hosts site.yml

查看ceph版本

[root@node1 ~]# ceph --version

查看集群信息

[root@node1 ~]# ceph -s

集群健康状态报错信息：

[root@node1 ~]# ceph health detail
HEALTH_WARN 1 pools have too few placement groups; mons are allowing insecure global_id reclaim
POOL_TOO_FEW_PGS 1 pools have too few placement groups
Pool cephfs_data has 8 placement groups, should have 32
AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED mons are allowing insecure global_id reclaim
mon.control has auth_allow_insecure_global_id_reclaim set to true
mon.node1 has auth_allow_insecure_global_id_reclaim set to true
mon.node2 has auth_allow_insecure_global_id_reclaim set to true
mon.node3 has auth_allow_insecure_global_id_reclaim set to true

解决办法：

设置资源池cephfs_data的pg数量为32

[root@node1 ~]# ceph osd pool set cephfs_data pg_num 32

禁止不安全模式

[root@node1 ~]# ceph config set mon auth_allow_insecure_global_id_reclaim false