Ceph cluster setup

Ceph is the undisputed leader in software based storage solution. A minimun of three nodes is required for the ceph cluster setup. Each node should contain a hard drive for the os installation vda and another one to participate in the ceph cluster at minimun vdb

install cephadm

Ceph installation begin with installation of cephadm. It can be installed using apt

sudo apt install cephadm

update the ceph repo

update the cephadm to the latest version

sudo cephadm add-repo --release tentacle
sudo adduser ceph
sudo cephadm install

bootstrap the cluster

once the latest version is installed bootstrap the cluster with

sudo cephadm bootstrap --mon-ip 192.168.200.160

install dependencies

use the same os as the master node on the other nodes and install the following dependencies.

sudo apt install lvm2 podman

editing ssh configuration allow root login. Edit /etc/ssh/sshd_config

PermitRootLogin yes

and restart ssh service using sudo systemctl restart ssh

copy ssh key from master to others

copy the ssh key from the master node to other nodes with the following command

sudo ssh-copy-id -f -i /etc/ceph/ceph.pub root@192.168.200.161

Note

used the following workaround to copy the pub key as a normal user and move it to .ssh/authorized_keys for the root user.

sudo mkdir -p /root/.ssh
sudo chmod 700 /root/.ssh
sudo cp /home/ubuntu/.ssh/authorized_keys /root/.ssh

add nodes to the ceph cluster

once ssh access has been ensured on the nodes. add the nodes to the ceph cluster. On the master

sudo /usr/sbin/cephadm shell
ceph orch host add ceph-02 192.168.200.161
ceph orch host add ceph-03 192.168.200.162

view hosts added to the cluster

root@ceph-01:/# sudo ceph orch host ls

create osd on the ceph master

sudo /usr/sbin/cephadm shell
ceph orch daemon add osd ceph-01:/dev/vdb
ceph orch daemon add osd ceph-02:/dev/vdb
ceph orch daemon add osd ceph-03:/dev/vdb

list the osds

sudo /usr/sbin/cephadm shell
ceph osd tree

create a user to access cephfs

create the user and get the auth key with

sudo /usr/sbin/cephadm shell
ceph auth get-or-create-key client.fuser
ceph auth print-key client.fuser

mount the filesystem

install ceph-common to support mounting cephfs

sudo apt install ceph-common

mount the fs using the command

sudo mkdir /mnt/testing
sudo mount -t ceph 192.168.200.160,192.168.200.161,192.168.200.162:/ /mnt/testing -o name=fuser,secret=<s>,mds_namespace=testing