Ceph Docker uses etcd. We use a simple docker-compose file to deploy it within our local network premises as of:
etcd1: image: quay.io/coreos/etcd container_name: etcd1 hostname: etcd1.circus.dev volumes: - /srv/docker/volumes/etcd/data-1:/data restart: always ports: - "5001:5001" - "7001:7001" extra_hosts: - "precortex.circus.dev:192.168.1.30" environment: ETCD_NAME: 'etcd1' ETCD_DATA_DIR: '/data' ETCD_ADVERTISE_CLIENT_URLS: 'http://precortex.circus.dev:7001' ETCD_LISTEN_CLIENT_URLS: 'http://0.0.0.0:7001' ETCD_INITIAL_ADVERTISE_PEER_URLS: 'http://precortex.circus.dev:5001' ETCD_LISTEN_PEER_URLS: 'http://0.0.0.0:5001' ETCD_INITIAL_CLUSTER_TOKEN: 'circus-etcd' ETCD_INITIAL_CLUSTER: 'etcd1=http://precortex.circus.dev:5001,etcd2=http://precortex.circus.dev:5002' etcd2: image: quay.io/coreos/etcd container_name: etcd2 hostname: etcd2.circus.dev volumes: - /srv/docker/volumes/etcd/data-2:/data restart: always ports: - "5002:5002" - "7002:7002" extra_hosts: - "precortex.circus.dev:192.168.1.30" environment: ETCD_NAME: 'etcd2' ETCD_DATA_DIR: '/data' ETCD_ADVERTISE_CLIENT_URLS: 'http://precortex.circus.dev:7002' ETCD_LISTEN_CLIENT_URLS: 'http://0.0.0.0:7002' ETCD_INITIAL_ADVERTISE_PEER_URLS: 'http://precortex.circus.dev:5002' ETCD_LISTEN_PEER_URLS: 'http://0.0.0.0:5002' ETCD_INITIAL_CLUSTER_TOKEN: 'circus-etcd' ETCD_INITIAL_CLUSTER: 'etcd1=http://precortex.circus.dev:5001,etcd2=http://precortex.circus.dev:5002'
All following components make use of it.
docker run --rm -it -e ETCDCTL_PEERS=http://192.168.1.30:7001 -e MON_IP=192.168.1.30 -e MON_NAME=precortex -e CLUSTER=circus -v /srv/docker/volumes/ceph/rootfs/etc/ceph:/etc/ceph ceph/config
docker run --net=host \ -v /srv/docker/volumes/ceph/rootfs/etc/ceph:/etc/ceph \ -v /srv/docker/volumes/ceph/rootfs/var/lib/ceph/:/var/lib/ceph/ \ -e MON_IP=192.168.1.30 \ -e CEPH_PUBLIC_NETWORK=192.168.1.0/24 \ -e KV_TYPE=etcd \ -e KV_IP=192.168.1.30 \ ceph/daemon populate_kvstore -e KV_PORT=7001 \
adding key /auth/cephx with value true to KV store adding key /auth/cephx_require_signatures with value false to KV store adding key /auth/cephx_cluster_require_signatures with value true to KV store adding key /auth/cephx_service_require_signatures with value false to KV store adding key /global/max_open_files with value 131072 to KV store adding key /global/osd_pool_default_pg_num with value 128 to KV store adding key /global/osd_pool_default_pgp_num with value 128 to KV store adding key /global/osd_pool_default_size with value 3 to KV store adding key /global/osd_pool_default_min_size with value 1 to KV store adding key /global/mon_osd_full_ratio with value .95 to KV store adding key /global/mon_osd_nearfull_ratio with value .85 to KV store adding key /mon/mon_osd_down_out_interval with value 600 to KV store adding key /mon/mon_osd_min_down_reporters with value 4 to KV store adding key /mon/mon_clock_drift_allowed with value .15 to KV store adding key /mon/mon_clock_drift_warn_backoff with value 30 to KV store adding key /mon/mon_osd_report_timeout with value 300 to KV store adding key /osd/journal_size with value 100 to KV store adding key /osd/osd_mkfs_type with value xfs to KV store adding key /osd/osd_mkfs_options_xfs with value "-f -i size=2048" to KV store adding key /osd/osd_mon_heartbeat_interval with value 30 to KV store adding key /osd/pool_default_crush_rule with value 0 to KV store adding key /osd/osd_crush_update_on_start with value true to KV store adding key /osd/osd_objectstore with value filestore to KV store adding key /osd/filestore_merge_threshold with value 40 to KV store adding key /osd/filestore_split_multiple with value 8 to KV store adding key /osd/osd_op_threads with value 8 to KV store adding key /osd/filestore_op_threads with value 8 to KV store adding key /osd/filestore_max_sync_interval with value 5 to KV store adding key /osd/osd_max_scrubs with value 1 to KV store adding key /osd/osd_recovery_max_active with value 5 to KV store adding key /osd/osd_max_backfills with value 2 to KV store adding key /osd/osd_recovery_op_priority with value 2 to KV store adding key /osd/osd_client_op_priority with value 63 to KV store adding key /osd/osd_recovery_max_chunk with value 1048576 to KV store adding key /osd/osd_recovery_threads with value 1 to KV store adding key /osd/ms_bind_port_min with value 6800 to KV store adding key /osd/ms_bind_port_max with value 7100 to KV store adding key /client/rbd_cache_enabled with value true to KV store adding key /client/rbd_cache_writethrough_until_flush with value true to KV store adding key /mds/mds_cache_size with value 100000 to KV store
/global/osd_pool_default_size has to be set to 2 for our environment. Let's do so by invoking
curl -sL http://localhost:7001/v2/keys/ceph-config/ceph/global/osd_pool_default_size -XPUT -d value="2" | jq .
in so we can continue to deploy the remaining components. First the monitor MON for this site's instance:
docker run -d --name ceph-mon --net=host \ --restart=always \ -v /srv/docker/volumes/ceph/rootfs/var/lib/ceph/:/var/lib/ceph/ \ -e MON_IP=192.168.1.30 \ -e MON_NAME=precortex \ -e CEPH_PUBLIC_NETWORK=192.168.1.0/24 \ -e KV_TYPE=etcd \ -e KV_IP=192.168.1.30 \ -e KV_PORT=7001 \ ceph/daemon mon
Then the OSD services which maintain the physical storage devices:
docker run -d --name=ceph-osd-sda --net=host \ --privileged=true \ --restart=always \ --pid=host \ -v /srv/docker/volumes/ceph/rootfs/var/lib/ceph/:/var/lib/ceph/ \ -v /dev/:/dev/ \ -e OSD_DEVICE=/dev/sda \ -e OSD_FORCE_ZAP=1 \ -e KV_TYPE=etcd \ -e KV_IP=192.168.1.30 \ -e KV_PORT=7001 \ ceph/daemon osd
docker run -d --name=ceph-osd-sdc --net=host \ --privileged=true \ --restart=always \ --pid=host \ -v /srv/docker/volumes/ceph/rootfs/var/lib/ceph/:/var/lib/ceph/ \ -v /dev/:/dev/ \ -e OSD_DEVICE=/dev/sdc \ -e OSD_FORCE_ZAP=1 \ -e KV_TYPE=etcd \ -e KV_IP=192.168.1.30 \ -e KV_PORT=7001 \ ceph/daemon osd
We've used a revision of the latter to initialize the second disk's partitions, as FORCE_ZAP didn't work initially. Returning to its use then allowed to use the newly initialized disk layout.
docker run -d --name=ceph-osd-sdc --net=host \ --privileged=true \ --restart=always \ --pid=host \ -v /srv/docker/volumes/ceph/rootfs/var/lib/ceph/:/var/lib/ceph/ \ -v /dev/:/dev/ \ -e OSD_DEVICE=/dev/sdc \ -e OSD_TYPE=disk \ -e KV_TYPE=etcd \ -e KV_IP=192.168.1.30 \ -e KV_PORT=7001 \ ceph/daemon osd