MongoDB集群部署 - 带访问控制的分片副本集
1. 前言
Ceilometer将meter、event等数据保存在MongoDB中,之前将MongoDB部署在控制节点上,使用三副本模式,时间长了发现meter数据爆炸式增长,区区2T的磁盘捉襟见肘,而想删除旧数据,需要执行db.repairDatabase()命令才能真正回收磁盘空间。
虽然按官方说法,MongoDB 3.2版本以后默认使用的WiredTiger Storage Engine,在执行db.repairDatabase()时不需要额外空间,可实际操作时发现它会重建索引,保存在storage.dbPath/_tmp目录下,剩余的200GB容量完全不够用来创建1.7TB数据的索引,容量用完,进程就卡住了。本机上又没有空闲的磁盘用来补充容量,这个节点上的MongoDB服务基本算是废了。
幸好Ceilometer不是核心服务,大不了把数据全删了重来。但是考虑到它的数据量太大,很有可能会影响到控制节点的性能,还是把MongoDB单独拎出来部署更为保险。搜了一圈,发现目前MongoDB集群的部署方案都是分片+副本集,我在这之上补充了权限验证的配置步骤,整理成这篇文档。
2. 环境
使用了三台服务器,部署三个分片,每个分片三副本。实际上分片数量可以是任意个,试主机性能而定。各个分片之间是完全相互独立的,一个database的数据只会落在一个分片上。
服务器:10.212.36.38、10.212.36.39、10.212.36.40
系统:CentOS Linux release 7.2.1511 (Core)
MongoDB:v4.0.0
部署结构如下表所示:
10.212.36.38 |
10.212.36.39 |
10.212.36.40 |
mongos: 27017 |
mongos: 27017 |
mongos: 27017 |
config: 27018 |
config: 27018 |
config: 27018 |
shard01: 27101 |
shard01: 27101 |
shard01: 27101 |
shard02: 27102 |
shard02: 27102 |
shard02: 27102 |
shard03: 27103 |
shard03: 27103 |
shard03: 27103 |
mongos是对外提供服务的进程,本身不保存数据,而是将请求转发到分片;config保存集群配置数据;shard是实际存储数据的服务,每个shard相互独立,由mongos调度。
3. 安装MongoDB
3.1 添加yum源
cat > /etc/yum.repos.d/mongodb-org-4.0.repo << EOF [mongodb-org-4.0] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.0/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-4.0.asc EOF
3.2 安装
# yum -y install mongodb-org
3.3 删除mongod服务
由于不是通过软件包内置的服务启动mongod进程,删除mongod这个服务避免误启动,也可以保留服务文件作为参照。
# systemctl disable mongod # rm –f /usr/lib/systemd/system/mongod.service # systemctl daemon-reload
4. 部署高可用MongoDB集群
4.1 准备配置文件
每台服务器上都运行monogs、config、shard01、shard02、shard03服务,分别对应一个配置文件,统一将配置文件存放在/etc/mongodb/目录下。
# mkdir /etc/mongodb/ # chown –R mongod:mongod /etc/mongodb/
将config和shard的数据保存在/data/mongodb/目录下。
# mkdir -p /data/mongodb/{config,shard01,shard02,shard03}/data /data/mongodb/mongos # chown –R mongod:mongod /data/mongodb/
日志统一存放在/var/log/mongodb/目录下。
# mkdir /var/log/mongodb # chown –R mongod:mongod /var/log/mongodb/
/etc/mongodb/shard01.conf
# where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/shard01.log # Where and how to store data. storage: dbPath: /data/mongodb/shard01/data journal: enabled: true wiredTiger: engineConfig: cacheSizeGB: 20 # how the process runs processManagement: fork: true pidFilePath: /data/mongodb/shard01/mongodb-shard01.pid timeZoneInfo: /usr/share/zoneinfo # network interfaces net: port: 27101 bindIp: 0.0.0.0 unixDomainSocket: pathPrefix: /var/run/mongodb #operationProfiling: replication: replSetName: ussmongo-shard01 sharding: clusterRole: shardsvr
View Code
/etc/mongodb/shard02.conf
# where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/shard02.log # Where and how to store data. storage: dbPath: /data/mongodb/shard02/data journal: enabled: true wiredTiger: engineConfig: cacheSizeGB: 20 # how the process runs processManagement: fork: true pidFilePath: /data/mongodb/shard02/mongodb-shard02.pid timeZoneInfo: /usr/share/zoneinfo # network interfaces net: port: 27102 bindIp: 0.0.0.0 unixDomainSocket: pathPrefix: /var/run/mongodb #operationProfiling: replication: replSetName: ussmongo-shard02 sharding: clusterRole: shardsvr
View Code
/etc/mongodb/shard03.conf
# where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/shard03.log # Where and how to store data. storage: dbPath: /data/mongodb/shard03/data journal: enabled: true wiredTiger: engineConfig: cacheSizeGB: 20 # how the process runs processManagement: fork: true pidFilePath: /data/mongodb/shard03/mongodb-shard03.pid timeZoneInfo: /usr/share/zoneinfo # network interfaces net: port: 27103 bindIp: 0.0.0.0 unixDomainSocket: pathPrefix: /var/run/mongodb #operationProfiling: replication: replSetName: ussmongo-shard03 sharding: clusterRole: shardsvr
View Code
/etc/mongodb/config.conf
# where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/config.log # Where and how to store data. storage: dbPath: /data/mongodb/config/data journal: enabled: true # how the process runs processManagement: fork: true pidFilePath: /data/mongodb/config/mongodb-config.pid timeZoneInfo: /usr/share/zoneinfo # network interfaces net: port: 27018 bindIp: 0.0.0.0 unixDomainSocket: pathPrefix: /var/run/mongodb #operationProfiling: replication: replSetName: ussmongo-config sharding: clusterRole: configsvr
View Code
/etc/mongodb/mongos.conf
systemLog: destination: file logAppend: true path: /var/log/mongodb/mongos.log processManagement: fork: true # pidFilePath: /data/mongodb/mongos.pid # network interfaces net: port: 27017 bindIp: 0.0.0.0 unixDomainSocket: pathPrefix: /var/run/mongodb sharding: configDB: ussmongo-config/10.212.36.38:27018,10.212.36.39:27018,10.212.36.40:27018 setParameter: diagnosticDataCollectionDirectoryPath: /data/mongodb/mongos/diagnostic.data/
View Code
4.2 准备服务文件
使用下面的命令可以启动MongoDB进程:
# mongod --quiet -f /etc/mongodb/shard01.conf
但是每个节点上都要启动五个进程,相当麻烦,也不便于后续维护。为了一劳永逸,以服务的方式运行MongoDB,使用systemctl管理MongoDB服务。
/usr/lib/systemd/system/mongo-shard@.service
[Unit] Description=MongoDB Database Shard Service After=network.target Documentation=https://docs.mongodb.org/manual PartOf=mongo-shard.target [Service] User=mongod Group=mongod Environment="OPTIONS=--quiet -f /etc/mongodb/shard%i.conf" EnvironmentFile=-/etc/sysconfig/mongod ExecStart=/usr/bin/mongod $OPTIONS PermissionsStartOnly=true Type=forking TasksMax=infinity TasksAccounting=false [Install] WantedBy=mongo-shard.target
/usr/lib/systemd/system/mongo-config.service
[Unit] Description=MongoDB Database Config Service After=network.target Documentation=https://docs.mongodb.org/manual PartOf=mongo.target [Service] User=mongod Group=mongod Environment="OPTIONS=--quiet -f /etc/mongodb/config.conf" EnvironmentFile=-/etc/sysconfig/mongod ExecStart=/usr/bin/mongod $OPTIONS PermissionsStartOnly=true Type=forking TasksMax=infinity TasksAccounting=false [Install] WantedBy=mongo.target
/usr/lib/systemd/system/mongos.service
[Unit] Description=MongoDB Database Service After=syslog.target network.target PartOf=mongo.target [Service] User=mongod Group=mongod Environment="OPTIONS=--quiet -f /etc/mongodb/mongos.conf" ExecStart=/usr/bin/mongos $OPTIONS Type=forking PrivateTmp=true LimitNOFILE=64000 TimeoutStartSec=180 [Install] WantedBy=mongo.target
为了便于批量管理,创建target文件:
/usr/lib/systemd/system/mongo-shard.target
[Unit] Description=mongo shard target allowing to start/stop all mongo-shard@.service instances at once PartOf=mongo.target [Install] WantedBy=mongo.target
/usr/lib/systemd/system/mongo.target
[Unit] Description=mongo target allowing to start/stop all mongo*.service instances at once [Install] WantedBy=multi-user.target
载入服务:
# systemctl daemon-reload # systemctl enable mongo-shard@01 # systemctl enable mongo-shard@02 # systemctl enable mongo-shard@03 # systemctl enable mongo-config # systemctl enable mongos # systemctl enable mongo-shard.target # systemctl enable mongo.target
现在就可以方便地管理MongoDB服务了:
# systemctl start mongo-shard.target # 启动所有shard服务
# systemctl start mongo.target # 启动所有shard、config、mongos服务
4.3 配置副本集
config和shard服务本质上都是mongod进程,将他们都配置为三副本模式。下面的操作可以在三个节点中的任意一个上执行,只需要执行一遍。
config副本集:
# mongo --port 27018 > use admin > config = { ... _id : "ussmongo-config", ... members : [ ... {_id : 0, host : "10.212.36.38:27018" }, ... {_id : 1, host : "10.212.36.39:27018" }, ... {_id : 2, host : "10.212.36.40:27018" } ... ] ... } > rs.initiate(config);
shard01副本集:
# mongo --port 27101 > use admin > config = { ... _id : "ussmongo-shard03", ... members : [ ... {_id : 0, host : "10.212.36.38:27101" }, ... {_id : 1, host : "10.212.36.39:27101" }, ... {_id : 2, host : "10.212.36.40:27101" } ... ] ... } > rs.initiate(config);
shard02副本集:
# mongo --port 27102 > use admin > config = { ... _id : "ussmongo-shard02", ... members : [ ... {_id : 0, host : "10.212.36.39:27102" }, ... {_id : 1, host : "10.212.36.40:27102" }, ... {_id : 2, host : "10.212.36.38:27102" } ... ] ... } > rs.initiate(config);
shard03副本集:
# mongo --port 27103 > use admin > config = { ... _id : "ussmongo-shard03", ... members : [ ... {_id : 0, host : "10.212.36.40:27103" }, ... {_id : 1, host : "10.212.36.38:27103" }, ... {_id : 2, host : "10.212.36.39:27103" } ... ] ... } > rs.initiate(config);
4.4 配置分片路由
mongos对外提供服务,是集群的入口。需要先将分片添加到mongos配置中:
# mongo --port 27017 > use admin > sh.addShard("ussmongo-shard01/10.212.36.38:27101,10.212.36.39:27101,10.212.36.40:27101") > sh.addShard("ussmongo-shard02/10.212.36.39:27102,10.212.36.40:27102,10.212.36.38:27102") > sh.addShard("ussmongo-shard03/10.212.36.40:27103,10.212.36.38:27103,10.212.36.39:27103") > sh.status();
到这里集群已经能够提供服务了。mongos是无状态的,在三个mongos之上配置负载均衡,就完成了MongoDB高可用集群的部署。
5. 启用访问控制
线上环境集群不可能使用免认证的方式,都要开启安全认证。MongoDB在开启了访问控制后,只有一次添加用户的机会,此后的操作都需要先认证通过。为了方便,我们先添加用户,然后再开启访问控制。
5.1 添加用户
连接上mongos添加的用户会保存在config副本集中,但是不会保存到shard副本集,因此添加用户的操作需要分别在config、shard01、shard02、shard03上执行。
config副本集:
# mongo --port 27018 > use admin > db.createUser( ... { ... user: "admin", ... pwd: "admin", ... roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase", "clusterAdmin"] ... } ... )
shard01副本集:
# mongo --port 27101 > use admin > db.createUser( ... { ... user: "admin", ... pwd: "admin", ... roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase", "clusterAdmin"] ... } ... )
shard02副本集:
# mongo --port 27102 > use admin > db.createUser( ... { ... user: "admin", ... pwd: "admin", ... roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase", "clusterAdmin"] ... } ... )
shard03副本集:
# mongo --port 27103 > use admin > db.createUser( ... { ... user: "admin", ... pwd: "admin", ... roles: ["userAdminAnyDatabase", "dbAdminAnyDatabase", "readWriteAnyDatabase", "clusterAdmin"] ... } ... )
5.2 启用访问控制
1) 创建秘钥文件
启用访问控制之后,外部访问MongoDB服务需要进行身份验证,而mongos访问config和shard服务则是通过配置的秘钥文件。
# openssl rand -base64 756 >/data/mongodb/ussmongo.key # chmod 0600 /data/mongodb/ussmongo.key # chown mongod:mongod /data/mongodb/ussmongo.key
将密钥文件复制到所有节点上。
2) 添加security配置
mongos的配置文件添加如下配置:
security:
keyFile: /data/mongodb/ussmongo.key
config和shard的配置文件添加如下配置:
security:
authorization: enabled
keyFile: /data/mongodb/ussmongo.key
3) 重启服务
在所有节点上重启所有MongoDB服务:
# systemctl restart mongo.target
至此带访问控制的MongoDB高可用集群就部署完成了。
参考资料
Configuration File Options — MongoDB Manual