centos7 memcached+magent+keepalived集群
111,222均部署keepalived,magent,memcached
keepalived 111为主机，222为备机
其中，111上magent以本地memcache为主，222为备用
222上magen以本地memcache为主，111为备用
安装memcached
安装：yum -y install memcached 查看配置：cat /etc/sysconfig/memcached, 通过ip+下图port可远程连接
PORT=”11211″
USER=”memcached”
MAXCONN=”1024″
CACHESIZE=”64″
OPTIONS=””
启动： systemctl start memcached
停止： systemctl stop memcached
重启： systemctl restart memcached
连接memcached测试
cmd命令窗口：telnet 10.0.0.111 11211
(回车)
ERROR
add aaa 0 0 4
1111
STORED
get aaa
VALUE aaa 0 4
1111
END
安装libevent(略，下次装时补充)
安装magent
mkdir /opt/magent cd magent wget https://files.cnblogs.com/files/lurenjia1994/magent.tar.gz tar xzvf magent.tar.gz make进行安装
可能错误1：
magent.c: 在函数‘writev_list’中: magent.c:729:17: 错误：‘SSIZE_MAX’未声明(在此函数内第一次使用) if (toSend > SSIZE_MAX || ^ magent.c:729:17: 附注：每个未声明的标识符在其出现的函数内只报告一次 make: *** [magent.o] 错误 1
解决方法：
vi ketama.h 在头部加入以下代码 #ifndef SSIZE_MAX #define SSIZE_MAX 32767 #endif
可能错误2：
gcc -Wall -g -O2 -I/usr/local/include -m64 -c -o magent.o magent.c gcc -Wall -g -O2 -I/usr/local/include -m64 -c -o ketama.o ketama.c gcc -Wall -g -O2 -I/usr/local/include -m64 -o magent magent.o ketama.o /usr/lib64/libevent.a /usr/lib64/libm.a gcc: 错误：/usr/lib64/libevent.a：没有那个文件或目录 gcc: 错误：/usr/lib64/libm.a：没有那个文件或目录 make: *** [magent] 错误 1
解决方法：
ln -s /usr/lib64/libm.so /usr/lib64/libm.a
ln -s /usr/lib64/libevent.so /usr/lib64/libevent.a
检查是否安装成功
[root@localhost magent]# cp magent /usr/bin/ [root@localhost magent]# magent please provide -s "ip:port" argument memcached agent v0.6 Build-Date: Aug 22 2018 15:40:29 Usage: -h this message -u uid -g gid -p port, default is 11211. (0 to disable tcp support) -s ip:port, set memcached server ip and port -b ip:port, set backup memcached server ip and port -l ip, local bind ip address, default is 0.0.0.0 -n number, set max connections, default is 4096 -D don't go to background -k use ketama key allocation algorithm -f file, unix socket path to listen on. default is off -i number, set max keep alive connections for one memcached server, default is 20 -v verbose
启动magent
[root@centos7 magent]# magent -u root -n 51200 -l 10.0.0.111 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211 [root@centos7 magent]# ps -aux|grep magent root 32007 0.0 0.0 11948 616 ? Ss 15:47 0:00 magent -u root -n 51200 -l 10.0.0.111 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211 root 32466 0.0 0.0 112720 972 pts/1 S+ 15:48 0:00 grep --color=auto magent
测试magent
1.连接magent节点
现象：通过magent节点可以取到所有主节点的数据
telnet 10.0.0.111 12000 UNSUPPORTED COMMAND get aaa 前面在测试memcache连接设置的aaa,这里取值成功 VALUE aaa 0 4 1111 END set b 0 0 4 1234 STORED
2.连接主节点
现象：可以取到所有magent节点存入的数据
ERROR get aaa VALUE aaa 0 4 1111 END get b VALUE b 0 4 1234 END set c 0 0 4 cccc STORED
3.连接备份节点
现象：可以取到所有magent节点存入的数据
ERROR get aaa END get b #只能取到在magent节点设置的值，其他取不到 VALUE b 0 4 1234 END get c END
4.关掉主节点的memcache进程
现象：通过magent节点可以取到所有备份节点的数据
[root@centos7 magent]# systemctl stop memcached [root@centos7 magent]# ps -aux|grep memcached root 30827 0.0 0.0 112720 972 pts/1 S+ 16:05 0:00 grep --color=auto memcached cmd连接magent节点 telnet 10.0.0.111 12000 UNSUPPORTED COMMAND get aaa END get b VALUE b 0 4 1234 END get c END
5.再开启主节点memcached
现象：取不到前面设置的所有值，因为重启以后，主节点数据全部清空，虽然备份节点有数据，但是主节点在备份节点不会生效
UNSUPPORTED COMMAND get a END get aaa END get b END get c END
安装keepalived(当10.0.0.111上的magent挂掉时，自动切换到10.0.0.222的magent)
yum install keepalived
修改keepalived配置
本文配置为在这个基础上添加的配置，请忽略多余部分keepalive+nginx 热备跟负载均衡
keepalived配置
! Configuration File for keepalived global_defs { notification_email { #acassen@firewall.loc #failover@firewall.loc #sysadmin@firewall.loc } #notification_email_from Alexandre.Cassen@firewall.loc #smtp_server 192.168.200.1 #smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr #vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_script chk_nginx { script "/etc/keepalived/chk_nginx.sh" interval 2 } vrrp_script chk_magent { script "/etc/keepalived/chk_magent.sh" interval 2 } vrrp_instance VI_1 { state MASTER interface ens192 virtual_router_id 51 priority 50 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.233 } notify_master "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！111成为主机！ 报！111成为主机！" notify_fault "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！111已挂！ 10.0.0.111故障！请尽快修复！" track_script { chk_nginx } } vrrp_instance VI_2 { state MASTER interface ens192 virtual_router_id 52 priority 50 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.234 } track_script { chk_magent } }
其中/etc/keepalived目录下的chk_magent.sh
#!/bin/bash run=`ps -C magent --no-header | wc -l` if [ $run -eq 0 ]; then magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211 sleep 3 if [ `ps -C magent --no-header | wc -l` -eq 0 ]; then systemctl stop keepalived fi fi
遇到问题1
绑定不了虚拟vip排查，没有发现虚拟ip 10.0.0.234
只有vrid 51的组播，没有52的组播 [root@centos7 keepalived]# tcpdump -nn -c 20 -i any host 224.0.0.18 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 10:46:43.008689 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:44.009046 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:45.009174 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:46.009534 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:47.009703 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:48.010830 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:49.010960 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:50.012050 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:51.012178 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:52.012727 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:53.012828 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:54.013334 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:55.013481 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:56.013858 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:57.014023 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:58.014457 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:46:59.014604 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:47:00.015002 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:47:01.015815 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 10:47:02.016914 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 20 20 packets captured 20 packets received by filter 0 packets dropped by kernel
View Code
查看日志：cat /var/log/message
似乎是因为脚本导致vip2启动失败
Aug 23 11:17:52 centos7 Keepalived_vrrp[8399]: Sending gratuitous ARP on ens192 for 10.0.0.233 Aug 23 11:17:52 centos7 Keepalived_vrrp[8399]: Sending gratuitous ARP on ens192 for 10.0.0.233 Aug 23 11:17:52 centos7 Keepalived_vrrp[8399]: Sending gratuitous ARP on ens192 for 10.0.0.233 Aug 23 11:17:52 centos7 Keepalived_vrrp[8399]: Sending gratuitous ARP on ens192 for 10.0.0.233 Aug 23 11:17:53 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:17:55 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:17:57 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:17:59 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:01 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:03 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:05 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:07 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:09 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:11 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:13 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:15 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:17 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:19 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:21 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:23 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:25 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:27 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:29 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:31 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 11:18:33 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15
View Code
难道他是先运行脚本，再开启vip2?
于是把脚本中的10.0.0.234改成10.0.0.111，再重启keepalived，再查看ip，发现已经有10.0.0.234了
[root@centos7 keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:0c:29:c8:03:3f brd ff:ff:ff:ff:ff:ff inet 10.0.0.111/24 brd 10.0.0.255 scope global noprefixroute ens192 valid_lft forever preferred_lft forever inet 10.0.0.233/32 scope global ens192 valid_lft forever preferred_lft forever inet 10.0.0.234/32 scope global ens192 valid_lft forever preferred_lft forever inet6 fe80::7e90:403b:465:db6c/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:70:99:88 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:70:99:88 brd ff:ff:ff:ff:ff:ff
View Code
于是在vip1中设置两个虚拟ip：10.0.0.233，10.0.0.234，vip2中不设置虚拟ip，检测脚本启动magent时绑定10.0.0.234，经测试10.0.0.111上部署成功
遇到问题2
10.0.0.222启动keepalived时，magent似乎启动失败
排查：
1.关掉10.0.0.111上的keepalived
2.给111上的keepalived.conf跟chk_magent.sh覆盖掉222对应文件
3.启动
systemctl start keepalived
4.检查地址
[root@localhost keepalived]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:0c:29:59:17:90 brd ff:ff:ff:ff:ff:ff inet 10.0.0.222/24 brd 10.0.0.255 scope global noprefixroute ens192 valid_lft forever preferred_lft forever inet 10.0.0.233/32 scope global ens192 valid_lft forever preferred_lft forever inet 10.0.0.234/32 scope global ens192 valid_lft forever preferred_lft forever inet6 fe80::5e69:554d:d28e:dbb5/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:59:38:b4 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:59:38:b4 brd ff:ff:ff:ff:ff:ff
View Code
5.检查magent服务
[root@localhost keepalived]# ps -aux|grep magent root 23443 0.0 0.0 115296 1476 ? S 15:21 0:00 /bin/bash /etc/keepalived/chk_magent.sh root 23457 0.0 0.0 112720 968 pts/0 S+ 15:21 0:00 grep --color=auto magent 通过./chk_magent.sh可以正确启动 正确启动应该是这样 [root@localhost keepalived]# ./chk_magent.sh [root@localhost keepalived]# ps -aux|grep magent root 24142 0.0 0.0 11948 612 ? Ss 15:22 0:00 magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211 root 24185 0.0 0.0 112720 972 pts/0 S+ 15:22 0:00 grep --color=auto magent
View Code
6.检查系统日志
Aug 23 15:22:19 localhost Keepalived_vrrp[22337]: /etc/keepalived/chk_magent.sh exited due to signal 15 Aug 23 15:22:19 localhost setroubleshoot: failed to retrieve rpm info for /usr/bin/magent Aug 23 15:22:19 localhost setroubleshoot: SELinux is preventing /usr/bin/magent from name_bind access on the tcp_socket port 12000. For complete SELinux messages run: sealert -l 7a3a5f2e-5328-4dd4-88d6-bdefea289425 Aug 23 15:22:19 localhost python: SELinux is preventing /usr/bin/magent from name_bind access on the tcp_socket port 12000.#012#012***** Plugin bind_ports (92.2 confidence) suggests ************************#012 脚本启动失败了，似乎是防火墙
View Code
7.关掉SELinux
[root@localhost keepalived]# setenforce 0 [root@localhost keepalived]# getenforce Permissive
View Code
8.重启keepalived
[root@localhost keepalived]# systemctl stop keepalived [root@localhost keepalived]# pkill magent [root@localhost keepalived]# ps -aux|grep magent root 27598 0.0 0.0 112720 972 pts/0 S+ 15:32 0:00 grep --color=auto magent [root@localhost keepalived]# systemctl start keepalived [root@localhost keepalived]# ps -aux|grep magent root 27647 0.0 0.0 11948 616 ? Ss 15:32 0:00 magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211 root 27699 0.0 0.0 112720 972 pts/0 S+ 15:32 0:00 grep --color=auto magent 果然是防火墙！
View Code
9.SELinux放出magent
比较复杂，以后再补
View Code
最终配置
10.0.0.111上的keepalived.conf
! Configuration File for keepalived global_defs { notification_email { #acassen@firewall.loc #failover@firewall.loc #sysadmin@firewall.loc } #notification_email_from Alexandre.Cassen@firewall.loc #smtp_server 192.168.200.1 #smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr #vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_script chk_nginx { script "/etc/keepalived/chk_nginx.sh" interval 2 } vrrp_script chk_magent { script "/etc/keepalived/chk_magent.sh" interval 2 } vrrp_instance VI_1 { state MASTER interface ens192 virtual_router_id 51 priority 50 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.233 10.0.0.234 } notify_master "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！111成为主机！ 报！111成为主机！" notify_fault "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！111已挂！ 10.0.0.111故障！请尽快修复！" track_script { chk_nginx } } vrrp_instance VI_2 { state MASTER interface ens192 virtual_router_id 52 priority 50 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { # 10.0.0.234 } track_script { chk_magent } }
View Code
10.0.0.111上的chk_magent.sh
#!/bin/bash run=`ps -C magent --no-header | wc -l` if [ $run -eq 0 ]; then magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211 sleep 3 if [ `ps -C magent --no-header | wc -l` -eq 0 ]; then systemctl stop keepalived fi fi
View Code
10.0.0.222上的keepalived.conf
! Configuration File for keepalived global_defs { notification_email { #acassen@firewall.loc #failover@firewall.loc #sysadmin@firewall.loc } #notification_email_from Alexandre.Cassen@firewall.loc #smtp_server 192.168.200.1 #smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr #vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_script chk_nginx { script "/etc/keepalived/chk_nginx.sh" interval 2 } vrrp_script chk_magent { script "/etc/keepalived/chk_magent.sh" interval 2 } vrrp_instance VI_1 { state BACKUP interface ens192 virtual_router_id 51 priority 30 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.0.0.233 10.0.0.234 } notify_master "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！222成为主机！ 报！222成为主机！" notify_fault "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！222已挂！ 10.0.0.222故障！请尽快修复！" track_script { chk_nginx } } vrrp_instance VI_2 { state BACKUP interface ens192 virtual_router_id 52 priority 30 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { # 10.0.0.234 } track_script { chk_magent } }
View Code
10.0.0.222上的chk_magent.sh
#!/bin/bash run=`ps -C magent --no-header | wc -l` if [ $run -eq 0 ]; then magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.222:11211 -b 10.0.0.111:11211 sleep 3 if [ `ps -C magent --no-header | wc -l` -eq 0 ]; then systemctl stop keepalived fi fi
View Code
参考文档
版权声明：本文为lurenjia1994原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。