三、LVS 持续性连接
LVS的持久连接：
持久连接即是不考虑LVS的转发方法，确保所有来自同一个用户的连接转发到同一个RealServer上。
lvs持久连接适用于大部分调度算法。当某一种请求需要定向到一个real server 时，就要用到持久连接一般应用到：ssl（http.https等）、ftp。-p //表示此连接为持久连接N //表示维持此持久连接的时间。默认6分钟。当超过这个时间后，如果网页还没有关掉，仍处于激活状态，重新复位时间为2分钟。
持久连接的类型：
1.PCC(persistent client connector,持久用户连接)同一个用户所有的请求在超时范围之内都被定位到同一个RealServer上，这个时候在指定端口的时候使用的是0端口，就是所有的请求都转发出去。
[root@localhost ~]# ipvsadm -A -t 10.40.0.51:0 -s rr -p
[root@localhost ~]# ipvsadm -a -t 10.40.0.51:0 -r 192.168.1.11 -g
[root@localhost ~]# ipvsadm -a -t 10.40.0.51:0 -r 192.168.1.12 -g
[root@localhost ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.40.0.51:0 rr persistent 360
-> 192.168.1.11:0 Route 1 0 0
-> 192.168.1.12:0 Route 1 0 0
2.PPC(persistent port connector)用户的所有请求在超时范围内按照端口定位到不同的RS上。，只对一个服务进行持久链接。
[root@localhost ~]# ipvsadm -A -t 10.40.0.51:22 -s rr -p
[root@localhost ~]# ipvsadm -a -t 10.40.0.51:22 -r 192.168.1.11:22 -g
[root@localhost ~]# ipvsadm -a -t 10.40.0.51:22 -r 192.168.1.12:22 -g
[root@localhost ~]# ipvsadm -A -t 10.40.0.51:80 -s rr -p
[root@localhost ~]# ipvsadm -a -t 10.40.0.51:80 -r 192.168.1.12:80 -g
[root@localhost ~]# ipvsadm -a -t 10.40.0.51:80 -r 192.168.1.11:80 -g
[root@localhost ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.40.0.51:22 rr persistent 360
-> 192.168.1.11:22 Route 1 0 0
-> 192.168.1.12:22 Route 1 0 0
TCP 10.40.0.51:80 rr persistent 360
-> 192.168.1.11:80 Route 1 0 0
-> 192.168.1.12:80 Route 1 0 0
3.防火墙标记：把相关联的端口在防火墙上打上同样的标记，用户在访问两个相关联的服务的时候，就会定位到同一个RealServer上。
4.FTP connection：由于ftp使用的是两个端口号，所以需要单独列出来。
参考文献：
https证书及CA创建过程
假设ca放在192.168.1.15上，httpd服务器是192.168.1.11、192.168.1.12
1、创建CA,并生成自签证书PEM（windwos系统的格式通常是DER）
[root@ 192.168.1.15 CA]# (umask 077; openssl genrsa 2048 >private/cakey.pem)
Generating RSA private key, 2048 bit long modulus
........+++
........................................+++
e is 65537 (0x10001)
[root@ 192.168.1.15 CA]# openssl req -new -x509 -in private/cakey.pem -out cacert1.pem -days 3650
Generating a 2048 bit RSA private key
........................................+++
....................................................+++
writing new private key to 'privkey.pem'
Enter PEM pass phrase:
140021996894024:error:28069065:lib(40):UI_set_result:result too small:ui_lib.c:869:You must type in 4 to 1024 characters
140021996894024:error:0906406D:PEM routines:PEM_def_callback:problems getting password:pem_lib.c:111:
140021996894024:error:0907E06F:PEM routines:DO_PK8PKEY:read key:pem_pk8.c:130:
[root@localhost CA]# openssl req -new -x509 -key private/cakey.pem -out cacert1.pem -days 3650
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:HENAN
Locality Name (eg, city) [Default City]:ZZ
Organization Name (eg, company) [Default Company Ltd]:MAGEEDU
Organizational Unit Name (eg, section) []:TECH
Common Name (eg, your name or your server's hostname) []:ca.magedu.com
Email Address []:ca@magedu.com
[root@ 192.168.1.15 CA]# touch index.txt
[root@ 192.168.1.15 CA]# echo “01”>serial
2、创建服务器的证书签发请求CSR
[root@192.168.1.12 ~]# cd /etc/httpd/conf/&& mkdir ssl&& cd ssl &&(umask 077;openssl genrsa 1024 >httpd.key)
Generating RSA private key, 1024 bit long modulus
................++++++
......................................++++++
e is 65537 (0x10001)
[root@192.168.1.12 /etc/httpd/conf/ssl]# openssl req -new -key httpd.key -out httpd.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:HENAN
Locality Name (eg, city) [Default City]:ZZ
Organization Name (eg, company) [Default Company Ltd]:MAGEEDU
Organizational Unit Name (eg, section) []:TECH
Common Name (eg, your name or your server's hostname) []:test.magedu.com
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@192.168.1.12/etc/httpd/conf/ssl]$ls
httpd.csr httpd.key
3、CA颁发给服务器证书
[root@ 192.168.1.15 CA]# scp -r 192.168.1.12:/etc/httpd/conf/ssl/httpd.csr /root/
root@192.168.1.12's password:
httpd.csr 100% 647 0.6KB/s 00:00
[root@ 192.168.1.15 CA]# openssl ca -in /root/httpd.csr -out /root/httpd.crt -days 265
Using configuration from /etc/pki/tls/openssl.cnf
unable to load number from /etc/pki/CA/serial
error while loading serial number
140408525461320:error:0D066096:asn1 encoding routines:a2i_ASN1_INTEGER:short line:f_int.c:215:
[root@ 192.168.1.15 CA]# ls
cacert1.pem certs httpd.csr newcerts privkey.pem
cacert.pem crl index.txt private serial
[root@ 192.168.1.15 CA]# echo "01">serial
[root@192.168.1.15 CA]# cat serial
01
[root@localhost CA]# openssl ca -in /root/httpd.csr -out /root/httpd.crt -days 265
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Mar 14 12:42:02 2017 GMT
Not After : Dec 4 12:42:02 2017 GMT
Subject:
countryName = CN
stateOrProvinceName = HENAN
organizationName = MAGEEDU
organizationalUnitName = TECH
commonName = test.magedu.com
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
A8:9F:C2:4C:1D:64:A1:A3:3E:07:C5:2D:81:73:7E:4B:55:F8:CD:A0
X509v3 Authority Key Identifier:
keyid:A3:7A:FD:8A:E0:F0:FF:E0:5F:F1:BE:F8:11:BA:7A:BD:53:8E:58:38
Certificate is to be certified until Dec 4 12:42:02 2017 GMT (265 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
4、http的服务器安装配置证书、安装mod_ssl模块。
[root@192.168.1.12 ~]# scp 192.168.1.15:/root/httpd.crt /etc/httpd/conf/ssl/
root@192.168.1.15s password:
httpd.crt 100% 3066 3.0KB/s 00:00
[root@192.168.1.12 /etc/httpd/conf/ssl]# yum install mod_ssl
5、修改/etc/httpd/conf.d/ssl.conf配置文件。
# Server Certificate:
# Point SSLCertificateFile at a PEM encoded certificate. If
# the certificate is encrypted, then you will be prompted for a
# pass phrase. Note that a kill -HUP will prompt again. A new
# certificate can be generated using the genkey(1) command.
#SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateFile /etc/httpd/conf/ssl/httpd.crt
# Server Private Key:
# If the key is not combined with the certificate, use this
# directive to point at the key file. Keep in mind that if
# you've both a RSA and a DSA private key you can configure
# both in parallel (to also allow the use of DSA ciphers, etc.)
#SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
SSLCertificateKeyFile /etc/httpd/conf/ssl/httpd.key
# General setup for the virtual host, inherited from global configuration
DocumentRoot "/var/www/html"
#ServerName www.example.com:443
6、对于多台实现负载均衡功能的httpd服务器，可以直接将ssl.conf和证书复制到其他httpd服务器上即可。
[root@192.168.1.12 /etc/httpd/conf/ssl]# scp -rp /etc/httpd/conf/ssl 192.168.1.11:/etc/httpd/conf/
The authenticity of host '192.168.1.11 (192.168.1.11)' can't be established.
RSA key fingerprint is ed:f5:8b:3a:70:4e:d6:7d:16:59:aa:52:14:34:5d:1f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.11' (RSA) to the list of known hosts.
root@192.168.1.11's password:
httpd.crt 100% 3066 3.0KB/s 00:00
httpd.csr 100% 647 0.6KB/s 00:00
httpd.key 100% 891 0.9KB/s 00:00
[root@192.168.1.12 /etc/httpd/conf/ssl]# scp -rp /etc/httpd/conf.d/ssl.conf 192.168.1.11:/etc/httpd/conf.d/
root@192.168.1.11's password:
ssl.conf 100% 9568 9.3KB/s 00:00
[root@192.168.1.12 /etc/httpd/conf/ssl]#
[root@192.168.1.12 /etc/httpd/conf/ssl]# netstat -tunlp |grep [80,443]
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1344/sshd
tcp 0 0 :::80 :::* LISTEN 3300/httpd
tcp 0 0 :::22 :::* LISTEN 1344/sshd
tcp 0 0 :::443 :::* LISTEN 3300/httpd
参考文献：
常见ipvs的持久链接，方式为防火墙标记：
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 10.40.0.51 -p tcp --dport 80 -j MARK --set-mark 9
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 10.40.0.51 -p tcp --dport 443 -j MARK --set-mark 9
[root@localhost ~]# iptables -L -t mangle
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- anywhere 10.40.0.51 tcp dpt:http MARK set 0x9
MARK tcp -- anywhere 10.40.0.51 tcp dpt:http MARK set 0x9
MARK tcp -- anywhere 10.40.0.51 tcp dpt:https MARK set 0x9
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
[root@localhost ~]#
[root@localhost ~]# iptables -t mangle -D PREROUTING 2
[root@localhost ~]#
[root@localhost ~]# ipvsadm -A -f 9 -s rr -p
[root@localhost ~]# ipvsadm -a -f 9 -r 192.168.1.11 -g
[root@localhost ~]# ipvsadm -a -f 9 -r 192.168.1.12 -g
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 9 rr persistent 360
-> 192.168.1.11:0 Route 1 0 0
-> 192.168.1.12:0 Route 1 0 0
[root@localhost ~]# ipvsadm -Lc
IPVS connection entries
pro expire state source virtual destination
TCP 00:58 SYN_RECV 10.40.0.208:call-logging 10.40.0.51:http 192.168.1.12:http
TCP 00:17 SYN_RECV 10.40.0.208:ms-v-worlds 10.40.0.51:http 192.168.1.12:http
TCP 00:18 SYN_RECV 10.40.0.208:ncr_ccl 10.40.0.51:http 192.168.1.12:http
TCP 00:58 SYN_RECV 10.40.0.208:vytalvaultpipe 10.40.0.51:http 192.168.1.12:http
TCP 00:54 SYN_RECV 10.40.0.208:pclemultimedia 10.40.0.51:http 192.168.1.12:http
IP 05:51 NONE 10.40.0.208:0 0.0.0.9:0 192.168.1.12:0
TCP 00:17 SYN_RECV 10.40.0.208:ema-sent-lm 10.40.0.51:http 192.168.1.12:http
[root@localhost ~]#
LVS在director服务器的防火墙工作流程
NAT :
进入：PREROUTING-OUTPUT-POSTROUTING
返回：PREROUTING-FORWARD-POSTROUTING