한 개의 SBD Disk, 2개 노드로 구성된 Linux Pacemaker Cluster 구성 방법
SBD Disk는 iSCSI 대상 서비스로 구성
호스트명 및 IP 주소 (각 호스트에서 작업해야 하는 거를 색깔로 구분했으니 헷갈리지 않도록 하세요~)
- iSCSI 대상 서버: sbd01 / 192.168.45.131
- Node 1번: hanadb01 / 192.168.45.132
- Node 2번: hanadb02 / 192.168.45.133
※ /etc/hosts 파일에 각 호스트명-IP주소가 등록되어 있어야 함
※ 아래의 모든 과정은 root 계정으로 진행함
1. SBD 서버에 iSCSI 대상 서비스 준비
iSCSI 대상 패키지 설치
sbd01:~ # zypper install targetcli-fb dbus-1-python sbd
iSCSI 대상 서비스 활성화 및 시작
sbd01:~ # systemctl enable targetcli
Created symlink /etc/systemd/system/remote-fs.target.wants/targetcli.service → /usr/lib/systemd/system/targetcli.service.
sbd01:~ # systemctl start targetcli
iSCSI 디바이스를 저장할 디렉터리 생성
sbd01:~ # mkdir /sbd
iSCSI 디바이스 생성
sbd01:~ # targetcli backstores/fileio create sbdhanathd /sbd/sbdhanathd 50M write_back=false
Created fileio sbdhanathd with size 52428800
sbd01:~ # targetcli iscsi/ create iqn.2020-02.hanathd.local:hanathd
Created target iqn.2020-02.hanathd.local:hanathd.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
sbd01:~ # targetcli iscsi/iqn.2020-02.hanathd.local:hanathd/tpg1/luns/ create /backstores/fileio/sbdhanathd
Created LUN 0.
sbd01:~ # targetcli iscsi/iqn.2020-02.hanathd.local:hanathd/tpg1/acls/ create iqn.2020-02.thd-hana-0.local:thd-hana-0
Created Node ACL for iqn.2020-02.thd-hana-0.local:thd-hana-0
Created mapped LUN 0.
sbd01:~ # targetcli iscsi/iqn.2020-02.hanathd.local:hanathd/tpg1/acls/ create iqn.2020-02.thd-hana-1.local:thd-hana-1
Created Node ACL for iqn.2020-02.thd-hana-1.local:thd-hana-1
Created mapped LUN 0.
sbd01:~ # targetcli saveconfig
Configuration saved to /etc/target/saveconfig.json
생성된 iSCSI 디바이스 확인
sbd01:~ # targetcli ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 0]
| o- fileio ................................................................................................. [Storage Objects: 1]
| | o- sbdhanathd ............................................................... [/sbd/sbdhanathd (50.0MiB) write-thru activated]
| | | o- alua ................................................................................................... [ALUA Groups: 1]
| | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
| o- rbd .................................................................................................... [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 2]
| o- iqn.2020-02.hanathd.local:hanathd ................................................................................. [TPGs: 1]
| | o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| | o- acls .......................................................................................................... [ACLs: 2]
| | | o- iqn.2020-02.thd-hana-0.local:thd-hana-0 .............................................................. [Mapped LUNs: 1]
| | | | o- mapped_lun0 ........................................................................... [lun0 fileio/sbdhanathd (rw)]
| | | o- iqn.2020-02.thd-hana-1.local:thd-hana-1 .............................................................. [Mapped LUNs: 1]
| | | o- mapped_lun0 ........................................................................... [lun0 fileio/sbdhanathd (rw)]
| | o- luns .......................................................................................................... [LUNs: 1]
| | | o- lun0 ......................................................... [fileio/sbdhanathd (/sbd/sbdhanathd) (default_tg_pt_gp)]
| | o- portals .................................................................................................... [Portals: 1]
| | o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
o- vhost ............................................................................................................ [Targets: 0]
o- xen-pvscsi ....................................................................................................... [Targets: 0]
iSCSI 서비스 포트 오픈
sbd01:~ # firewall-cmd --permanent --add-port=3260/tcp
success
sbd01:~ # firewall-cmd --reload
success
2. 클러스터 노드 양쪽에 iSCSI 디바이스 연결
클러스터 노드 각각에 sbd 패키지 설치
hanadb01:~ # zypper install sbd
hanadb02:~ # zypper install sbd
클러스터 노드 각각에 iSCSI와 SBD 서비스 활성화
hanadb01:~ # systemctl enable iscsid
hanadb01:~ # systemctl enable iscsi
hanadb01:~ # systemctl enable sbd
hanadb02:~ # systemctl enable iscsid
hanadb02:~ # systemctl enable iscsi
hanadb02:~ # systemctl enable sbd
클러스터 노드 각각에 iSCSI 초기자 이름 변경 (위에서 만들었던 ACL과 일치하도록)
hanadb01:~ # vi /etc/iscsi/initiatorname.iscsi
##
## /etc/iscsi/iscsi.initiatorname
##
## Default iSCSI Initiatorname.
##
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator. The InitiatorName must be unique
## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.
#InitiatorName=iqn.1996-04.de.suse:01:a6d4ca7a1659
InitiatorName=iqn.2020-02.thd-hana-0.local:thd-hana-0
hanadb02:~ # vi /etc/iscsi/initiatorname.iscsi
##
## /etc/iscsi/iscsi.initiatorname
##
## Default iSCSI Initiatorname.
##
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator. The InitiatorName must be unique
## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.
#InitiatorName=iqn.1996-04.de.suse:01:a6d4ca7a1659
InitiatorName=iqn.2020-02.thd-hana-1.local:thd-hana-1
iSCSI 서비스 다시 시작
hanadb01:~ # systemctl restart iscsid
hanadb01:~ # systemctl restart iscsi
hanadb02:~ # systemctl restart iscsid
hanadb02:~ # systemctl restart iscsi
iSCSI 디바이스 연결
hanadb01:~ # iscsiadm -m discovery --type=st --portal=192.168.45.131:3260
192.168.45.131:3260,1 iqn.2020-02.hanathd.local:hanathd
hanadb01:~ # iscsiadm -m node -T iqn.2020-02.hanathd.local:hanathd --login --portal=192.168.45.131:3260
Logging in to [iface: default, target: iqn.2020-02.hanathd.local:hanathd, portal: 192.168.45.131,3260]
Login to [iface: default, target: iqn.2020-02.hanathd.local:hanathd, portal: 192.168.45.131,3260] successful.
hanadb01:~ # iscsiadm -m node -p 192.168.45.131:3260 --op=update --name=node.startup --value=automatic
hanadb02:~ # iscsiadm -m discovery --type=st --portal=192.168.45.131:3260
192.168.45.131:3260,1 iqn.2020-02.hanathd.local:hanathd
hanadb02:~ # iscsiadm -m node -T iqn.2020-02.hanathd.local:hanathd --login --portal=192.168.45.131:3260
Logging in to [iface: default, target: iqn.2020-02.hanathd.local:hanathd, portal: 192.168.45.131,3260]
Login to [iface: default, target: iqn.2020-02.hanathd.local:hanathd, portal: 192.168.45.131,3260] successful.
hanadb02:~ # iscsiadm -m node -p 192.168.45.131:3260 --op=update --name=node.startup --value=automatic
iSCSI 연결 확인 및 디바이스명 확인
hanadb01:~ # lsscsi
[0:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda
[0:0:1:0] disk VMware, VMware Virtual S 1.0 /dev/sdb
[0:0:2:0] disk VMware, VMware Virtual S 1.0 /dev/sdc
[0:0:3:0] disk VMware, VMware Virtual S 1.0 /dev/sdd
[0:0:4:0] disk VMware, VMware Virtual S 1.0 /dev/sde
[0:0:5:0] disk VMware, VMware Virtual S 1.0 /dev/sdf
[2:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0
[3:0:0:0] disk LIO-ORG sbdhanathd 4.0 /dev/sdg
hanadb02:~ # lsscsi
[0:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda
[0:0:1:0] disk VMware, VMware Virtual S 1.0 /dev/sdb
[0:0:2:0] disk VMware, VMware Virtual S 1.0 /dev/sdc
[0:0:3:0] disk VMware, VMware Virtual S 1.0 /dev/sdd
[0:0:4:0] disk VMware, VMware Virtual S 1.0 /dev/sde
[0:0:5:0] disk VMware, VMware Virtual S 1.0 /dev/sdf
[2:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0
[3:0:0:0] disk LIO-ORG sbdhanathd 4.0 /dev/sdg
위에서 확인한 디바이스명으로 디바이스 ID 확인
hanadb01:~ # ls -l /dev/disk/by-id/scsi-* | grep sdg
lrwxrwxrwx 1 root root 9 Feb 20 14:30 /dev/disk/by-id/scsi-1LIO-ORG_sbdhanathd:71c1643e-337e-44c7-80a2-9f9b4e1c9602 -> ../../sdg
lrwxrwxrwx 1 root root 9 Feb 20 14:30 /dev/disk/by-id/scsi-3600140571c1643e337e44c780a29f9b4 -> ../../sdg
lrwxrwxrwx 1 root root 9 Feb 20 14:30 /dev/disk/by-id/scsi-SLIO-ORG_sbdhanathd_71c1643e-337e-44c7-80a2-9f9b4e1c9602 -> ../../sdg
hanadb02:~ # ls -l /dev/disk/by-id/scsi-* | grep sdg
lrwxrwxrwx 1 root root 9 Feb 20 14:31 /dev/disk/by-id/scsi-1LIO-ORG_sbdhanathd:71c1643e-337e-44c7-80a2-9f9b4e1c9602 -> ../../sdg
lrwxrwxrwx 1 root root 9 Feb 20 14:31 /dev/disk/by-id/scsi-3600140571c1643e337e44c780a29f9b4 -> ../../sdg
lrwxrwxrwx 1 root root 9 Feb 20 14:31 /dev/disk/by-id/scsi-SLIO-ORG_sbdhanathd_71c1643e-337e-44c7-80a2-9f9b4e1c9602 -> ../../sdg
3. SBD 디바이스 생성
첫 번째 클러스터 노드에서 SBD 디바이스 생성 (두 번째 클러스터 노드도 상관 없음)
hanadb01:~ # sudo sbd -d /dev/disk/by-id/scsi-3600140571c1643e337e44c780a29f9b4 -1 60 -4 120 create
Initializing device /dev/disk/by-id/scsi-3600140571c1643e337e44c780a29f9b4
Creating version 2.1 header on device 3 (uuid: 991a88bd-5552-4147-9769-aad55cd81c3d)
Initializing 255 slots on device 3
Device /dev/disk/by-id/scsi-3600140571c1643e337e44c780a29f9b4 is initialized.
SBD 설정 파일에 디바이스 ID 입력
hanadb01:~ # vi /etc/sysconfig/sbd
## Type: string
## Default: ""
#
# SBD_DEVICE specifies the devices to use for exchanging sbd messages
# and to monitor. If specifying more than one path, use ";" as
# separator.
#
SBD_DEVICE="/dev/disk/by-id/scsi-3600140571c1643e337e44c780a29f9b4"
<중략>
hanadb02:~ # vi /etc/sysconfig/sbd
## Type: string
## Default: ""
#
# SBD_DEVICE specifies the devices to use for exchanging sbd messages
# and to monitor. If specifying more than one path, use ";" as
# separator.
#
SBD_DEVICE="/dev/disk/by-id/scsi-3600140571c1643e337e44c780a29f9b4"
<중략>
softdog 구성 파일 생성하고 모듈 로드
hanadb01:~ # echo softdog | tee /etc/modules-load.d/softdog.conf
softdog
hanadb01:~ # modprobe -v softdog
insmod /lib/modules/4.12.14-lp151.28.36-default/kernel/drivers/watchdog/softdog.ko
hanadb02:~ # echo softdog | tee /etc/modules-load.d/softdog.conf
softdog
hanadb02:~ # modprobe -v softdog
insmod /lib/modules/4.12.14-lp151.28.36-default/kernel/drivers/watchdog/softdog.ko
4. 클러스터 설치/구성
클러스터 노드 각각에 필요한 패키지 설치
hanadb01:~ # zypper install socat fence-agents ha-cluster-bootstrap
hanadb02:~ # zypper install socat fence-agents ha-cluster-bootstrap
첫 번째 클러스터 노드에서 SSH 키 생성
hanadb01:~ # ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): <엔터>
Enter passphrase (empty for no passphrase): <사용할 패스워드 입력>
Enter same passphrase again: <사용할 패스워드 입력>
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:HSi+/ew4ASGu/tcg7pOFrfJAvNlLcQhC7suM6iiQeqw root@hanadb01
The key's randomart image is:
+---[RSA 2048]----+
| . |
|o . . . |
|.... .... . |
|.o ..o.. . . |
| oo.ooo.S . |
|*.o+oo+o. |
|+*+.o=.o.. |
|= =o=.. o+ |
|E+ ==o .o+ |
+----[SHA256]-----+
첫 번째 클러스터 노드에 생성된 퍼블릭 키 확인
hanadb01:~ # cat /root/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUJtAbc0S8ykp5W6GVCo50waJcKmgO3nVhskYs4n4h2GnIe+OAiuf31trmfvH0NRLnHSuioFE49IVokOFZGLgG07CrQ2PqzRgnTg4gpRyIQLjtHtxl2FSQ8TSADDByHxHZJd8uNMfOnlST3dPtjRHgL28FkRTMUTxha5Z5U4fKTFTim3aHUn2gQg8VNtrfqh7iJ1xOcXuQ1Cq+QBZzDoa6HEzHxLWPH/IKjWQJ8Y9EuQ5wzZ9O7dsu/I5o4jdG2kvsO/eBJx6lMCknC1BYEND+hsbT/Wh4bxlyPVHey6jASeXjaRAkCr7LcO9pwuMTZAp0QyMrc/iCtwgR7Z6JIOOl root@hanadb01
두 번째 클러스터 노드에 SSH 키 생성
hanadb02:~ # ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): <엔터>
Enter passphrase (empty for no passphrase): <사용할 패스워드 입력>
Enter same passphrase again: <사용할 패스워드 입력>
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:tyugxCANxbCBsHjrRMtlzSPyfpM+biYDqCO33+qU6i8 root@hanadb02
The key's randomart image is:
+---[RSA 2048]----+
|*+. |
|++. o |
|+o+ + + |
|.+oB . . |
| o=o. S . |
|.o..o... . . |
|. .o+.+. . |
|+ Eo+++.. . |
|.++*=B+. .. |
+----[SHA256]-----+
두 번째 클러스터 노드에 생성된 퍼블릭 키 확인
hanadb02:~ # cat /root/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUL2IpIyJ1ZSMNEkGeOA9oktia5s7+0K+Fh5CIKpa9fRHhboXVtNbfdiflenNEIuFXsayI9Z1cLsw02lXBiJTKpI6DP+myOeyFYoKvagf+WcwGG5YBYtmaHvGRtjrsE17lpgQxXDm/+YJEYrDwvnAjLSDBx8Mm59mXz4VYj88f/tkdSL0YaJeI84JBs1/FlpOS2RCwRqdvxtp+2fPRRhVQEUN0PnmHYJuO54qHw1TqRT2LECwIjcfZpX8mvJ7xRuSlRGku1KV/9i0Qk2nnYYUT+5ZHLRru8fgBeyYSMCr3geplKJfhEruT22UoFjLxiUTkFiblBnOk9nyK0qTzhdIr root@hanadb02
첫 번째 클러스터 노드에서 생성된 퍼블릭 키를 두 번째 클러스터 노드에 복제
hanadb02:~ # vi /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUJtAbc0S8ykp5W6GVCo50waJcKmgO3nVhskYs4n4h2GnIe+OAiuf31trmfvH0NRLnHSuioFE49IVokOFZGLgG07CrQ2PqzRgnTg4gpRyIQLjtHtxl2FSQ8TSADDByHxHZJd8uNMfOnlST3dPtjRHgL28FkRTMUTxha5Z5U4fKTFTim3aHUn2gQg8VNtrfqh7iJ1xOcXuQ1Cq+QBZzDoa6HEzHxLWPH/IKjWQJ8Y9EuQ5wzZ9O7dsu/I5o4jdG2kvsO/eBJx6lMCknC1BYEND+hsbT/Wh4bxlyPVHey6jASeXjaRAkCr7LcO9pwuMTZAp0QyMrc/iCtwgR7Z6JIOOl root@hanadb01
두 번째 클러스터 노드에서 생성된 퍼블릭 키를 첫 번째 클러스터 노드에 복제
hanadb01:~ # vi /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUL2IpIyJ1ZSMNEkGeOA9oktia5s7+0K+Fh5CIKpa9fRHhboXVtNbfdiflenNEIuFXsayI9Z1cLsw02lXBiJTKpI6DP+myOeyFYoKvagf+WcwGG5YBYtmaHvGRtjrsE17lpgQxXDm/+YJEYrDwvnAjLSDBx8Mm59mXz4VYj88f/tkdSL0YaJeI84JBs1/FlpOS2RCwRqdvxtp+2fPRRhVQEUN0PnmHYJuO54qHw1TqRT2LECwIjcfZpX8mvJ7xRuSlRGku1KV/9i0Qk2nnYYUT+5ZHLRru8fgBeyYSMCr3geplKJfhEruT22UoFjLxiUTkFiblBnOk9nyK0qTzhdIr root@hanadb02
첫 번째 클러스터 노드에 클러스터 설치
hanadb01:~ # ha-cluster-init -u
/root/.ssh/id_rsa already exists - overwrite (y/n)? n
Configuring csync2
Generating csync2 shared key (this may take a while)...done
csync2 checking files...done
Configure Corosync (unicast):
This will configure the cluster messaging layer. You will need
to specify a network address over which to communicate (default
is eth0's network, but you can use the network address of any
active interface).
Address for ring0 [192.168.45.132] <엔터>
Port for ring0 [5405] <엔터>
Configure SBD:
If you have shared storage, for example a SAN or iSCSI target,
you can use it avoid split-brain scenarios by configuring SBD.
This requires a 1 MB partition, accessible to all nodes in the
cluster. The device path must be persistent and consistent
across all nodes in the cluster, so /dev/disk/by-id/* devices
are a good choice. Note that all data on the partition you
specify here will be destroyed.
Do you wish to use SBD (y/n)? y
SBD is already configured to use /dev/disk/by-id/scsi-3600140571c1643e337e44c780a29f9b4 - overwrite (y/n)? n
Hawk cluster interface is now running. To see cluster status, open:
Log in with username 'hacluster', password 'linux'
WARNING: You should change the hacluster password to something more secure!
Waiting for cluster........done
Loading initial cluster configuration
Configure Administration IP Address:
Optionally configure an administration virtual IP
address. The purpose of this IP address is to
provide a single IP that can be used to interact
with the cluster, rather than using the IP address
of any specific cluster node.
Do you wish to configure a virtual IP address (y/n)? y
Virtual IP []192.168.45.134
Configuring virtual IP (192.168.45.134)....done
Done (log saved to /var/log/ha-cluster-bootstrap.log)
두 번째 클러스터에서 조인
hanadb02:~ # ha-cluster-join
Join This Node to Cluster:
You will be asked for the IP address of an existing node, from which
configuration will be copied. If you have not already configured
passwordless ssh between nodes, you will be prompted for the root
password of the existing node.
IP address or hostname of existing node (e.g.: 192.168.1.1) []192.168.45.132
Retrieving SSH keys - This may prompt for root@192.168.45.132:
/root/.ssh/id_rsa already exists - overwrite (y/n)? n
No new SSH keys installed
Configuring csync2...done
Merging known_hosts
Probing for new partitions...done
Address for ring0 [192.168.45.133] <엔터>
Hawk cluster interface is now running. To see cluster status, open:
Log in with username 'hacluster'
Waiting for cluster....done
Reloading cluster configuration...Password:
done
Done (log saved to /var/log/ha-cluster-bootstrap.log)
보안을 위해 각 노드에서 hacluster 계정의 패스워드 변경 (패스워드 초기값은 linux)
hanadb01:~ # passwd hacluster
New password: <사용할 패스워드 입력>
Retype new password: <사용할 패스워드 입력>
passwd: password updated successfully
hanadb02:~ # passwd hacluster
New password: <사용할 패스워드 입력>
Retype new password: <사용할 패스워드 입력>
passwd: password updated successfully
5. 클러스터 구성 확인
참고자료:
https://documentation.suse.com/sle-ha/12-SP4/pdf/book-sleha_color_en.pdf
https://documentation.suse.com/sle-ha/12-SP4/pdf/art-ha-install-quick_color_en.pdf
'Linux' 카테고리의 다른 글
| GRUB? (0) | 2020.03.18 |
|---|---|
| RAID? (0) | 2020.03.18 |
| netstat: command not found 해결 방법 (arp, route, iptunnel, ipmaddr, ifconfig 동일) (0) | 2020.01.19 |
| openSUSE의 릴리즈 정보를 SLES로 바꾸기(속이기) (0) | 2020.01.17 |
| 리눅스 파티션 및 파일시스템 생성 (0) | 2020.01.16 |