SingleStore HA(High Availability) 설정 (선택사항)

문서번호 : 11-234059

HA 구성시 주의 사항

•

안정적인 HA 구성을 위해서 MA(Master Aggregator) 노드를 단독 호스트에서 구성하고, CA 노드도 다른 호스트에 구성하는 것을 권장합니다.

•

HA의 동작은 MA가 수행합니다. MA노드가 정상 Online 상태가 아닐 경우 HA는 동작될 수 없습니다. 

만약, MA와 Leaf노드가 하나의 호스트에서 기동되고 있다면, 그 호스트에 장애가 발생할 경우 HA는 동작될 수 없습니다.
이때, CA노드가 있어, 관리자가 CA노드를 수동으로 승격(AGGREGATOR SET AS MASTER; Link) 시키면, 즉시 HA가 동작하게 됩니다.

•

HA 활성화를 위해서는 Leaf 노드가 2 개 이상이어야 하며, 총 개수가 짝수 이어야 합니다. (단, 활성화 이후 노드 변경시는 무관)

HA 활성화 절차

•

더 많은 정보는 SingleStore 의 Managing High Availability 를 참고하시기 바랍니다.

sdb-admin enable-high-availability 명령을 통해 HA 기능을 활성화

$ sdb-admin enable-high-availability --yes
Toolbox will perform the following actions to upgrade the cluster to redundancy 2
  · Configure the cluster to enable redundancy

  · Remove leaf nodes that need to be re-assigned to availability group 2
    - remove leaf node 10.41.1.176:3307
    - remove leaf node 10.41.8.45:3308

  · Add back existing leaf nodes and reassign them to availability group 2
    - add leaf node 10.41.1.176:3307 into availability group 2
    - add leaf node 10.41.8.45:3308 into availability group 2

  · Restore redundancy on each database
    - Restore redundancy on database "test"

Would you like to continue? [Y/n]:
Automatically selected yes, non-interactive mode enabled

✓ Set cluster redundancy level to 2
✓ Removed leaf 10.41.1.176:3307 from cluster
✓ Removed leaf 10.41.8.45:3308 from cluster
✓ Re-added leaf 10.41.1.176:3307 to cluster
✓ Re-added leaf 10.41.8.45:3308 to cluster
✓ Executed `RESTORE REDUNDANCY ON "test"`
Operation completed successfully
Bash
복사

클러스터의 파티션 균형 재조정(rebalance; 필요시 MA 에서 실행)

새 Leaf 노드 추가 후 HA 를 활성화한 경우라면, 마스터 파티션을 균형 있게 재조정 하십시오.

-- 모든 database rebalance
singlestore> REBALANCE ALL DATABASES;

-- 특정 database만 rebalance
singlestore> REBALANCE PARTITIONS ON <database_name>;

## shell 명령어 예시
$ singlestore -s -uroot -p -e "REBALANCE ALL DATABASES"
Bash
복사

HA 상태 확인

•

방법 1) 노드 목록에서 가용그룹(Availability Group)이 1, 2 로 나눠졌는지 확인합니다.

$ sdb-admin list-nodes
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| MemSQL ID  |    Role    |    Host     | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| CA4A5F8A37 | Master     | 10.41.8.45  | 3306 | Running       | True         | 7.8.2   | Online         |                    | 0.0.0.0      |
| A557CFC555 | Aggregator | 10.41.1.176 | 3306 | Running       | True         | 7.8.2   | Online         |                    | 0.0.0.0      |
| 7660E2E74C | Leaf       | 10.41.1.176 | 3307 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
| 0EC9E69070 | Leaf       | 10.41.1.176 | 3308 | Running       | True         | 7.8.2   | Online         | 2                  | 0.0.0.0      |
| 14EFEEE995 | Leaf       | 10.41.8.45  | 3307 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
| A3FB1307DA | Leaf       | 10.41.8.45  | 3308 | Running       | True         | 7.8.2   | Online         | 2                  | 0.0.0.0      |
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
Bash
복사

•

방법 2) 생성된 test 데이터베이스의 파티션 정보가 Master/Slave 로 나눠져 있는지 확인합니다. (Aggregator 에서 실행)

singlestore> show partitions on test;
+---------+-------------+------+--------+--------+
| Ordinal | Host        | Port | Role   | Locked |
+---------+-------------+------+--------+--------+
|       0 | 10.41.8.45  | 3307 | Master |      0 |
|       0 | 10.41.8.45  | 3308 | Slave  |      0 |
… 중략 …
|      31 | 10.41.1.176 | 3308 | Master |      0 |
|      31 | 10.41.1.176 | 3307 | Slave  |      0 |
+---------+-------------+------+--------+--------+
Bash
복사

HA 비 활성화 절차

노드 삭제 (가용그룹 2 전체)

sdb-admin remove-leaf 명령을 통해 같은 가용그룹(Availability Group) 2 에 속해 있는 모든 Leaf 노드를 제거해야 합니다. 가용그룹 1 은 제거할 수 없습니다.

# 가용그룹 2의 LF3
$ sdb-admin remove-leaf --yes --memsql-id 0EC9E69070
Toolbox will perform the following actions on host 10.41.8.45:
  · Run 'memsqlctl remove-leaf --host 10.41.1.176 --port 3308'

Would you like to continue? [Y/n]:
Automatically selected yes, non-interactive mode enabled

✓ Successfully ran 'memsqlctl remove-leaf'
Operation completed successfully

# 가용그룹 2의 LF4
$ sdb-admin remove-leaf --yes --memsql-id A3FB1307DA
Toolbox will perform the following actions on host 10.41.8.45:
  · Run 'memsqlctl remove-leaf --host 10.41.8.45 --port 3308'

Would you like to continue? [Y/n]:
Automatically selected yes, non-interactive mode enabled

✓ Successfully ran 'memsqlctl remove-leaf'
Operation completed successfully
Bash
복사

노드 삭제 확인

$ sdb-admin list-nodes
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| MemSQL ID  |    Role    |    Host     | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| CA4A5F8A37 | Master     | 10.41.8.45  | 3306 | Running       | True         | 7.8.2   | Online         |                    | 0.0.0.0      |
| A557CFC555 | Aggregator | 10.41.1.176 | 3306 | Running       | True         | 7.8.2   | Online         |                    | 0.0.0.0      |
| 7660E2E74C | Leaf       | 10.41.1.176 | 3307 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
| 14EFEEE995 | Leaf       | 10.41.8.45  | 3307 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
| 0EC9E69070 | Unknown    | 10.41.1.176 | 3308 | Running       | True         | 7.8.2   | Recovering     |                    | 0.0.0.0      |
| A3FB1307DA | Unknown    | 10.41.8.45  | 3308 | Running       | True         | 7.8.2   | Recovering     |                    | 0.0.0.0      |
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------
Bash
복사

redundancy_level 값 1 로 설정

$ sdb-admin update-config --key redundancy_level --value 1 --set-global --yes
Toolbox is about to run 'memsqlctl update-config --key redundancy_level --value 1 --set-global' on the following nodes:
    - On host 10.41.8.45:
      + CA4A5F8A37D8FA229A58ED77F6EC94E35895826E 

Would you like to continue? [Y/n]:
Automatically selected yes, non-interactive mode enabled

✓ Updated configuration on host 10.41.8.45
+------------------------------------------+------------------------------------------+---------+
|                 MemsqlID                 |                 Message                  | Result  |
+------------------------------------------+------------------------------------------+---------+
| CA4A5F8A37D8FA229A58ED77F6EC94E35895826E | Updated configuration on host 10.41.8.45 | changed |
+------------------------------------------+------------------------------------------+---------+
Bash
복사

제거된 노드를 다시 가용그룹 1에 추가(선택사항)

# 가용그룹 2의 LF3
$ sdb-admin add-leaf --yes --password <dbpw> --memsql-id 0EC9E69070
Toolbox will perform the following actions on host 10.41.8.45:
  · Run 'memsqlctl add-leaf --host 10.41.1.176 --port 3308 --user root --password ●●●●●●'

Would you like to continue? [Y/n]:
Automatically selected yes, non-interactive mode enabled

✓ Successfully ran 'memsqlctl add-leaf'

# 가용그룹 2의 LF4
$ sdb-admin add-leaf --yes --password <dbpw> --memsql-id A3FB1307DA
Toolbox will perform the following actions on host 10.41.8.45:
  · Run 'memsqlctl add-leaf --host 10.41.8.45 --port 3308 --user root --password ●●●●●●'

Would you like to continue? [Y/n]:
Automatically selected yes, non-interactive mode enabled

✓ Successfully ran 'memsqlctl add-leaf'
Operation completed successfully

## 클러스터 상태 확인
$ sdb-admin list-nodes
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| MemSQL ID  |    Role    |    Host     | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| CA4A5F8A37 | Master     | 10.41.8.45  | 3306 | Running       | True         | 7.8.2   | Online         |                    | 0.0.0.0      |
| A557CFC555 | Aggregator | 10.41.1.176 | 3306 | Running       | True         | 7.8.2   | Online         |                    | 0.0.0.0      |
| 7660E2E74C | Leaf       | 10.41.1.176 | 3307 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
| 0EC9E69070 | Leaf       | 10.41.1.176 | 3308 | Running       | True         | 7.8.2   | Recovering     | 1                  | 0.0.0.0      |
| 14EFEEE995 | Leaf       | 10.41.8.45  | 3307 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
| A3FB1307DA | Leaf       | 10.41.8.45  | 3308 | Running       | True         | 7.8.2   | Recovering     | 1                  | 0.0.0.0      |
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
Bash
복사

클러스터의 파티션 균형 재조정 및 고아 파티션 정리(MA 에서 실행)

singlestore> clear orphan databases;
Query OK, 0 rows affected (2.48 sec)

singlestore> rebalance all databases;
Query OK, 1 row affected (3.92 sec)
Bash
복사

클러스터의 노드 확인

$ sdb-admin list-nodes
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| MemSQL ID  |    Role    |    Host     | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| CA4A5F8A37 | Master     | 10.41.8.45  | 3306 | Running       | True         | 7.8.2   | Online         |                    | 0.0.0.0      |
| A557CFC555 | Aggregator | 10.41.1.176 | 3306 | Running       | True         | 7.8.2   | Online         |                    | 0.0.0.0      |
| 7660E2E74C | Leaf       | 10.41.1.176 | 3307 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
| 0EC9E69070 | Leaf       | 10.41.1.176 | 3308 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
| 14EFEEE995 | Leaf       | 10.41.8.45  | 3307 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
| A3FB1307DA | Leaf       | 10.41.8.45  | 3308 | Running       | True         | 7.8.2   | Online         | 1                  | 0.0.0.0      |
+------------+------------+-------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
Bash
복사

일자	작성자	비고
2022.04.27	min
2022.04.29	min	leaf node 2개 추가
2022.07.27	kkh	주의사항 추가
2022.08.18	kkh	rebalance 내용 변경
2022.09.01	wee	orphan, rebalance 순서 변경

A-platform