Elasticsearch 高可用集群搭建
Elasticsearch集群
Elasticsearch 集群是一组连接在一起的一个或多个 Elasticsearch 节点实例。Elasticsearch 集群的强大之处在于跨集群中的所有节点分配任务、搜索和索引。
Elasticsearch 集群中的节点可以分配不同的工作或职责：
- 数据节点——存储数据并执行与数据相关的操作，例如搜索和聚合
- 主节点——负责集群范围的管理和配置操作，例如添加和删除节点
- 客户端节点——将集群请求转发到主节点，将数据相关请求转发到数据节点
- 摄取节点——用于在索引之前预处理文档
默认情况下，每个节点都会自动分配一个唯一标识符或名称，用于管理目的，并且在多节点或集群环境中变得更加重要。
Elasticsearch 集群部署
docker部署Elasticsearch集群
准备三台服务器，信息如下：
节点1: 192.168.1.101
节点2: 192.168.1.102
节点3: 192.168.1.103
节点1运行如下命令，启动elasticsearch服务：
docker run -d --name elasticsearch-1 --hostname elasticsearch-1 -p 9200:9200 --net iot \
-e node.name=elasticsearch-1 \
-e cluster.name=elasticsearch-1 \
-e discovery.seed_hosts=elasticsearch-2,elasticsearch-3 \
-e cluster.initial_master_nodes=elasticsearch-1 \
-e bootstrap.memory_lock=true \
-e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
--add-host elasticsearch-2:192.168.1.102 \
--add-host elasticsearch-3:192.168.1.103 \
elasticsearch:7.16.1
节点2运行如下命令，启动elasticsearch服务：
docker run -d --name elasticsearch-2 --hostname elasticsearch-2 -p 9200:9200 --net iot \
-e node.name=elasticsearch-2 \
-e cluster.name=elasticsearch-2 \
-e discovery.seed_hosts=elasticsearch-1,elasticsearch-3 \
-e cluster.initial_master_nodes=elasticsearch-1 \
-e bootstrap.memory_lock=true \
-e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
--add-host elasticsearch-1:192.168.1.101 \
--add-host elasticsearch-3:192.168.1.103 \
elasticsearch:7.16.1
节点3运行如下命令，启动elasticsearch服务：
docker run -d --name elasticsearch-3 --hostname elasticsearch-3 -p 9200:9200 --net iot \
-e node.name=elasticsearch-3 \
-e cluster.name=elasticsearch-3 \
-e discovery.seed_hosts=elasticsearch-1,elasticsearch-2 \
-e cluster.initial_master_nodes=elasticsearch-1 \
-e bootstrap.memory_lock=true \
-e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
--add-host elasticsearch-1:192.168.1.101 \
--add-host elasticsearch-2:192.168.1.102 \
elasticsearch:7.16.1
基于单机Elasticsearch集群部署
1.环境准备
2.1.安装jdk环境
1.检查jdk是否安装：java -version
2.如果没安装，请安装JDK1.8及以上版本
2.2.部署docker容器
1.检查docker是否安装：docker -v
2.如果没安装，请安装docker容器
2.3.安装docker-compose编排工具
1.检查docker-compose是否安装：docker-compose -v
2.如果没安装，请安装docker-compose工具
2.3.获取elasticsearch镜像
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.2
2.4 设置参数vm.max_map_count
max_map_count设置为262144是es启动最低要求的堆大小。
1.输入命令： vim /etc/sysctl.conf
2.添加参数 : vm.max_map_count=262144
3.查询是否添加完成：grep vm.max_map_count /etc/sysctl.conf
4.生效：/sbin/sysctl -p
2.创建docker network
docker network create elastic
3. 创建docker-compose.yml
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es02,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es03
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data02:/usr/share/elasticsearch/data
networks:
- elastic
es03:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
container_name: es03
environment:
- node.name=es03
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02,es03
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data03:/usr/share/elasticsearch/data
networks:
- elastic
volumes:
data01:
driver: local
data02:
driver: local
data03:
driver: local
networks:
elastic:
external: true
4.Elasticsearch集群部署测试
单机部署完成测试：
集群部署完成测试：
基于多机Elasticsearch集群部署
1.集群机器准备
准备三台机器、其中一个主节点和两个子节点。
IP：192.168.146.128 主节点：es01
IP：192.168.146.129 子节点：es02
IP：192.168.146.130 子节点：es02
服务器配置：均为单个四核处理器、1G运行内存、20G磁盘内存
2.各机器环境准备
2.1.安装jdk环境
1.检查jdk是否安装：java -version
2.如果没安装，请安装JDK1.8及以上版本
2.2.部署docker容器
1.检查docker是否安装：docker -v
2.如果没安装，请安装docker容器
2.3.安装docker-compose编排工具
1.检查docker-compose是否安装：docker-compose -v
2.如果没安装，请安装docker-compose工具
2.3.获取elasticsearch镜像
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.2
3.Elasticsearch集群部署
当各机器环境准备完成后，通过docker-compose依次部署到各集群机器中。
3.1 elasticsearch部署准备
// 每台机器增加目录配置文件
1.创建es配置文件路径： mkdir -p /home/soft/elasticsearch/config
2.创建es数据文件路径： mkdir -p /home/soft/elasticsearch/data
3.创建es日志文件路径： mkdir -p /home/soft/elasticsearch/logs
4.创建es插件文件路径： mkdir -p /home/soft/elasticsearch/plugins/ik
// 下载镜像
1. docker拉去es镜像： docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.2
// 下载分词器
1. 切换存储路径： cd /home/soft/elasticsearch/plugins/ik
2. 下载分词器： wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.5.2/elasticsearch-analysis-ik-7.5.2.zip
3. 解压： unzip elasticsearch-analysis-ik-7.5.2.zip
4. 删除 rm -rf elasticsearch-analysis-ik-7.5.2.zip
//vm.max_map_count参数配置
1.输入命令： vim /etc/sysctl.conf
2.添加参数 : vm.max_map_count=262144
3.查询是否添加完成：grep vm.max_map_count /etc/sysctl.conf
4.生效：/sbin/sysctl -p
3.2 创建elasticsearch.yml文件
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
3.3 主节点配置,节点名称为es01
docker-compose.yml
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
container_name: es01
restart: always
environment:
- node.name=es01
- network.host=0.0.0.0
- cluster.name=es-docker-cluster
- network.publish_host=192.168.146.128
- discovery.seed_hosts=192.168.146.128,192.168.146.129,192.168.146.130
- cluster.initial_master_nodes=192.168.146.128
- bootstrap.memory_lock=true
- TAKE_FILE_OWNERSHIP=true
- "ES_JAVA_OPTS=-Xms128m -Xmx128m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- /home/soft/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /home/soft/elasticsearch/data:/usr/share/elasticsearch/data
- /home/soft/elasticsearch/plugins:/usr/share/elasticsearch/plugins
- /home/soft/elasticsearch/logs:/usr/share/elasticsearch/logs
ports:
- 9200:9200
- 9300:9300
3.4 子节点配置,节点名称为es02
创建docker-compose.yml文件
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
container_name: es01
restart: always
environment:
- node.name=es02
- network.host=0.0.0.0
- cluster.name=es-docker-cluster
- network.publish_host=192.168.146.130
- discovery.seed_hosts=192.168.146.128,192.168.146.129,192.168.146.130
- cluster.initial_master_nodes=192.168.146.128
- bootstrap.memory_lock=true
- TAKE_FILE_OWNERSHIP=true
- "ES_JAVA_OPTS=-Xms128m -Xmx128m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- /home/soft/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /home/soft/elasticsearch/data:/usr/share/elasticsearch/data
- /home/soft/elasticsearch/plugins:/usr/share/elasticsearch/plugins
- /home/soft/elasticsearch/logs:/usr/share/elasticsearch/logs
ports:
- 9200:9200
- 9300:9300
3.5 子节点配置,节点名称为es03
创建docker-compose.yml文件
version: '2.2'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
container_name: es01
restart: always
environment:
- node.name=es03
- network.host=0.0.0.0
- cluster.name=es-docker-cluster
- network.publish_host=192.168.146.129
- discovery.seed_hosts=192.168.146.128,192.168.146.129,192.168.146.130
- cluster.initial_master_nodes=192.168.146.128
- bootstrap.memory_lock=true
- TAKE_FILE_OWNERSHIP=true
- "ES_JAVA_OPTS=-Xms128m -Xmx128m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
volumes:
- /home/soft/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- /home/soft/elasticsearch/data:/usr/share/elasticsearch/data
- /home/soft/elasticsearch/plugins:/usr/share/elasticsearch/plugins
- /home/soft/elasticsearch/logs:/usr/share/elasticsearch/logs
ports:
- 9200:9200
- 9300:9300
4.Elasticsearch集群部署测试
单机部署完成测试：
集群部署完成测试：
5. Elasticsearch 集群 API
- 查询集群信息
curl -XGET 'localhost:9200/_cluster/health?pretty'
- 查询集群状态
curl -XGET 'localhost:9200/_cluster/state?pretty'
- 集群统计
curl -XGET 'localhost:9200/_cluster/stats?human&pretty'
-
节点统计
-
所有节点
curl -XGET 'localhost:9200/_nodes/stats?pretty' -
特定节点
curl -XGET 'localhost:9200/_nodes/node-1/stats?pretty' -
仅索引统计信息
curl -XGET 'localhost:9200/_nodes/stats/indices?pretty' -
结构获取任何单个节点的任何特定指标
curl -XGET 'localhost:9200/_nodes/stats/ingest?pretty' -
结构获取任何多个节点的任何特定指标
curl -XGET 'localhost:9200/_nodes/stats/ingest,fs?pretty'
-
-
节点信息
- 检索单个节点
curl -XGET 'localhost:9200/_nodes/?pretty' - 检索多个节点
curl -XGET 'localhost:9200/_nodes/node1,node2?pretty' - 检索插件
curl -XGET 'localhost:9200/_nodes/plugins curl -XGET 'localhost:9200/_nodes/ingest
- 检索单个节点
-
任务管理
- 待处理的集群任务
curl -XGET 'localhost:9200/_cluster/pending_tasks?pretty'- 正在执行的集群任务
curl -XGET "localhost:9200/_tasks -
远程集群信息
curl -XGET 'localhost:9200/_remote/info?pretty'
- 投票配置排除
curl -X DELETE 'localhost:9200/_cluster/voting_config_exclusions?pretty'
