Adding a New Node to an Elasticsearch 8.17.10 Cluster
This document explains how to add a new node to a running ApinizerEsCluster, assuming Elasticsearch has already been installed on the new servers according to the installation guide.
There are two distinct reasons for adding a node. Do not confuse them:
1. Capacity (horizontal scaling): The disk is full and more storage is needed. The solution is to add a data node (node.roles: ["data"]). No master node is required. Whenever the disk fills up, you scale out by adding another data node.
2. High Availability (fault tolerance): A cluster always has one active master node that manages it. If this node fails and no replacement can be elected, the entire cluster stops. H.A. means that even if the master fails, the remaining nodes automatically elect a new master and keep the cluster running. This requires the number of master-eligible nodes to be odd and at least 3. The solution is to add a master node. (A master node holds no data and does not relieve disk pressure.)
Accordingly, there are two typical situations:
- The organization is not yet H.A. and wants to become H.A.: increase the number of masters to 3 (add master nodes). For a small or medium cluster, the most practical approach is 3 nodes, all with
["master","data"]. - H.A. already exists (3 masters) and only storage is needed: add data nodes only. Do not increase the number of masters; 3 is sufficient and the count must remain odd.
1) How Many Nodes Are Required for High Availability?
Elasticsearch has two separate forms of high availability, and both must be satisfied.
a) Master H.A. (keeping cluster management alive)
Master election works by majority (quorum): quorum = (number of master-eligible nodes / 2) + 1.
| Topology | Master-eligible | H.A. |
|---|---|---|
| 1 node | 1 | ❌ Single point of failure |
| 1 master + 2 data | 1 | ❌ Master is still a single point of failure |
| 2 masters | 2 | ❌ Cluster stops if one fails (split-brain risk) |
| 3 nodes (all master+data) | 3 | ✅ Tolerates the loss of one node — minimum H.A. |
| 3 masters + N data | 3 | ✅ For larger clusters |
Rule: the number of master-eligible nodes must be odd and at least 3. A "1 master + 2 data" topology does not provide H.A. on the master side.
b) Data H.A. (no data loss if a data node fails)
This requires replica ≥ 1 on your indices and at least 2 data nodes, so that a copy of each shard is held on a different node.
- Small/medium cluster: 3 nodes, all with
["master","data"]. This provides both the 3-master quorum and data redundancy in one step. This is the most practical H.A. setup. - Large cluster: 3 dedicated masters
["master"]plus as many data nodes["data"]as needed. This prevents heavy query and indexing load from interfering with master election.
2) Different VLAN, or a Load Balancer / NAT in Between
- All master nodes must reside at the primary site, on the same VLAN (a low-latency network). Master election is highly sensitive to latency. A node on a different VLAN, or behind a load balancer/NAT, must not be given the master role; such nodes must always be
node.roles: ["data"]. - A load balancer is for HTTP (9200) client traffic only. Inter-node transport (9300) traffic is never placed behind a load balancer; it goes directly over the nodes' real IP addresses. For this reason,
discovery.seed_hostsmust contain the master nodes' real IPs, not a VIP/LB address. - On a different VLAN, the firewall must allow port 9300 bidirectionally; if clients are on a different network, port 9200 must also be open.
- If behind NAT, the node must advertise an externally reachable address via
transport.publish_host/transport.publish_port. Otherwise, even after joining, other nodes cannot connect to it.
Summary: Nodes on the same VLAN at the primary site may be ["master","data"] (or separated); nodes on a different VLAN, or behind an LB/NAT, must be ["data"] only.
Adding a node is a join operation, not a fresh installation. For this reason, three steps from the installation guide are skipped on the new node:
- Certificates are not generated; they are copied from an existing node (
certutilis not run). elasticsearch-setup-passwordsis not run (built-in users already exist in the cluster).cluster.initial_master_nodesanddiscovery.type: single-nodeare not set on the new node.
Ports 9200 (HTTP) and 9300 (transport) must be open bidirectionally between the new nodes and the master nodes.
3) Copying the Certificates
The elasticsearch-certutil commands from the installation guide are not run. All nodes in the cluster share the same certificate. The certificate files from an existing node are copied to the new node, to the same path.
# 1) Create the certs directory on the new node
mkdir -p /opt/elasticsearch/elasticsearch-8.17.10/config/certs/
# 2) Run on the existing (primary) node; copies the certificates to the new node
scp /opt/elasticsearch/elasticsearch-8.17.10/config/certs/elastic-certificates.p12 \
elasticsearch@<NEW_NODE_IP>:/opt/elasticsearch/elasticsearch-8.17.10/config/certs/
scp /opt/elasticsearch/elasticsearch-8.17.10/config/certs/elastic-stack-ca.p12 \
elasticsearch@<NEW_NODE_IP>:/opt/elasticsearch/elasticsearch-8.17.10/config/certs/
scp /opt/elasticsearch/elasticsearch-8.17.10/config/certs/elastic-certificates.crt \
elasticsearch@<NEW_NODE_IP>:/opt/elasticsearch/elasticsearch-8.17.10/config/certs/
# 3) Fix permissions on the new node
sudo chown -Rf elasticsearch:elasticsearch /opt/elasticsearch/elasticsearch-8.17.10/config/certs
sudo chmod -Rf 775 /opt/elasticsearch/elasticsearch-8.17.10/config/certs
4) Configuring elasticsearch.yml on the New Node
The following configuration is applied on the new node only.
sudo vi /opt/elasticsearch/elasticsearch-8.17.10/config/elasticsearch.yml
cluster.name: ApinizerEsCluster
node.name: "<NEW_NODE_IP>"
network.host: "0.0.0.0"
http.port: 9200
# For a primary-site master: node.roles: ["master","data"] (or ["master"])
# For a data / remote node: node.roles: ["data"]
node.roles: ["data"]
# List the IP addresses of the master nodes in the cluster
discovery.seed_hosts: ["<MASTER_NODE_IP>:9300"]
# Do NOT set cluster.initial_master_nodes or discovery.type: single-node
path.data: /data/elastic-data/
path.repo: ["/data/elastic-snapdata"]
# Security
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# HTTP SSL
xpack.security.http.ssl:
enabled: true
keystore.path: certs/elastic-certificates.p12
truststore.path: certs/elastic-certificates.p12
# Transport SSL
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/elastic-certificates.p12
truststore.path: certs/elastic-certificates.p12
# CORS settings
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-methods: OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers: X-Requested-With, X-Auth-Token, Content-Type, Content-Length
This list contains the addresses the new node contacts first in order to find the cluster. Because discovery proceeds through master-eligible nodes, you list the IP addresses of the master nodes here, not those of data nodes.
If the cluster has more than one master, list all of them. If only a single master is listed and that node is down, the new node cannot join the cluster.
Specifying the port is not mandatory (the default transport port is 9300), but writing it as "<IP>:9300" is clearer.
When network.host is set to the node's own IP address, Elasticsearch attempts to listen only on that address. If the address is not defined on the server, or if traffic arrives on a different interface, the service fails to start with BindException: Cannot assign requested address.
Setting 0.0.0.0 makes Elasticsearch listen on all interfaces, avoiding this problem. If you want to pin the address that nodes advertise to each other (the publish address):
network.bind_host: 0.0.0.0
network.publish_host: "<NEW_NODE_IP>"
You can verify this in the startup logs via the line publish_address {<IP>:9300}, bound_addresses {0.0.0.0:9300}.
In jvm.options, the heap value may be set up to half of the server's RAM and must not exceed 32 GB.
sudo vi /opt/elasticsearch/elasticsearch-8.17.10/config/jvm.options
-Xms16g
-Xmx16g
5) No Changes Required on the Master Node
Discovery is one-directional: the new node reaches the master via discovery.seed_hosts and sends a join request. The master does not need to know about the new node in advance.
Therefore, when adding a new data node:
- Do not add the new data node to the master's
discovery.seed_hostslist (it serves no purpose). - Do not restart the master node.
The master's seed_hosts list should contain master nodes only.
The rule above applies only when adding data nodes. When a new master-eligible node is added to the cluster, the masters must be able to discover one another. For this reason, the discovery.seed_hosts list of every master node must contain the IP addresses of all masters (including the new one).
For example, in a 3-master setup, each master's seed_hosts list would be:
discovery.seed_hosts: ["<MASTER_1_IP>:9300", "<MASTER_2_IP>:9300", "<MASTER_3_IP>:9300"]
For this change to take effect, the affected master nodes are restarted one at a time (each fully rejoining the cluster before the next is restarted). Data nodes are not added to this list.
This setting only takes effect when a cluster is bootstrapped for the very first time. Once the cluster has formed, Elasticsearch ignores this line and reads the voting configuration from the cluster state on disk. Leaving it in the configuration therefore causes no harm to a running cluster.
Even so, removing it is recommended. The reason: if that node's path.data directory is ever deleted, or the node is rebuilt from scratch, and this line is still present, the node will bootstrap a new, separate cluster instead of joining the existing one.
No restart is required to remove it; the change takes effect on the next startup.
# cluster.initial_master_nodes: ["<MASTER_NODE_IP>"] <-- remove this line
6) Starting the New Node
sudo systemctl daemon-reload
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
sudo systemctl status elasticsearch
elasticsearch-setup-passwords is not run. The built-in users (elastic, kibana, etc.) already exist in the cluster; running this command on the new node would overwrite the existing passwords.
7) Verification
After starting the new node, check from the master without restarting any node:
curl -k -u elastic "https://<MASTER_NODE_IP>:9200/_cat/nodes?v"
Example output (1 master+data, 2 data nodes):
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
<DATA_NODE_2_IP> 12 63 1 0.01 0.14 0.22 d - <DATA_NODE_2_IP>
<DATA_NODE_1_IP> 6 98 1 0.06 0.07 0.26 d - <DATA_NODE_1_IP>
<MASTER_NODE_IP> 53 30 2 0.16 0.09 0.07 dm * <MASTER_NODE_IP>
In the node.role column, d means data and m means master-eligible. The * in the master column indicates the currently elected master.
If the new node does not appear in the list, inspect its logs:
sudo journalctl -u elasticsearch -f
If a connection cannot be established, check the discovery.seed_hosts IP address, port 9300, and the certificates.