Ana içeriğe geç

Elasticsearch Possible Issues and Solutions

Stopping Log Writing Due to Disk Fullness​

Warning

When Elasticsearch is installed with default settings, it warns when the disk it is on reaches 85% fullness and stops log writing when it reaches 90%.

Solution​

Space needs to be freed on the disk or the disk needs to be enlarged. After this operation, the disk is notified that it is ready for writing again with the following command:

curl -X PUT "<ELASTIC_IP>:9200/_all/_settings?pretty" \
  -H 'Content-Type: application/json' \
  -d '{
    "index.blocks.read_only_allow_delete": null
  }'

Disk Watermark Settings​

Since this situation can lead to high amounts of space being unusable on large disks, it is recommended to be set specifically for your servers. These limits can be updated by giving disk size as percentage or numeric.

Setting with Numeric Limit​

curl -X PUT "<ELASTIC_IP>:9200/_cluster/settings" \
  -H 'Content-Type: application/json' \
  -d '{
    "transient": {
      "cluster.routing.allocation.disk.watermark.low": "100gb",
      "cluster.routing.allocation.disk.watermark.high": "80gb",
      "cluster.routing.allocation.disk.watermark.flood_stage": "50gb",
      "cluster.info.update.interval": "1m"
    }
  }'

Setting with Percentage Limit​

curl -X PUT "<ELASTIC_IP>:9200/_cluster/settings" \
  -H 'Content-Type: application/json' \
  -d '{
    "transient": {
      "cluster.routing.allocation.disk.watermark.low": "90%",
      "cluster.routing.allocation.disk.watermark.high": "93%",
      "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
      "cluster.info.update.interval": "1m"
    }
  }'
Note

You should also enter the same values in the configuration file of Elasticsearch that you continue to run instantly afterwards. This is necessary to prevent your settings from being lost in case of a possible restart of the application.

Settings to be added to elasticsearch.yml file:

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 93%
cluster.routing.allocation.disk.watermark.high: 95%

Highlight Error in Kibana​

Error Message​

x of y shards failed: The data you are seeing might be incomplete or wrong. 
The length of [X] field of [Y] doc of [<INDEX_NAME>] index has exceeded [1000000] 
- maximum allowed to be analyzed for highlighting

Reason​

The data size limit that Elasticsearch can perform highlight operation for each record comes as 1,000,000 characters by default. This is the optimal value Elasticsearch has determined for JVM RAM usage and search speed.

Solution​

This setting can be increased with the following command. If you don't know your data size, you can adjust it to a suitable limit by increasing this value gradually.

curl -XPUT "<ELASTIC_IP>:9200/.ds-apinizer-log-apiproxy-AAAA-000*/_settings" \
  -H "Content-Type: application/json" \
  -d '{
    "index": {
      "highlight.max_analyzed_offset": 2000000
    }
  }'

"I/O Reactor Status: STOPPED" Error on API Traffic Screens​

Error Message​

Request cannot be executed; I/O reactor status: STOPPED

Reason​

RAM limits used by Elasticsearch need to be increased.

Solution​

This setting can be increased from the jvm.options file. It is recommended not to exceed half of total RAM amount.

sudo vi /opt/elasticsearch/elasticsearch-7.9.2/config/jvm.options

Add the following lines to the file:

-Xms8g
-Xmx8g

Restart Elasticsearch to apply changes:

systemctl restart elasticsearch

API Traffic Logs Cannot Be Written to Elasticsearch​

Symptom​

Records for recent requests are missing from the API traffic screens and the following lines are repeated in the gateway (worker) application log:

ERROR ConnectorElasticSearchIndex - Elasticsearch traffic log could not be indexed for connection [<CONNECTION_ID>]: ...
WARN  ConnectorElasticSearchIndex - Elasticsearch I/O reactor stopped (...) for connection [<CONNECTION_ID>]. Client evicted ...

Reason​

When the Elasticsearch connection becomes temporarily unresponsive (the "I/O Reactor Status: STOPPED" case above, a cluster restart, a network outage), Apinizer closes the connection and builds it again. Requests that arrive while it is being rebuilt do not find a ready connection.

Behaviour​

  • API calls are not affected. Writing the traffic log neither delays nor blocks the response returned to the client.
  • The record is not lost. If a Failover Connector is defined for the connection, the traffic record that could not be written is written there instead. If no Failover Connector is defined, the record is dropped and a separate warning line is written to the log.
  • Rebuilding is bound to a cool-down. After a failure, the connection is rebuilt at most once every 10 seconds. This prevents the error storm that would otherwise be caused by every single request triggering its own rebuild under load; attempts inside the cool-down window are skipped silently and appear at DEBUG level only. It is therefore normal for the warning line above not to repeat more often than once every 10 seconds.

Verification Steps​

  1. Verify the state of the Elasticsearch cluster; apply the "I/O Reactor Status: STOPPED" and "Stopping Log Writing Due to Disk Fullness" sections on this page.
  2. Make sure a Failover Connector is selected for the Elasticsearch connection definition; otherwise traffic records produced during an outage are not kept.
  3. If the warning lines continue after Elasticsearch has recovered, check the address, credentials and index name in the connection definition.

If the Log Line Shows No Stack Trace​

When the Java virtual machine produces the same error many times, it eventually shortens the stack trace and only the error itself remains in the log line. For diagnostic work that requires the full stack trace, the gateway (worker) application can be started with the -XX:-OmitStackTraceInFastThrow option. This option is for diagnostics only and should be removed once the problem is resolved.

Maximum Shards Limit Error​

Error Message​

Elasticsearch exception [type=validation_exception, reason=Validation Failed: 
1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]

Reason​

_cluster.max_shards_per_node limit has been reached. It is necessary to increase the number of data-holding nodes, reduce the number of shards in the cluster, or increase the shard limit on the system.

Solution 1: Increasing Data Node Count​

Note

The recommended solution for this problem is to increase the number of data-holding nodes.

Solution 2: Increasing Shard Limit​

Since increasing data nodes may not always be possible, manually managing shards is also a usable solution:

curl -XPUT http://<ELASTICSEARCH_IP>:9200/_cluster/settings \
  --header "Content-Type:application/json" \
  -d '{
    "persistent": {
      "cluster.routing.allocation.total_shards_per_node": 2000,
      "cluster.max_shards_per_node": 2000
    }
  }'

Trigger rollover operation:

curl http://<ELASTICSEARCH_IP>:9200/apinizer-log-apiproxy-<INDEX_KEY>/_rollover
Warning

This method is NOT RECOMMENDED as it will cause loss in old logs. It should only be used as a last resort.

curl -XDELETE http://<ELASTICSEARCH_IP>:9200/apinizer-log-apiproxy-<INDEX_KEY>-<INDEX_NUMBER>

Unassigned Shards - CLUSTER-RECOVERED​

Reason​

It may not be able to distribute shards after a possible server restart or file loss.

Solution Steps​

1. Checking Node Status​

It is necessary to ensure that all Elasticsearch nodes are running and there is no file loss.

The config/elasticsearch.yml file is checked on Elasticsearch master node. IPs of other nodes can be found here and it should be ensured that they are also running. You cannot see these nodes with GET /_nodes request while connected nodes are closed.

2. Checking Cluster Status​

Status of nodes, cluster, and shards is checked with the following commands:

# List nodes
curl "<ELASTICSEARCH_IP>:9200/_nodes"

# Get shard allocation explanation
curl "<ELASTICSEARCH_IP>:9200/_cluster/allocation/explain"

# Check shard status
curl "<ELASTICSEARCH_IP>:9200/_cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state"

3. Reactivating Allocation​

Distribution on nodes is reactivated with the following command:

curl -XPUT "<ELASTICSEARCH_IP>:9200/_cluster/settings?pretty" \
  -H 'Content-Type: application/json' \
  -d '{
    "transient": {
      "cluster.routing.allocation.enable": true
    }
  }'

4. Reroute Operation​

When the above command is not sufficient, this operation is forced with the following command:

curl -XPOST "<ELASTICSEARCH_IP>:9200/_cluster/reroute?retry_failed=true&pretty"

NAS Disk Mount Access Error​

Reason​

When trying to connect to NAS disk, file system access permissions need to be set.

Solution​

1. Getting Elasticsearch User ID Information​

id elasticsearch

Example output:

uid=1000(elasticsearch) gid=1000(elasticsearch) groups=1000(elasticsearch),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),101(lxd)

2. Mounting NAS Disk​

Mount operation is performed with the following command by giving appropriate permissions to mount NAS disk:

sudo mount -t cifs \
  -o rw,uid=1000,gid=1000,file_mode=0755,dir_mode=0755,username=elasticsearch,password=1234 \
  //192.168.111.248/LogApinizer \
  /home/data/

3. Permanent Mount Settings​

Add the following lines to the fstab file for the mount operation to be permanent:

sudo vi /etc/fstab

Line to add:

//192.168.111.248/LogApinizer /home/data cifs rw,uid=1000,gid=1000,file_mode=0755,dir_mode=0755,username=elasticsearch,password=1234 0 0
En son tarihinde güncellendi