Monitoring Elasticsearch Nodes for Low Disk space

Scenario: You login to Kibana and notice there’s no new logs within the past 15 minutes, the last time you received a log was 18 hours ago. You then run a query against Elasticsearch and notice your indexes are red. Finally you run a df -h command on Linux and notice your disk space is at 100%, you then blame yourself because you knew you could create an email alert to get alerted on such events.

How do we accomplish this?

We will create an automated Cron job to check and monitor for disk space in Elasticsearch every 59 minutes, and if the disk capacity is over 97%, we will receive an email from our server.

Setup email notification

We will first install mailx which is a fairly easy setup.

Step 1: Install mailx

yum install mailx

Step 2: Edit configuration

vi /etc/mail.rc

set smtp=youremailserver.domain.com:25
set nss-config-dir=/etc/pki/nssdb/

Those are really the only setting you need.

Next, you may send a test email to ensure that this works.

Run the following command:

echo “Hey this is a test email” | mailx -s “Test email” pablo@domain.com

Note: You should get an authenticated email account from your exchange administrator and enter if it’s necessary (This is more secure); otherwise, you’ll need to add a relay on your exchange server.

If you didn’t get the test email, tell your exchange admin to do the following:

Step 1: Launch Exchange Management Console

Step 2: Select Hub Transport

Step 3: Under “Receive connector” find your Anonymous Relay Option and click on it.

Step 4: Navigate to the “Network” tab, and click on “add”
Step 5: Add the IP address of your elasticsearch  and Apply. 

“Disk space” monitoring script

We will be creating a bash script to monitor the disk space in our server to ensure that we don’t run out of space and stop receiving logs for our Elasticsearch server.

I created a folder called scripts under /opt/

Step 1: Create Bash script

vi /opt/scripts/Diskspace.sh

Then paste the following:

#!/bin/bash

usage=`df -h | awk '{print $5}' | head -n 2 | tail -1 | sed 's/[\.%-]//g'`
if [ $usage -ge 97 ]
then
  echo "Disk Space is over 97%, Resolve issue." | mail -v -s "DISK USAGE : Alert - ElasticsearchServerName" Pablo@email.com
else
  echo
  echo "The disk space is normal"
fi

Save script as Diskspace.sh

What does this script do?

The script will run the df  -h to return the current disk space percentage and if it’s greater than 97% capacity it will return the message “Disk Space is over 97%, Resolve issue.” You may customize as you please.

Setting up Cron

Cron is a time-based job scheduler (Think scheduled task in windows), that allows you to scheduled any kind of bash script. We will be installing it and then adding our script to ensure that it runs in an hourly basis.

Step 1: Install Cron

sudo yum install cronie

Step 2: Configure

vi /etc/crontab
59 * * * * root sh /opt/scripts/Diskspace.sh > /opt/scripts/Diskpace_log.txt

the 59 * * * *  means that this script will run every 59 minutes. If you need more info on Cron you can visit this page.

This is it!

Final Notes:

• You should set this up for all of your elasticsearch servers (If you don’t have X-pack)

In the next article we’ll go over Logstash, monitoring our elasticsearch indexes and also ensuring our ports are listening.