Nagios NRDP – Testing The Configuration

The final part of this series to to test the configuration to ensure that the remote Nagios server triggers the send_nrdp global event commands and that the central Nagios server receives them and changes the states accordingly.

For the purposes of this testing I stopped the nagios services on both service so that the checks are performed and I can show the log entries on both servers as well as initial states and so on.

• After starting the nagios service on the central Nagios server the host and services will show as PENDING as shown below:

Central Server – Initial Host and Services state

• After starting the nagios service on the remote Nagios server, the first check performed is for the host. The host changes to UPand below you can see the host becomes UP on the central Nagios server:

• In the remote nagios logs /usr/local/nagios/var/nagios.log we can see that the Global Event handler command is triggered:

[1576510881] GLOBAL HOST EVENT HANDLER: NAG-REMOTE;UP;HARD;1;global_send_nrdp

• In the central nagios logs /usr/local/nagios/var/nagios.log we can see the passive check has been received:

[1576510886] PASSIVE HOST CHECK: NAG-REMOTE;0;(null)

• As the service checks are performed on the remote Nagios server their state is updated on the central Nagios server as shown below:

• In the remote Nagios server logs /usr/local/nagios/var/nagios.log we can see the global event handler commands have been triggered:

[1576510952] SERVICE ALERT: NAG-REMOTE;Current Load;OK;HARD;4;OK - load average: 0.00, 0.01, 0.05
[1576510952] GLOBAL SERVICE EVENT HANDLER: NAG-REMOTE;Current Load;OK;HARD;4;global_service_send_nrdp
[1576510989] SERVICE ALERT: NAG-REMOTE;Root Partition;OK;HARD;4;DISK OK - free space: / 15468 MB (88.92% inode=99%):
[1576510989] GLOBAL SERVICE EVENT HANDLER: NAG-REMOTE;Root Partition;OK;HARD;4;global_service_send_nrdp
[1576511027] SERVICE ALERT: NAG-REMOTE;Current Users;OK;HARD;4;USERS OK - 1 users currently logged in
[1576511027] GLOBAL SERVICE EVENT HANDLER: NAG-REMOTE;Current Users;OK;HARD;4;global_service_send_nrdp
[1576511031] HOST ALERT: test;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.04 ms
[1576511031] GLOBAL HOST EVENT HANDLER: test;UP;HARD;1;global_send_nrdp
[1576511064] SERVICE ALERT: NAG-REMOTE;SSH;OK;HARD;4;SSH OK - OpenSSH_7.4 (protocol 2.0)
[1576511064] GLOBAL SERVICE EVENT HANDLER: NAG-REMOTE;SSH;OK;HARD;4;global_service_send_nrdp
[1576511102] SERVICE ALERT: NAG-REMOTE;HTTP;WARNING;HARD;4;HTTP WARNING: HTTP/1.1 403 Forbidden - 5179 bytes in 0.001 second response time
[1576511102] GLOBAL SERVICE EVENT HANDLER: NAG-REMOTE;HTTP;WARNING;HARD;4;global_service_send_nrdp
[1576511139] SERVICE ALERT: NAG-REMOTE;Swap Usage;OK;HARD;4;SWAP OK - 100% free (2047 MB out of 2047 MB)
[1576511139] GLOBAL SERVICE EVENT HANDLER: NAG-REMOTE;Swap Usage;OK;HARD;4;global_service_send_nrdp
[1576511177] SERVICE ALERT: NAG-REMOTE;Total Processes;OK;HARD;4;PROCS OK: 72 processes with STATE = RSZDT
[1576511177] GLOBAL SERVICE EVENT HANDLER: NAG-REMOTE;Total Processes;OK;HARD;4;global_service_send_nrdp
[1576511181] SERVICE ALERT: NAG-REMOTE;PING;OK;HARD;4;PING OK - Packet loss = 0%, RTA = 0.03 ms
[1576511181] GLOBAL SERVICE EVENT HANDLER: NAG-REMOTE;PING;OK;HARD;4;global_service_send_nrdp

• In the central nagios logs /usr/local/nagios/var/nagios.log we can see the passive checks have been received:

[1576510956] PASSIVE SERVICE CHECK: NAG-REMOTE;Current Load;0;(null)
[1576510996] PASSIVE SERVICE CHECK: NAG-REMOTE;Root Partition;0;(null)
[1576511036] Error: Got host checkresult for 'test', but no such host can be found
[1576511036] PASSIVE SERVICE CHECK: NAG-REMOTE;Current Users;0;(null)
[1576511066] PASSIVE SERVICE CHECK: NAG-REMOTE;SSH;0;(null)
[1576511106] PASSIVE SERVICE CHECK: NAG-REMOTE;HTTP;0;(null)
[1576511106] SERVICE NOTIFICATION: nagiosadmin;NAG-REMOTE;HTTP;WARNING;notify-service-by-email;HTTP WARNING: HTTP/1.1 403 Forbidden - 5179 bytes in 0.001 second response time
[1576511106] SERVICE NOTIFICATION: NickPage;NAG-REMOTE;HTTP;WARNING;notify-service-by-email;HTTP WARNING: HTTP/1.1 403 Forbidden - 5179 bytes in 0.001 second response time
[1576511106] SERVICE ALERT: NAG-REMOTE;HTTP;WARNING;HARD;1;HTTP WARNING: HTTP/1.1 403 Forbidden - 5179 bytes in 0.001 second response time
[1576511146] PASSIVE SERVICE CHECK: NAG-REMOTE;Swap Usage;0;(null)
[1576511186] PASSIVE SERVICE CHECK: NAG-REMOTE;PING;0;(null)
[1576511186] PASSIVE SERVICE CHECK: NAG-REMOTE;Total Processes;0;(null)

Previous Page                                                                                                                                         Next Page

Nagios NRDP – Configuring The send_nrdp Global Event Commands

This following pages of this article explain how to set up the global event commands on the remote Nagios server NAG-REMOTE to use the send_nrdp.py client and send check results to the central Nagios server NAG-CENTRAL.

Remote Server – Configuring a Global Host Event  Command

To configure a Global Host Event Command on the remote Nagios server NAG-REMOTE perform the following steps:

• SSH to NAG-REMOTE as a normal user
• Switch to root in by executing the command below:

su

• Edit /usr/local/nagios/etc/objects/commands.cfg

vi /usr/local/nagios/etc/objects/commands.cfg

• Go to the end of the file and add in a new command as shown below, substituting the {Your Nagios server address here} with the IP Address of your Nagios Server with NRDP installed and configured and substituting {Your nrdp token here} with the token you created when setting up NRDP:

define command {
 command_name global_host_send_nrdp
 command_line /usr/bin/printf "%b" "$HOSTNAME$\t$HOSTSTATEID$\t$HOSTOUTPUT$" | $USER3$/send_nrdp.py -u http://{Your Nagios server address here}/nrdp/ -t {Your nrdp token here}
}

• Save and exit the file
• Edit /usr/local/nagios/etc/nagios.cfg

vi /usr/local/nagios/etc/nagios.cfg

• Search for the line #global_host_event_handler=somecommand and change it to the following:

global_host_event_handler=global_host_send_nrdp

• Save and exit the file
• Restart the nagios service

Remote Server – Configuring a Global Service Event Command

The next piece of configuration on the Remote Server is to configure a Global Service Event Command which will then be used to send check results to the Central Nagios server via send_nrdp.py.

To configure  a Global Host Event Command perform the following steps:

• Edit /usr/local/nagios/etc/objects/commands.cfg

vi /usr/local/nagios/etc/objects/commands.cfg

• Go to the end of the file and add in a new command as shown below, substituting the {Your Nagios server address here} with the IP Address of your Nagios Server with NRDP installed and configured and substituting {Your nrdp token here} with the token you created when setting up NRDP:

define command {
 command_name global_service_send_nrdp
 command_line /usr/bin/printf "%b" "$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATEID$\t$SERVICEOUTPUT$" | $USER3$/send_nrdp.py -u http://{Your Nagios server address here}/nrdp/ -t {Your nrdp token here}
}

• Save and exit the file
• Edit /usr/local/nagios/etc/nagios.cfg

vi /usr/local/nagios/etc/nagios.cfg

• Search for the line #global_service_event_handler=somecommand and change it to the following:

global_service_event_handler=global_service_send_nrdp

• Save and exit the file
• Restart the nagios service

Previous Page                                                                                                                                          Next Page

Nagios NRDP – Configuring Passive Checks

This following pages of this article explain how to set up the passive checks on the central Nagios server NAG-CENTRAL which will receive the host check results for NAG-REMOTE and also a service via send_nrdp.py.

To configure a passive host and check definition perform the following steps:

• Edit the nagios.cfg configuration file /usr/local/nagios/etc/nagios.cfg by executing the command below:

vi /usr/local/nagios/etc/nagios.cfg

• Find the object configuration file for localhost by searching for localhost.cfg
• Add a new line as shown below to add a new object configuration file named passive-hosts.cfg

cfg_file=/usr/local/nagios/etc/objects/passive-hosts.cfg

• Save and exit the file
• Create a new object configuration file named passive-hosts.cfg under /usr/local/nagios/etc/objects

vi /usr/local/nagios/etc/objects/passive-hosts.cfg

• Add the text below to define NAG-REMOTE

define host {
  use            passive-host
  host_name      NAG-REMOTE
  alias          NAG-Remote
  address        10.100.100.200
  hostgroups     passive-hosts
}

• Add the text below to define a passive service check for CPU Usage on NAG-REMOTE

define service {
   use                     passive_service_check
   host_name               NWO-REMOTE
   service_description     Current Load
   check_command           check_dummy!2!"No Passive Results Received"
   contact_groups          admins
   notification_period     24x7
   flap_detection_enabled  0
   event_handler_enabled   0
   notification_interval   1440
   notification_period     24x7
}

• Save and exit the file
• Restart the nagios service by executing the command below:

systemctl restart nagios

• Go to the nagios web page http://NAG-CENTRAL/nagios and logon
• Check under Hosts and there will now be new host named NAG-REMOTE showing a status of UNKNOWN.

Previous Page                                                                                                                                          Next Page

Nagios NRDP – Setting Up The Test Environment

This following pages of this article explain how to set up the test environment used for this series of wiki pages. After installing and testing NRDP the next step is to set up a simple test environment where the remote Nagios server NAG-REMOTE and its services are configured for passive monitoring on the central Nagios server NAG-CENTRAL.

For the purposes of this article the hostname will be changed from localhost to NAG-REMOTE on the remote Nagios server NAG-REMOTE. This is being changed as otherwise the $HOSTNAME$ variable sent as part of the check results by the send_nrdp.py client will be localhost.

The following sections of this article will consist of the following:

• Remote Server – Changing the hostname
• Remote Server – Changing the initial state for hosts and services
• Central Server – Configuring a passive host template
• Central Server – Configuring a passive service template

Remote Server – Changing the hostname

The first step is to rename the Remote Nagios server’s hostname in the configuration file localhost.cfg to ensure that send_nrdp.py sends the correct hostname to the central Nagios server.

To change the hostname perform the following on the remote Nagios server NAG-REMOTE:

• SSH to NAG-REMOTE as a normal user
• Switch to root in by executing the command below:

su

• Edit the localhost.cfg configuration file /usr/local/nagios/etc/objects/localhost.cfg by executing the command below:

vi /usr/local/nagios/etc/objects/localhost.cfg

• Change the host_name value from localhost to NAG-REMOTE as shown below:

define host {
   use        linux-server
   host_name  NAG-REMOTE
   alias      NAG-Remote
   address    127.0.0.1
}

• Change the host_name entries for the linux-servers hostgroup definition and all services by executing the following in vi:

:%s/localhost/NAG-REMOTE/

• Save and exit the file
• Restart the nagios service by executing the command below:

systemctl restart nagios

Remote Server – Changing the initial state for hosts and services

The next step is to change the initial state settings for hosts and services on the remote Nagios server NAG-REMOTE.  Changing these to Unreachable for Hosts and Unknown for Services will ensure that the first check the remote server does triggers the send_nrdp global event commands and updates the central Nagios server.  In the default configuration, the remote server will not send check results for the initial checks to the central server which means they will show as PENDING until a state change occurs on the remote Nagios server.

This could probably be achieved by setting the initial state on the central Nagios server so that they are presumed to be OK but I prefer for them to turn to the correct state when first checked on the remote Nagios server.

To change the initial states for the hosts and services on the remote Nagios server perform the following:

• Edit the templates.cfg configuration file /usr/local/nagios/etc/objects/templates.cfg by executing the command below:

vi /usr/local/nagios/etc/objects/templates.cfg

• Find the generic-host definition and add the line initial_state shown below to set hosts initial state to Unreacheable:

define host {
   name                          generic-host 
   notifications_enabled         1
   event_handler_enabled         1
   flap_detection_enabled        1 
   process_perf_data             1
   retain_status_information     0
   retain_nonstatus_information  1
   notification_period           24x7
   initial_state                 u
   register                      0
}

• Find the generic-service definition and add the initial_state line shown below to set services initial state to Unknown:

define service {
   name                           generic-service
   active_checks_enabled          1 
   passive_checks_enabled         1 
   parallelize_check              1 
   obsess_over_service            1 
   check_freshness                0 
   notifications_enabled          1
   event_handler_enabled          1
   flap_detection_enabled         1
   process_perf_data              1
   retain_status_information      0
   retain_nonstatus_information   1
   is_volatile                    0
   check_period                   24x7
   max_check_attempts             3
   check_interval                 10
   retry_interval                 2
   contact_groups                 admins
   notification_options           w,u,c,r
   notification_interval          60
   notification_period            24x7
   initial_state                  u
   register                       0
}

• Save and exit the file
• Restart the nagios service by executing the command below:

systemctl restart nagios

Central Server – Configuring a passive host template

The next step is to create a passive host template on the central Nagios server NAG-CENTRAL which can then be used to create the host definition for NAG-REMOTE and receive the host check results via NRDP.

To configure a passive host template, perform the following on the central Nagios server NAG-CENTRAL:

• SSH to NAG-CENTRAL as a normal user
• Switch to root in by executing the command below:

su -i

• Edit the template.cfg configuration file /usr/local/nagios/etc/objects/template.cfg by executing the command below:

vi /usr/local/nagios/etc/objects/template.cfg

• Go to the end of the file and add the following text to configure a passive host definition template definition:

define host {
 name                              passive_host
 notifications_enabled             1
 event_handler_enabled             1
 retain_status_information         0
 retain_nonstatus_information      1
 flap_detection_enabled            1
 process_perf_data                 1
 active_checks_enabled             0
 passive_checks_enabled            1
 notification_period               24x7
 check_period                      24x7
 notification_interval             120
 notification_options              d,u,r
 contact_groups                    admins
 max_check_attempts                10
 check_interval                    5
 retry_interval                    1
 register                          0
}

• Save and exit the file

Central Server – Configuring a passive service template

The next step is to create a passive service template on the central Nagios server NAG-CENTRAL which can then be used to create the service definitions for NAG-REMOTE and receive the service check results via NRDP.

To configure a passive service template, perform the following on the central Nagios server NAG-CENTRAL:

• Edit the template.cfg configuration file /usr/local/nagios/etc/objects/template.cfg by executing the command below:

vi /usr/local/nagios/etc/objects/template.cfg

• Go to the end of the file and add the following text to configure a passive service definition template definition:

define service {
 name                      passive_service_check
 register                  0
 max_check_attempts        1
 check_interval            1
 retry_interval            1
 active_checks_enabled     0
 passive_checks_enabled    1
 notifications_enabled     1
 check_freshness           1
 freshness_threshold       93600
 check_period              24x7
}

• Save and exit the file

Previous Page                                                                                                                                          Next Page

Nagios NRDP – Installing and testing NRDP

This page of this series explains how to install and configure NRDP (Nagios Remote Data Protocol) on a central Nagios Core server and a remote Nagios Core server.

Full information can be found on the Nagios support site page NRDP – Installing NRDP From Source.

Central Nagios Server Install

The NRDP install on the Central Nagios server will be performed fully as this will be where the passive checks reside and the Remote Nagios server will send check results for processing.

To install NRDP on to the Central Nagios Server NAG-CENTRAL perform the following steps:

• SSH to NAG-CENTRAL as a normal user
• Switch to root in Raspian by executing the command below:

sudo -i

• Update the software and install php-xml by executing the commands below:

apt-get update

apt-get install -y php-xml

• Download the source and unpack by executing the commands below:

cd /tmp

wget -O nrdp.tar.gz https://github.com/NagiosEnterprises/nrdp/archive/1.5.1.tar.gz

tar xzf nrdp.tar.gz

• Create the directory /usr/local/nrdp which will store the PHP files, copy the files, and change the permissions by executing the commands below:

cd /tmp/nrdp-1.5.1/

mkdir -p /usr/local/nrdp

cp -r clients server LICENSE* CHANGES* /usr/local/nrdp

chown -R nagios:nagios /usr/local/nrdp

•  Edit the file /usr/local/nrdp/server/config.inc.php

vi /usr/local/nrdp/server/config.inc.php

• Locate the section below:

$cfg['authorized_tokens'] = array(
 //"mysecrettoken", // <-- not a good token
 //"90dfs7jwn3", // <-- a better token (don't use this exact one, make your
 own)
);

• Add the token (ThisIsMyTestNRDPToken) is the token used in this test environment) which is to be used to this section of the config file as shown below:

$cfg['authorized_tokens'] = array(
 "ThisIsMyTestNRDPToken",
 );

• Save and exit the file
• Copy the NRDP Apache config file by executing the command below:

cp nrdp.conf /etc/apache2/sites-enabled/

• Restart Apache by executing the command below:

systemctl restart apache2.service

Remote Nagios Server Install

The NRDP install on the remote Nagios server will only be performed up to the copying of the client files as these are all that is needed on the remote Nagios server.

To install NRDP on to the remote Nagios server NAG-REMOTE perform the following steps:

• SSH to NAG-REMOTE as a normal user
• Switch to root in by executing the command below:

su

• Update the software and install php-xml and php-json by executing the commands below:

yum update

yum install -y php-xml

yum install -y php-json

• Download the source and unpack by executing the commands below:

cd /tmp

wget -O nrdp.tar.gz https://github.com/NagiosEnterprises/nrdp/archive/1.5.1.tar.gz

tar xzf nrdp.tar.gz

• Create the directory /usr/local/nrdp which will store the PHP files, copy the files, and change the permissions by executing the commands below:

cd /tmp/nrdp-1.5.1/

mkdir -p /usr/local/nrdp

cp -r clients server LICENSE* CHANGES* /usr/local/nrdp

chown -R nagios:nagios /usr/local/nrdp

Testing NRDP

The final part of this page in the series is to test NRDP is working both locally on the central Nagios server and from the remote Nagios server also.

Central Nagios Server Testing

To test that NRDP is installed and configured correctly perform the following steps:

• Navigate to NRDP page of the central Nagios server NAG-CENTRAL http://nag-central/nrdp
• If it hasn’t opened to the Submit Check Result tab select it
• Enter the token defined in the Central Nagios Server Install section which in this example was ThisIsMyTestNRDPToken.
• Click on the Submit JSON Check Result button at the bottom of the page and it will show OK
• SSH to NAG-CENTRAL as a normal user view the /var/log/messages log file by executing the following command:

tail /var/log/messages

• In the log there should be two entries for the check result tests as shown below:

Dec 5 08:43:32 NAG-CENTRAL nagios: Error: Got host checkresult for '', but no such host can be found
Dec 5 08:43:32 NAG-CENTROL nagios: Error: Got host checkresult for '', but no such host can be found

• These show that NRDP has received the test check results but cannot process them as no hostname was given.

Remote Nagios Server Testing

To test that NRDP is installed and configured correctly to send check results from the remote Nagios server perform the following steps:

• SSH to NAG-REMOTE as a normal user
• Switch to root in by executing the command below:

su

• Go to the /usr/local/nrdp/clients directory and execute the command below substituting NAG-CENTRAL for your central Nagios server address and ThisIsMyTestNRDPToken with the token defined in the Central Nagios Server Install section :

./send_nrdp.py -u http://NAG-CENTRAL/nrdp/ -t ThisIsMyTestNRDPToken --host=another-host --service="PASSIVE-CPU-CHECK" --state=1 --output="Server is DOWN"

• SSH to NAG-CENTRAL as a normal user view the /var/log/messages log file by executing the following command:

tail /var/log/messages

• In the log there should be two entries for the check result tests as shown below:

Dec 5 09:36:02 frontroom nagios: Error: Got host checkresult for 'another-host', but no such host can be found

• This shows that the remote Nagios server has sent a check result using the send_nrdp.py client script and that the central Nagios server has received the check result but cannot process it as there is no host defined named another-host.

Previous Page                                                                                                                            Next Page

Nagios NRDP – Introduction And Prerequisites

Introduction

Nagios is a an open source monitoring solution for network devices and servers. Nagios NRDP (Nagios Remote Data Protocol) is designed to be a flexible data transport mechanism and processor for Nagios, built on top of web technologies which are becoming a more popular transport mechanism. It is designed with a simple and powerful architecture that allows for it to be easily extended and customized to fit individual users’ needs.

This following pages of this article explain how to configure distributed monitoring in Nagios using NRDP to send check results from a separate Nagios Core server to Passive Hosts and Services on a central Nagios Core server. This configuration is to simulate utilising a separate Nagios monitoring server for a separate data centre or customer data centre with the check results being fed back to a central NOC or managed service providers monitoring server.

Test Environment

The test environment for this set of articles consists of the following:

• Central Nagios Core monitoring server
  • General Details
    • Name – NAG-CENTRAL
    • IP – 192.168.0.100
  •  Specification
    • Raspberry Pi Model 2
    • Raspian Buster Lite
  • Nagios Core 4.4.5 installed and basic configuration completed
  • Apache installed and nagios integration completed

• Firewall
  • General Details
    • FIREWALL-REMOTE
    • External IP – 192.168.0.215
    • Internal IP – 10.100.100.215

• Remote Nagios Core monitoring server
  • General Details
    • Name – NAG-REMOTE
    • IP – 10.100.100.200
  •  Specification
    • ESXi VM, 2vCPUS, 2GB RAM, 20GB Disk
    • CentOS 7 64-Bit
  • Nagios Core 4.4.5 installed and basic configuration completed
  • Apache installed and nagios integration completed

The NAG-REMOTE server sits behind the Remote Firewall with no access allowed from the NAG-CENTRAL server.  The FIREWALL-REMOTE is configured to allow outbound communication on TCP 80 from NAG-REMOTE to NAG-CENTRAL.

Installation Overview

The configuration is broken down in to the following steps which are explained in the subsequent sections of this article :-

• Installing and testing NRDP
• Setting up the test environment
• Central Server – Configuring Passive checks
• Remote Server – Configuring a send_nrdp Global Event Commands

Prerequisites

In order to install and configure NRDP for distributed monitoring the following pre-requisites must be met :-

Central Monitoring Server

• Nagios Core 4.4.5 installed and base configuration performed
• Apache installed and base Nagios configuration performed

Remote Monitoring Server

• Nagios Core 4.4.5 installed and base configuration performed
• Apache installed and base Nagios configuration performed

Previous Page                                                                                                                            Next Page

Nagios Distributed Monitoring Using NRDP

• Nagios NRDP – Introduction And Prerequisites
• Nagios NRDP – Installing And Testing NRDP
• Nagios NRDP – Setting Up The Test Environment
• Nagios NRDP – Configuring Passive Checks
• Nagios NRDP – Configuring The send_nrdp Global Event Commands
• Nagios NRDP – Testing The Configuration