Monitoring Cached Websites

2023/10/26 by Leave a comment

Websites that use CDN or caching services like Cloudflare or Amazon Cloudfront can be a little tricky to monitor. You need to make sure all the regular website stuff is working along with additional monitoring for the backend web server and the caching service itself.

NodePing has you covered on both ends: the regular website monitoring and some powerful features specifically for monitoring CDN websites.

The Basics

When someone wants to connect to your website, there are 4 things that have to be working correctly:

  1. DNS
  2. Routing
  3. SSL
  4. Web Server response

DNS

If your DNS servers aren’t available, your visitor’s browser won’t be able to translate the FQDN in the URL to a routable IP address so monitoring each of your nameservers is vital to website availability.

Create a DNS check for each nameserver and be sure the query for your FQDN is being answered.

Routing

Now that the browser has the IP address of a web server, it needs to be able to reach out across the interwebs and request content. Incorrect routing and packet loss can make your website unreachable.

Use our PING and MTR checks to ensure that routing is working and that there is no packet loss.

We often get questions from site owners when the monitoring says their site is failing, but they can get to it fine from their device. Upstream connectivity issues are often the reason, and having the PING or MTR check in place and running from different geographical regions can help identify those troubles quickly.

SSL

Visitors expect websites to use industry best-practices for security, including transport encryption using TLS/SSL. You don’t want them to see that embarrassing “Unsafe” warning when they hit your website because your SSL certs are expired or incorrectly configured.

Create an SSL check to warn you before the cert expires.

Web Server Response

If everything mentioned above is firing on all cylinders, the web server will respond to the visitor’s request and reply with the expected content.

Using NodePing HTTP Content checks, you can verify that the web server is returning the expected HTTP response code and content.

With the basics of website monitoring nailed down, there’s a couple of additional challenges that caching services create that need special monitoring.

Monitor the Back-end Server

If your back-end server isn’t functioning, your CDN or caching service will continue to respond normally, at least for a while. But you’ll want to know right away if that back-end server is offline. You won’t be able to monitor that back-end server using the regular FQDN in the URL because it points to the caching service, not your back-end server.

NodePing can monitor a back-end server using an IP in the URL. The IP address can be either an IPv4 or an IPv6 address.

Example: https://192.168.1.1/index.html

Example: https://%5Bfe80::ec4:7aff:fe06:c186]/index.html

Note: When using IPv6, use square brackets around the IPv6 address.

To get the SSL to respond properly, use an HTTP Advanced check and send a special request header of “Hostname” set to the FQDN of the website.

Example: “Hostname” “example.com”

Cache-busting

To make sure the caching service or CDN is able to communicate properly with your back-end server, you need to send an HTTP request to the service with a URL it hasn’t cached. That will force the service to talk with your back-end server to get fresh content. To do that, it has to be a different URL each time it’s monitored. This is called cache-busting.

NodePing has a cool cache-busting feature on the HTTP Advanced check that will slightly change the URL each time it monitors so that it always causes the caching service to talk with your back-end server.

To use cache-busting, modify the URL query string. Add a non-essential element with the value of “{{now}}”. NodePing will replace that value with a millisecond timestamp each time the check is run.

Example URL: https://nodeping.com/?cachebusting={{now}}

When the URL is run, it will look like: https://nodeping.com/?cachebusting=1697232757035

Each time it is run, the value will be different: https://nodeping.com/?cachebusting=1697232816021

Since each URL is unique each time, there will be no cached entry and the caching service will hit your back-end server on each check run. If the service isn’t able to reach your back-end server, it should return a 522 error or something similar, which will make the NodePing check fail and alert you to the issue.

Website Monitoring

Using a CDN or caching service with your website can speed things up but it can also make things break in ways that basic website monitoring may miss. NodePing’s features allow you to ensure that your back-end web server is up and running and that your caching service is operating as expected.

If you don’t yet have a NodePing account, please avail yourself to our 15-day, free trial. You’ll see why those who know, use NodePing.

Filed under DNS, HTTP, Monitoring, MTR, PING, SSL, website monitoring Tagged with , , ,

Database Monitoring with NodePing

2023/09/26 by Leave a comment

In our recent post ‘Beyond “Is It Up?” – Website Monitoring should be Comprehensive‘ we talked about how website monitoring should be wider than just measuring if a website responds to requests.  If you are responsible for website availability or performance you should of course be monitoring if the website responds in a timely manner, but you also should be monitoring status codes in the response, validating certificates and domain expiration, and DNS.  In today’s post I wanted to extend that discussion to another critical component of your infrastructure: monitoring database accessibility and performance.  Not only are databases a critical part of your web infrastructure, NodePing’s database monitoring can be a key element in your overall infrastructure monitoring that goes well beyond the web.

Monitoring Database Connectivity and Accessibility

As with any server monitoring, the first layer to consider is connectivity and accessibility.  Depending on your network infrastructure, this might mean some combination of PING, MTR, and PORT checks.  These check types can tell you if the database is accessible from places it should be, and make sure it remains inaccessible when it shouldn’t be.  In fact, it is a good idea to monitor your firewalls and routers with these check types.  At its most basic level, the PORT check tells us if your database servers are listening and responding on the expected ports.  The flip side of this is also important.  If your database should not be accessible from the Internet, running a check that monitors that can be just as important to make sure it stays that way. 

In testing connectivity, the PORT check can test just about any service.  For databases specifically, NodePing adds protocol aware monitoring for databases that respond to HTTP requests, as well as check types that understand the specifics of connecting to Redis, MySQL, PostgreSQL, and MongoDb databases.

Querying HTTP based databases with advanced HTTP check types

For databases with native HTTP support, using NodePing’s HTTP based check types can be really useful for monitoring the database performance, as well as monitoring specific responses.  For example, internally we have used our own monitoring to keep track of CouchDB databases for years.  This is useful for answering questions like 

  • Is the database responding in a reasonable time frame?
  • Is the database accessible using SSL/TLS, and is the certificate valid or about to expire?
  • Is the database listening and accessible to non-encrypted ports?  In our use case, this answer needs to be “No,” so NodePing’s capability to monitor that a port is not accessible is important here.
  • Are Couchdb views responding?  We keep some indexes that don’t get constant use warm with monitoring checks.
  • Are the actual values returned within expected ranges?  NodePing’s HTTP Parse check can make an HTTP request and then monitor specific fields within a JSON response to make sure it is within the expected range.  This is particularly useful in monitoring a Couch view that uses reduce.

Running Queries on MySQL, PostgreSQL, and MongoDb

NodePing has specialized check types that can monitor MySQL, PostgreSQL, and MongoDB specifically.  For these databases, NodePing’s monitoring can run a user-defined query and report on the response.  This allows you to run a custom query to track performance metrics, data counts, or set up alerts based on whether the responses are within a defined range.  If the expected response is a string, you can use regex to match the response to ensure it is what is expected.  These responses can be used to trigger notifications, graph values in your database, and store the data for future reference and analytics.

Security considerations for database monitoring

We already mentioned a few of the considerations around the security implications of database monitoring.  The database queries discussed above can use authentication where applicable.  For HTTP connections, as well as MySQL and PostgreSQL, the checks can be used to verify the encryption and certificates.  But a key element in monitoring databases is that in the vast majority of cases the databases should not be universally accessible.  NodePing’s AGENT based checks allow you to run these checks from your own networks, which means you get the full robust monitoring of NodePing’s checks, without opening your database ports to the outside world. In addition, our PUSH check type allows you to script database verification and push the results into NodePing for alerts and reporting.  This suite of check types and options together allow you to do robust monitoring for your databases without compromising your security.  In addition, we recommend that you use authentication to provide the monitoring with the least required access to do the necessary monitoring.

Monitoring Should be Comprehensive

From the outset NodePing has sought to provide website and server monitoring at a price point that allows you to run all the checks you need cost-effectively.  That’s because we believe that monitoring should be comprehensive.  You should be monitoring all of the components of your infrastructure in a way that when you are notified, you know immediately where a sysadmin response is needed.  Database monitoring is a critical component of this.  NodePing’s database monitoring can be a key part of your overall solution to ensure your services are all working as they should, and quickly tell you where the problem is in order to minimize disruptions.

Give it a try!  NodePing has a free, 15-day trial, so there is no risk to trying it out.  We think you’ll be impressed by how easy it is to get comprehensive monitoring in place, including for your databases, and how cost-effective and timesaving it can be as a part of your overall infrastructure management.

Filed under MongoDB, Monitoring, PostgreSQL, Redis, SSL, Using NodePing Tagged with , , , ,

Monitoring the Server Room with NodePing – Part 3: Temperatures

2023/09/19 by Leave a comment

In Part 2, I configured my Raspberry Pi to monitor temperature, humidity, and pressure with the Sense Hat. When I encountered issues with the Sense Hat needing weird tricks to properly measure temperature, I looked for alternative sensors. I found the DHT22/AM2302 sensors on Amazon to try out. I’ve noticed with some of these DHT22 sensors that you need to add an extra resistor along the way to the Pi’s GPIO pins. I like this one in particular because the resistor is integrated already. I grabbed my Pi from my server room and took it back home to configure it.

The DHT22 Sensor

I won’t go much into detail on this sensor, but I found that compared to the other common DHT11 sensor, this one is most ideal because it has better accuracy and can be had for a similar price. So, I went with the DHT22 sensor. My package contained 2 DHT22 sensors, and six different colored cables. I wound up with orange, red, and brown cables. I connected the orange cable to power, red to data, and brown to ground. Below you can see how I connected the sensor to my Pi’s pinout. When looking at Raspberry Pi’s documentation, I connected orange to pin 2 (5V), red to pin 7 (GPIO 4), and brown to 9 (ground). If you intend on connecting the sensor to different GPIO pins, there will be some tweaking of the module needed on your part later.

The Software

I needed to install only a single piece of software to get this to work, in addition to a Python library. The software and Adafruit’s DHT Python library which I installed this way:

python3 -m pip install adafruit-circuitpython-dht
sudo apt-get install libgpiod2

User Setup

As in Part 2, I had to add my user to some groups to get my module working so I could collect temperatures and humidity:

$ usermod -aG video input gpio i2c pi

Creating the PUSH Check

Before I can start submitting metrics to NodePing, I first need to create the check for me to submit results to. The check looks like this:

Below in the Fields section, the names should be:

  1. pidht22.temperature
  2. pidht22.humidity

Adjust the min/max values accordingly to what you consider a safe temperature and humidity. NodePing provides a DHT22 module for PUSH checks to work with the Raspberry Pi called “pidht22”. Since the DHT22 sensor does not have a neat LED array like the Sense Hat I used, it will only submit values to NodePing, and not do any of the cool LED displaying of information. If that is something you want, I suggest following our Part 2 guide with the Sense Hat.

Configuring the Client

Now that the check is created, I moved on to installing the PUSH client software on the Pi. To get the PUSH clients code, I can visit the GitHub page to download a zip file or I can use git to fetch the code directly using the following command:

$ git clone https://github.com/NodePing/PUSH_Clients.git

From here, I made a copy of the Python3 client. A convention I tend to follow is to create a folder named something like “server_room_dht22-202306261809OGK26-QLJGGVM1” where name is the label of the PUSH check in NodePing, and the check ID that is generated when you create that check in the NodePing web interface.

I created those folders using the command:

$ mkdir -p push-clients/dht22PUSH-202306261809OGK26-QLJGGVM1

You can call your folder whatever you want. I use this so I can easily find the right PUSH check in NodePing that corresponds to this PUSH client.

I will be using the pidht22 module to monitor my server room environment.

I copied the Python3 client code into the directory I made and cd’3d into that directory for easier editing:

$ cp -r PUSH_Clients/Python3/NodePingPython3PUSH/* push-clients/server_room_dht22-202306261809OGK26-QLJGGVM1/
$ cd push-clients/server_room_dht22-202306261809OGK26-QLJGGVM1/

To make the PUSH check work, I have to edit 2 files:

  1. config.ini
  2. metrics/pidht22/config.py

At the end of config.ini, there is a modules section. It should contain this information:

[modules]
pidht22 = yes

You also want to include your check ID and checktoken in the server section:

Next is the config.py file for the module. This file has two variables:

  1. UNIT – set to “F” for Fahrenheit or “C” for Celsius
  2. PIN – The GPIO pin to use for data (if you follow this guide and connect to the same pins I did, you don’t have to change this variable)

The client should be configured now. I will now test it by running:

python3 NodePingPythonPUSH.py --showdata

This will let me see the data that would be sent to NodePing without actually sending it. This is just for testing and is a good way to make sure I edited those files correctly without impacting any uptime stats for the check in NodePing.

Lastly, I need to set up cron to run my PUSH client on a regular interval. I want this one to run every minute, so the cron line will look like the one below. To edit my cron tab, I run crontab -e and add the following entry:

* * * * * python3 /home/pi/push-clients/server_room_dht22-202306261809OGK26-QLJGGVM1/NodePingPythonPUSH.py

Now my check is running every minute and submitting my server room climate info to NodePing. This will allow me to track temperatures and humidity in the server room and notify me if the room is too hot/cold, humid/dry.

Ready to Roll

With the new sensor tested and running properly with NodePing’s PUSH checks, I gave it some days to trial run at home before I brought it back to my networking room to monitor temperatures and humidity.

Testing atop the home server

One thing I have noticed with this DHT22 sensor is how steady temperature reporting is. Since I had to calibrate my check for the Sense Hat to account for CPU temperatures, the temperatures were always within +/- 3 degrees of the actual temperature. With this DHT22 sensor, since I have the sensor away from the Pi, the temperature reporting is more steady and makes for cleaner looking results. You can even see from the graph where the humidity dropped when I opened my window, as well as when I had stopped submitting results during testing.

Conclusion

In the end, I found the DHT22 sensor to be a better choice for temperature monitoring . The price is lower and the results are more reliable, and I don’t have to worry about the processor on the Pi messing with the sensor. Additionally, I can use any case I choose and have access to the rest of the GPIO pins if I choose to add more to the Pi.

If you find this to be useful to you and want to try this configuration for yourself you can find all the code needed on the NodePing github. If you don’t yet have a NodePing account, please sign up for our free, 15-day trial.

Filed under Monitoring, PUSH Tagged with , , ,

Probe Server Change [FR]

2023/09/06 by Leave a comment

The following probe server will be changing IP addresses on 2023-09-13

Paris, France (FR) – FR will change from
195.154.167.97 / 2001:bc8:2327:110::10
to
163.172.52.132 / 2001:bc8:2327:101::10

Please adjust your firewalls appropriately if you whitelist so your checks do not fail because of the probe IP address changes.

An always current and updated list of all the IP addresses for our probe servers can be found in the FAQ, a text file, and via DNS query, probes.nodeping.com.

[UPDATE – 2023-09-13 11:11GMT-7] – IP change complete.

Filed under Announcement, IP Change Tagged with ,

Beyond “Is It Up?” – Website Monitoring should be Comprehensive

2023/09/05 by Leave a comment

When it comes to monitoring websites, the question most often asked is, “Is the site up?” While this is certainly an essential aspect, the answer hardly paints the whole picture. True website monitoring involves a plethora of factors that can affect the user experience and performance. These factors can often be hidden, and your user’s experience of you site might not be the same as what you are seeing from your network. With NodePing’s suite of tools, you have the ability to dig deep and understand the vital aspects of your website’s functionality. Let’s explore these considerations.

DNS Monitoring

Domain Name System (DNS) is the backbone of internet navigation, converting human-friendly URLs into IP addresses. Monitoring DNS health is crucial as an unresponsive DNS can render your site unreachable. Problems with DNS can be hidden by caching, and we are often asked why we are notifying about a site that seems to be working from the owner’s perspective. The answer is often that the site owner’s browser or DNS caching is making the site appear to be working when in fact for people who haven’t been on the site recently it appears to be offline because of DNS problems. NodePing offers robust DNS checks to ensure that your DNS servers are resolving correctly.

Monitoring Status Codes

Like with DNS, just checking with a browser can also miss situations in which the web server is actually responding with an error because the modern browsers try to show the page if they can. We often get messages from customers who’s website appears to be working but is actually returning status codes that indicate errors on the site. Even if the site looks right in your browser at the moment, you need to know if it is returning a status code in the 500 range indicating the server is throwing an error. Many content management systems or frameworks also return a visible page with a 404 Not Found status. NodePing’s HTTP checks watch for status code problems with your site.

Similarly, it is important to know if your site is properly following redirects. On some checks, you want the monitoring to follow the redirect to ensure that is getting the final page, and that page is responding with a 200 status code. You may also want to test specific URL’s for the 302 response as well. NodePing’s HTTP Advanced check allows you to ensure that a URL is returning a specific redirect response code.

SSL Certificate Validation

SSL certificates encrypt data transferred between users and your servers. Monitoring and receiving warnings about certificate expirations help you maintain trust and protect sensitive user information. Many of our check types include SSL validation, and and there is also a specialized SSL check that warns you if a certificate has a problem, as well as notifying you that your certificate will expire in a certain number of days. With NodePing, stay ahead with timely reminders and validations.

Domain Registration Expiration and the WHOIS Check

Keep track of your domain registration status with NodePing’s WHOIS checks. Understanding the ownership and registration details ensures that you stay in control of your domain and can prevent unexpected downtime.

Monitoring Other Services on the Host

If your website relies on additional services like databases or caching servers, monitoring them alongside the main site is essential. Integrating these checks into your monitoring strategy ensures that all parts of your site are functioning seamlessly.

CDN & Proxy Consideration

Content Delivery Networks (CDNs) and proxies enhance site performance but can complicate monitoring. By monitoring the back-end site directly, NodePing allows you to quickly pinpoint whether the issue lies with the CDN, helping you react quickly to any problems.

Tying It Together

Every notification your receive from your monitoring system should be actionable. Otherwise it becomes noise, and you either waste time or start ignoring alerts and miss important events. It is important to monitor every aspect of your site, but you don’t necessarily want ten notifications when the site is down. NodePing allows you to set a check as being dependent on another check, so you won’t get a stack of notifications if the dependent checks fail together.

Automated Diagnostics

NodePing’s has both on demand and automated diagnostics tools that provide extra insights when your site is down, supplying valuable information to help troubleshoot and resolve issues more efficiently.

Conclusion

Monitoring a website involves much more than merely checking if it’s up. With tools like NodePing, you can dive into a multitude of factors that contribute to your site’s performance and reliability. By understanding and keeping tabs on DNS, redirects, SSL certificates, domain registration, host services, CDN considerations, and more, you ensure a smooth user experience and robust site functionality.

At NodePing, we’re committed to helping you monitor your website from all angles. Get in touch with us to learn how you can take your website monitoring to the next level. If you don’t have an account yet, give it a try with our 15-day, free trial.

Filed under diagnostics, DNS, HTTP, Using NodePing, WHOIS Tagged with , ,

Monitoring the Server Room with NodePing – Part 2: Sense HAT

2023/08/29 by Leave a comment

In Part 1, I talked about getting started with monitoring my server room with NodePing’s PUSH and AGENT checks and the Raspberry Pi. On my continued path of server room monitoring I wanted to look into environment monitoring with the Raspberry Pi and NodePing’s PUSH checks. To follow up, I sourced a Sense Hat so I could bring a simple way to monitor my climate and get alerts on the metrics with NodePing. The Sense Hat has the capability of monitoring temperature, humidity, and atmospheric pressure. In this post, I will look at how you can integrate the Sense Hat into server room monitoring, configure notifications, and the pros and cons I found when testing the Sense Hat.

The Hardware

To start, the Sense Hat has to be installed. My Sense Hat came with some extra screws and standoffs, as well as a GPIO riser. I’ll use them and a compatible case to secure my Raspberry Pi and Sense Hat. Once the pieces were assembled together, I installed Raspberry Pi OS to a microSD card. The easiest way is to use the Raspberry Pi Imager like in the previous blog post. I put the formatted microSD card in the Pi and connect Ethernet and power.

The Software

There is only one extra dependency that I needed, and it may even be installed on your Raspberry Pi already. To be sure, I installed it with this command:

$ sudo apt install sense-hat

That is all that is needed in addition to what is in the base installation of Raspberry Pi OS.

Creating the PUSH Check

To start using the Sense HAT metrics with NodePing, I signed into NodePing and created a PUSH check. The check looks like this:

Below in the Fields section, the names should be:

  1. pisensehat.temp
  2. pisensehat.humidity
  3. pisensehat.pressure

Adjust the min/max values accordingly to what you consider a safe temperature. NodePing provides a Sense Hat Python 3 module for PUSH checks to monitor temperature, humidity, and atmospheric pressure called “pisensehat”. Not only will it submit the values to NodePing, but it also takes advantage of the Sense HAT’s LED array and will display basic environmental information so you can visually see the temp/humidity/pressure on the display.

User Setup

Out of the box, the first user I made belonged to the right groups to use the Sense Hat. However, I wanted to use a different user to run my PUSH client, so I ran this usermod command to add the pi user to the necessary groups to work with the Sense Hat:

$ usermod -aG video input gpio i2c pi

Configuring the Client

Now that the check is created, I moved on to installing the PUSH client software on the Pi. To get the PUSH clients code, I can visit the GitHub page to download a zip file or I can use git to fetch the code directly using the following command:

$ git clone https://github.com/NodePing/PUSH_Clients.git

From here, I made a copy of the Python3 client. A convention I tend to follow is to create a folder named something like “server_room_climate_202306261808OGK26-JEZ6ENNW” where name is the label of the PUSH check in NodePing, and the check ID that is generated when you create that check in the NodePing web interface.

I created those folders using the command:

$ mkdir -p push-clients/server_room_climate_202306261809OGK26-09S996LV

You can call your folder whatever you want. I use this so I can easily find the right PUSH check in NodePing that corresponds to this PUSH client.

I will be using the pisensehat module to monitor my server room environment. The pisensehat module does a bunch of different things. It:

  1. Collects temperature, humidity, and pressure values
  2. Formats the data to send to NodePing
  3. Lets you control the LED array
    • Turn it on/off
    • Rotate the LED array
    • Shows the status of your configured ranges for temperature(T), humidity(H), and air pressure(P):
      • Red = higher than the safe configured range
      • Green = within the safe configured range
      • Blue = lower than the safe configured range

I copied the Python3 client code into the directory I made and cd’d into that directory for easier editing:

$ cp -r PUSH_Clients/Python3/NodePingPython3PUSH/* push-clients/server_room_climate_202306261809OGK26-09S996LV/
$ cd push-clients/server_room_climate_202306261809OGK26-09S996LV/

To make the PUSH check work, I have to edit 2 files:

  1. config.ini
  2. metrics/pisensehat/config.py

At the end of config.ini, there is a modules section. It should contain this information:

[modules]
pisensehat = yes

You also want to include your check ID and checktoken in the server section

Next is the config.py file for the module. Configure this to your own needs. Mine contains this information:

# True if you want to output colors to LED array
LED_ON=True

### ROTATION
# Rotate 0,90,180,270 degrees to change orientation of LED array
LED_ROTATION=90

### TEMPERATURES
# C for Celsius or F for Fahrenheit
UNIT="F"
# Colors status
MIN_OK_TEMP=60
MAX_OK_TEMP=90
# The SenseHat temp is off by a little but can be corrected
# to compensate for the CPU temp contributing some heat
TEMP_CALIBRATION = 1.0

### HUMIDITY
MIN_OK_HUM=30
MAX_OK_HUM=70

### PRESSURE
MIN_OK_PRESSURE=29
MAX_OK_PRESSURE=31

Important information of note:

  • If you want the LED array to show you info, set this to True
  • If the LED output is upside down you can change its rotation by modifying LED_ROTATION
  • If you are submitting temps in Celsius, set UNIT="C"
  • The min/max for TEMP, HUM, and PRESSURE should match what was entered when you created the PUSH check on NodePing. These values will determine the color output on the LED display
  • I will talk more about TEMP_CALIBRATION later

The client should be configured now. I will now test it by running:

python3 NodePingPythonPUSH.py --showdata

This will let me see the data that would be sent to NodePing without actually sending it. This is just for testing and is a good way to make sure I edited those files correctly without impacting any uptime stats for the check in NodePing.

Lastly, I need to set up cron to run my PUSH client on a regular interval. I want this one to run every minute, so the cron line will look like the one below. To edit my cron tab, I run crontab -e and add the following entry:

* * * * * python3 /home/pi/push-clients/server_room_climate_202306261809OGK26-09S996LV/NodePingPythonPUSH.py

Now my check is running every minute and submitting my server room climate info to NodePing. This will allow me to track temperatures and humidity in the server room and notify me if the room is too hot/cold, humid/dry, and if the air pressure is too high/low.

Note on TEMP_CALIBRATION

While creating the pisensehat module for Python3, I noticed an odd problem with the Sense Hat that turns out to be a well known issue. The temperature sensor on the Sense Hat will change with the temperature of the processor. So if for example, you have a load average of 0.0 on the Raspberry Pi, the temperatures may be just 5 to 10F hotter than the actual ambient air. However, if the load goes up, the processor heats up and the temperature sensor will read much hotter. This will result in inaccurate readings for the room.

To best offset the temperature problem, I included a TEMP_CALIBRATION number in the config.py file. This is a best-effort method of calibrating the sensor to work with the current room temperature and account for the heat produced by the Pi’s processor. In my testing while running the NodePing AGENT and the pisensehat PUSH check, setting the value to 1.0 was relatively consistent with my other thermometers. The number could be different for you, and it is important to know that if your Pi’s load increases, so will your reported temperatures.

I have tried another sensor to work around the temperature issue with the Sense Hat, but I will save that for Part 3.

Another Issue

Another issue I came across in my Sense Hat testing was with the pressure sensor. The very first reading after boot seemed to report back a 0 no matter what. To fix this issue, the module reads from the pressure sensor two times to ensure it does not get and submit a 0.

Pi in the Wild

I have the Pi ready to go now. I took it to one of my networking rooms so I could monitor the environment in there. This room isn’t in any sort of datacenter so the conditions are more harsh with limited temperature and humidity regulation. The Raspberry Pi is perfect for this location so I can know if the air conditioning goes off, or if the humidity gets too low in the drier winter climate.

I connected the Pi to the switch and power and not long after starting up, the cron job started checking the environment and showed all green on the LED array. Success!

With the Raspberry Pi in place and monitoring my server room, I not only get the power of NodePing PUSH checks running on the Pi, but I also have an AGENT running to do some additional internal monitoring, as well as monitoring outbound connectivity.

If the temperature, humidity, or pressure fall outside of my configured ranges, I will be promptly alerted of the issue with the notifications I configured for the PUSH check. Additionally, if I am on premise, I can view the LED display to see if any of those environmental metrics is too high or low.

Check Status Reports

I like to have a visual graph of my environmental status so I created a status report page to get a visual representation in my browser of the environmental info from my check.

Now I have a live graph I can use to watch what is happening in my server room at this moment and over the last few hours. This is helpful for me to watch trends in my temperature, for example.

More to Come

Thanks to its versatility and availability of software packages, there is a lot that I can do with the Raspberry Pi to monitor my services and my surroundings. In Part 3, I will swap out the Sense Hat for a DHT22 sensor for monitoring instead of the Sense Hat to see if I can get better temperature readings without relying on mathematically offset values.

If you don’t yet have a NodePing account, please sign up for our free, 15-day trial.

Filed under PUSH, System Monitoring Tagged with , , , ,

SMTP Monitoring with NodePing

2023/08/22 by Leave a comment

In the world of email communication, Simple Mail Transfer Protocol (SMTP) plays a crucial role in ensuring the seamless delivery of messages. However, like any other technology, SMTP is not immune to issues that can disrupt email flow and affect business operations. To maintain a healthy and reliable email infrastructure, it is essential to monitor SMTP servers continuously. In this blog post, we will explore how NodePing can be used to monitor for SMTP functionality, packet loss, blacklisting, deferred queues, and MX/SPF records.

NodePing is a versatile and powerful server monitoring service that allows businesses to monitor their infrastructure’s performance and uptime. With its extensive range of monitoring checks, NodePing provides an excellent solution for monitoring SMTP servers and ensuring they are operating optimally.

Monitoring SMTP Functionality

Verifying the functionality of your incoming SMTP server is crucial to ensure that it can receive emails without any hiccups. NodePing’s SMTP check allows you to periodically test your SMTP server by trying to send a test email to a designated email address. If the test email is accepted, it indicates that your SMTP server is functioning correctly. In case of failures, like timeouts or server errors, NodePing will promptly alert you, enabling you to troubleshoot and rectify the issues.

Monitoring Packet Loss

Packet loss can severely impact the performance of your SMTP server and lead to email delivery delays or failures. NodePing’s ICMP PING check is a valuable tool to monitor packet loss and routing issues to to your SMTP server. By regularly performing ping tests, you can assess packet loss trends and determine whether network-related issues are affecting your email delivery. If a failure is seen, NodePing automated diagnostics will send you MTR results so you can quickly troubleshoot where the issue originates. Addressing packet loss problems promptly will lead to a smoother email experience for your users.

Monitoring SMTP Blacklisting

Blacklisting can be detrimental to email delivery, as it prevents messages from reaching their intended recipients. NodePing’s RBL check allows you to monitor your SMTP server’s IP addresses against popular DNS-based blacklists (RBLs) such as Spamhaus and Barracuda. By configuring blacklisting checks at regular intervals, you can quickly identify if your server’s IP addresses have been blacklisted, enabling you to take immediate action to resolve the issue and maintain a good sender reputation.

Monitoring Deferred Queues

When your SMTP server is unable to deliver emails to the recipient’s mailbox immediately, it sits in the deferred queue. If emails in your deferred queues are piling up, you likely have a delivery issue. NodePing’s PUSH check can watch your deferred queues and send you notifications if they rise above what you’re comfortable with; allowing you to investigate and resolve the underlying problems before they escalate.

Monitoring MX Records

MX (Mail Exchange) DNS records play a crucial role in email delivery by specifying the mail servers responsible for receiving emails for a domain. NodePing’s DNS check allows you to monitor the MX records of your domain to ensure they are correctly configured and that your DNS servers are responding with those records properly. Regular checks of MX records help you keep incoming mail flowing.

Monitoring SPF Record

Your SPF record is actually a TXT DNS record that specifies which servers are allowed to send email from your domain. If that record is missing, compromised, or your DNS servers aren’t responding, sending email may be blocked or delayed. NodePing’s DNS check will make sure your SPF record is available and hasn’t been hacked. That will keep your outgoing mail flowing.

In conclusion, email monitoring is essential to ensure the reliability and efficiency of your message communications. NodePing provides a comprehensive suite of checks that empower you to monitor for blacklisting, SMTP functionality, packet loss, deferred queues, along with MX and SPF records. By leveraging NodePing’s monitoring capabilities, you can proactively identify and address issues affecting your SMTP server’s availability and performance, leading to better email deliverability and improved customer satisfaction.

Remember, a robust email infrastructure is the backbone of modern businesses, and investing in reliable monitoring tools like NodePing with automated diagnostics is a step towards a smoother and more efficient communication system.

Start monitoring your SMTP servers with NodePing today by signing up for our free, 15-day trial and stay one step ahead of any potential email delivery challenges!

Filed under DNS, Email, Monitoring, PING, SMTP, uptime Tagged with , , , , , , , , , ,

Leveraging NodePing’s Web Monitoring for Optimal Website Uptime

2023/08/15 by Leave a comment

Hello, NodePing community and all those striving for flawless website uptime!

In this digital age, achieving uninterrupted website availability is paramount. At NodePing, we provide the tools you need to ensure your website doesn’t just exist—it thrives. This blog post will dive into our range of HTTP-oriented server monitoring checks and how they can address real-world tech business needs.

1. HTTP Check: Your Go-To for Website Availability Monitoring

Our HTTP Check provides a reliable pulse on your website’s status. It connects to your site at regular intervals, confirming that it’s up and responding with a 200 status code within the defined timeframe.

Use case: Online retailers understand the importance of 24/7 website availability. With our HTTP Check, you get an early warning of any potential outages, allowing you to promptly address issues and maintain a seamless shopping experience for your customers.

2. HTTP Content Check: More than Just a Pulse

Our HTTP Content Check takes things a step further by examining the actual content of your site’s response. In addition to checking if your site is up, this check looks for specific content within the response, based on a string or regular expression you provide. It can also make a negative match, making sure that a specific string or regular expression match does not exist on the page. This is great for looking for database errors appearing on the page, for example.

Use case: If you are monitoring a WordPress or other CMS site, you may need to verify that the site is showing specific content on the page, and not just responding with a “No Posts” message.

3. HTTP Advanced Check: Comprehensive Website Monitoring

The HTTP Advanced Check allows for full customization of your website monitoring needs. It’s the perfect tool for comprehensive server monitoring, from request methods and response headers to posted fields and data.

Use case: Tech start-ups with complex SaaS products can benefit from the HTTP Advanced Check. This tool can ensure all components of your site, including complex forms or API endpoints, are functioning as intended.

Leveraging the Suite of Monitoring Tools

One of the big strengths of Nodeping’s monitoring services is how they can all work together to make sure all the pieces are working. NodePing’s checks are designed to work in harmony. If you’re responsible for a web site, it is important to know if the DNS is working properly as well. You might also want to monitor the health and expiration date of your SSL certificates on your sites. And you can use our automated diagnostics to put information about why services are failing their checks directly in your inbox. For example, this can include an MTR to help diagnose when your web host is having connectivity problems. A layered approach provides the most comprehensive picture of your website’s health.

At NodePing, we understand that every website is different, and that’s why we’ve designed a range of checks to meet all kinds of needs. From small blogs to complex e-commerce platforms, our checks can help you ensure your site stays up and running.

So why wait? Dive into our documentation, explore our checks, and make sure your website’s heartbeat is as steady as ever. We also have a free trial 15 day trial. And remember, we’re here to help. If you have any questions or need assistance, don’t hesitate to get in touch!

Filed under HTTP Tagged with , ,

Monitoring the Server Room with NodePing – Part 1

2023/08/08 by Leave a comment

Monitoring services inside the server room is pretty easy with NodePing. I’m going to show you how I can keep an eye on my server room and its services using NodePing PUSH and AGENT checks. In this scenario, the room only contains network devices, so I chose to stand up a Raspberry Pi. It will also let me add temperature and humidity sensors in addition to monitoring stuff inside the firewall.

NodePing offers two different check types to allow me to do on-premise system, sensor, and network monitoring: PUSH and AGENT checks.

PUSH checks allow me to send heartbeats from any server to NodePing. I can also use the metric tracking to do things like monitor load, disk space, memory usage, and more. Basically any numeric value I can read or generate on a box can be submitted, tracked, and trigger notifications.

NodePing AGENT checks allow me to configure NodePing probe functionality, installed and maintained by me and available only to my NodePing account. I can use my AGENTs to run other NodePing checks from anywhere, even inside my private network, without needing to open up any firewall ports.

The PUSH and AGENT checks can run on a wide variety of hardware. In this blog post, I would like to show you how I configure NodePing on-premise monitoring tools to work on a Raspberry Pi.

Setting Up

To get started with setting up the Raspberry Pi, I use the Raspberry Pi Imager to configure an appropriate version of RaspberryOS. I can use either the 32-bit or 64-bit options. Watch this quick video from Raspberry Pi on how to use their installer. The Imager can be obtained from their website. Below, I chose to use the default, and very popular, Raspberry Pi OS 32-bit. You can also select the 64-bit option as well as the Lite versions which are designed for headless installations if you don’t need a desktop environment. NodePing PUSH and AGENT checks work with all the different flavors of Raspberry Pi OS.

I intend on using this Pi in a headless configuration, without a keyboard or monitor always connected, so I go into the “Advanced Options” page in the imager and enable SSH. I’ll also need to configure a user, password, and necessary networking. It will need internet access either via wifi or ethernet to send check results to NodePing. I’m going to use one of the Lite versions in this case to reduce the amount of extra software that gets installed.

Enabling SSH, authentication settings, and setting a username and password.

Once the microSD card is imaged, I insert it into my Pi and fire it up… waiting for it to boot. I’m able to SSH in! Good start.

From here, I do some further SSH hardening and require the user I configured to have the password entered when I do any sudo commands.

Additionally, I update the packages on my Pi.

$ sudo apt update
$ sudo apt upgrade

Installing the PUSH Clients

PUSH checks can submit any numeric metrics to NodePing using an HTTP POST, and be used to trigger notifications of the PUSH check. The PUSH payload schema is discussed in the PUSH check documentation. I can also use one of the pre-defined clients available on GitHub. Currently, NodePing provides PUSH clients written in POSIX shell, Python, and Powershell. The POSIX shell scripts and Python scripts will work on the Raspberry Pi. If I want to use a different programming language to create my own client I might want to use the existing code as a reference.

I’m going to go with the POSIX client. To get the PUSH clients code, I can visit the GitHub page to download a zip file or I can use git to fetch the code directly using the following command:

$ git clone https://github.com/NodePing/PUSH_Clients.git

From here, I can either use the clients from the directory that was cloned, or make a copy of the individual client elsewhere. A convention I tend to follow is to create a folder named something like “pi_push_metrics_202306261808OGK26-JEZ6ENNW” where name is the label of the PUSH check in NodePing, and the check ID that is generated when you create that check in the NodePing web interface.

I created those folders using the command:

$ mkdir -p push-clients/pi_push_metrics_2023-06261809OGK26-JEZ6ENNW

You can call your folder whatever you want. I use this so I can easily find the right PUSH check in NodePing that corresponds to this PUSH client.

Configuring a PUSH Check

Now I’m going to copy the POSIX client from the git clone I made into the new folder I created above.

Installation instructions are available for each of the client types. For now, I will show you a quick demo of how to configure a PUSH client that will monitor free memory, and 1 and 5 minute load averages. I configured my check like so in the NodePing web interface:

This check will pass so long as the free memory is greater than 100MB and less than 3800MB, 1 min load is between 0 and 4, and 5 min load is between 0 and 2.5. This will help me keep tabs on when my system load is too high, and if my Pi is using more or less memory than expected. For the client on the Pi, I am interested in a few files:

  1. NodePingPUSH.sh
  2. moduleconfig
  3. The modules directory

I have to ensure NodePingPUSH.sh and the other .sh files are executable by this system user. For the moduleconfig file I only want to run the memfree and load module so I simply add those in the moduleconfig file.

The contents of the file are only the 2 lines “load” and “memfree”.

For the NodePingPUSH.sh file, I need to add my check ID and its checktoken. These are found in the check drawer.

The only variables needing editing are CHECK_ID, and CHECK_TOKEN.

After configuring those files I will run:

$ sh NodePingPUSH.sh -debug

This will let me see the data that would be sent to NodePing without actually sending it. This is just for testing and is a good way to make sure I edited those files correctly without impacting any uptime stats for the check in NodePing.

Lastly, I need to set up cron to run my PUSH client on a regular interval. I want this one to run every minute, so the cron line will look like the one below. To edit my cron tab, I run crontab -e and add the following entry:

* * * * * /bin/sh /home/pi/push-clients/pi_push_metrics_202306261809OGK26-JEZ6ENNW/NodePingPUSH.sh -l

The -l at the end of that line will make the script log to a NodePingPUSH.log file in that same directory. You can modify the logfilepath variable in NodePingPUSH.sh if you want to have your logs go to a different folder.

NodePing AGENT

PUSH checks are great for pushing arbitrary metrics into NodePing. However, if I want to run one of our standard check types in my network, without creating ingress holes in my firewall, NodePing AGENT checks are the way to go. Once the AGENT software is installed, I’ll be able to run checks directly on this Pi.

Installing NodeJS

To setup the AGENT, I need NodeJS and npm installed. Raspberry Pi OS makes this simple with a single command on the Pi:

$ sudo apt install nodejs npm

With those pieces of software installed, I can clone the AGENT software git repo to get that code. The instructions can always be found in the GitHub repository, but for now, here’s the short version of the commands I ran as an example:

$ git clone https://github.com/NodePing/NodePing_Agent.git
$ cd NodePing_Agent
$ npm install
$ node NodePingAgent.js install 202306261809OGK26-1QFZPD19 31PF34KZ-JQ4S-43FP-8L6J-20Q46NFRPT07

The Check ID and Checktoken in the forth command example above are used to tie this instance of the AGENT running on my Pi to the AGENT check I created in NodePing. They can be found with the check information on the NodePing site:

The node NodePingAgent.js install command will set up the AGENT to run, and at this point I can now create checks on NodePing and assigned them to run on this AGENT. To do that, when I create a check in NodePing, in the “Region” dropdown, I select the name of the AGENT I made. When that check starts to run, I should begin to see it is being run from my AGENT, not the normal NodePing probes.

Here’s an example where I’m pinging an interval server, using the private IP address, from the AGENT on my Pi.

Note that the Region is set to the label we gave to the AGENT check. Now with the AGENT running, I can monitor anything I choose from this Pi, whether the servers are internal or external.

Diagnostics

I want to take advantage of NodePing’s Automated and On-demand Diagnostics so I want to start the Diagnostics Client on my Pi as well using the following command:

node DiagnosticsClient.js >>log/DiagnosticsClient.log 2>&1 &

The Diagnostics Client will allow NodePing to get MTRs, DNS, and other diagnostics to send me when my checks assigned there fail. It makes troubleshooting so much faster and easier.

This is Only the Beginning

Now I have a Raspberry Pi all ready to have NodePing collect a wide variety of metrics with my PUSH check and run as a custom NodePing probe with my AGENT check. I can use them to do all sorts of cool things.

In Part 2, I’ll add some hardware sensors to this Pi to measure temperature and humidity of my rack so alerts can be sent if either goes outside of a safe range. Fun stuff coming.

If you don’t yet have a NodePing account, please sign up for our free, 15-day trial.

Filed under AGENT, Node.js, PUSH, SysAdmin, System Monitoring Tagged with , ,

Finding the Best Server Providers

2023/08/01 by Leave a comment

Great services need great boxes to run on. How do we know if a server or VPS host is performant and reliable?

We use dozens of different hosts for NodePing and our standards for performance and reliability are really high. There are many SaaS out there that host only on AWS. Putting all your eggs in one basket is nice for billing but would make our service fragile and vendor-dependent. We spread our boxes around to make it resilient and better represent the Internet’s disperate architecture for monitoring.

We have to take new boxes out and put them through their paces; kick the tires and make sure they’re solid. This is how we test out a new provider before we use a dedicated server or VPS.

Blacklisted

As soon as we have our IP assignments from the provider, we check to make sure the IPs aren’t listed in any spam blacklists using NodePing RBL checks. Most of our hosts don’t send any actual email but our public probes do a lot of SMTP connections to ensure our customers’ mail servers are functioning properly. If the IPs are blacklisted, we’ll need a clean IP from the provider or cancel and look elsewhere.

We’ll leave this RBL check running once an hour to make sure it doesn’t get listed half way through our testing period.

Blacklisted IPs can be a good indicator of provider quality even if the server won’t be sending any email. A provider that can’t keep spammers out of their service is unlikely to be able to keep a reliable network.

Incoming Traffic

Solid networks can be hard to find. We test for inbound packet-loss and routing issues using NodePing PING checks. We’ll sometimes test from a few different geographical regions to ensure global routing is stable. Anything less than 100% uptime for 30 days is unacceptable for us. If the provider had announced planned maintenance well in advance, we’d use NodePing’s maintenance feature to ensure the uptime stats remained accurate despite planned outages. In our decade-plus experience, a network that sees even one episode of packet-loss or route failure is going to continue to see them and isn’t stable enough for our use.

We’ll do the same for IPv6 addresses as routing and packet-loss can be independent of the IPv4 stack. Some providers have a hard time keeping their IPv6 blocks broadcasted and we’ve seen IPv6 completely fail while IPv4 continued to function normally.

We enable automated diagnostics for all our PING checks so we can see where on the route the packet-loss or routing failure is happening. Getting immediate MTRs can show us the weak links in a network and if we see issues with some of the usual suspects, we will for sure dump it. Yes, I’m looking at you, Cogent!

Outbound Traffic

Sometimes a network issue seems to only impact outbound routing. We use the AGENT functionality to assign additional PING checks to originate from the server being tested towards some of the other servers it would be connecting to if it’s moved into production. The AGENT software will run NodePing checks just like the public probes but originating from our test host. It’s a great way to detect outbound packet-loss and routing issues from the server. Again, anything less than 100% uptime on this test and the service isn’t going to make muster.

System Load

The performance of a VPS can be greatly impacted by issues outside our control. Two of the most frequent system load issues we’ve seen on VPS are noisy neighbors and host server backups.

A good provider won’t oversell their VPS host servers and will suspend anyone who is abusing more than their fair share of resources. If we end up on a box with noisy neighbors, the system load on our VPS will likely spike, starving our processes from getting the CPU, memory, networking, or storage I/O they need to function properly.

We’ve also come across providers where we saw system load rise every Saturday around midnight (GMT) for 30 mins or so. Turned out their backup process was overwhelming the disks and causing load issues on all the VPS on the host.

These types of issues are simple to find using PUSH checks that monitor the system load. Since we aren’t using these boxes for anything yet, we have to set the thresholds pretty low to detect load issues caused by resource starvation. This is one test that we’ll give a bit of slack to a provider if it fails though. Noisy neighbors or hungry backups can happen to any provider and we’ll give them a chance to find and address the cause. If it keeps happening though, pull the plug on that provider. It’ll just be worse once you start using the machine and an ongoing headache trying to get their support to do anything about it.

If a server can keep humming along for 30 days without any of the checks above failing, there’s a pretty good chance that provider and network are going to be solid and reliable. I hope this look into our vetting process will help you with your provider search for those elusive reliable networks and servers.

If you don’t yet use NodePing, please sign up for our free, 15-day trial and see for yourself how our monitoring can increase your uptime.

Filed under AGENT, diagnostics, Monitoring, MTR, NodePing, PING, PUSH, RBL, SMTP, System Monitoring, uptime Tagged with , , , , ,

← Older posts

Newer posts →