New Performance Summary Report and Updated Public Reports

Web site developers spend a significant amount of time and effort optimizing the site so that it loads quickly and performs well for their visitors. All of that effort is wasted if the web server performs poorly. One of the key uses for website monitoring is keeping an eye on web server performance to make sure that piece of the visitor’s experience is working optimally.

NodePing Monitoring Results Summary ReportWe have implemented a new report to help with this task. The Performance Summary report shows the minimum, maximum, and average response time for a site over an hour. By default, the report shows the last 31 days of performance data. As with the results report, you can change the number of results shown by editing the number on the report’s URL.

Of course, this isn’t limited to just web site performance. This report is available for all monitoring on NodePing, so you can see the same thing for ping results, ssh checks, email checks, and the whole range of other service monitoring we provide. In particular, this type of information is useful in ping tests to routers to watch connectivity performance over time.

We have also adjusted the way our “Public” reports feature works. The summary and results reports are now available if you are logged in to your NodePing account, even if the “Public” access is turned off. The “Public” toggle still controls whether the report is visible to visitors who are not logged in. We also changed the URL’s to better reflect that it is the same report with a different format. The results reports are now all at /reports/results/ (although the old URL’s will continue to work). You can retrieve the data in JSON or CSV format by adding ?format=csv or ?format=json to the URL. For CSV output, you can add a file name to the URL as well for convenience (so the URL would end in /filename.csv?format=csv). Documentation can be found in our reports documentation.

We are continuing to work on improving and expanding our reports. Please let us know what you think, and what you want to be able to see from the monitoring reports. Our continuing goal is to make NodePing not only the most cost effective, but also the most useful monitoring service anywhere.

Twitter Notifications

NodePing is happy to announce our newest notification method – twitter direct messages.  The ability to receive a twitter direct message is a great addition to our current notification system that already includes unlimited email, international SMS, and voice calls.

Twitter notifications are in testing at this point.  They are available on all accounts so please do kick the tires and let us know how things work for you at support@nodeping.com.

You’ll need to follow @NodePing in order to get direct message alerts.  Then add your twitter handle in your contact record and in your check’s notification section and we’ll send you a private and discrete ‘direct message’ (not an embarrassing public tweet) when that check goes down and again when it comes back up.

Let us know in the comments how this new notification type is working for you and what you’d like to see added next – instant message (IM), HTTP POST to url, carrier pigeon, etc?

Monitor Streaming Audio

We’re happy to introduce our new audio check type.  Now NodePing can monitor HTTP streaming audio servers and notify you via email, SMS, and even voice alerts when your audio streams go offline.

The new audio check is available now and can be used to watch HTTP streaming audio services like ICEcast and SHOUTcast servers.  Set the target of the check to the URL specified inside your pls file.  If your pls file looks like:

NumberOfEntries=1
File1=http://example.com:8002/

Set the check target to “http://example.com:8002/“.  The check briefly connects to the stream and reads the headers returned to determine if the audio steam is up or down.

Many thanks to all who gave their “+1” to this new check type.  Your feedback and suggestions help us prioritize which enhancements and new check types our development team works on. If you’ve got a need for a specific check or an enhancement to an existing check, please let us know in the comments below or email us at support@nodeping.com.

Monitoring password protected websites

Some of our customers have asked us to add basic authentication to our HTTP checks. They want to be able to check the availability of web pages that are protected from public access by a login. So, we have enhanced our HTTP and HTTP Content checks to support basic authentication.

This means that when you set up HTTP or HTTP Content checks in NodePing’s website monitoring service, you can now include a username and password in the URL. The format is username:password@host. We already supported both http and https requests and arbitrary ports in the URL. The following URL examples are all in a valid format for NodePing HTTP and HTTP Content checks (although these are fictitious, don’t actually use these URL’s in your checks), with this enhancement rounding out the list:

People who use this feature should be aware that HTTP basic authentication is not secure, which is one of the reasons we had not included it until now. This has required a small change to our Terms of Service to point out that we aren’t responsible for confidential information that is included in checks such as this. If you choose to include a username and password in your checks, you should take normal precautions to protect your data, including making sure that the login used for the checks is limited to no more access than is needed for the check, and avoid reusing passwords.

If you have any questions about basic authentication in HTTP or HTTP content checks, or about any aspect of our website monitoring, please don’t hesitate to ask us by emailing info@nodeping.com.

DNS Monitoring for Both Sides

DNS monitoring, like a coin, has two sides: “What does my DNS server say?” and “What does ‘public’ DNS say?”  With NodePing server monitoring, you can ask both questions.

Our DNS check allows you to send a query of a specific type to your DNS server (or a public DNS server) and test the response against a string you define.  For example, you can verify that your website domain resolves to your web server’s static IP address and have NodePing send you an email or SMS alert when either the server or the response fails.

DNS queries can be made for the following types and the response verified:

  • A
  • CNAME
  • MX
  • NS
  • PTR
  • SOA
  • TXT

You can find more info on the DNS checks and our other check types in our documentation.

If you don’t have a NodePing account yet, try out our new DNS monitoring checks for free with a 15-day trial.

6 Sources of Residual Income for the Web Developer

Most web developers I know are hired for site creation, or re-creation and when the job is done, that’s the end of the revenue from that client until something is broken or some other change needs to happen. But with a little extra effort on your part, you can transition a ‘project’ client into residual income. Not only will this keep a steady flow of cash coming in, but you’ll be able to more easily maintain those valuable relationships with your clients so when the next site or re-design happens, you’ll be the one they call.

Consider adding or packaging a few of the following ‘services’ with your development pitch.

  1. Hosting
    Chances are, you’re already doing some of this. The ability to hand over a turn-key website solution makes it easy to add the ‘web hosting’ line item to your bill. Given the low cost of shared hosting like Bluehost or  HostGator, you can easily charge a modest monthly hosting fee and make a nice margin on it. If you don’t ‘do’ hosting – sign up as an affiliate on a hosting company you recommend. Affiliate programs typically give you a commission for sending them customers. It’s easy to do and doesn’t cost anything. Heck, the two links in this paragraph are affiliate links – so go sign up! <grin>
  2. Monitoring
    If the site is worth developing and hosting, it’s worth keeping an eye on. No matter who the host is, your client (and you, if you’re hosting it – see #1) should be the first ones to know if it has gone down. Get yourself a website monitoring account at NodePing (come on, you saw that plug coming a mile away). For only $10/month, you can set up 1000 URLs to keep an eye on. Resell some of those checks to your client – charge them whatever you think is fair and they’re willing to pay. Set your client up as a ‘Notifications Only‘ contact and they’ll get an email or SMS whenever the site goes offline – and when it comes back up too. With NodePing’s new public reports, you can create a URL on your branded website and iframe in the public report so your client can bask in the glow of their uptime graph. If you don’t know how to iframe one page into another, you’re not the target audience of this blog post – but just in case, here’s a link on how to iframe.
  3. SSL Certificates
    If your client’s site has a login form, they’ll need an SSL certificate. The ease of wireless ‘sidejacking’ using Firesheep and similar tools, you should know better than to have a non-SSL login form. Like hosting you can either resell SSL certificates directly or sign up to be an affiliate and earn a commission (usually a percentage) of your client’s spending. Unlike hosting, it takes quite a bit more hoop jumping to become a reseller but with the price of certs and the fact that they need to be re-issued on a regular basis it’s worth the effort.
  4. Payment Processing
    The least fun part of developing ecommerce solutions is the payment gateway integration. Typical reseller/affiliate programs with payment gateways include a percentage of the setup fee, a percentage of monthly fee, and even a per-transaction cut. While Braintree is definitely our preferred gateway here at NodePing (great API and low costs), they unfortunately don’t have a reseller/affiliate program so you may want to stick with some of the old guards like Authorize.net.
  5. Backups
    Your client will need ‘offsite’ backups of their sites and databases. Most of the cheaper hosting companies don’t provide adequate backups, which provides you with an opportunity to sell a much-needed service to your client. Drive Headquarters has a generous referral program and offers both client and server backup capabilities that are very script-able. Just be sure you know how to restore the site from those backups.
  6. Timeshare in Vegas
    OK, this one is just for laughs.  Timeshares are like boats – everybody I’ve ever known who bought one, has also sold one.

If you started adding auxillary services like those above, you could easily see a $100/month/client jump in revenue for just a wee bit of your time in administrating… multiply that times your current client base and you just got a nice raise!  Felicidades!

More importantly, providing extra services will help you maintain a relationship with your client, so the next time they need a $7k makeover on their site, yours is the only bid they’ll consider… after all, you’re doing so much for them already.

So there are 5 ways you can leverage your web development relationship with your client into a bit of residual income.  It’s by no means an exhaustive list.  If you’ve got better channels for residual income or maybe just excellent suggestions for the resell-able or referr-able services listed above, share it with us in the comments.

Public status reports of monitoring results

One of the features that we get asked about fairly often is public reports for monitoring results. It has been on our todo list from the start, and starting today they are available.

Of course, not every check on your list is something you want out in public. So this feature can be turned on individually for each check. Each check that you enable for public reports will have its own URL.

We’ve intentionally made these reports fairly plain, without much branding. This makes them suitable for framing or other methods you could use to embed them in your own site.

These reports will see further enhancements in the near future. As always your feedback is welcome.

Eight things you could do with monitoring checks on 1000 targets

With NodePing, you get checks on up to 1000 targets or services for one flat rate. NodePing’s 1000 service limit is designed to take the lid off of the kinds of limitations you might face with other service providers that charge more for adding checks or services. Once more checks don’t cost you more, what could you do with them? Every once in a while one of our customers has that moment where they realize how much they can do with NodePing that they couldn’t do when adding check targets raised the price. Here are eight things 1000 check targets can allow you to do that you might not do on other services.

  1. Monitor all your web sites and all basic services. OK, this one isn’t very creative, but it has to be said. If you are responsible for your business’s web sites then you need to know if they are down. Web sites that are down are not generating revenue, or if they are internal sites are not enabling your business to operate. If a site is worth having, then it is worth monitoring. This is the main reason people use monitoring in the first place. This goes beyond just web sites. If you are responsible to make sure that a service is available to customers or employees, you should monitor it so that you know immediately if it is unavailable, before someone complains.
  2. Our tongue-in-cheek tag line is “All your nodes are pinged by us,” but why not? With NodePing, now you can ping them all. If you don’t need notifications on all of them, just turn that off on a host by host basis, but you’ll have availability and uptime stats on everything.
  3. Monitor that a web page is showing the right information. This is called a web content check. Some web applications and content systems don’t return a proper 404 error, so to a normal HTTP check the page might appear to be up. A HTTP Content check makes sure the site is up by checking that it contains what you are expecting it to contain. It is often good to set the content to be checked as something that appears in all your pages, such as your copyright statement. This way if the text on the page changes during the normal course of business, your check will still pass.
  4. Monitor that the wrong text isn’t appearing on the page. Some web pages contain dynamic text. This is particularly the case for pages that show feeds, or your most recent news items. We’ve all gone to a site that should have a page with a list of articles or posts, but instead shows a database error or some kind of “No articles found” message. If that’s not what you want people to see, but you don’t know what text to check for because you don’t know what articles will appear, a check that makes sure the page does not contain specific text is the way to go.
  5. Along the same lines, since you have plenty of checks you might want more than one check on the same URL. If you need to watch for more than one error message, or check that multiple widgets or blocks on the page are populating correctly, why not check them all?
  6. Simple cron replacement. Many times web applications have a process that needs to run every so often, maybe every hour or every minute. These are often accessible by hitting a URL. This is often done by using curl or wget in a cron job, but it is easier to set up a check to hit the URL at the right interval. We use this to keep couchdb views fresh. Similarly, it can be used to replace Drupal’s cron job requirements.
  7. Check API’s and other HTTP interfaces. These often don’t get monitored, but they can be a key piece of your business. The HTTP Content check doesn’t care what kind of body the response has, and it will happily check for your text in JSON or XML as well as in HTML. You can monitor that a CouchDb server is saying “Welcome,” for example, or hit a URL that returns a reduced view and look for the value you expect in the results. The same idea applies to SOAP interfaces as well.
  8. Monitor other monitoring. Many systems have a status page that says how services on that host are doing. Frequently they’ll have an OK message, or an ERROR message will appear when things go wrong. HTTP Content checks can be used to watch these pages and send notifications if the wrong thing appears or does not appear on those pages. Both the “Contains” and “Does not contain” options for content checks are useful on this one.

There are many more things you could do with 1000 checks that you might not even consider doing with other services. We plan to add more check types to increase the utility of the service even more. What other things could you think of doing if you aren’t limited by artificial constraints imposed by services that charge by the target service or URL?

Monitoring Services Are Poised for a Shake Up

Server monitoring and website monitoring services cost too much and are overly complicated.

Over the past several months we have built and launched NodePing’s site and server monitoring service. Part of that process involved looking at the other companies in this market niche, and finding the opportunities for offering a service that fills a gap in what is currently being provided to customers. What we have found has confirmed our original reasons for starting NodePing.

There are a lot of companies offering site and server monitoring services. However, our experience as consumers of these services was that it was hard to find a provider that did what we needed at a reasonable price, and I think our experience is probably typical. Where’s the disconnect? We wanted a service that would allow us to watch twenty to thirty sites and services for a reasonable price. It is easy for a small to medium business to get to a couple of dozen services needing monitoring. Most companies have at least one or two web sites that need to be available all the time for their customers. Many also have two or three web sites used internally for collaboration and sharing or publishing information to employees (Intranets). Throw in a DNS server or two, a mail service, an accounting system, a key router or two, and you are quickly into double digits on the number of services that need to be checked.

IT departments used to run software like Nagios for this type of thing, and that is still a good option in many cases. Nagios provides a wider set of checks than a typical SaaS monitoring service, there are lots of specialized plugins available, and it is not all that difficult to write custom plugins. If you need specialized checks, a system like Nagios is probably the best bet. On the other hand, while Nagios is free software, running it is not free. It requires a server to run on. Typically you want monitoring to run on separate infrastructure from your normal servers, which often means leasing a server or using a VPS service. Doing this inexpensively typically runs $50-100 a month, and involves a non-trivial amount of technical expertise and work to setup, tune, and maintain. That’s not a huge amount of money, but it is not free.

External providers offer similar services. The majority of companies need HTTP, SMTP, and PING checks. These are the primary checks provided by the bulk of the monitoring as a service industry. These types of services don’t cost much to run. With today’s opportunities to build and deploy cloud based services in cost effective ways that scale well, the cost of these types of services should be fairly low. That’s not currently the case.

A quick search turns up a lot of companies offering these services. Many of them offer “free” or inexpensive services. “Free” monitoring is typically provided for one to five URLs, often with fifteen to thirty minute intervals. That is basically useless. If it is ok for a service to be down for 30 minutes without getting a notification, you probably don’t need monitoring. In my opinion, a price “plan” isn’t a serious offer unless they offer the service in intervals of five minutes or less at that price. Getting beyond that unhelpful “Free” level, many providers start charging by the URL or address you want to monitor. One company prominently advertises checks starting at $1, but again that’s one URL in thirty minute intervals, and it costs $11 for that URL check if you want to do check it every minute. Paying per check or per URL quickly gets expensive. It is not uncommon to find special price calculators on the sites of this kind of provider, which is itself a hint that the pricing is too complicated. At these prices, a fairly typical small to mid-sized company could easily find themselves spending hundreds of dollars a month on monitoring.

There are more competitive options out there. These companies typically cost $40-$60 for a reasonable number of addresses and services. These prices probably save you money compared to running monitoring yourself using something like Nagios. Plus, you don’t need to deal with setting up and maintaining the software. That’s a pretty good deal.

However, it still doesn’t need to cost that much. With modern hosting and technology, the cost per check and even per customer to run these types of services is very low. In fact, just about the only cost of running a service like this that is attributable to an individual account is the credit card processing. All the rest of the costs scale, and are spread in ways that actually decrease per account as you scale up. Unless they are just running very inefficient systems, the total overhead for the companies charging $40-$60 per month (not to mention the ones costing hundreds) should be less than $4 per customer. Of course, the companies advertising “Free” services are also spending dollars a click to get those accounts, and that easily becomes the biggest expense. Meanwhile, allowing their customers to add additional checks or URLs to an account costs the provider pennies. Pricing based on adding checks or URLs is a model completely detached from the economics of running the service.

Experience in running IT departments and talking to system administrators tells us that there are a lot of services that should be checked if best practices were followed that aren’t getting checked. Many companies that use external providers check their company’s primary site, but when adding checks means adding overhead costs (or just the work load), secondary and internal sites don’t get checked. This means that there are millions of services that should be monitored that aren’t getting monitored at all. Companies are just reacting to complaints when something goes down.

To us, this smelled like opportunity. It is not simple to set up a solid monitoring service. However, once the technology, infrastucture and processes are in place, it is a service that scales. The margin stays fairly stable even if you let customers use it as much as they need. This calls for a flat rate model.

Our biggest problem is that we have entered a market that is saturated by misinformation. Buyers assume that this type of service costs at least $40 for a reasonable level of monitoring, and often lots more. They expect to see low entry prices that don’t really meet anybody’s needs, followed by much higher prices for the real service. This becomes a marketing challenge. When shopping for these services, NodePing’s price of a flat $10 for monitoring sounds like one of the entry point bait ads. We say “$10 to monitor up to 1000 services in 1 minute intervals” and people ask “Yes, but what do we really get, and what’s it cost if we actually need to do real world monitoring?”

NodePing’s services really cost $10 a month. Period. There are no add-ons, no “X is available at additional cost”. We set 1000 services as the maximum because we don’t want to monitor IBM’s network (no offense to IBM intended). Our target is small to medium sized businesses, and we want them to monitor everything they want to monitor for one reasonable price. If this model works, maybe others will also move to flat rates. That’s great. We’d be happy to help make the monitoring world make more sense and be more cost effective for businesses. We think we have a solid technology stack and a great service, and we can do quite well even if other providers compete with us directly on price. Until then, there are few if any major providers that really provide the services that our customers need anywhere close to our price.

Monitoring services cost too much and are too complicated. We think this market is set for a change, similar to how the cloud has impacted other technology services. This shift will be a significant benefit to small and medium sized companies that need these services, and it is a fantastic opportunity to providers poised to provide the services the customers need at truly competitive scale and rates. NodePing has positioned itself to provide the services businesses need at a fantastic, flat-rate price.

10 Common Server Monitoring Mistakes

Server monitoring is an essential part of any business environment that has services.  Even if you don’t have your own servers and use cloud-based services, you’ll want to know about downtime.  You don’t want to find out your web site is down from customers and you don’t want your boss to be the one to point out the email server has wandered off into the weeds.  Done properly, server monitoring alerts those responsible for the services the minute they’re unavailable, allowing them to respond quickly, getting things back up and running.

David and I have been responsible for servers and server monitoring for years and have probably made nearly all the mistakes possible while trying to do it properly.  So listen to the war stories from a couple of guys with scars and learn from our mistakes.

Here are 10 common server monitoring mistakes we’ve made.

1. Not checking all my servers

Yeah it seems like a no-brainer but when I have so many irons in the fire, it’s hard to remember to configure server monitoring for all of them.  Some more commonly forgotten servers are:

  • Secondary DNS and MX servers.  This ‘B’ squad of servers usually gets in the game when the primary servers are offline for maintenance or have failed.  If I don’t keep my eye on them too, they may not be working when I need them the most.
  • New servers.  Ah, the smell of fresh pizza boxes from Dell!  After all the fun stuff (OS install, configuration, hardening, testing, etc) the two most forgotten ‘must-haves’ on a new server are the asset tag (anybody still use those?) and setting up server monitoring.
  • Temporary/Permanent servers.  You know the ones I’m talking about.  The ‘proof of concept’ development box that was thrown together from retired hardware that has suddenly been dubbed as ‘production’.  It needs monitoring too.

2. Not checking all services on a host

We know most failures take the whole box down but if I don’t watch each service on a host, I could have a running website while FTP has flatlined.

The most common one I forget is to check both HTTP and HTTPS.  Sure, it’s the same ‘service’ but the apache configuration is separate, the firewall rules are likely separate, and of course HTTPS needs a valid SSL certificate.  I’ve gotten the embarrassing calls about the site being ‘down’ only to find out that the cert had expired.  Oh, yeah… I was supposed to renew that, wasn’t I.

3. Not checking often enough

Users and bosses have very little tolerance for downtime.  A lesson learned when trying to use a cheap monitoring service  that only provided 10 minute check intervals.  That’s up to 9.96 minutes of risk (pretty good math, huh?) that my server might be down before I’m alerted.  Configure 1 minute check intervals on all services.  Even if I don’t need to respond to them right away (a development box that goes down in the middle of the night), I’ll know ‘when’ it went down to within 60 seconds which could be helpful information when slogging through the logs for root cause analysis later.

4. Not checking HTTP content

Standard HTTP check is good… but the ‘default’, ‘under-construction’ Apache server page has given me that happy 200 response code and a green ‘PASS’ in my monitoring service just like my real site should.  Choose something in the footer of the page that doesn’t change and do an HTTP content matching check on that.  Don’t use the domain name though – that may show up in the ‘default’ page too and make that check less useful.

5. Not setting the correct timeout

Timeouts for a service are very subjective and should be configurable on your monitoring service.  Web guys tell me our public website should load under 2 seconds or our visitors will go elsewhere. If my HTTP service check is taking 3.5 seconds, that should be considered a FAIL result and someone should be notified.  Likewise, if I had a 4 second ‘helo’ delay configured in my sendmail, I’d want to move that timeout above that.

Timeouts set to high let my performance issues go unnoticed; timeouts set too low just increase my notification noise. It takes time to tweak these on a per-service level.

6. Not realizing external and internal monitoring are different

When having an external monitoring service watch servers behind my firewalls, I may need to punch some holes in said firewall for that monitoring to work properly.  This can be a real challenge sometimes as many monitoring services use multiple locations and then dynamically pick one to monitor my servers making it hard to maintain a white-list of their IPs or hostnames to let in my network.

Another gotcha I’ve run into is resolution of internal and external DNS views.  If these aren’t configured properly, you’ll most likely get lots of ‘down’ notifications for hosts that are simply unreachable.

7. Sensitivity too low/high

Some servers or services seem more prone to having little hiccups that don’t take the server down but may intermittently cause checks to fail due to traffic or routing or maybe the phase of the moon. Nothing’s more annoying than a 3AM ‘down’ SMS for a host that really isn’t down.  Some folks call this a false positive or flapping- I call it a nuisance.  Of course I should jump every time a single ping looses its way around the interwebs and every SMTP helo goes unanswered – but reality sets in and a more dangerous condition might occur – I may be tempted to start ignoring notifications because of all the false positives.

A good monitoring service handles this nicely by allowing me to adjust the sensitivity of  each check.  Set this too low and my notifications for legitimate down events take too long to reach me but set it too high and I’m swamped with useless false positive notifications.  Again, this is something that should be configured per service and will take time to tweak.

8. Notifying the wrong person

Nothing ruins a vacation like a ‘host down’ notification.  Sure, I’ve got backup sysadmins that are covering it but I forgot to change the service so notifications get delivered to them and not me.

Another thing I’ve forgotten to take into consideration is notification time windows.  John’s always the first in the office at 6AM, he should get the alerts until Billy shows up at 9AM because we all know Billy is useless until he’s had that first hit of coffee.

9. Not choosing the correct notification type

Quick on the heels of #8 is knowing which type of notification to send. Yeah, I’ve made the mistake of configuring it to send email alerts when the email server is down.  Critical server notifications should almost always send via SMS.

10. Not whitelisting the notification system’s email address

Quick on the heels of #9 (we’ve got lots of heels around here) is recognizing that if I don’t whitelist the monitoring service’s email address – it may end up in the bit bucket.  Mental note – dang, all out of mental note paper.

Bonus!

11. Paying too much

I’ve paid hundreds of dollars a month for a mediocre monitoring service for a couple dozen servers before.  That’s just stupid.  NodePing costs $10 a month for 1000 servers/services at 1 minute intervals and we’re not the only cost effective monitoring service out there.  Be sure to shop around to find one that fits your needs well.  Know that most services are charging way too much though.

They say a wise man learns from his mistakes but a wiser man learns from the mistakes of the wise man.  Nuff said, true believer.