Monday, 3 October 2016

Post Change Checks Automation

Checking changes, ever had one of those changes that should be simple then after something wasn't working and took hours to track down.

like when one server in the farm is not running because someone forgot to start it, or network subnet was wrong on a firewall change so some things work and others don't ?

Well if you have don't worry, your not alone, now if you've invested some time in good monitoring you might be able to check for those things quickly, or perhaps you could just add to the change process a post change check.

Today I'm going to show the benefits of scripting some post change checks.
like is the network connection ok, are is the application running, etc

Part one is the network ok.
There are normally a few things to check on the network level

1) DNS - this might not be important to you if the application server uses only IP resolution, however, I like to use names as it makes network changes more dynamic.

2) Packets/Ports - of you have ping that will tell you some basic network connection however if there is a firewall you need to know if the port the application is communicating with is open.

3) Are common services available, can you reach NTP, DNS, LDAP/Active Directory and Databases.

This can be done with batch script, for the most part, however, there are some limits on windows that you can't check if the ports are open, however, you can check most things, for example here is one to check that your local internet is working.

@echo off
ping -n 1 | find "TTL"
if not errorlevel 1 set error=ok
if errorlevel 1 set error=fail
nslookup | find "Addresses"
if not errorlevel 1 set error1=ok
if errorlevel 1 set error1=fail
ping -n 1 | find "TTL"
if not errorlevel 1 set error2=ok
if errorlevel 1 set error2=fail
nslookup | find "Addresses"
if not errorlevel 1 set error3=ok
if errorlevel 1 set error3=fail
echo Result: Local connection %error%
echo Result: Local DNS %error1%
echo Result: Remote connection %error2%
echo Result: Remote DNS %error3%

One of the most common issues is when your ISP has DNS servers failing so you can see that not only do I check DNS on the router but I then check the result against Googles Open DNS server, proving local and remote connectivity.

If you have Windows 8 or Windows 2012 and higher you can use PowerShell Test-NetConnection this can check if ports are open, unlike batch without needing third party tools.

#check connection to dns
Test-NetConnection -ComputerName -Port 53 -InformationLevel Detailed | Select-Object RemotePort, TcpTestSucceeded
#http lookup
Test-NetConnection -ComputerName -CommonTCPPort HTTP -InformationLevel Detailed | Select-Object RemotePort, TcpTestSucceeded
#dns lookup
Resolve-DnsName -Server –Type A | Select-Object IPAddress
#check running service
Get-Service -Name "vss" -ComputerName "localhost"
#check service account user is not locked out, and connection to active directory
Get-ADUser IIS_ServiceAccount -Properties * | Select-Object LockedOut

With Linux this can be done much easier using NetCat or Nmap to get the results, these can also be used with Windows, however, Nmap needs a reboot so I'd recommend using NetCat if you have the choice.

Now obviously the list of checks need to be custom to your needs however with these simple example hopefully, you will be able to create some quick post change checks.

For some of you, this will include message queues and status of jobs however, for the most part, you'll already have some monitoring to help you with this.

No comments: