Sunday, 11 September 2016

Capacity Management

Is its possible that you can have a second network on your existing hardware for free?
most likely yes.

Looking withing most network you will find servers that are under utilized such as file and print servers, and these make ideal servers for building a second network.

If you haven't started the migration to virtual servers yet!
It because you are still running older operating systems as nearly all new ones have this as an option.
Where is the benefit in this...? well the real benefit is that you can put all that hardware to a good use example.

In the past you will have had server that are loaded maybe 10% CPU and other that are 60% or more now this unequal loads was directly related to the task they where performing such as running application and/or file servers.

By having multiple server running on one physical host we can make full use of the resource and even share the load of other more heavily utilized servers, by adding another virtual server to the farm.

OK before we get into the many ways you can improve your network by doing this lest look at some prerequisites, number one you need to have enough physical resources to support this and as the base operating systems is recommended to be a clean system to avoid stability issues that means you will have to factor a in one more operating system (no mater what they say its still going to use some I/O and RAM)

The largest overhead in visualization is I/O and RAM with most quad core systems now the CPU can handle the load very well but after using the system for a while the issue of I/O is often the first to come up.

Disk orations are always an issue with large amounts of data and bottle neck is the area are more common these days that before, remember that virtual server are using virtual devices not physical ones so its always best to get you counters from the base OS that is really interacting with the hardware.

Some basic figures where provided on Disk I/O related to SAN's by HP a few years ago, they stated that you should have one 4Gbps HBA connection for every 250GB of data on a highly used system, when visualizing your server you loading many operating systems and accessing data for all of them so depending on the nature of the system this can be very high I/O access, database servers and file servers use the highest I/O, while authentication server use the highest CPU.

So here is a example load for a virtual server, you have two servers and two operating systems might be the same OS with different applications but the general is that you'll end up with six servers, you have two base systems and four virtual servers, lets say you have SQL servers and Active Directory servers.

Place domain controller on server A with SQL server A
then place SQL server B and Domain controller B on server B this way you have mixed the I/O and CPU loads between the servers.

Now VMware resource manager does this very nicely and while Hyper-V can also do this its frankly not as polished as VMware so you might have to manually balance the resources.

But we not finished, what about resource spikes? what happens if the SQL server gets deadlocks and the CPU load goes up??? will our domain controller freeze... in short yes... because the most important thing is that you must setup some resource management on the base OS so that this can't happen.

I always like to run the virtual servers for a good weekend before setting the limits on the resource management so that I have some idea where to place it.
60 percent CPU for the Domain controller and 30 percent for the SQL server was one of the best one I had so far, the domain controller had some 90,000 users so you can imagine it was quite busy on the other hand the SQL server was not used for ETL jobs or OLAP is it was more I/O running reports and adhoc query's with very low CPU load.

The layering of systems allow for better usage, however its good to remember that failures happen to so if you have 5 servers layered like this you need a 6th in case something happens to one of the 5.
You can scale up like this as well, so for 10 servers 2 are for fail-over, also try to keep servers away from one another if you can, so if you have 12 servers try not to have more than two in the same blade enclosure, this way you avoid single enclosure killing the farm.

Keep in mind you should also write down the growth plan, when will you need to add servers and when would an application become too big for virtual server, this can become more interesting than it sounds as you have to go ask the application owned if that application can be farmed out over more servers or if a migration to larger hardware will be needed.

Ideally you should have those answer to hand and check them at least once a year that they haven't changed as with good planning you will know what the upper limit is and when the expected growth will come from.

SSH config for users

Let me ask you the question are you still using the ssh admin@host.com or more likely using port forwarding like this ssh -L 5900:localhost:5900 admin@host.com

If so, STOP right now just stop, there is a so much better way its call ssh config.
user's configuration file (~/.ssh/config)
system-wide configuration file (/etc/ssh/ssh_config)

For the moment since I'm not the sharing kind when it comes so connection details we'll focus on the user configuration file stored in ~/.ssh/config now if you've never used it before then the file doesn't exist so there are a number of ways to create it using vi or nano
But we'll use touch ~/.ssh/config

Now you should have an empty file, let's give some example what can be put there.

Host server1
     HostName server1.company.com
     User minecraft
     Port 4242
     IdentityFile ~/.ssh/server1.key


Host server2
     HostName 192.168.1.100
     User root
     IdentityFile ~/.ssh/server2.key

What this now lets you do is use just the ssh server1 or ssh server2, not only is that shorter and easier to remember it also means you don't need to remember switches like -p and -i and there is no reason to stop there more complex configurations can be used for matching domains and IP ranges.


Now here is a larger example using few kinds of setups both with ports and port forwarding and timeout settings.

### default for all ##
Host *
     ForwardAgent no
     ForwardX11 no
     ForwardX11Trusted yes
     User minecraft
     Port 22
     Protocol 2
     ServerAliveInterval 60
     ServerAliveCountMax 30

## override as per host ##
Host server1
     HostName server1.company.com
     User minecraft
     Port 4242
     IdentityFile /nfs/shared/users/nixcraft/keys/server1/id_rsa

## Home nas server ##
Host nas
     HostName 192.168.1.100
     User root
     IdentityFile ~/.ssh/nas01.key

## Login AWS Cloud ##
Host aws.apache
     HostName 10.20.3.4
     User wwwdata
     IdentityFile ~/.ssh/aws.apache.key


## Forward all local port 3128 traffic to port 3128 on the remote vps1.company.com server ##
## $ ssh -f -N  proxy ##
Host proxy
    HostName vps1.company.com
    User anonnymus
    IdentityFile ~/.ssh/vps1.key
    LocalForward 3128 127.0.0.1:3128

You can also use ranges on the hosts so that anything matching that range can be used
Host *.company.com
Host 192.168.0.?

You can also pre-configure connections via a server that has network access like a proxy or gateway to the network.

Host *.company.com
  User admin
  Port 4444
  IdentityFile ~/.ssh/aws.apache.key
  ForwardAgent yes
  ProxyCommand ssh accessable.company.com nc %h %p

Now for any server I ssh to that ends company.com will be forward first to the server call accessible then on to the target server, meaning that you don't have type very long ssh commands to reach a server.

Not that forwarding has to be enabled for the last example on the ssh server.

Friday, 9 September 2016

Installing Software Remotely Using Powershell

If you have Powershell on your network it might be time to make use of it, now I am not saying replace the psexec if you have been using it, as that is still a great tool, and on large networks with servers both with and without Powershell its sometimes the best way.

Nevertheless if your one of the lucky few with only a new network, this is how you can use PowerShell to do that nice installer thing I once showed on ps execs.

First you will need to make sure both the server and client should have powershell installed.
And psremoting enabled.

Start PowerShell as administrator and fire below command.

Enable-PSRemoting 

Now you might want to do this quickly using psexec

psexec \\[computer name] -u [admin account name] -p [admin account password] -h -d powershell.exe "enable-psremoting -force"

You can also replace "\\[computer name]" with an IP address, or even "@C:\[path]\list.txt to automatically enable psRemoting on a big list of computers
With PSRemoting enabled you can run scripts remotely on any computer that you wish.

For example:

Invoke-Command -Command {\\servershare\Softwares\Setup.exe /parameter:01 /parameter 2 } -computerName (Get-Content "c:\webservers.txt")

Or you can go one step more and create the list of computers dynamically using the active directory.

Invoke-Command -command { dir } -computerName ( Get-ADComputer -filter * -searchBase “ou=Web,dc=company,dc=pri” | Select-Object -expand Name )

This is where the power over psexec starts as you can use objects in active directory to determine where to install the software.

Virtualizing Active Directory

Along time ago I wrote that is was a good idea to have virtual active directory servers, as this is a very quick way to recover in a disaster recovery.

What I forgot to mention at the time are the things you need to think about to have in place for this to work.
For example, Microsoft doesn't like supporting you unless the platform is hyper-v however VMware will support you.

but sorry to say you have limited or no support on other platforms.

Also to avoid dirty writes and this is something I hope you have already done for your databases and application servers that are virtualized, to disable the write cache.
This should be less of an issue if you are using a SAN.

Last but not least do test the restores, create at least one isolated VLAN to restore active directory to so that you are sure current backup works, and you can do this at least once a month as finding out you have a corrupt active directory and can't restore it is a nightmare you don't want to ever have.
That said the benefits of being able to do restores quickly and being able to script even the disaster recovery tests make this it worth it.

As an example, a disaster recovery test used to take 6 hours for active directory restoring it and then being able to bring up applications.
With scripting and backups on the SAN, virtual tape library, it was now done with only a few commands in under 40minutes.

Dynamic code generation bad for apps?

While using HSQLDB, H2 and others let code be created dynamically without your JAVA developer needing to think hard about the database it does come with some overheads.

First, the query in now created by a program that doesn't understand the intended outcome and sometimes because of this will create repeated queries for the same set of information, this doesn't show up in small unit testing, however, can become a large performance bottleneck when dealing with many thousands of transactions.

Second, these programs do not use any performance best practices and can be hard to link the java query to the actual SQL statement that is executed on the Database layer.

Such things that are overlooked are:
  • Network traffic caused by long query statements.
  • Slow and N+1 query issue.

With this in mind does such development style help?  Well yes if you're writing a program that is small and has a very small database.
If on the other hand you're expecting it to grow to a larger size and will continue to do so over years to come, you might have just shot yourself in the foot, as debugging performance issues will become a nightmare, this is not to say that the caching functions are not useful, however do not rely on them to write good queries for you.

Wednesday, 7 September 2016

ssh keys how big should they be

I was asked recently how big should an ssh key be, the answer is simple as big as you can support.
The reason I say as big as you can support is not only the larger the key the harder it will be to break but also because you will most likely be limited by some device on your network that doesn't support larger than 4096bit keys.

For example, I try to run 8192bit everywhere I can and one of the places that I've found that I can't is phones, however, this is more of an app issue than the phone itself.

Some of you would ask why not a larger key like 16384bit while others would ask why larger than 2048.

Well, the answer is simple in both counts 2048bit is now standard for most systems meaning it will be the first one that people try to break, this doesn't make it any less secure however it does mean more people are trying to break it.

As for the 16384bit will apart from the overhead on the connection, depending on the speed of computer on each side it can make the connection unreliable and painful to use.

So I split the difference and when with 8192bit key, so far I can say the connections are stable and I've a good feeling about security, however, I still have another 2048bit key that I use for online sites that don't yet support 8192bit and I'm sad to say that doesn't look like it will change for at least another two years.

Slow Database

As we move to larger datasets we have improved processors, disk I/O and CPU, however, we are still held back mostly by our own code.

We know that full table scans are bad and we do our best to avoid them, most of the time, and if you have a good DBA he/she will find any that come up over time, but there is another thing that can slow down the data when a large number of queries are overlapping.

Locks
Now locks are a perfectly natural thing in a database for data consistency and that is a good thing, nevertheless, this makes sense when changing data it's unnecessary for retrieval of information such as selects that normally make up the bulk of database queries.

SELECT doesn't hold the exclusive lock on pages rather it sets shared lock(S) on the pages to read and other transactions can't modify the data while shared locks exist(but can read the data by placing another shared lock). So it is expected that your SELECT blocks any updates.

So let's paint an example, if you have a website with 100,000 users some 10,000 might be online at the same time so that is some overlapping selects as many of the users view or query the same information, now there are many caching and other smart things you can do at the middleware layer to reduce this however at some point the query will still get to the database and at that point you don't want them waiting because the row is locked by update or insert is running and the rows/table/pages are locked.

So the pain here is when an update and a select are running the select uses a shared lock but the update and insert use exclusive lock.

Now one option would be to use nolock on all of your select statements problem solved right ?
Well not really as now you have reads on incomplete data and also its bad habit to get into as once you start using them you might put one on a "insert/update/delete" statement and then you have a lovely corrupt database.
Also if you have a good DBA he/she will have flagged any statement with nolock in the same way they will flag select * as they are not best practice.

So what to do?  Well, the answer is more simple... drum roll, please... ISOLATION LEVEL now this gives you the more options to read data both committed or uncommitted depending on your need, MySQL, Oracle and Microsoft SQL all support Isolation Level, so you can now control what needs locks and what doesn't.

This is not only best practice but the way you should use your database.

Big Data Means Less Workers ?

Now we might all have heard about big data, but for some 90% of people, this means?
Well based on what I hear from people when I ask them people think DNA research or understanding space or NSA and CIA thanks to spying one the public.

Amazingly almost no one has heard of or even knows about IBM Watson  and that's a little bit of a shames as an example of what big data can do is never better than Doctor Watson.

Now Watson because has access to all the information both medical and drug-related has a higher than your GP success for diagnosis and treatment and think about it, he doesn't need sleep and learns all the current medical practices in real time so is never working without of data information.

So this I think is a great medical tool but let's explore this for a moment in other fields, Architecture, Electronics Design, Tax and Government workflows, IT Support, Clothing and much more.
All of these need many things to be known and change over time and because of the complexities are hard to master but with big data intelligence this is no longer an issue as all of that information can be on hand at once.

So let's give an example TDP is too high for your next model of laptop because engineer overlooked something, this results in a product recall with big data this could have been prevented.

With clothing the colour runs because of the kind of dye that is used and you need to set the label correctly, this could be done without checking by big data.

IT support and Government follows workflows that will get you the end result this could be done by big data without the need for humans.

Some of you are by now starting to think so what jobs are left as many of these things are done by humans would no longer be needed.  Well, it's not all bad news, first of all, computers still can't create something new so we need people to think of new things.
Second computers still can't interact with people very well so for that face to face time we need people.

What big data is good for is understanding complex things better and avoiding human mistakes that happen when things are overlooked.