Tuesday, July 12, 2016

The Phoenix Project - a book review - of sorts!

There is a book doing the rounds at the moment called "The Phoenix Project" which talks a lot about new ideas coming into the IT world around Kanban and devops.


I've recently read it and I'll admit that it's got some interesting ideas but like any work delivery method these are ideas that must be embraced at a higher level than just in IT and the book does comment on this as well, unfortunately, the book is a bit of an ideal vision of how everyone gets on board with the kanban method of working and how it changes things for the better.

I have no disagreements with Kanban, it's a good method to keep on top of things and I'm trying to make myself more disciplined in it's usage by having my workload in Trello and using that to prioritise.

For those of you who don't know what Kanban is, the best way I can describe it is to think of a collection of five paper trays which are labelled In, Action, Filing, In progress and pending.

In - This is new work that has arrived and needs to be prioritised
Action - This is Work in Progress, it needs to be done but is in the queue - whilst in the queue it's not generating any value.
Filing - This is work that is completed but it just needs to be signed off/filed away
In progress - This is work that is being done.
Pending - This is work that is waiting on someone else, much like action, it's in a queue but in this case waiting for a third party.

In Trello, I have columns (the kanban)  that follow a similar pattern. This way, I know what I'm waiting on, what needs to be done and what is coming in. It's just my way of keeping myself organised.

The phoenix project does a nice job of introducing Kanban and the whole work in progress flow although one of the characters is rather annoying as a deus ex machina who is there to kick the protagonist in the head and lead him down the right route.

And that's the big problem with this book. It does a great job of introducing kanban and devops to the world of it, it cannot be faulted for that but it shows everyone happily pitching in and helping out when they see it work and there is the problem, in the real world it doesn't take much for people to slip back into old habits and screw themselves over.

I've worked for quite a few companies in my years and I've done CMM, ITIL, and now it's all about Kanban and Devops.

Now, don't get me wrong, I actually highly approve of ITIL, Kanban and devops but once again, it's needs to be pushed from the top with EVERYONE getting involved to make kanban work. If you only have one or two people doing it then it's going to fall apart very quickly for the department. However, this is not a reason to not do this sort of thing on some sort of personal level, just to keep yourself organised.

One of the most important lists I have in trello is called "known issues" where I make a note of an issue that I've spotted. This only needs to be a few words or a picture of the issue but it's come in handy a few times in there past where something has needed to be done and existing issues have had to be dealt with first but then I'm one of these annoying people who believes in taking notes so I think I'm more predisposed to working in a type of Kanban style anyway.

Either way, I do recommend giving the book a passing glance. It's the first time I've seen a workflow method written down as a story so it's worth a look just for that and you never know, you might pick up a few tips.

Tuesday, July 05, 2016

NTP in a virtualised world

Let me start this off by saying that I love NTP. The whole way the protocol has been designed is truly elegant and it is such an important protocol that is often neglected that I thought I'd put together a blog article on how I configure NTP and why.

Before that, it's important to go through a few things about how NTP works, if you're familiar with NTP, feel free to skip to the next bit.

Basics of NTP

The first thing to note is that NTP relays time in UTC format

If you think about it, NTP has to be ignorant of timezones. It's whole job is to keep accurate time and timezones will just upset that as there are so many of them. Better to just keep to something like UTC internally and have the OS deal with the timezone.

One question this always generates is "What happens if I point my UK server at a US time zone source?" - Because NTP doesn't care about timezones, those NTP servers in the US will have the exact same time as those based in the UK and across the rest of the world. It's up to the operating system to sort out the time zone so yes, pointing NTP at servers in another country is fine and it's not going to force all your machines into the time zone of the country where the NTP server is! 

Another thing to realise is that NTP is hierarchical. Each time server in the chain is said to be at a particular stratum level. Stratum 0 is an atomic clock. Stratum 1 and 2 would be NTP servers around the world that you can connect to. Your internal time source (if you use one) would be stratum 3 or more likely, stratum 4. There is a very good explanation of it all here https://ntpserver.wordpress.com/2008/09/10/ntp-server-stratum-levels-explained/

It's also key to note that NTP takes into account the latency involved in contacting an NTP server. This means that even connecting to NTP servers around the world you should find that your time is still within 500ms of reference time and even accurate down to less than 100ms.

It's also always a good idea to provide multiple external NTP servers, in testing, I've found that three are optimum as three allows for one to ruled invalid by NTP cross checking the three servers and it allows for NTP to use some clever math to offset both the latency of all three and to average out the time received from all three to ensure that the time you're getting is as close to reference time as it can be. (see, I said that NTP was elegant!). In testing, it was often possible to get time on the server to within 10ms around 90% of the time and within 100ms 100% of the time.

Now that I've gone through some of the aspects of how NTP works, another key question is "Does this apply in a virtualised world"? The answer is Yes, but with a few caveats to watch out for.

NTP issues to watch out for:

1. Circular time referencing.



The first issue is to watch for is around circular time. This occurs when a source of time is set up as a virtual machine and is pulling it's time from the host which in turn pulls it's time from the VM. At this point the whole hierarchy breaks time as the hypervisor is both a recipient and a giver of time. This is something that needs to be avoided as it'll cause no end of issues as there is no way to correct for clock drift as neither server is authoritative in terms of the NTP hierarchy.

This is why it is vitally important to turn OFF the ability to pull time from the host on ANY server that participates in any sort of NTP hierarchy. This sort of issue is most often seen when the VM is running as the active directory PDC emulator. In that case it's always best that BOTH the hypervisor and the DC's pull time from an external source such as the pool NTP servers.

2. Invalid time in ESXi



The screenshot above is a very important one, it shows that NTP is configured and running but that for some reason the time on the host is WRONG which is why it's highlighted in red. This can often be because it's not possible for the host to connect to the NTP server, this is commonly see when external NTP is used and it's blocked on the firewall.

Why does accurate time matter on the host if VMWare tools/integration tools are turned off?

Even if such tools turned off, the VM has to ask the hypervisor for time under two special conditions, the first is when the VM is powered on. Because a VM doesn't have a CMOS, it has absolutely no idea what the time is when it's first powered on. The only place it can get it from is the host. If the time on the host is wrong then you've already got a problem and the VM isn't even started yet! This is key when you're dealing with Active Directory which, by default, needs time to be within 5 minutes of UTC across the domain (AD also ignores timezones internally), if isn't then you're going to have authentication issues.

The second case is during VMotion/Live migration tasks. For a fraction of a second, that VM does not exist on the old host or the new host. When the host your transferring it to takes on responsibility for the VM, the VM has to ask the host for the time. Again, if it's wrong then you have issues.

Those are the two major misconfigurations/issues I've seen the most in virtualised environments with the circular time setup being the most common. Getting NTP right even in a small network is key to avoiding strange authentication issues and other problems.

Monday, June 13, 2016

Have you used Chocolatey?

If you've not come across Chocolatey then it's certainly worth a look. Those of you who have used Linux will be familiar with yum and/or apt-get, well, this is the Windows version of that software.

As a Windows admin for some years, I've used tools such as Nlite and Ninte to create custom builds and automated installs. I've also used Windows GPO's to install software or to make software available in the Add/Remove programs list but nothing quite compares to the ease of which chocolatey allows software to be installed.

The way it all works will be very familiar to Linux admins. Chocolatey uses a repository where all the install files live and then a very simple command will allow for the necessary software to be installed as long as you have an internet connection to the repository.

It's also possible to setup an internal repository so that you can install both your own software and third party software from a trusted internal resource as there is always a risk that someone has uploaded a malicious installer to Chocolatey.

Details about the process for using Chocolatey are here and I do encourage you to give it a go. If you use automated/unattended installations then using Chocolatey to install applications not only makes sure that you've got the latest versions but that you also have a relatively simple upgrade method.

Some have asked why I'm so interested in this sort of technology and it's simply because I've had a bit of a revelation of late. That revelation is around automation and devops.
I suspect that most IT folk have unattended scripts for installing Windows and I also suspect that many have a few scripts floating around that they reinvent when the can't find the original.
Devops is changing all of that, there are hundreds of tools that are there to simplify all of this and it's my firm believe that tools like Chocolatey are part of a huge cultural change coming to IT. Check it out, it's going to be the future.


Tuesday, May 03, 2016

Exploring Windows 2016 TP5 - HyperV

Windows 2016, TP5 just came out and, after largely ignoring the previews, this one is looking rather good so I thought it was time to give it a bit of attention starting with Hyper-V.

Windows 2016 has some impressive improvements to Hyper-V, in fact, some of them look like they may well give VMWare a bit of a run in so it'll be interesting to see how things stack up once 2016 has had time to be deployed to a few datacenters.

My first test for any new Windows based OS is to test out the WIM file deployment through WDS. With Windows 2016 TP5, this worked perfectly and it even allowed me to use the same unattend file I created for Windows 2012 R2

My first test in HyperV was to migrate a machine from Windows 2012 R2 Hyper-V to 2016 TP5, as this is going to be something a lot of IT departments will look at first, after all, if you can't get your VM's into 2016 then the take up will almost certainly be slower.

I was a bit surprised when the move didn't work, it generated an error saying I never had permissions which was a bit strange.


Out of randomness, I added the 2016 TP5 server to the MMC on the 2012 R2 server and tried the move again and it worked. It seems that moving from 2012 R2 to 2016 is fine as long as the move is run from the 2012 R2 MMC which strikes me as a little strange and certainly something worth watching out for when the full version is released.
A move from one 2016 server to another went without error.

Aside from that little bit of strangeness, the move from 2012 R2 to 2016 worked pretty flawlessly.

Upgrading a VM's configuration from 2012 R2 (Version 5) to 2016 (version 7.1) is straight forward and takes no time at all but like VMWare, it has to be done whilst the machine is powered off.



Along with the version 7.1 VM you also get something new, production checkpoints. These are actually snapshots that are Microsoft approved to be used in a production environment. Microsoft don't say how long a VM can be used with checkpoints. Personally, I'd still avoid them being used for more than a few days as that will cause slowdowns for large VM's.




One other improvement that has been long overdue is the ability to add a vnic to a live VM. This is something that has been in VMWare for years and yet was strangely absent from Hyper-V until now.

After using Hyper-V in 2016 for a few hours, I'm impressed with the changes, many of which are long overdue to bring Hyper-V onto a level playing field with VMWare. The MMC for it though is still many times more clunky that that offered by VCentre.






Wednesday, April 20, 2016

Just how good is IISCrypto?

I've played around with IIS Crypto a fair bit, for those who don't know it, it's a freeware application that can make changes to the registry to restrict the protocols that are used by IIS in order to secure it and avoid the SSL sites being affected by vulnerabilities such as poodle, drown and so on.

 but I wondered just how good an application is it?

Well, after some testing, it's pretty damn good.

I had some free credit at Azure so I decided to spin up an IIS VM and run a the QualSys SSL testing against it to see how it fairs out of the box.

The test I ran was a very simple test, I spun a Windows 2012 R2 box up at Azure, installed IIS and connected to it over it's IP address to do basic validation. I then set up a DNS pointer to it and grabbed an SSL cert from StartCom and once all configured, the default IIS page was available over SSL.

So, out of the box, with SSL enabled, how does IIS fare according to Qualsys?

Actually, not too badly:



It rated a "C" with the server being vulnerable to Poodle.


Running the IIS Crypto tool and selecting "Best Practices" removed a whole list of ciphers and protocols. A reboot was required which was slightly annoying but as this changes the registry it's understandable.

A quick test on Qualsys again and we get a nice 'A' rating:


An 'A' is good but it's not an 'A+' which I'd have liked to have seen, unfortunately, I didn't have the time to do any further testing but a quick google and I did see an article from Scott Helme about adding in the "strict transport security" header into IIS which I'd liked to have tried but wasn't able to. I suspect that Scott is spot on here and this will get a coveted A+ from Qualsys.

This is all very straight forward and simple for a single IIS site but if you have multiple sites on the same server then you're going to need to test each and every site as multiple sites are a key requirement to make a server vulnerable to drown, add in allowing SSL 2 or 3, RC4 as a cipher and still using SHA-1 certs and you get........


Avoid the 'F' grading! Look after your protocols no matter where the web server is hosted.

Thursday, July 09, 2015

David Cameron wants to Ban Encryption - 2

You can see my original blog article here.

In a follow up speech David Cameron reiterated his desire to break encrpytion on the UK's IT systems and in a show of the "special relationship" between the UK and the US, America joined in on decrying encryption as the root cause of all terrorism.

On Wednesday FBI director James Comey briefed both the Senate Judiciary Committee and the Senate Intelligence Committee about the problems encryption is causing the FBI and others at stopping ISIL, drug dealers, pedophiles, and other unsavory types. He described the situation as communications "going dark" for law enforcement. 
Comey said that there were firms that provided encrypted end-to-end communications and yet still had the ability to read such messages as they travel through their servers, but declined repeatedly to say who these companies were or how their systems worked.
Source - The Register

At the risk of over speculating on what the eventual plans will be, I really cannot see either country outright banning encryption or even suggesting weakening it. They know that Encryption is what drives eCommerce which is vital to both countries financial standing and it drives things like "Digital Britain" which, in turn, allows for more access to local councils and for more things to be done online reducing the costs of providing those services at a traditional over the counter environment.

It seems to be highly likely that the UK and the US will push forward with a plan to have some sort of encryption Master key and they'll somehow require companies to register that key with them. In essence, they'll be building a snoopers database which will become one of the most important hacking targets ever created. Should the database be compromised then the Government will have just given the keys of the digital kingdom to every terrorist out there.

Imagine what IS or any other disgruntled group could do if they can intercept commercial traffic?
Imagine the chaos they could be caused if hackers got hold of it?
And it won't just be bedroom hackers or terrorists, it will be government trained hackers of foreign nations who want to get hold of commercial data. Companies would be ruined overnight if this goes ahead.

Even if banks are excluded from this master key database it won't matter because crackable encryption will mean that things like credit card data will still be readable by people who have access to those master keys.

And that brings me to my final point, what about systems outside of the UK and the US? The cloud allows me to spin up servers in Asia and other places. If those places have don't have similar laws to the UK and US then people are free to set up strong encryption and not hand over the master keys, business will almost certainly be invited to move to places like Singapore, Dubai and China with promises of being able to conduct business securely.

Ironically, the UK Government sponsors a site called "cyberstreetwise" which have a page at https://www.cyberstreetwise.com/cyberessentials/ which offers a nice little badge should a company pass a questionnaire. One of the things in that questionnaire is related to using encryption to ensure secure digital communication.

All my questions to cyberstreetwise and to the Conservatives have gone unanswered which is funny as cyberstreetwise promotes itself with the line "please do ask us anything you need to know" and then ignores any difficult questions about the very Government that bank rolls it.

This proposed ban is going to be a very silly affair, many places have highlighted just how impractical this is. My own favourite can be read here.

Friday, July 03, 2015

How to do DNS correctly

Time and again people seem to be doing DNS outside of best practice rules so I thought it might be a good time to go through how DNS works, what DNS best practice is (with regards to a windows environment) and why it's like that.

In a nutshell, the most common mistake I see with DNS configuration is this:



This configuration is put in place for one of two reasons:

1. It is there to resolve external addresses should the internal fail.
2. It's there to provide internet access should the internal fail.

Point 1 is the most common that I come across and it's very wrong because that is not how DNS works.
When a name query is run, DNS will ask the first name server to handle it, if that name server replies and says "I don't know what that name is that you've sent me", that's it. DNS will not ask the second DNS server because it has a valid reply. Yes, negative replies are valid replies. They are even cached locally for a period of time. All of this is covered in RFC 2308.

Point 2 appears to make some sense, if the internal DNS server dies then queries to it fail but, hey, at least people can still get on the internet - right?
Well yes but.... Every now and then that first DNS server is going to be too busy to reply so the client will ask the second DNS server. If the query is for an internal resource then the second DNS server won't know about it and suddenly you've got this weird condition where a client appears to be refusing to ask the internal DNS server and nothing internally is being resolved, again this is due to the cached negative responses covered in point 1.

Best practice is always to have your clients use internal DNS servers and it's always best practice to have two internal DNS servers.

The second big configuration error that I see is people using internal name servers in the forwarders. This is utterly pointless as the forwarder is there to handle queries that your internal DNS servers cannot. So, internally if you ping www.google.com your internal DNS servers won't know what that is so will pass it on to the forwarder.
If your forwarders are just internal servers then the query will either take a long time to complete (i.e. until it gets out of the network) or it'll just fail.

In summary, Internal DNS server IP addresses for clients, forwarders on the DNS servers for everything else. Stick to that and DNS shouldn't ever be a problem.