Ramblings of a Sysadmin

Blackberry dual outage highlights the need for redundancy in enterprise systems

2011-10-12T09:40:00.001+01:00

This week Blackberry has been hit by two outages, both of which appear to be caused by single points of failure (SPOF) within the RIM infrastructure.

In the news today Blackberry said "The messaging and browsing delays... in Europe, the Middle East, Africa, India, Brazil, Chile and Argentina were caused by a core switch failure within RIM's infrastructure" (Source: http://www.bbc.co.uk/news/technology-15243892).They also said "

"Although the system is designed to failover to a back-up switch, the failover did not function as previously tested," (Source: http://news.cnet.com/8301-30686_3-20118882-266/international-blackberry-outage-continues/)

This immediately causes me to ask a few questions:

Why wasn't the fail over triggered manually?
What was missed in the testing of the switches fail over?
Was this an existing issue?
When was the DR plan last tested?
Had changes been made which invalidated the DR plan?

And this is the major point of DR testing. You can't over everything so sometimes you will have to learn from failures that impact you and incorporate those failure modes into future testing but you should also have the ability to be able to manually failover to be able to quickly recover from a systems problem.
You also have to be absolutely aware of changes that are made which could affect your DR plans and this means every change has to be screened to ensure that you aren't creating an SPOF or that if you are then everyone is aware of it and plans are put forward to plug that gap.

The key in any major outage is to get the system back up, even if it means failing over manually - however, any steps taken to recover the service should be noted in an emergency change request of some description and once this is done and the systems have been recovered it is vital that the change notice is thoroughly reviewed to find out both what went wrong and what could go wrong because recovering from an outage is one thing but it's all for naught if that recovery leads to a potential problem which will bite you later on.
ITIL processes teach a lot of this and implementing these practices can be a pain but its a choice. You either suffer the pain of the paperwork or the pain of the outage.

At least if potential problems are known about they can be more easily dealt with when they appear and bite you and they will appear.

The mobile industry is very much a cut throat industry and this dual outage with Blackberry will do them no good at all because others will seize upon it as a sign of Blackberries weak infrastructure and they will be right.

To recover from this Blackberry need to do a through review of their systems and DR processes and ensure that if this happens again they have the ability to recover from it very rapidly. They are, after all, reliant on their userbase for their income and they have failed a major test.

Looking at Home Storage systems - OpenFiler

2011-09-02T16:27:00.001+01:00

Like many geeks I've got a considerable amount of storage at home - currently, that's around 7TB split between various storage systems and it doesn't include space provided by Hard Drives in various bits of hardware I have scattered around the study.

After HP introduced their Microserver offer I decided to get one simply to throw 4 x1TB Hard drives I had floating around into it just to provide additional storage and I thought that Openfiler booting from a 16GB USB stick would be perfect for this job.

It wasn't. I've found that openfiler is a bit cumbersome to get to grips with, the menus don't link from one section to another. So, for example, if you want to create a Windows share it's not obvious where you go - there is nothing for Windows or CIFS on the main menu and once you find it you'll often find that there is a pre-requisitie you need to configure first and thats on a different menu so you have to start again!

None of this would be a major issues except for one thing.

It's slow. Really, really painfully, awfully slow.

I know that the Microserver ships with just 1GB of RAM and that an openfiler system really needs 2GB minimum but menus should not take 5 minutes+ to respond to a click and I notice that other people have been complaining about the same issues.

So, for me Openfiler works, is clunky and slow. I'm going to replace it this weekend with FreeNAS.

The issue with antivirus software (2)

2011-06-30T21:44:00.001+01:00

News has emerged of a new botnet set up that is trying to be indestuctible thanks to hiding in the MBR. According to the BBC article 'Code that hijacks a PC hides in places security software rarely looks and the botnet is controlled using custom-made encryption.' then goes on to say The virus installs itself in a Windows system file known as the master boot record. This file holds the list of instructions to get a computer started and is a good place to hide because it is rarely scanned by standard anti-virus programs.

Excuse me? MBR viruses are not exactly a new thing. They existed back in Novell days and it was a pain because you'd have to shutdown netware to get to the DOS area to fix the damn thing. To me, this is yet again pointing out the flaw of AV software. It's being lazy and not doing it's job properly.

AV Software is basically arse about face. It scans for things that SHOULD NOT be there whereas it should be scanning for things that SHOULD BE there and considering everything else a threat. It really shouldn't be too much difficulty to have a database of common windows files and the most popular applications/games/utilities in use today along with MD5 hashes and scan against those to ensure the integrity of the system.

Both Vista and Win7 go some way to doing this with things like UAC but UAC needs to be a little more friendly and more granular to configure. If UAC could be configured to stop things editing start up locations without user consent and from modifying key system attributes then Anti-Virus software could start it's very welcome decline into obscurity.

The issue with antivirus software

2011-06-07T16:13:00.000+01:00

I hate anti-virus software. I really do hate the stuff. This is not a mere dislike but an actual hatred.

The reason for this is quite simple. In IT security terms any security you deploy needs to do it's job with minimal fuss. Too much fuss and the security system outweighs its usefulness and after many tussles with anti-virus software I have come to the conclusion that AV software is a waste of time.

AV software is still far too reactive. It absolutely must have the latest definition files to have any hope of finding anything bad trying to infect the machine and even with all the heuristics switched on they don't seem to have much luck.

As an example, I do all my web browsing in a sandbox thanks to a nice tool called Sandboxie. This tool allows for a sandbox to be created which will contain any downloads, requested or otherwise, in the sandbox. This means that if a virus gets onto the machine it'll be contained and this exact scenario happened to me not too long ago thanks to a mistyped URL. Examining the contents of the sandbox I saw a very suspicious file which I submitted to VirusTotal. The results from that site are below.

Only four anti-virus programs all with the latest definitions actually spotted a harmful file. The others would have quite happily allowed the application to run and wreck havoc. Not good at all.

It is my belief that the best security is no longer in anti-virus software but in applications which prevent suspicious activity just like the UAC tools Microsoft are now introducing but this technology needs to go further and it should be possible to have as part of the boot process a system which scans active files to ensure that no changes have happened since the last boot and if required revert or delete those files.

Along with these systems I firmly believe that production computers, that is, office computers with email and corporate applications need to be locked down much tighter. Server hardening and desktop hardening need to move forward and better security is needed for portable devices so that they can only work on specific systems. The whole desktop security culture needs a huge revamp and anti-virus software needs to be consigned to the same bin as the floppy disk.

Cloud Computing - Amazon Outage

2011-04-22T14:59:00.000+01:00

Of course, just as I extol the virtues of cloud computing and talk about how much I've moved into the cloud Amazon suffers an outage. Opps.

Well, yes and no to the opps. A lot of people have been saying that as it was one of five Amazon datacenters that suffered this outage systems and services should have automatically recovered at another site.
Well, that's not true. you have to remember that Amazon operates it's data centers just like virtual copies of a real datacenter.
What I mean by that is that if you have a service you host in your own data center then lose that data center you'll lose the service. It's up to you as the admin/developer/owner of the service to make sure that you have redundancy set up in another location be that another Amazon datacenter or a datacenter under your own control.

As I said in my previous blog posting, cloud computing is not a panacea and you have to be careful how you use it. This outage is a classic case in point of that comment.

Renaming local administrator accounts - good or bad?

2011-04-20T13:39:00.000+01:00

A lot of the time I hear the following statement 'Renaming the local administrator account makes it secure'.

No, it doesn't. Renaming the local administrator account just leaves you with a renamed local administrator account. It only makes it secure from people who are too dumb to read SID's but overall adds very little in the scheme of security.

In Windows, the local administrator account, no matter what it is named will always have a SID ending -500. Guest is -501

With that information and a couple of tools you can list out the local accounts, find the administrator and attack the account. Of course, if you have physical access to the hard drive and the drive doesn't use any form of encryption there are plenty of password reset tools out there.

Moving into the cloud

2011-04-10T15:29:00.001+01:00

Without realising it I've found myself moving more things into the cloud. I'm not exactly reliant on the stuff that's in the cloud but I'm certainly using more services out there and it would be an inconvience if I lost those services. I guess you could say that the cloud has been creeping up on me.

It started out just after I got married. The photographer used digital media for the wedding photos and provided them on a DVD. Obviously these needed to be stored somewhere safe and the idea of keeping the DVD in the house where I could lose, throw it away without realising it (would be hard but this is ME I'm talking about) or potentially lose it in an accident/fire/theft not things you want to think about but you must when you are talking about this sort of data.

So, I started looking around for offsite storage. Inititally the thought was to rent a location or something and leave a copy of the DVD there. A bit like a safe deposit box but with easier access then I came across Amazon's S3. This is cloud storage. Absolutely massive cloud storage at that with unlimited space for the user - you pay for what you use and when you look at the amount of data you hold that you really do need backed up it isn't that much.

I have a rule for backup data, I'll only back up data that can't easily be recreated or downloaded. So documents, excel work, password databases, etc, etc and I do this on a monthly or semi-monthly basis with the occasional ad-hoc backup for something specific. So far I'm paying just a few dollars a month for the service and that translates into less than £5 a month. Is your data worth that?

Alongside S3 you've also got Amazon's EC2 (elastic cloud compute) basically virtual servers that you can use. Amazon give you administrator or root access to the machine and lets you get with it. Whilst the server is on you are paying for it. Whilst it's off you pay for the storage. This provides a really nice environment for scenario testing or for externally hosting something a web provider won't allow or like. For example, I'm using an Amazon EC2 service to host a Quake server - just for experimental purposes you understand!

Finally, the last cloud enabled application that I've found that I can't live without is DropBox. For me, this is a killer cloud application.
Dropbox provides an online file storage solution. That's all it does but it does it in such a clever and useful way that it's now invaluable.

All you do is install it and by default you get 2GB for free which you can see via my documents/my dropbox. I've got this installed at home and at work and what it means is that I can drop a document into dropbox and have it available for viewing/editing at home.

This avoids all the complications of having to remember to copy a file to a USB stick and of taking (and possibly losing) the USB stick on the train or in the back of a Taxi. It's out there in the cloud.
The way dropbox works is simply to sync everything in the My Dropbox folder back to the dropbox servers. this means that it'll even work when offline and simply sync the files up when you have an internet connection available again which is an invaluable method and what Microsofts Briefcase and offline files and folders were supposed to have provided.

There have been a lot of questions around security though - i.e. how secure is dropbox and my response to this is simple - it's in the cloud so you need to be careful. Do not put any confidential data on it or if you do encrypt before hand.

Like anything cloud computing is a nice idea and can be used for many things but it's not a panacea and you need to be careful with how you use it.

Windows 2008 Course this week

2011-03-28T11:33:00.000+01:00

On a Windows 2008 course this week. It seems to be a rehash of stuff that I've covered before but it's a week out of the daily office grind. The tea on the course is disgusting and hopefully I'll learn something....

Which reminds me. I really should update my PXE article.

New goal - We want to be CMM level 3

2011-01-25T11:49:00.000Z

For those who don't know CMM stands for 'Capability Maturity Model'. It's a set of standards designed to help an organisations software processes to the point that a software project is easily repeatable. What this actually means is simply the common sense practices will take place instead of the normal panic.
So, right at the start of a software development project proper specifications would be provided, changes to the spec would be monitored and, if required, the schedule for the project will be changed based on known schedule impacts due to schedule changes.
All of this is based on existing projects and on one key item - lessons learnt which should always take place towards the end of a project and be recorded and reviewed for the next project.

CMM levels have next to no bearing on IT management processes. For the IT side of things you have ITIL.

I can sort of see what they are trying to do here, like any set of standards CMM is pretty much common sense.

Briefly, CMM breaks the software development process down into these levels:

Initial (chaotic, ad hoc, individual heroics) - the starting point for use of a new process.
Managed - the process is managed in accordance with agreed metrics.
Defined - the process is defined/confirmed as a standard business process, and decomposed to levels 0, 1 and 2 (the latter being Work Instructions).
Quantitatively managed
Optimizing - process management includes deliberate process optimization/improvement.

So the basics of CMM can be applied to any process but it's really designed for software development.

Another thing that made me both laugh and cry was when I asked about the 'level 3' were supposed to meet and to confirm that it actually was CMM level three I was told by the meeting chair that he wasn't exactly sure when measurements were being used but knew that by just doing the DR processes for each site we would be granted level 3.

Of course, as soon as the documents are written they are out of date. We also have several data centre moves coming up in seven months so I feel an exercise in futility coming on.

If you are having a meeting please have an agenda and control your meeting

2011-01-14T06:00:00.001Z

One of my big complaints is that many people do not know how to control a meeting. You see this more in meetings where outside consultants can hijack the meetings and steer them away from the original meeting points because they can and they see a chance for more business.

The key to running a good meeting is to have an agenda and stick to it. It's actually very hard to do this because as a meeting chair, you don't want to be rude and tell someone to keep to the agenda. Guess what? As meeting chair it's ok to do this and others will thank you.

I'm going to borrow a tip from a colleague:

Everything is either:
   1. An action (write down who for, what and deadline)
   2. An update / fact (write down for future reference)
   3. An open question (see (a))
   4. Something not currently relevant (write down under a section titled “park” or (“to come back to)

Number 4 on that list is probably your most powerful tool as a meeting chair. If someone starts to go off on a tangent you let them have their say for a minute or two, make a note of it then say 'OK, That's interesting and has been noted. Let's move on'. If they come back to the point you just point to the note in the meeting minutes and say 'We've noted that as a parked comment, moving on'.

This way you can stick to the agenda and keep the meeting focused.

Exporting from Exchange mailboxes to PST files is a false economy

2011-01-11T10:34:00.001Z

I'm sure most network admins have come across that one user who has a huge multi-gigabyte mail file that absolutely cannot have a single email deleted from it.
In these instances the answer many IT departments fall back on is to export some of the data to a PST and put the PST file into another location.

This is a false economy. Please avoid it!! It's a false economy because you'll end up taking up more space. Exchange and most other mail systems have a pretty neat feature called Single Instance Storage. Put simply, if I send a 5MB email to 100 people it'll take up 5MB on the mail server as 100 pointers to a shared object will actually be sent. Total disk space used? Maybe 10MB for the mail, pointers, sent/received information and other overheads.

If those 100 people then export that mail to a PST file it'll take up 500MB (5MB*100) and that data is probably on the users home drive, local drive, pen drive, somewhere else so not only have you lost control of the location of what really could be sensitive data you've also lost the single instance capability of your mail system and wasted space on other systems.

If the user can't be persuaded or taught to delete emails then I'd be inclined to leave them be. At worst I'd give them a storage instance all to themselves. If you are hosting data on a SAN or NAS you should be able to move those storage instances around to best balance the disk space requirements without sacrificing single instance and having some protection around keeping confidential mail items in one place.

Next 6 months targets

2011-01-08T02:10:00.001Z

At the start of the year the company I work for sets targets for the next six months. These targets often have no bearing on the overall strategy or direction of the team but are down some managers whim or based on a technical article they read that morning.
The targets also often don't follow on from one 6 month period to the next.

For example, one of my targets for the previous 6 months was to dcoument suggestions for a re-org of the OU's in Active Directory. One wasn't actually necessary but upper management caught site of the AD layout and thought it looked a bit untidy. They never asked any questions or sought to understand the layout or even ask if a reorg of the OU's would bring any benefits.

Anyway, one nice result of this was that it was an easy piece of work yet for this bunch of 6 months work I don't actually have 'Implement the suggested OU re-org' as a target which strikes me as just odd.

In the next six months I have to write a document providing options for green computing in the data centers and write another exploring the option of using Windows 2008 HPC in place of Windows 2008 Standard never mind that the licence agreement for HPC specifices that it shouldn't be used in this way.

Sudden Truth

2011-01-06T14:25:00.000Z

It's a rather sad state of affairs when you realise that the greatest achievement you can claim in a day where a meeting wasn't involved is going to the toilet.

Problem Ownership

2010-10-19T15:22:00.001+01:00

I've been helping out with some issues found on some development workstations. Fairly simple things like the user not being able to change wallpaper and stuff like that. When the image leaves the team that do the images they are quite locked down but as these are going to be development we want them unlocked a little.

Simple enough, Stick them into an OU, add in a GPO that unlocks the setting, job done.

The problem occurs when something crops up that not easy to fix in a GPO or not something we can fix in a GPO. You see, this is where it all gets convoluted because the team I'm in can only set GPO's for development machines and even then it's computer GPO's and not user GPO's - I'm sure some are asking if there is a difference? and the answer is oh hell yes.

Anyway, had a classic of a problem today. the group everyone was denied permissions to c:\documents and settings. Not pretty. The troubleshooting went a little like this:

User: "I get access denied to c:\documents and settings"
Service desk: "You have a development machine?"
User: "Yes"
Service desk: "not our problem, talk to the R&D GPO people, its a GPO issue"
Me: "How can it be a GPO issue? We don't have any GPO's that muck around with permissions"
Service Desk: "GPO Issue"
Me: "What about the Corporate images? Do they have the same problem? If not then it's not GPO but a user issue but a regular service desk call"
Service Desk: "GPO Issue"
User: "I just want it fixed" (Understandably)
Me: "We don't own these images and are just lending a hand, I can change the permissions if that'll help?"
Service Desk "YES!! Via GPO!"
Me: *headdesk*
Me: "No, as a one off but we need to have a look at the image"
Service Desk: "GPO Issue and we are not allowed to talk to the team that makes the images"
Me: "Excuse me? Why not? If you find a problem with the image what do you do?"
Service Desk:"Reimage the machine and part of the contract states we can't talk to the imaging team".
Me: "..."

So now I'm talking to the imaging team or rather I've sent them an email. It seems that our wonderful contract with the service desk doesn't actually allow them to own a problem so they just shove it away or just reimage a machine knowing that it's a temporary fix (and netting them £350 in the process).

*sighs*

This is the most important project we have ever done

2010-10-14T22:42:00.001+01:00

I'm sure those words have been echoed in many IT departments up and down the country.

With those words at the start of a meeting I was under the impression that "The most important project" would actually have some PRINCE-2 methodogy behind it but six months in to the project and I'm being told that no one has actually documented the requirements, set the win conditions or anything else so now there are a group of people running around trying to deliver something that is the most important project we have every done with with no clear 'win' to end the project on.

Maybe it's the suspicious part of me but I really can't imagine that the people on this project haven't thought of this? Surely, they must have asked for requirements and all that? I can only surmise that it's being done so that those at the 'coal face' part of the project can be refused bonuses as the 'work didn't met the set criteria' which will, of course, be written after the project to make the managers look good and the staff look bad.

Did I say I was paranoid?

LASIK Part Two

2010-07-09T12:47:00.003+01:00

The post surgical advice the laser specialists give you is to go home and sleep but I found this rather impossible to do. My eyes felt like I'd been standing in the middle of a circle of smokers and I couldn't rub them, they also ached and were very bloodshot, they looked like I'd pulled a all night drinking session but didn't feel tired so I couldn't sleep.

My vision was hazy and it was difficult to keep my eyes open because the light hurt a little so in the end I plugged in headphones and listened to LBC radio over the internet. This worked well because it's a chat station so it was easy to mentally get involved with the discussion and keep my eyes closed. About 9pm that evening my appetite started to come back so krys (my wife) very kindly bought Pizza.

It must have been something of a surreal sight, sitting there on the bed, curtains drawn, lights out, munching pizza listening to radio over the internet :) The blurb says that 2-4 hours after surgery your eyes will feel better. Mine took 5 hours but the change at 5 hours was remarkable, I could see, my eyes ached a little but as long as I avoided bright light I was fine.

For the first seven nights after eye surgery they give you a funky eye mask to wear. This is designed to stop you accidentally poking or rubbing your eyes. It takes a few goes to get it sitting comfortably but it works well and that evening I had a good nights sleep even in goggles!

Another thing to note, for the first 7 days you are on a cocktail of eyedrops, anti-biotic, anti-inflammatory and artificial tears. I must admit that I never thought I'd need the artificial tears as my eyes used to stream at the drop of the hat but since laser surgery that's pretty much stopped.

The next day I was able to get up, use the computer for a little bit and walk around with near perfect vision. I was impressed, after 20 minutes of using the computer I had to walk away though as my eyes were still a bit light sensitive. It was at this point I noticed something wrong, my distance vision was perfect but my close up vision wasn't right.
My right eye was fine but my left eye hard marked double-vision which caused me slight panic until Krys very kindly slapped me on the head and told me to calm down as it wasn't even 24 hours post surgery. Also, I had a follow up appointment that day so it was best to see what they had to say.

That afternoon I was able to walk from home to the local optical express (about a km) although I needed sunglasses even though it was cloudy. My eyes were still very light sensitive.
At optical express I was seen by the same person who did the pre-op assessment and I have to say that throughout the whole process she has been awesome.
So, This first post op assessment was quite positive with the exception of inflammation and dryness. The inflammation being caused when I squinted just before they created the flap. I was told to use more of the Pred Forte (anti-inflammatory agent) in the left eye only and to use a lot more of the blink drops (artificial tears). The good news was that there were no striae (bubbles in the cornea due to flap movement) and an eye test showed that my right eye was 20/20 whilst my left was 20/30 - not perfect but a lot better than before surgery.

The next appointment was supposed to be a week later but was set for the end of the week as they wanted to check on the eye dryness.

Anyway, rather than go on about each visit to the optician I'll summarise it quickly. In two months I've been seen 4 times with a 5th appointment due in another three weeks or so. One appointment was cancelled due to the optometrist getting caught up on the M20 after a very nasty accident but it was rescheduled for the next day.

The second visit showed that the anti-inflammatory had worked but I still had slight ghosting in the left eye. This was due to excessive dryness which was still causing problems so I was given some gel to use of a night to help keep my eyes moist.

2 more check-ups on and I'm finding that my eyes are still light sensitive but they always were prior to surgery so that's not a big surprise, they are not as dry as they were but still dry so I make sure I've got the artificial tears on me at all times. The blurb says that your vision will continue to improve and it's true. 8 weeks on and I suspect my left eye is 20/20 now so I'm looking forward to a follow up eye test to see if it's true. The follow up care from Optical Express has been second to none. The apologies (and free cup of tea) when the optometrist was stuck in traffic was lovely and they have been more than generous with additional blink drops and other sorts of eye gel supplies.

The two hardest bits for me were finding something to do the evening after surgery and not being able to get my face wet in the shower or wash my eyes in the sink for the first seven days.
Your eyes and eyelashes need cleaning after the eyedrops because they can be messy, the way you have to do it is with a face cloth and kettle boiled water (let it cool down obviously!) then you VERY CAREFULLY dab your eyes and lashes - It works but it is cumbersome. The reason for this is simply because you can't risk any sort of bacteria taking hold on the eye or worse under the cornea.

So that's about it for my experiences. The first evening was hard going, the next day better and since then my vision has been perfect in my right and near my perfect in my left and improving. Not having glasses has given me a whole new way of looking at the word and I do mean that literally, I no longer have the frames of the glasses in the way of my vision, my night vision and peripheral vision are incredible. I did have halos for the first few weeks but they faded quickly and as I type this its very hot outside so I know I'm going to need additional eye drops, especially in my left eye, and I'm making sure my sunglasses are close to hand but that's it, the procedure, for me at least has been a resounding success and I am very grateful for the opportunity to see without glasses.

So, one last question - Will I need glasses again?

Insufficient information to answer that. There is a chance that as I get older I'll need reading glasses. This is nothing to do with laser eye surgery it's just a fact of life and something to do with the back of the eye changing so it cannot be fixed with a laser.

What about short-sightedness? Could it return?

Yes it could. That's also a fact of life. My eyes should stay at 20/20 for 10-20 years, after that they might deteriorate again, if they do I can get them lasered again for no additional cost which is a very fair deal to me and depending on my age at that point I might not bother but that's something to decide in the future.

LASIK Part one

2010-07-09T09:43:00.006+01:00

Two months ago I underwent LASIK eye surgery to correct my short-sightedness. Several people have been asking for more details and what the downsides of it all are so I thought it might be easier to write it all down and point people here.

First of all, I'll provide a little bit of background. I started wearing glasses when I was about 11 or 12 and over the years I became more dependant on them to the point that I couldn't see a thing without them. Before I had LASIK I was -4.75 in my left and -4.5 in my right. My eyes were getting worse at the rate of around -0.25 every four years.

I was one year into a new prescription and on leave when I spotted an offer from Optical Express to go for a free laser eye surgery consult so I decided to go along and see what they had to say.
The consult itself was an interesting experience with my eyes measured every which way I could think of and a few I couldn't.
Part of the process included putting eye drops in that forced my pupils to dilate fully and stay like that for a few hours, going home after that wasn't fun because there was far too much light coming into my eye but that discomfort only lasted a few hours. The tip here is TAKE SUNGLASSES!!

So, with the consult out of the way and with a rather expensive bill for the actual LASIK procedure I had something to think about. The actual cost was nearly 10 times more than the adverts promised but it was because my eyesight was so bad and my pupil size so big. This meant that the only option I had was all laser LASIK with wavefront guided technology, this wasn't an issue as I was already only interested in all laser LASIK. I'll explain the reason in later entry when I explain how the eye is shaped during surgery.

Anyway, after a discussion with my wife about the procedure and the cost it was duly booked 2pm in for the Bluewater shopping centre which is about 15 miles from where we lived. The nice thing about Optical Express is all the follow up work could be done at my local Optical Express, only the surgery had to be done in a specialist centre.

So, the day comes and Krys drives me up to Bluewater, we have lunch.... rather she has lunch as for some reason my appetite has deserted me then the clock rolls round to 13:55 and off I go...

Now at this point I was half expecting them to take me in, shoot me with lasers and I go home but nothing is that simple I end up having to fill out a few more forms and sit there for half and hour.... and it was that 30 minutes which was a total killer, all thoughts going through my head about the bad side of LASIK, the warnings and I'm almost ready to make a run for the door but I manage to hold my cool and I'm duly summoned.... to have my eyes checked, again...
Turns out that one of the scans they took of my right eye wasn't good enough for the surgeon so it was to be repeated.
These scans are vital though as they show how light enters the eye for the wavefront laser to do it's job.

15:00 rolls around and I'm summoned in to speak to the surgeon who asks me if I fully understand the risks and if I'm ready to do this. A voice at the back of my head squeaks 'no' but my voice, probably an octave or two higher says yes and then it's time....

I get lead into the theatre and the room is BRIGHT white, I mean the sort of white you get in a Star Trek episode or something. it's GLEAMING. I get introduced to two people, one is a nurse the other an assistant laser specialist. I get to put on a funky hair net and they lay me down on one of the most comfy chairs ever.

Then the laser surgeon arrives, my heart rate goes up a notch and at this point, it's all down to business. Some eye drops go in to numb the eye, an eye patch goes over my left eye, something metal goes over my right eye, there is an INTENSE pressure on my right eye and I'm seeing all sorts of funky colours which I guess is down the pressure of this metal contraption pressing down hard and causing nerves or light cones to fire off randomly. It doesn't hurt, it's just a little odd an uncomfortable, they roll me to my right and tell me to hold very still and look at a white light, the light flickers and that's the flap in the cornea created. I think it was the assistant surgeon that then moved something over my eye which pulls back the flap and my eyesight goes straight to shit, everything is splodges of colour and at this point I'm grateful to whoever it was who said "you won't see much, don't panic it's normal" because my left eye is covered and my right eye is a blur.

Then I had the most disorientating thing I have ever experienced, bear in mind that my left eye is covered, my right eye has no cornea covering the pupil so I've no vision in my right, just blurs and then the damn world MOVED. Turns out the chair I'm laying on rotates between laser devices but I didn't know this, all I sense is the world moving and I've got no visual reference for it so I'm gripping the chair in a death like manner until I realise that this is normal and I'm not falling...

So, I'm now pointed at another laser machine and the assistant surgeon tells me to relax and look at the orange flashing point to which I respond "which one?!" as thanks to my now very blotchy vision I can see about 9, the assistant surgeon tells me that it doesn't matter and once again to relax so I try very hard to but it's a little difficult. The surgeon tells me that I need to hold very still and that there will be a smell of burning which is normal as it's just a byproduct of the machine, others have told me that it's the smell of the surface of the eye being ablated but I prefer the byproduct thought. Anyway, the surgeon puts his hand on my head quite firmly.
I'm once again told to relax then there is a loud clicking and the orange light pulses, in the background I hear the nurse count down from 30.
Now, what's amazing is as she goes '3...2....1' my vision clears, from 9 dancing orange spots to one very well defined one - already my vision is far better...
They move the chair again, something passes over my eye and then some eye drops are squirted in. My right eye is done.

The process is repeated for my left eye and I won't go through the details for it suffice to say that's it's almost the same. When they created the flap apparently I squinted a little and put too much pressure on my left eye which caused some temporary damage and swelling but I'll go into a little bit of detail about that later.

The whole process for both eyes took no more than five minutes, was uncomfortable, not painful and was performed by some really friendly people. Once the procedure was done I was lead into a darkened recovery room and told just to relax. In the room were a few paintings which I could SEE without glasses... amazing.

One of the nurses gives me a little goody bag of eye drops and I'm told very firmly "DO NOT rub your eyes". My eyes feel very gritty and there is a film over them which is all quite normal.

So, I give my wife a call, she comes in to collect me and drives me home, at this point I realise that Optical Express is right across from a store that sells... lamps, which are on, causing all sorts of halos and weird optical effects right in front of me. Who the hell thought it was a good idea to put a lamp story across the concourse from a Laser Surgery facility?!

The drive home was interesting as I couldn't look out the windscreen because it was just all too bright, the few glances I did steal were awesome because I could read car number plates without glasses!

That's probably enough rambling about the procedure, I'll cover the aftercare and some other things in later posts.
In summary it was one of the most uncomfortable and disorientating experiences of my life but it was very quite, certainly painless with results that I could literally see seconds after surgery.

Exploring the Job Market

2010-06-30T15:07:00.002+01:00

So it's that time of year again, appraisals - the part where you sit down with your manager, he tells you just how useful you are and how you don't quite get a bonus because you never exceeded the goals set despite the fact that exceeding the goals set would require changing the laws of physics. Part of this cycle of horror is "career development" which always makes me laugh as, so far, I've never been allowed to 'stretch myself' to the areas that I want to try my hand at yet every 6 months we go through the same dance...

Anyway, with the advent of the Appraisal coming I decided to start looking at the job market again an found that most of the ads were the same:

"Seeking person to look after servers, storage, VMware, service desk, support calls, other people, animals, wash boss types car, work under pressure to tight deadlines. URGENT!!"

And there it is. Three phrases I hate:

"Work under pressure" - basically means the boss will tell you that something needs doing five minutes after it should have been done. On a Friday. at 16:55.

"tight deadlines" - This is a negotiation that happens up and down the country. Boss type says project x will require six months. His boss says "great but you have 3 months" which then gets passed down to the team with some hokey speech about teamwork, pulling together and camaraderie.

"URGENT!!" - I hate this word more than any other in the English language. In nearly 16 years of IT work I've come across a handful of truly urgent situations. Most of the rest are because someone didn't plan, panicked and needs to cover it. Any job ad that shoves "urgent" in there is (in my mind anyway) incapable of planning for even the short term.

Depressingly most places seem to use the above three phrases interchanably. It seems that IT jobs are not real IT jobs unless those three phrases are used.

I think that maybe I'm getting just a little bit jaded with the IT sector these days as for something that's considered an Engineering discipline it doesn't seem to be maturing in the same way that the engineering sector has.

New Phishing scams in progress?

2010-04-22T10:20:00.003+01:00

I think we've all had those emails from the Banks "security" department and which threaten all sorts of mayhem and chaos if you don't go to the site right now and type in all your security details which then gets sent to some dodgy bloke in an internet cafe in deepest scam land, well, yesterday I saw a new twist on it in the form of two emails.

The first was from an IT recruitment agency and it was the normal "we are wonderful, we are awesome, send us your CV" and I didn't think much of it until I saw the email address under the link - it was a yahoo mailbox with what looked like one of those auto generated addresses.

The second came from "undisclosed" and was talking about a webmail upgrade. My hosting provider recently upgraded their webmail portal so at first I thought this was part of that work until I saw that it had been through a mail server in the ukraine then when I read the email more closely I saw that they were asking me to reply to them giving my email address and logon password. Now, my hosting providers security is atrocious as they use the account password for verification purposes (which must mean that staff can see it) so I can easily see people getting duped into sending this information. Of course, the reply to address wasn't a legitimate address and so was another phishing attempt.

I'm now wondering what other scams are out there... I'm also thinking of changing hosting providers due to their terrible security policy.

Common questions - Why doesn't NTP set the time?

2010-04-09T13:01:00.000+01:00

Because that's not how NTP works.

NTP will take the machines current time and compare it against a reference server (or servers) and then work out how much of an offset to apply.
If your machines clock is only half a second or so out then NTP may well make just one change to bring the machines clock into line with the reference server but if it's more than than then it will apply clock slewing to gradually bring the machines clock into line with the reference server.
In testing I've seen NTP on a a machine with 8 seconds lack from the reference time server take 15 minutes to slew to NTP time.

Why doesn't NTP just set the correct time in one go?

Because of our old favourite - security.

Lets imagine a scenario - User A logs on to the network at 09:00:00 (according to NTP time). His workstation is 10 seconds fast so according to his machine the logon is at 09:00:10 - this doesn't cause any issues for authentication as it's well within the allowed time range.

User B is an evil user. He has just captured all of User A's logon traffic.

NTP now corrects User a's machine clock to be 09:00:01 which is the same as the time on the NTP server.

User B replays User A's logon request. This request includes a timestamp. The authentication server (lets say a Windows Domain controller) accepts the request and compares timestamps, this new request from User B will APPEAR to be later than the original from User A and so the domain controller grants access.

The above is a very, very, very simplified view of domain logons, in the real world it wouldn't actually work due to how times are recorded and how password authentication actually works but it provides a simplistic view of why time skew is in place of a forced change.

Offsite Vendor presentation meeting

2010-03-29T13:01:00.001+01:00

So I've been meeting most of my targets and have been fairly organised work wise and this has now come back and bitten me because I've been told that I need to go to a vendor presentation meeting tomorrow miles from the main office and that it starts at 09:00 SHARP! (There was emphasis on the sharp bit, it's not me doing it for effect).

So, it's a very early start, trapse across country to hear a salesweasel enthuse about something I really don't care about and don't even have much knowledge with. The only think I can do for my own amusement is think up awkward questions and see if he (because it's always a him, no sexy nible women to push the sale unfortunately) and enjoy the 'free' lunch.

In other news, very soon I should have 168TB of shiny SATA disk to play with. This is far in excess of what we really need but it'll help collapse 20 disk shelves of 72GB disks into just 3 disk shelves of 1TB SATA. I really, really love SATA disks and think of the improvement in sizes, 72GB which were state of the art 6 years ago to 1TB today. It's quite incredible.

The drawback here is that of spindles, more data on less spindles means that different people could cause the filer to fight over access to the data. This can be fairly easily mitigated by dividing the aggregates (the physical disks) in sections that will have specific usage patterns.
In many ways I'm looking forward to this project as it will really help consolidate a lot of the NetApp filer works I've been doing and provide a lot of room for growth.

Catching up

2010-02-01T13:01:00.004Z

I've been a bit remiss in not updating my blog for several months. A few combinations of a sudden urgent project, the wife having knee surgery and a flood at home all conspired to steal a lot of my time and then, of course, the inevitable run up to Christmas with all the fun and games of Christmas shopping in the snow.

I have been thinking about what to do with the blog, should I continue it? should I change it or should I just close it? I'm not playing with enough new technology to bring anything wonderful and astounding to the site and my time is ever more limited but I'd like to keep the blog going with a few war stories and other things.

Thoughts?

Upcoming Week

2009-09-28T08:10:00.005+01:00

I already know it's going to be one of "those" weeks, but then I guess it always is, there is always something that turns a nice quiet, productive week into a nightmare. This week hasn't even started properly yet and it already has the feel of problems.

Put simply, this week is going to be all about shuffling data from one location to another, the sheer joys of being taken over and having to comply with another companies standards. It doesn't help that our permissions are a total disgrace. If you think of the standard rules of users into groups and assign the groups permissions.. well, we haven't done that. we don't even come close so permissions changes are proving to be a stumbling block.

We also have a subsidiary company that has been bought out by another company. Data transfer to the new location is proving to be another interesting experience, I cannot, for the life of me, understand why people cannot get organised and actually work out before hand what it is they need and so this is leading to a lot of last minute requests... situation normal I suppose.

Add this to the fact that Netapp have discovered another bug in their firmware this time and this bug is a good one. It seems that only on the 3070 cluster the bug will result in a reboot of the filer leading to a panic. Nice. I had this happen to one of the filers in Cambridge and now it seems that the only way to fix it is by being onsite with a laptop and a serial cable....

More updates coming soon and I'm thinking of changing this blog into more of a war stories blog with the occasional bit of technical information rather than leaving it a few months between updates.

TaskCoach

2009-07-07T12:01:00.003+01:00

There are a lot of little tools out there which can allegedly help you get more organised, better with time management, etc, etc. Personally I find most of the tools are more awkward to use and so consume time that is better spent on actually do the stuff that needs doing.

However, I did find TaskCoach which is a simple enough .exe file that I can around on USB stick, any task that needs doing I stick into there. It's possible to organise by category and to set deadlines on. It's short of a few features I would like, for example, no software seems to get the idea of chained-tasks, that is where one task has to be completed before another starts so as soon as you flag the first task as complete the second appears.
TaskCoach does have quite a powerful category system for organising items into different groupings and I'm using this to organise work tasks by week number. This is a very useful way of seeing what I've got on for the week and what can be deferred to a later week.

All in all it's a lovely little tool. Now, all I need is a checklist tool.....

Why change control can be a Bad Idea

2009-07-05T21:28:00.001+01:00

I'm sure that many people have had to endure the torture that is a change control process. In short, change control is a process whereby changes to a system have to be approved by a change control panel.
Generally, the panel is a group of people who probably don't know about the system and/or don't much care about it so you have to be quite vocal in why and change might be needed.
The actual ideal behind change control is to moderate changes to a system in such a way that should a system fail or have problems it should be possible to use the change control tool to work out what recent changes had been applied and undo those changes or research/test to see if those changes could be the root cause the issue. Sounds ideal doesn't it? In reality it never works that way.

In the long run a change control tool can actually do more damage that it's designed to prevent. How come?

Let's say that you have a website which has a fairly minor bug. Let's say that you know that the change control process will take two weeks to follow it's winding path and that you will need to invest about four hours to write and represent what is, at best, a 20 minute change.

What do you do?

Do you spend the time and fix the bug or do you forget about it and press on with the several hundred other things that's on your list?

Let's say you pick the second option (and I don't blame you if you do because I've done that) and several months later that small issue could explode to be a big issue.

And that's why change control systems need to be as flexible as possible otherwise what appear to be minor changes will be quietly shelved and in the long run that can lead to a major incident.

I'll provide some suitably altered real world examples in a future article.

Marriage!

2009-07-04T12:01:00.001+01:00

It's certainly been a busy previous six months. Not only the company take over but I got married last week.

The amount of work required for a wedding reminds me of one of those projects with an unmissible deadline except in this case the deadline really is unmissible!! Once a date is set, that's it. Full steam ahead until the big day and wow, what a day. For me, the big day was June 29th and it was too hot. Mind you, we had some lovely photos taken down by the river and then off for the honeymoon in Paris.

Now that I'm back at home things feel like they are calming down but I'm sure the next major piece of work isn't far away.... why do I have the feeling that my wife wants to have most of the rooms redecorated......!

Anyway, here is to you Krys.

Software is not a panacea - Part 2

2009-06-10T12:01:00.002+01:00

In the previous article I raised the fictional scenario of a company wanting to automate a timesheet submission process. In this article I'd like to touch on some of the project processes that would be used by the majority of companies.

Generally, most companies will start off with the sensible process of evaluating existing software packages, looking at what's out there and maybe even seeing what other companies use. After a period of time a sensible company will come to the conclusion that there is no one piece of software that fits their requirements and so their requirements must change as well as some processes. This is a key point as every company likes to think that they are unique and so around that uniqueness certain process have appeared so when it comes to upgrade or computerise those processes they are reluctant to change them.

However, back here in the real world most companies will do one of three things, they will

Abandon the idea
Buy the commercial package closet to their requirements and get it customised
Hire a developer to write a bespoke piece of software

Of the above three options the first is the best and safest but at this point many companies make another fundamental mistake. They never document the issues found or the reason for the project to be abandoned. This means that often someone else will reopen the project 6 to 12 months later, reinvestigate options and then select option 2 or 3.

Option 2 is an interesting one, surely there can't be much wrong with making some customisations could there?
Well, it depends. If the software is designed to allow those customisations then go ahead. However, may companies will want to alter certain business logic (e.g. maybe three people would have to approve a timesheet and the system, by design, only allows a maximum of two.
Quite often a company will purchase development skills and get the codebase changed to support what they require. This causes a problem when upgrades are required or if a security hole is discovered as often the customised verison will break when patches for the mainline system are applied if it's even possible to apply them at all.
Now the company ends up in a situation where they like and want the features in the next version but are tied to an old version due to the customisations, often they will have to face the choice of staying with the customised version, migrating to the new version or paying out to get the customisations in the new version.

Option 3 opens up all sorts of interesting possibilities for problems and complications to occur but I'll save that one for another blog

Software is not a panacea - Part 1

2009-03-24T12:01:00.000Z

There have been many times where I've come across situations where people have this mistaken belief that software will fix all their ills. Surely, anyone with a few years experience know that software is merely a tool designed to fix a problem the way the programmer/designer intended it to be fixed and not the way that you expect it to be fixed.

There is often a huge disparity in the way a company wants a problem to be fixed compared to the way the software actually fixes it and this leaves the company with three options, namely:
a. Change the processes to fit the software.
or
b. Change the software to fit the process.
or
c. Write their own software.

Let's take the classical scenario of a time recording and billing application. Let's say that company X records time on paper sheets which then get passed to finance to generate the bills and send out to the clients. Let's say that the company bills on 15 minute intervals based on a client code.

Obviously, the above scenario is crying out for some sort of automation, the amount of time and money that can be saved with an online tool means it's worth investing in the hardware and IT departments time to get such a system installed.

In the next article I'll take a closer look at the process many companies follow for such a project.

Gameforge supports theft

2009-02-26T12:01:00.000Z

Up until a few days ago I played ogame. This is a browser based online game where thousounds of people interact to steal and trade resources. It was the sort of game that you could spend 10-20 minutes on throughout the course of a day and provided a welcome respite from work.
My finacee also played it for similar reasons. She played for almost a year and I did for 2 and a half years.

That is, until the other day. Because I was helping out my financee (which is within the rules) and because she was on the same IP address (also within the rules) when the resources arrived (against the rules) we both got banned until 2036. Now, I'm not disputing the ban. We both violated one small part of the terms and conditions. The penalities for that are a permanent ban which seems a little draconian but that's how it is.

So, Ban in place I decided to ask for my money back as I've got 8 months left to run on the account only to be told to go away.

So, bewary of online games especially of places like gameforge who will happily take your money and then ban you for an infraction.

This to me is theft - a bought for service is not being provided, they won't transfer the subscription to another account and they won't refund me.

So, thanks to Gameforge I'll not be trusting any MMO ever again.

Company Takeover

2009-02-14T12:02:00.002Z

The company where I work has just been taken over by a much larger organisation. I'm not going to say where I work for now as enough has been plastered over the technical press but I'll have a few things to say on the matter much later on.

It's an interesting time for both good and bad reasons. Obviously the economic problems have hit both companies hard so the usual bans on travel and overtime have come into force. This presents a problem for us IT Department types who have been given a lot of work to do around the integration and are going to have to take time off in lieu for it. At the end of the year I think they will have many staff out as they will have leave that they will have to take. Hell, they will probably buy it off us!!

The current plan is to re-ip all the devices as the company that has taken us over use the same IP range as we do and this is right on the back of recently relocating them all to new data centres do yet more out of hours work and then fixing things that break.

The Chinese have a curse "May you live in interesting times" and for the next few months those times sure will be interesting....

Busy Project Time (2).........

2009-01-15T12:01:00.000Z

So, It's January 2009 and not much has changed, it's still busy project time. Unsurprisingly the server migration which absolutely, totally HAD to be completed by the end of December 2008 wasn't. How the project manager expected 79 servers and a lot of filer data to be moved in less than three months was beyond me but as with so many projects here the project managers are under the thumb from higher ups to deliver so often the way they do this is to pass on only good news to the higher ups whilst pressurising the staff to do the job no matter the cost...

Still, as the company is being taken over there is hope that this will change. Time will tell. With a little luck I'll be back blogging my normal nonsense very soon.

Busy Project Time.........

2008-10-17T12:01:00.000+01:00

Just when you think it's all going to be quiet and maybe it will be a good time to get those niggling little tasks out of the way and to be able to sit down and write some decent blog articles someone comes up with the idea of decommissioning a server room to save on power. So now I'm involved in a project that requires the relocation of about 4TB of data to another filer, including updating and moving the servers that use the filer data....

Yes, it's going to be a busy few months.

And a project manager just asked me if I needed any help installing IIS...... Sometimes I'd rather be doing anything else than working in IT.

Some DNS Tips

2008-09-15T12:01:00.000+01:00

Several times in just the past week I've had to deal with DNS entries that have made things a touch more painful than they should have been so I thought it might be time for me to jot down a few notes on how DNS should be configured to save IS people's sanity!

First up the DNS servers themselves. You should always have a primary and secondary which generally, speaking are two different DNS servers at your ISP's location. If two are not available you should consider switching ISP's. Personally, I use three. Two from my ISP and one from OpenDNS. This way, should the ISP change for any reason and/or should access be denied to the ISP's DNS servers I've got a third, totally separate service available to me.

Next up, A records. These should always point to the IP address of the server in question and they should always use the hostname of the server. Sure, this can lead to some unfriendly names but it's really handy to know the proper hostname of the server. If you want to use something 'pretty' then use CNames. When you create the A record make sure the PTR record is also created in the reverse look up zone. This way, when you are trying to work out what physical server a CName is all you have to do is a reverse lookup against the IP address.

MX Records should also have two internal/DMZ based mail servers which they can deliver to and a third at the ISP which can retry delivery to your internal servers at a later date.

These are simple tips and they (or variants of them) can be found as best practice advice for standard DNS configurations.

Understanding your environment

2008-09-08T12:01:00.001+01:00

A practical demonstration of why understanding your environment is vital occurred a few evenings ago when some NetApp filer\domino work went wrong. A little bit of background first, domino data is stored on a NetApp filer which is shared using nfs. This is mounted by the domino server and it all (most of the time) works.

For some reason this particular server running Domino (let's call him Bob) was showing high i/o stats, although the server itself was responding fine. The filer (Nutkins) wasn't reporting any problems but it was deemed that Nutkins had to be at fault. There are a lot of connections to Nutkins after all and in fairness the mount point is living in an aggregate that is unbalanced in terms of i/o profile so the decision was made to create a new aggregate decided for Bob. Simple enough to do. For those not filer aware an aggregate is a collection of physical disks. In giving Bob his own aggregate it dedicated 8 spindles to the Domino data. More than enough to remove any i/o bottleneck.

Now, Nutkins itself has a very cool piece of technology called snapmirror. A snapmirror was duly setup and Nutkins began copying the data to its new home.

So, the big evening arrives. The paperwork is signed (in blood, naturally). The changes authorised, the servers poised....... A hush descends and the commands to stop Domino are typed into Bob......... and Domino promptly hangs.

Red flag 1 - when a manager says "oh, it always does that. Just issue kill -9 and everything will be fine, well except that a few databses might be corrupt" it's probably time to start worrying. However, the final snapmirror is initiated and the last 140mb of changes are copied (in 22 seconds no less, not even enough time to get a cup of tea). The snapmirror is then quiesed and broken. This makes the destination for the snapmirror writable. Over to the unix admin and a few key clicks later the export is mounted and Bob was started.........
Or not. Seems that a small fact was missed. Bob not only has data stored on Nutkins but also has a local directory for crash dump logs.

Red-flag 2 - when Bob's admin doesn't know the configuration of Bob's setup it is probably time to start panicking. Anyway, a tappety-tap of the keyboard and the directory is created. Oh, lets stop and start Bob hoping red flag 1 doesn't pop up. Mr. Unix issues the command and on the screen "server shutdown. Bob_stop not found". Ok, so did it shut down or not? Ps -ef | grep lotus and nope, nothing running. Red flag 1 avoided! So, start Bob and..... Nothing. Not happy. Hmmm. Time to fail back, something isn't understood\not working.. So Mr. Unix does his stuff and...... No Bob. Seems red flag 1 corrupted the data then the final snapmirror copied corrupt data. Also seems that the shutdown script has at least one bug in it which causes a loop to fail when the script is executed.

Anyway, to cut a long story short we backed out and made the change a few days later. There are several lessons learnt here mostly revolving around documentation, standarisation and knowing your environment. I'll leave it as an excercise to the reader to work out the rest!

AD Find

2008-08-21T12:01:00.001+01:00

AD Find is the second of the two tools I managed to find in the same week. This little tool weighs in at just 700K for the download and about 2mb for the actual file. This tool does exactly what it says, it finds things in Active Directory. The clever part about it is it's possible to say exactly what you want to get back and the format it should be in.
As an example, a few weeks back I had the issue with Bindview not liking non-ASCII characters.

Now, the version of Bindview that's being used where I work is a very old NT4 only aware application which means it will update the SAMAccountName attribute but not the display name.

This isn't a problem as there is a workflow from an HR application which deals with all of that, all bindivew should be doing is delegated group permissions (and yes, I know it's much easier in AD but thats a war story for another time).

Anyway, I was curious to know how many SAMAccountNames didn't match up with display names so I used ADFind to display the CN, Samaccountname, mail, firstname and lastname fields in a CSV format which could then be processed by a filer in Excel. Much quicker than messing around with the native Active Directory tools.

AD Explorer from Sysinternals

2008-08-21T12:01:00.000+01:00

Sometimes it's possible to stumble upon a tool and wonder just how you would have gotten a task accomplished without it. Last week I had the good fortune to stumble upon two such applications right at the time when I needed them most. I did consider buying a lottery ticket that evening!

The first one is AD Explorer and it's from sysinternals and it's exactly what it says, a explorer tool for Active Directory. It allows viewing, searching and editing of the AD in ways that are far superior to Active Directory Users and Computers. I suspect the only thing that AD users and computers can do (or do better) that this tool cannot are password changes, logon hour restrictions and limiting logon ID's to specific computers.

One very nice feature this tool has is the ability to take a snapshot of an Active Directory and compare it to another snapshot. Doing this shows just how many changes occur in the AD in just a few days. It's also a great way to see how many differences accumulate between your production and test active directory environments.

Overall this is a fantastic tool and one I'll be using when the MS technotes require delving into some obscure key via ADSIEdit. I'll also be using it in place of tools like Softerras LDAP browser unless I need to something LDAP specfic.

Why Total Cost of Ownership is a fallacy

2008-08-01T12:01:00.001+01:00

If I have one more potential supplier try and sell me something on the lie that it will "reduce TCO" I will not only scream but I will beat them to death with a CAT 5 cable.

Total Cost of Ownership (TCO) is one of those almost unmeasurable values that seems to have pride of place in the salespersons portfolio. How do they KNOW a new system (with it's associated equipment, licensing and training costs) will work out cheaper than the old one?
The idea is that newer systems have better support so rather than training someone in an older system and maybe having to buy in more expensive skills more legacy systems it works out cheaper to upgrade or replace with the latest model.

I don't disagree that for some systems which are truly legacy such the old DOS or OS/2 application may well work out cheaper in the long run but the one thing that will truly reduce TCO?

Understand your systems.

Take time to test and document the fixes.

Use your call logging system as a knowledge base.

These three tips alone will truly reduce TCO.

VMWare course

2008-07-15T12:01:00.001+01:00

For much of this week I'm on a VMWare course for the second half of my VMWare training. This part of the course is titled Deploy, Secure and Analyse. The course itself is to prepare me for a server consolidation project that the company I work for is kicking off.
The project invovles several VMWare clusters, a Hitachi SAN and blades. Lots of flashing lights and new technology to ~~break~~ support.

Legacy Systems and a very handy SQL comparrison Tool

2008-07-13T16:16:00.002+01:00

On Friday, I had the "pleasure" of having to get a legacy system up and running.
This system was originally introduced to allow users in the business to manage group membership for projects they had ownership of. The idea was that it would cut down user calls to the service desk by about 10% and allow the project managers themselves to get a speedier turn around for new starters.
Sounds fine in theory and in the world of NT4 it wasn't a problem. Move on to the world of Active Directory and things are a little different. The legacy system (Bindview v4.6) has been superceded about 5 times over but we can't just install the latest version. Trust me on this, the latest version is fine but there are many design decisions and compromises as well as several rejections for upgrading the system from a few years back that have all combined to lead to the current problem.

The actual problem was an interesting one. The system was complaining whenever anyone tried to edit a group. A restore of the back end SQL database fixed the problem until the next domain sync occurred when the database would corrupt itself again.

Obviously, the sync was pulling something from the domain that it didn't like.
For the first attempt at a fix I fired up SQL Trace which records every single SQL statement that goes to a selected database. The neat thing about Trace is that it's possible to point the trace results to a SQL database itself and then filter it to get rid of stuff you know isn't going to help - such as SQL agent tasks and so on.
Trace left me with a multi-variable SQL script spanning over 4,000 lines and quite difficult to read or even test so I decided that the next best thing was to restore the working database to new a database name and then find a tool to compare every object on the bindview user table to see what was different between the restore and the one that synced with the domain and promptly broke.

AdeptSQL was the third tool I tried and whilst it has a very simplistic point and click interface it's incredibly powerful for comparing two SQL databases. Once the comparison is done you get two side-by-side windows which represent the two databases. Changes are highlighted by colour - Red for deletions, Blue for new and black for no changes.
This left me with a 2,000 list of changes, deletions and amendments in the database.
AdeptSQL also lets you filter things out and by using these features I eventually tracked the problem down to the description field of two user accounts.
These accounts had spurious characters in them which Bindview being rather old and totally ASCII prompt fell over on. Removing these and waiting for a resync solved the problem.

Whilst AdeptSQL helped me solve that particular problem there is still the problem of this legacy system updating Active Directory whilst not being active directory aware which leads to some other fun and games with the display name versus the SAMAccount name but more on that in a later article.

Build your own NAS

2008-06-30T12:01:00.001+01:00

Things have really moved on in terms of storage. Not so long ago the largest hard drive you could buy for a home PC was a 200GB IDE. Today, 1TB SATA hard drives are available for less than £100 from my favourite hardware website AUT Direct.

I'll admit that I couldn't resist for long and as I've got a tower PC with 6 IDE hard disks in which are not doing anything at present it was just too much of a lure and I've ordered up 4 1TB disks.
The plan is to replace four of the IDE disks with these 1TB SATA drives and I've bought the necessary SATA drive bays to making swapping them out easier if needed.

As the motherboard is quite old I also purchased two SATA cards which will be able to handle the SATA disks.

The tower also has two IDE disks on an IDE expansion card. This was originally for the OS but I'm going to pull that
and put one of the SATA cards in it's place. The IDE disks are small (either 10 or 20GB) which I'm going to bin and replace with two 250GB IDE disks.

In total the box will have about 4.5TB raw storage capability. I need to configure the 4 SATA drives as RAID 5 in case of a failure. I also want to configure the two 250GB IDE's as RAID1 for the same reason but testing in in VMWare showed it wasn't quite that easy.

The operating system of choice will be OpenFiler. This OS supports all sorts of storage options including CIFS, NFS and iSCSI. It's free and actually supports more than some hardware solutions such as the Buaffalo terrastation I recently bought!

Even so, When finished and configured with the RAID arrays the box should be able to support an impressive 3.2 or so TB or usable storage.

A fun little project......!

Issues upgrading Domain Schema to 2003

2008-06-29T12:01:00.001+01:00

So I'm probably a little behind in upgrading my home networks domain schema to support Windows 2003 but better late than never!
The process itself was smooth enough once I'd corrected some problems on the machine but the upgrade logs were not the most helpful troubleshooting aid I've come across.
One particular error had me stumped for a few days:

"Error code: 0x57 Error message: The parameter is incorrect.."

No indication of which parameter it was but as it occurred when checking security descriptors and many blog articles refer to missing security ACL's on GPO's I had a look at those and sure enough, Enterprise admins was missing some rights so I fixed those up and....... the same problem. At this point I'd admit to a lot of head scratching. The event logs didn't shed much light until I realised that the security event logs were not accessible. Sure enough, somehow the ACL's on the security event logs had lost all their rights. Resetting these and then rebooting allowed the process to complete perfectly.

ITIL Overview Training

2008-06-20T10:23:00.005+01:00

The company I'm currently working at have decided that ITIL is the way forward. Yes, after several years of different ideas, options, tests and other madness they want to adopt the official ITIL framework over a period of 6-7 months.

Now, whilst I think that ITIL is a good idea and yes, I am something of a convert to the whole ITIL structure I think that the nature of the user/customer base here is simply one that won't tolerate the ITIL way of doing things because it will require them to become more proactive and less reactive. I really do believe that many IT departments are products of the greater company in which they find themselves. Have a company that's reactive and unstructured then your IT department will be as well because it fits in to the business model.

Still, the training was interesting if a little dry and I picked up a few things on Problem Management and Root Cause Analysis. Something I'm very interested in because of the way it deals with problems and provides permanent documented fixes. This is something I'll go into in more detail in a later blog.

As for ITIL here, well.... I really do hope it works but I can see it being a somewhat half-hearted implementation unless the business are prepared to be a little more structured.

The final thing I'll say on ITIL is that it's a nice framework with a focus on how IT should be run but it doesn't address any sort of approach for bringing it into the business. I know that ITIL practitioners will say that this is because each business is different but it would be nice to read some success stories and find out just how they implemented ITIL and what order they implemented it.

Posting update

2008-06-09T12:01:00.000+01:00

Yes I know I've not posted for a bit. No excuses and I promise I will try to be good for here on in!

Lot's of changes at work and enough material to fill the blog every day for a year but I do need to actually get on with writing some of it down!

One article a week from here on in. Not a new years resolution but a start of summer resolution.

InfoSec 2008

2008-04-28T12:01:00.000+01:00

Well, After some false starts involving problems with London Undergrounds District Line I made it to Olympia and to Infosec 2008. The event itself is a good one for picking up the latest trends in security and seeing a few demo's of various products and as always there was some good stuff to see there.

For example, Sophos have come on in leaps and bounds and I was most impressed with their new AV console. It can also do NAP (where a machine is quarantined until it means a specific criteria for patches and AV).

The Sophos solution also has a web based applet which can be deployed to guest machines (i.e. visitors). The classic here was the sales guy who was demonstrating it was telling me just how clean the solution was "It uninstall's without a trace so we don't change a THING on the users machine" he extolled. Hmm. But if it doesn't met the policy then the remediation servers will be the only ones the user can see. This allows the user to update AV definitions and patches. Now, if we can't touch a visitors machine then what's the point? It's a nice technology but worthless for that reason.
Guest machines should be in an isolated vlan with only net access. They should not only be isolated from the production network but from each other as well.

The Microsoft seminar was superficial but I did learn a few things about their NAT offering in Windows Server 2008 and it does look useful. Certainly on the "to test" list.

Overall, I came away from Infosec slightly underwhelmed. There didn't seem to be any new technologies or ideas that made me feel "yes, I like this. This is a good way forward". The last time I had that feeling was with Splunk and I still think that about the product. I do wonder if security is falling into something of a rut just waiting for the next big attack.......

nLite Automated builds

2008-04-14T12:01:00.002+01:00

I'm a big fan of unattended builds and I've been using them for over five years now. The process of creating an unattended build can be somewhat hit and miss so using something like VMWare to test the final build is often an essential.

nLite has been around for a while but the last time I used it I found that the resultant build could be flaky and often just not work.
These issues seems to have been fixed with current version as it's remarkably easy to create a custom build and to add service packs, drivers and patches.

Overall I'm very impressed with the tool and at price tag which is free I really cannot complain!

Mac OS

2008-02-26T12:01:00.000Z

I'm not much of a Mac fan. This is simply because I don't have a need to use Mac's. I have friends who use and love the Mac book pro and I've seen a few being used on the train when I travel into and back home from work. I'm still not a fan though so never looked into Mac OS until a few days ago when I was testing out a new security tool for some due dilligence work that was required and a copy of Mac OS would have been very useful for testing.

Could I just go to Apple's site and download a trial? Nope. Not allowed. It seems insane to me that Apple have no ability to allow the regular intel user the ability to try out Mac OS without having to buy the hardware. This policy must be causing Mac sales.

Mail for Exchange documentation woes

2008-02-22T12:01:00.000Z

Too many times now I've come across badly written documentation. That is documentation that leaves you hanging wondering "what next?" or "where do I go from here?".

An example of this is the Mail for Exchange application on my Nokia E61. Having spent no less than 4 hours trying to get it to work and still having no joy I realized just how painful the documentation is. I'll cover the fun and games with Mail for Exchange in a later article but for now I just want to highlight how badly written the documentation is.
When configuring my phone to connect to my Exchange server over wireless I get an error "Communication error, retry later". The documentation has a section that reads "Troubleshooting - Errors you may receive" and lists that error with no fix or reason why that error is occurring.

Thanks Nokia.

If you are going to present the user with an error you should at least give the user and idea of what to do with it.

Get ready for a bumper patch Tuesday

2008-02-12T12:01:00.000Z

With no less than 12 security updates coming out of Microsoft later on today and Vista SP1 slated for February 15 there will be a lot of update servers groaning under the weight of so many updates to download so it's probably a good idea to ensure your WSUS servers have plenty of free disk space and are as up to date as possible now to ensure they download the minimum necessary during the next couple of weeks.

Call for Eicar V2

2008-02-04T13:01:00.000Z

Many years ago it was recognised that there existed a need to test AV software without throwing live viruses around and so the EICAR test file was developed as a safe way of testing that AV software was indeed working.
This was fine but I think there is now a need for an EICAR v2. Something that is NOT recognised by AV software by default. Why would this be of use?

Well, A scenario I had last week involved a virus getting onto NetApp filers. Now, Netapp will send the file to an AV scanner and get one of three responses back: clean, infected or timed out.
Clean means the file gets added to the clean list and will not be rescanned until the file changes.
In other words, if the file has a virus that the definitions do not pick up that file is NOT rescanned even if new definitions are released. This means a virus-infected file can get onto a NetApp system.

Having an EICARv2 test file will enable testing of the automatic clean-list clearing type of scenario and be very useful to the IS industry in general.

First Patch Tuesday of 2008

2008-01-04T12:01:00.000Z

The first patch Tuesday of the year is rolling around somewhat early this year as the second Tuesday is on the 8th. Microsoft has decided to be kind this month though as only one critical and one important patch are being released.

It seems though that Microsoft's RSS feed is suffering from a New year hangover as it's not been updated with the information on the website.

Data lost by Revenue and Customs

2007-11-20T12:01:00.000Z

The news story linked above talks about the UK Government losing 25 million records containing names, addresses, national insurance numbers and bank details.

Apparently the data was password protected but not encrypted, Now depending on the application used there may be some encryption there. I'm hoping that the data is an encrypted database that also has a password on it which is where the confusion is coming from but why do I have a feeling that it's just a CSV file?

The thing is, this is NOT NEWS. It's happened before, there have been reviews and procedures created yet it KEEPS happening. It happens in pretty much all companies and yet no one seems to care.

I, for the life of me, cannot work out why security is second fiddle. With word terrorism, bank fraud, phishing and everything else why am I and other members of the IT security industry still fighting an uphill battle? What is it going to take to get security onto the agenda?

The state of IT

2007-11-01T12:01:00.000Z

I came across the above article earlier today and I know that examples of the above problems are not just endemic to development process but instead seem to be buried deep into the very psyche of the majority of IT projects today.

I honestly would not been surprised to see Matt Allwright of BBC's Rogue Traders pop up at some of the meetings and accused the attendees of doing a shabby job and, of course, they would be right.

The classic in the above linked article is the very last email complaining that 'I'd love to write a dev env setup guide, but I just don't have the time!'. Hang, Didn't that email exchange basically list most of the steps needed? If there is time for the email exchange and time to waste someones time in scrabbling around for this information then the setup guide could have been written ages ago!!

We, as IT professionals are constantly subjected to these shabby practices and yet we don't accept them from other professionals so why should we in our own industry?

Snooping on Facebook user profiles is a 'staff perk'

2007-10-29T12:01:00.000Z

I'm not a huge fan of Facebook as I really don't see the point of sites like these. Generally, If I've not spoken to anyone in a number of years then there is a reason for it so I really don't want to hook up with them again thanks to facebook.

At the end of June I wrote that facebook users provide far too many personal details and were at risk of identify fraud. Well it seems that others are just catching on to this idea with several horror stories of exactly that in the media and then today The Register has this l ittle gem of a story.

So it seems privacy settings on facebook are absolutely meaningless and staff consider snooping a 'perk'. If users privacy is treated in such a cavalier fashion by those that administer the site I can for see a risk that users will become more blasé about risks surrounding identity theft which will, in turn, create an entire identity theft industry around facebook.

Centralised Logging

2007-10-25T12:01:00.000+01:00

One of the essential features for even a small network is a centralised logging solution.
Having a centralised logging tool makes for much easier trouble shooting as it becomes possible to review logs and search for related events or even search for the same event on separate machines, traditionally this has required quite expensive software such as HP open view in order to implement but a fairly new company might be about to put an end to that.

Enter Splunk, The 'Google of IT data'. This application will happily collect all sorts of different logs once configured and the configuration is not too difficult.

Splunk needs to be installed onto a Linux, Mac or Solaris environment although a Windows version is promised soon. As a workaround Splunk recommend that SNARE is installed on Windows servers. This software will convert event logs into syslog format and send them to a named server.

Putting Splunk in the center of you logging infrastructure as a syslog server and pointing all your syslog capable devices at it and then using SNARE to roll up event logs as syslogs which also get sent to Splunk is very easy to do. Within a few hours you have a surprising amount of data available to be searched by splunk.

And the price for all this information?

SNARE is free, Splunk is free if the amount of data you send to the Splunk server is less than 500mb a day although some of the features are limited.

I will admit to being a fan of Splunk after playing with it in VMWare. Over the next few weeks I'm going to describe how to configure a simple splunk installation for Linux, Windows, NetApp filers and Cisco switches.

NT4 Emulator Key

2007-10-02T12:01:00.000+01:00

If you happen to run a large Windows environment you might be familiar with the in place upgrade method of upgrading your domain to Active Directory. If you run a large Windows environment that spans several sites over a variety of links then you will know that an in place upgrade can be a pain.

The main problem with an in place upgrade stems from the fact that client machines will always prefer to talk to the Active Directory server instead of the Windows NT4 Backup Domain Controller. This means you can end up in a situation where a remote sites clients are traversing a poor link to authenticate against the Active Directory server and ignoring the local NT4 Domain Controller.

To work around this issue Microsoft provide a registry hack called the Windows NT4 Emulation key. If a DWORD key called NT4Emulator is created in HKLM/System/CurrentControlSet/Netlogon/Parameters and given the value of 1 is created then the server will 'pretend' to be a Windows NT4 server thus the client machines do not see any Active Directory domain controllers on the network and so will be quite happy to authenticate locally.

I'll cover this key and some of it's drawbacks in some later articles.

If in doubt, reboot........ the train........

2007-09-24T12:01:00.000+01:00

My journey into work is normally quite uneventful. Since the move out to Kent it generally takes 20 minutes longer but the journey is actually fairly pleasant. Today was the exception.

About 20 minutes into the journey the trains brakes come on pretty hard slamming the train to a stop and we sat there for a couple of minutes before the guard come onto the tannoy to explain that there was a problem with the trains brakes (really?!) and that there were going to try a fix... This is the point that they REBOOTED the train. I kid you not, the annunciator at both ends of the coach went out, the air con died and the lights all went out......... A few minutes in the quiet and everything came back on but I would have loved to have seen a BIOS start up message scroll across the annunciators!

As a side note in this case the fix didn't work and the train was taken out of service at Orpington but I swear that's the first time I've been on a train that's needed a reboot!

FSMO Confusion in multiple domains

2007-09-20T12:01:00.000+01:00

When I teach classes on Active Directory I will cover various domain models including the empty root domain model, this model has several security,delegation and political based benefits that I will cover in a future article suffice to say it uses two domains and the child domain is the production domain and the empty root just contains certain FSMO roles and forest-wide groups.

When I teach this model I will always ask the class to tell me how many FSMO roles there are and if the class is awake I will generally get the correct answer of five. I will then point to the empty root domain model and ask the class where the 8 FSMO roles should be placed, invariably I will get a look of confusion because there are only five.....

What a lot people forget is the minimum number of FSMO roles you can have in a domain is three and the maximum is five. Lets look at that empty root domain again - The empty root is just a windows domain that just happens to be the first in the domain to be created and as such will hold five FSMO roles. The roles are Schema Master, Domain Naming Master, PDC Emulator, RID Master and Infrastructure Master. The first two are forest wide so will only ever exist in one domain of the tree whereas the other three are domain wide and will exist in each and every domain created and this seems to be where the confusion comes in.
Your very first domain (the empty root in this example) will have FIVE FSMO roles, the child domain will hold THREE. Five+three equals eight which explains how you can have eight FSMO roles across two domains.

Creating a Default User Profile

2007-09-05T12:01:00.000+01:00

One of the things that annoys me about windows is the Default User profile. This is the profile that a new user who logs onto a machine (or server will get).
The way it works on NT, 2000, XP and 2003 is pretty much the same.
Under the documents and settings folder on 2000, XP and 2003 are a series of folders for each person that logs on to the machine.

Hidden in here is also a profile called 'Default User' and whatever is in here gets copied to the logon name of any NEW person that logs on.

Microsoft provide a somewhat tortuous way of customising this profile by creating an additional local user which is fine unless you have already spent time customising the profile you have logged on as.

Facing this situation yesterday I realised that the easiest fix is to just log off the machine which unloads the user profile and then you can copy the existing profile over the top of default user and so have a working default user profile in seconds..... Permissions might need to be adjusted as need be but it was a quick and painless way to take an existing and cofigured profile and make it a default.

Kensington Laptop Locks (2)

2007-08-24T12:01:00.000+01:00

As per the comment on my previous entry the issue of the Kensington laptop lock being pickable is actually an old one. I admit that I never realised this as it was only highlighted to me just over a week ago.

What I don't understand is why you would upgrade a perfectly good and working laptop lock? I checked with Kensington and found that they have know about the issue for THREE years and never issued a product recall. In the email conversation I have had with them they would not answer why they never issued a recall which is pretty bad going for a company that makes security devices.

Overall the lesson appears to be one of caution with laptop locks because they are not as secure as you may think.

Kensington Laptop Locks

2007-08-18T12:01:00.000+01:00

Earlier this week I was sent a link to a video on how to pick a Kensington laptop lock using nothing but a toilet roll.

It works.

After about 20 minutes practice it becomes very easy to pop the lock within 30 seconds. The process on how to do it is on a very well known video sharing site which I won't link to here for obvious reasons but you'll be able to find very easily with a Google search.

Project time Analysis

2007-08-14T12:01:00.000+01:00

After last weeks fun and games with project management and changing deadlines I've come to the conclusion that I really don't like MS Project. Maybe this is because I don't really know how to use it and I don't have the back end server infrastructure set up for it all to work.

Even after saying that I still don't like project. I don't like the way it requires dates. If a persons time was dedicated to a single project then fine, project will work better but when you have a person whose time you get in varying lengths and that length of time is unpredictable then project is unable to cope with working that way and breaks down.

I personally feel that what is needed is a time planner tool which can be used as a time base for tasks. This system would become a centralised store of tasks - what to know how long it takes to get a server? Not a problem, look it up and the tool will give you the average time it takes based on previous knowledge.

MS Project is not able to offer this, in fact no project tool I have used seems to be capable of offering this pooled knowledge and therefore all project plans are new, finger in the air guesses with no ability to draw on previous learned experiences.

Additionally, the tool I envision will be able to detect compare the tasks you are creating and suggest others based on previous projects.
Once your project tasks are drawn up it will be easy to see the amount of hours required and most of this information will be based on known information!

To me, this seems to be a common sense approach based on learnt knowledge yet no one seems to work this way and I can't find a tool that is able to do this type of time analysis?

Bad Project Management

2007-08-07T10:27:00.000+01:00

Sometimes this industry makes me want to scream. My old favourite the artificially tight deadline has been back in force this week with a project due to finish at the end of the month being shortened to the 23rd and now further short end to this Friday.
Obviously, in order to deliver the project will have to skip most if not all of the testing. Problems will occur in a very user facing environment and there will be no pre-learnt knowledge of failure modes which in turn means a very steep learning curve.

of course, chances are everything will be fine. Chances are testing won't uncover any major problems, chances are the testing can be deleted with no obvious impact to the systems.

however, without testing its impossible to know, without testing the little oddities that do crop up during the operation of a system cant be found or at least recognised.

a second really annoying part is that the project management tool we are using requires testing to be added so all the tests have been carefully thought about and added and now they won't be used.

this is my bosses boss demanding this so do I get on with it knowing the system will be inferior or refuse unless its properly tested?

At the end of the day I consider myself part of the engineering community with standards and a pride in my work so I will make a noise but fights like these end up leaving me drained, tired and wondering why the hell I still work in this sector.

This project doesn't need to deliver early. It just makes the stats look good and I am now fed up of working long days to fix something that should not have been delivered broken in the first place.

Security Industry Commentary

2007-07-31T12:01:00.000+01:00

Last night I watched the "Diana: Last Days of a Princess" documentary. I admit that I largely watched it just to moan about how much the British media is still concentrating on Diana but I was pleasantly surprised at how good the documentary was because they largely focused on the two bodyguards assigned to Diana and Dodi.

What really surprised me was the similarity between physical security and computer security. Both have a recommend set of practices or standard operating procedures. During the final days of Diana's life the documentary highlighted that the two bodyguards were physically exhausted and their recommendations for security practices had been ignored with the result of Diana and Dodi paying with their lives.

Now computer security isn't as hands on as computer security but there are startling similarities with the way people in both industries are treated. I still don't understand why we as security and IT professionals are hired and often ignored/overruled by management.
Obviously, there are some occasions when management have to do this to fit in with a company vision or similar which has not been fully cascaded to the business or for reasons of corporate confidentiality have to be kept quiet but this sort of practice happens all to often.

Careful when testing security

2007-07-26T12:01:00.001+01:00

I was amused to read that two Daily Mirror journalists where arrested when 'testing' security procedures on London Underground. Whilst it they say that they have a 'journalistic right' to expose security holes I personally think that it's just another type irresponsible disclosure where the people concerned don't have a chance to put right the problem before it's widely reported.

On another completly different note I was browsing You tube earlier today and found this fantastic clip of Bill Bailey and his rendition of the BBC News theme tune. I had the fortune to see Bill Bailey live once and the guy is a genuis.

Careful when testing security

2007-07-26T12:01:00.000+01:00

Is it time to scrap the password?

2007-07-24T12:01:00.000+01:00

As password protection gets more secure the actual passwords for users get weaker.

A paradox? Not so, let me explain.

A few years back the biggest problem was a password going 'over the wire' and being intercepted. This can easily been seen in telnet. Just capture a telnet session and you will see the password in plain text contained in the packet capture.

Obviously this is not very secure so the next step is to encrypt the password on the workstation and send the encrypted password to the server. This way the plain text version never left the workstation.
This was defeated in fairly short order because the level of encryption used was not very high.

The next step was to use encryption and a salt but this was also broken is short order thanks to the salt being included with the hash.

Currently, Kerberos does a wonderful job of making sure a password is secure. Not only is it encrypted on the client machine with a one-time hash but it never leaves the machine - only bits do - A bit like a bank asking you for five letters from your address at random locations to very its you. Kerberos also defeats a replay attack by encoding the time into the response. If the domain controller sees a client-side request with the same or an earlier time its rejected as a reply attack - this is why its vital to make sure your servers and clients all have the correct time.

So, how does this make our password weaker?

In many systems its possible to check the password complexity levels but it's getting harder to check to see if users are following a pattern, e.g. password1234, password1235, password1236 and so on because the password is NOT stored on the server and it's not possible to decrypt the hash the checking must be done on the client side machine - BUT, if the client side machine stores the password it will have weaker security than the domain controller and as it's easier to access a client side PC than a locked away domain controller the passwords should be stored on the domain controller.
BUT if the passwords are stored on the domain controller they must be decryptable or plain text otherwise certain complexity checks cannot be enforced.
This would again open up the password to packet sniffing attacks.

It seems that the password has run it's course but it's there something out there that's easy to use, doesn't require a heavy infrastructure (like RSA tokens) and offers two-factor authentication?

Increase in eCard scams

2007-07-17T12:01:00.000+01:00

A few weeks ago I got a fairly genuine looking ecard email but something about it triggered my suspicious so I did some further checking of the mail message and spotted that the from address didn't match the listed name and that the link for the ecard was an IP address rather than a website.
The link itself displays a message "If this card does not download in 15 seconds click this link" and the link takes you to an .exe file so obviously not a legitimate site!

This was about two weeks ago, since then the number of emails I'm getting with this type of exploit has been steadily increasing so it appears that there is a trend increase for this type of attack and it does make sense. This type of attack is less likely to be stopped by mail filtering, will not trigger an IPS alert and probably won't be stopped by local anti-virus software - Hopefully the .exe file that the site points to will be stopped by local AV.

I've downloaded the .exe but not had a chance to see what it does. I will be analysing it in more detail later on today and will blog back here in the meantime keep a watch out for dodgy ecard emails.

Reporting new spyware

2007-07-10T09:15:00.000+01:00

When setting up the network at the new house I fired up each machine to test out connectivity and other behaviour before putting it back on the network. One of the laptops was behaving a bit oddly so I put it to one side for further examination and carried on with the rest. The area where the computers are located is a real pain though as there is only one power point - Something that will need to be fixed as I need about 30 power points for the kit!

Looking at the laptop I went through some standard checks to see what was running at startup and so on, Nothing unusual popped out so I looked deeper into the services and once service I tried to disable gave me error so I had a look in the registry to find it calls itself tixqvawf in the registry and goes by a host of different names when you delete it.

I dug further and found that it launches a DLL called bkldbkl.dll which I sent off to Sophos and McAfee for analysis - Now this is where it get's interesting.
Sophos replied within 48 hours to say that it was spyware and that they would be issuing an IDE update would to cover it and the link is here.

McAfee sent back an automated reply to say that an AVERT researcher would be looking into it because the automated test could not find a match. Further down it said that automated testing would only occur if the original sample was in a password protected zip and that a researcher would only look into it if that was the case but the one I sent wasn't.
So now I've got two pieces of conflicting information.

I assume that because McAfee have not bothered to get back to me they require a password protected zip file but it's annoying that the sample wasn't rejected because it wasn't password protected. All a bit silly really and McAfee will miss out because of the poor wording when reporting new items.

Information Leakage via social networking

2007-06-30T12:01:00.000+01:00

I frequent the linkedin website and someone recently asked 'what constitutes an identity?'. I found this an interesting question because there are several ways to answer it ranging from philosophical to technical but there is also the flip side - how easy is it for someone to assume your identity?

Social networking sites like facebook and linkedin promote information sharing, you can list work and school experience as well as share interesting tidbits about yourself but I do have to wonder just how much of that information could be used for illegal or illicit gain? All it takes is someone determined to have the sites database and not only do you have all the information that's publicly available but you also have other restricted data (assuming that the data is not stored encrypted of course).

These sites have a lot to promote them, they are a great way of getting in touch with past colleagues and for making new contacts but in many ways they scare me with the amount of information people are happy to give away about themselves - The potential for social engineering attacks based on information contained in this sites must be huge.

Moving house

2007-06-27T12:01:00.000+01:00

Well, that was fun!

The move is over. The computers are almost all in place but there is a lot of little jobs that need doing, lots of rewiring for the computers - The study area only has a single plug which is not exactly ideal for all the computers!

It's been quite a week, Moved on Saturday, slept on Sunday then back to work Monday and Tuesday then on a course for Blue Coat Wednesday, Thursday and Friday.

The journey here is a bit more of a pain so I'm looking at jobs in the area - worth keeping an eye on all the options.

Normal blogging service will be resumed over the weekend with a look into reducing the number of services a default installation of Windows 2003 has.

As always, any suggestions for security focused articles please drop me a line.

Blog Updates for the next two weeks

2007-06-13T12:01:00.000+01:00

Blog updates for the next two weeks make be a little sporadic as I'm moving house and will have limited net access.

Once I'm up and running I've got a series of blog articles planned that I hope you will all find interesting. Here is a brief taste:

Running Windows using Minimal services

iSCSI on Windows, Linux and NetApp

Issues with P2V and domain controllers

Any suggestions for future articles always greatly received.

Profiting from Security vulnerabilities

2007-06-06T12:01:00.000+01:00

A new company is offering security researchers the chance to profit from discovering and coming up with innovative fixes for Security Vulnerabilities in products.

There web page (linked above) contains the following ominous paragraph:

We evaluate the vulnerability for the following criteria:

(a) Either the researcher or ourselves can suggest a method of fixing the vulnerability
(b) The fix is difficult to "design around"
(c) The fix can be protected by patents or other intellectual property.
(d) If the fix is adopted, it is easy for us to gain evidence that this has happened.

So, If I'm reading this right they are only interested in fixes that they can profit from? If a researcher discovers security hole and they can't patent a fix then they are not interested?
The next line is even more scary, How do they propose to gain evidence that a fix has been adopted? This suggests some sort of 'phone home' technology to report in to them that a particular machine has the fix installed.

Is it just me who can see several rather worrying aspects to this proposal?
1. It encourages the less ethical security researcher to profit from abusing a security hole if they believe or if they cannot make a profit from the fix
2. The phone home technology that's hinted about in clause D can be exploited - Lets say a security fix is removed by accident from a machine. That fix will then not be able to 'call home'. It its now possible for someone to review the database and see just what fixes are missing from machines rendering them vulnerable to a non-ethical employee!! This is totally unacceptable.

Annoyed at Symantec 'Trialware'

2007-06-01T12:01:00.000+01:00

I just tried to download a copy of Symantec Sygate 5.1 from Symantec's website. I've used and earlier version of the product and wanted to test out the current version just to see what it looked like - If we purchase it then it will be via an already establisehd partner.

When I clicked on the 'download trialware' link I got a THREE-PAGE form to fill in, I never use my details on these because I already get bugged by enough sales folk.

Imagine my amazement when I complete the process to get this message:

"To ensure that Symantec can provide any technical assistance needed for a smooth evaluation, a Sales representative will contact you within 3-5 business days to provide software download details, as well as help with product activation and implementation. Do you wish to continue?"

Right. How can a SALES Rep give me technical assistance?

Why does a SALES Rep need to contact me to allow me to download the software?

Why does it take 3 to 5 DAYS before the software can be downloaded.

I thought I'd give Symantec customer care a call about this, So I dial up 0870 2431003 which is the number listed on the website for customer services and get through to a normal digital dorothy phone menu, select option 3 for Symantec customer services and promptly get cut off.

It seems that Symantecs 'Customer Care' is basically them saying "We hate you. Go away".

Thanks Symantec. I'll make sure I do as little business with you as possible.

The power of the human mind........ to fail.

2007-05-29T12:01:00.000+01:00

The human brain is an amazing computer. It can store almost infinite quantities of data, it has near instant recollection to enable you to recognise people, places and perform the essential day to day actitives of breathing.

So why does it let us down in the middle of a crisis when you need to have that vital bit of information or when you remember seeing a tech note on the very problem you are fixing but can't recall the technical notes reference number.

The answer is simply to do with the way the human mind has evolved, Our technological prowess has evolved quicker than the brains ability to deal with this new landscape. When we, as a specics, were huddled in caves a crisis required the 'fight or flight' response and today when you have a crisis in the office that same reaction kicks in and all of a sudden you have trouble recalling technical details yet when the crisis passes you will be doing something else when the brain, still working on the problem in the background, will kick in.
The military actually have special training programmes to allow test pilots and others under extreme pressure to continue to think rationally. It is a very special SKILL.

so, why do we all have problems saying 'I don't know' and recording those oh so useful technical notes when we have the chance?

In the IS industry there seems to be a huge amount of pride in relying on ones memory to get people through a bad day. Checklists and procedures only seem to come into force for the day to day working practices - I have yet to see a company have an emergency procedures checklist.

Death of a Domain Controller

2007-05-23T12:01:00.000+01:00

My home network has just a single domain controller on it. This domain controller runs a few other servers such as spam checking email and so on.

A few weeks back when the server was started it would always do a disk check but it would never find any problems so as a precaution I backed up the system state data - I don't use two domain controllers because I just don't see the need for it on a small home network.

Good thing I took the backup as the server decided it had had enough and blue screened with 'INACCESSIBLE BOOT DEVICE'.

At first I thought the fix would be simple enough, Just build a new Windows 2000 Domain controller because the original was windows 2000 and run DCPROMO /ADV to restore but Windows 2000 Active Directory doesn't support it then I hit upon the idea of building a Windows 2003 domain controller and running DCPROMO /ADV but I don't know if it will be able to restore a Windows 2000 Active Directory database to Windows 2003 - Something I will test very soon.

The safest option and the one I'm following is to build a Windows 2000 server and drop it it into Active Directory Services restore mode and then restore the system state from backup and THEN DCPromo the new windows 2003 server.

The week from hell

2007-05-18T12:01:00.000+01:00

Ok, so it's more nine days than a week but you know how people say trouble comes in three's? Well, I've almost had three lots of three over the past nine days. I'll go into further explanations for some of the more technical ones but so far this is the list of problems from just over a week!

1. Power supplied died in my Freeview box
2. My DVD player died (Ok, It's been on the way out for a while)
3. Weekend works did not go as planned due to an oversight
4. I found two bugs in Data ONTap (NetApp's proprietary operating system). One caused a filer panic
5. My Domain controller at home died and the backup image for it is causing me problems
6. An upgrade of a server caused a database to go bad. That took three days to get back and it's still playing up a bit
7. I got run over by a cyclist who are (according to the police) running people over in order to snatch phones.
8. Work Laptop

FreeView Box
I have a Digivsion FVRT150 freeview box, This has an internal 80GB hard disk which is great for recording programmes and has worked flawlessly up until a week ago when all sorts of odd noises started coming from it which was caused by the unit not having enough power to work. Apparently it's a common problem and there is a site called XtendedPlay that sell replacement power supplies. I bought one and it's all working perfectly now

DVD Player
After having a lot of issues with videos I bought a combination DVD & Video player a couple of years back. The DVD player on this works but the mechanical rollers which allow the tray to eject have died so I bought a new slimline dedicated DVD player from Amazon. The prices on these have REALLY come down as it cost just £17.

Weekend Works
The company I work for planned a maintenance weekend to upgrade a FAS940 filer with a clustered FAS3070 unfortunately no one took into account the vFiler which is part of the existing filer.
For those that don't know NetApp, filers are basically bit storage cabinets of disks and they allow some clever tricks such as iSCSI, proper quotas and so on.
They also allow the creation of vFilers. A vFiler is a virtual filer or a 'filer within a filer' and its useful for segregating data. Unfortunately vFilers and VIF (which are multiple interfaces joined aggregated into one connection) don't work together. The new clustered environment was planned to use nothing but VIF's.
During the work the onsite engineer from NetApp decided to create a single VIF, that is a single connection as part of a VIF group so that when the vfiler was migrated elsewhere the connection it freed up would be able to be added into the VIF group and it should all just work.
Wrong. It turns out due to a probable bug in the Operating System a single interface in a VIF group will not work.

Second filer bug
One thing filers do quite well is pretend to be windows boxes (via CIFS or Samba if you prefer) or pretend to be linux/solaris boxes via NFS. Unfortunately there is a bug in the version of the operating system that we run which can, under rare occasions, cause the filer to panic in certain CIFS operations.
Somehow we triggered that situation and the filer crashed. Fortunately the cluster worked and the second filer head took over the load.

My domain Controller
In several articles I've said that I only run one domain controller and back it up roughly once a week as rebuilding the server is quite easy. Well, it looks like I might have to go back on that as my domain controller died in the week and I can't access the data in the backup. Fortunately I do have a VMWare image of the server which is working but DNS is broken so recovering the domain controller is proving to be 'fun'..... Obviously, I shall re-evaluate that second DC!

Database upgrade
During the aforementioned maintenance weekend the decision was made to install SQL 2000 SP4 onto all the SQL servers and onto all instances on the SQL servers. This went well but one system we use - bindview, which is a delegated access tool took a turn for the worse. It was then we found out that no one knows the password that bindview uses to talk to SQL. Ok, simple. Change the password and reset it in bindview but you can't do that without reinstalling the software and you can't reinstall the software on our bindview server because it's REALLY only meant for NT4 and not the active directory (but NT4 emulated) environment. One hell of a restore later (sever, database) and a clever hack of the hashed password out of sysxlogins and it was fixed but it was an interesting time.

Run Over!
On the way home the other night a cyclist went into the back of me then after a slinging a punch went dashing off. I reported it to the police and was told that it's becoming common. The idea is by riding into the back of someone they either knock the phone out of the persons hand or knock the person over and they can then grab the phone and ride off. Right now, I'm sporting a lovely set of cuts down the back of my leg which have all been treated and should heal up quickly.

Work Laptop
for some reason the laptop I use at work decided to go slow, Firefox locked the processor at 99%, killing it then locked another process at 99% and so on until winlogon locked the processor at 99% - Something was obviously interfering and causing problems. I'm now in the process of rebuilding the laptop.

And the week is not yet over!!

Roving Mars

2007-05-14T12:01:00.000+01:00

This Sunday I had the pleasure of visiting the IMAX to see the Roving Mars presentation. Whilst this is not in 3D the movie is pretty spectacular on the huge screen at the London IMAX - The delta II launch vehicle looks and sounds fantastic even if it is mostly CGI animation it really needs to be seen on the big IMAX screen with that surround sound system.

If you have a free hour and are close to an IMAX it's worth checking out.

Nimda, Slammer and the like

2007-05-11T12:01:00.000+01:00

Now that Microsoft have released a patch for the recent DNS RPC vulnerbility IT Admins should be deploying it as quickly as possible - I was talking to a friend about this today and we got to talking about how the threat landscape had changed over the years.

Many years ago a vulnerbility would be announced on bugtraq or the like, Microsoft would rush a patch out and then few people would deploy it - IT Admins would brief easy because a patch was out and things would continue.
Then the virus would hit. It would exploit a hole that had been patched MONTHS before hand. After the problem was fixed, the virus cleaned out and tools or a white paper written on how the bug worked and how slack Microsoft was in making products with security holes in.

Fast Forward a couple of years and look at the operating system. Its resonably secure out of the box, there are templates for making it more secure, there is COPIOUS amounts of documentation on locking it down. How many people ACTUALLY lock down a new server? How many apply the security templates or even take a template and modifty it? Show of hands?

Thought so.

Why do we as IT Admins wring our hands and blame Microsoft for all the security woes on the planet when they provide us with things like security templates that very few use?

The threat landscape has changed. It's highly unlikely there will ever be another SQL slammer, Nimda, love bug or code red style attack. It's just not worth it. With firewalls, IPS/IDS and Anti Virus all over the place writing a virus is actually quite difficult. It's even more difficult to get it unleashed on a network via email or similar because people are aware of it.

The new threat landscape comes from Information Disclosure. It's now routine for applications to phone home and send anonymous information 'back to base' in order to 'improve the application'. I do wonder just what information is sent back. I also wonder just how many applications turn this ability on and do NOT TELL THE USER.
Obviously, If a vendor gets caught sending back a bit too much information from your PC then they will look foolish and it will hurt their sales for a while but is this enough?

The single biggest abuser of the 'phone home' capability is spyware. The little applications that install from some websites. Some of this spyware is incredibly intelligent in how it hides itself and in what it selects to send home.

I firmly think that today, this is our biggest challenge.

3,000 test users

2007-05-10T12:01:00.000+01:00

Do you ever have need of a few hundred to a few thousand random names to populate your Active Directory in order to test something?

This is the requirement I had a few weeks back so I dug out about 3,000 random names from the 1901 census and threw them into a csv file that can be read by the addusers tool.

My names.csv file can be downloaded by clicking on the blog article link or by clicking here.

To get the users into active directory copy both adduseres.exe and names.csv to the root of your C: drive and then type in:

addusers /c c:\names.csv

addusers /? will give you a list of other options where you can set parameters for passwords and the like.

Rejoice for it's patch Tuesday

2007-05-09T12:01:00.000+01:00

Once again patch Tuesday rolls around and this time we have a total of 18 patches released across five security updates. It's good to see that MS07-029 is the much anticipated fix for the DNS RPC Vulnerability.

Slightly more worrying is another Internet Explorer Cumulative roll up. This patch covers Internet Explorer 5, 6 and 7. If fact, the bulliten goes as far as to say that the new Internet Explorer vulnerabiltiies are only rated as Moderate on Windows 2003 server but as Critical on Windows Vista. It's dissapointing to see Vista not being able to offer the same levels of protection as Internet explorers enhanced security mode on Windows 2003.

The law of the 6 P's

2007-05-04T12:01:00.000+01:00

When I first started out in the IT industry I was fortunate enough to work with someone who was incredibly well versed in Windows and taught me a lot about transitiing networks from NetWare to Windows.

One of the things he told me was 'The law of the 6 P's' which stands for:
Proper Planning Prevents Piss Poor Performance.

This is something that has stayed with me over the years. In different companies I've worked in I am still amazed at how little coordiantion and planning goes on. Warnings go ignored, procedures dont get followed (often because no one knows they exist) and eventually a crisis forces the IT dept to pull out all the stops to achieve something. Because much of the knowledge exists in a few peoples heads they are the ones that always get asked to fix things, sometimes this is self inflicted yet most of the time its because they have initative and get on with things.

A lot of the problem stems from a lack of quality. Everyone I work with in the IT industry wants to do a high quality job but often they are not allowed to because the client/boss/other wants its NOW. It's often a choice between doing something right and doing something right now.

Tiddly Wiki's

2007-05-02T12:01:00.000+01:00

For a couple of projects I'm working on it's nice to have 'scratchpad' type area where information can be quickly written and accessed. A wiki is perfect for this type of information because it can be easily uploaded and modified by people working on the project WITHOUT a need to purchase an horrendously expensive server and a copy of Groove or SharePoint.

GTDTiddly Wiki has some very nice features - It doesn't need to be installed as it's just an index.html file. Java is needed to add in the functionality but beyond that there is no server side configuration needed. I've not yet unleashed any of the three wiki's I have created on a server.

GTDTiddly Wiki is a nice little applet, With the right configuration I don't see why it couldnt be used on a server and as such you can use it for all sorts of quick and dirty project/note keeping websites and these sites can be developed in next to no time.

InfoSec

2007-04-26T12:01:00.000+01:00

I had the pleasure of attending this years infosec event at Olympia and it was a thoroughly enjoyable afternoon - even with the hard sell By some vendors!!

I wasn't aware of any particular focus to this years event, certainly there were vendors who duplicated other vendors products with password management systems seemingly the 'in' thing. At least three vendors had the same password management product just presented in a slightly different way.

One vendor has an interesting approach to the problem of single sign on - Imprivata, whom I've had the pleasure of dealing with before displayed their showpiece single sign on tool - This appliance is a very impressive piece of technology that not just does single sign on but can also integrate with the building management system make decisions on access based on where your door card is used.
For example, No local logon's allowed on the servers unless your swipe card has been detected as being used to access the server room, also no vpn access for your account if your pass HAS been used to gain access to the building.
With Imprivata's single sign on technology these types of rules can be used to build a very powerful and comprehensive access control layer

The second vendor I was impressed by was Secerno. They have an appliance that is desigend to sit in front of SQL servers and reject or accept SQL queries dependant on where the query originated.
For example, if you have a payroll system that HR have access to but you don't want anyone else running (or trying to run) queries against it you can just block ALL queries from other IP addresses/terminals, etc. This is very handy for the casual browser or for the SQL admin who wants to poke around confidential databases. Obviously, if the SQL admin can take a backup of the database and take it offsite then the security is broken but it's an interesting idea.

Another vendor. GFI may well have something of a niche product with their endpoint security software. Many vendors have endpoint software for managing USB, CD burners and so on but this is the first I have seen that claims to be able to do the range of mobile phones as well. It should be an interesting application to play with.

The final stand I have to mention is Microsofts where I got to meet Claire Smyth of Technet magazine and she is an absolute delight to talk to, Obviously very comitted to TechNet and the technet community. Just a few minutes at the Microsoft stand really gives you an idea of the passion these people have for thier products and the comittment they have to security.

Overall, The event was a lot of fun and I'd recommend it to anyone who has to deal with IT Security.

Display FSMO role holders

2007-04-25T21:01:00.000+01:00

Imagine the scene - You are consultant and have been asked to fix an Active Directory issue - One of the first things you need to find out is where all the FSMO roles live. You could go digging around in Active Directory Computers and Users, Domains and Trusts and Schema Master (remembering to register SCHMMGMT.DLL) or you could just run the script below.

Copy the script in the box below, save it as 'fmso-role-holders.vbs' then run it via cscript fmso-role-holders.vbs

Set objRootDSE = GetObject("LDAP://rootDSE")

Set objSchema = GetObject ("LDAP://" & objRootDSE.Get("schemaNamingContext"))
strSchemaMaster = objSchema.Get("fSMORoleOwner")
Set objNtds = GetObject("LDAP://" & strSchemaMaster)
Set objComputer = GetObject(objNtds.Parent)
wscript.Echo "Forest-wide Schema Master FSMO: " & objComputer.Name

Set objNtds = Nothing
Set objComputer = Nothing

Set objPartitions = GetObject("LDAP://CN=Partitions," & objRootDSE.Get("configurationNamingContext"))
strDomainNamingMaster = objPartitions.Get("fSMORoleOwner")
Set objNtds = GetObject("LDAP://" & strDomainNamingMaster)
Set objComputer = GetObject(objNtds.Parent)
wscript.Echo "Forest-wide Domain Naming Master FSMO: " & objComputer.Name

Set objDomain = GetObject ("LDAP://" & objRootDSE.Get("defaultNamingContext"))
strPdcEmulator = objDomain.Get("fSMORoleOwner")
Set objNtds = GetObject("LDAP://" & strPdcEmulator)
Set objComputer = GetObject(objNtds.Parent)
wscript.Echo "Domain's PDC Emulator FSMO: " & objComputer.Name

Set objRidManager = GetObject("LDAP://CN=RID Manager$,CN=System," & objRootDSE.Get("defaultNamingContext"))
strRidMaster = objRidManager.Get("fSMORoleOwner")
Set objNtds = GetObject("LDAP://" & strRidMaster)
Set objComputer = GetObject(objNtds.Parent)
wscript.Echo "Domain's RID Master FSMO: " & objComputer.Name

Set objInfrastructure = GetObject("LDAP://CN=Infrastructure," & objRootDSE.Get("defaultNamingContext"))
strInfrastructureMaster = objInfrastructure.Get("fSMORoleOwner")
Set objNtds = GetObject("LDAP://" & strInfrastructureMaster)
Set objComputer = GetObject(objNtds.Parent)
wscript.Echo "Domain's Infrastructure Master FSMO: " & objComputer.Name

Automated phone systems

2007-04-18T12:01:00.000+01:00

Working in the IT field means I'm often on the phone to different companies and the one thing I VERY quickly learn to hate are the automated call handling systems that it seems everyone has introduced.

There are several things that I truly hate about these menu systems but one of the main ones is how every damn system has a variant of "In order to serve you better our menu has changed" - err, how does you messing around with your MENU serve me? It just means I have to sit here and listen to you twittering on when you could serve me better by fixing the product in the first place!
In actually many people do the same as me, they listen to the menu options once and write them down for future use - Outlook's notes field is particularly handy for this.

The other big problem I have with these systems is how you have to negotiate a maze of options to finally get the dept you want only to be told "This department is closed". Frustrating doesn't even begin to describe it.

There has got to be something better than a digital Dorothy answering the phones for every company.

Further exploits for DNS Remote management vulnerability

2007-04-17T12:01:00.000+01:00

Last Friday Microsoft announced a new security vulnerability in DNS management, today the advisory was updated to include reports of an attack in the wild that is trying to exploit this hole.

Whilst the security hole is an issue I'm not overly concerned about it. Certainly I'm tracking it and Microsoft have provided a couple of work arounds for the issue but this is typical security hole that a good security policy would at best prevent and at worst severely restrict.

The explain what I mean let's have a look at the vulnerability in detail:

DNS is a server service that listens on port 53. All DNS servers have to listen on port 53 as it's part of the requirements for running DNS - Changing the port is not an option so that already opens up possible attack vectors and so you lock that down by accepting traffic on port 53 from just a limited range of IP addresses.

However, This new security hole isn't based around DNS as it works on port 53 it's a hole in how DNS accepts remote management requests over RPC. This is a very important thing to understand. Just because it's DNS it's not port 53.

RPC is another protocol that uses ports which cannot be changed and it's also been known as an attack vector for some years. Additionally, RPC is the protocol that allows access to things like the c: drive on most computers and this is one of many reasons that most ISP's block RPC port traffic.

I'm hoping that this goes someway to explaining how this exploit works, if you understand then you will understand what I'm about to say:

The biggest risk from this security vulnerability is from INSIDE the corporate network.

Got that? Good.

Any company that exposes RPC ports inside the network for any user on the Internet to access has already been attacked and has hopefully wised up.
The threat from this security hole is more related to remote management of DNS from internal networks. Most administrators will install the adminpak.msi tools onto their machines and then connect to the DNS server and manage them remotely.

A good security policy which includes locking down remote access to the server for management functions to a limited subset of users would render this type of hole useless before it even gets off the ground.
The fact that an exploit has been crafted which is not a proof of concept is proof itself of companies inabilities to take proper responsibility for the security of their infrastructure.
Windows has shipped with security templates since before Windows 2000 was released, the templates in Windows 2000 were FAR superior to those that you could get for NT but even Windows NT had a lock down tool.

Microsoft often gets blamed for poor security practices but most administrators are guilty of exactly the same. A good sense of security and a good lock down policy will mitigate against most attacks that we see these days.

Say goodbye to initative

2007-04-13T12:01:00.000+01:00

When I joined my first IT department a lot of things were new to me but I was expected to basically get on with it. Sure I could ask questions and I could see if there were any procedures but a lot of it was left up to me to learn and sort out.

Fast-forward thirteen years and we have the paper-MCSE who doesn't seem to have the ability to think for themselves. Too often people in IT Depts these days seem to expect to be spoon fed and led by the neck, there doesn't seem to be that spark of "I want to figure out why this is broken".
To many of today's IS staff seem content to just rattle through the same old checklist of items and in the process quite often alienate customers.

In many ways this is not the fault of the new helpdesk person coming into the organisation but instead its a problem endemic in the very way IT Departments work these days.
The problem is this - Many IT Departments have the same type of calls and these calls can be handled by someone with very little knowledge following a checklist. Not only does this make the call closure rate look good but also reduces costs as the person on the helpdesk can be paid less.

This is all fine for the very basic calls and issues but it starts to fall down for the more complex ones because some of the more complex problems cannot be fixed by following a checklist. Certainly a checklist can be used as a guide but real world troubleshooting needs more of a thorough approach than a checklist can provide.

Goodbye Easter, Hello Patch Tuesday

2007-04-10T12:01:00.000+01:00

Many of us have just gotten back into the office after a nice Easter break eating to much chocolate and too many hot cross buns but thanks to Microsoft you had better not get too complacent as another round of security patches is about to be unleashed.

Not content with just the out-of-band security patch released to counter the Animated Cursor vulnerability Microsoft are going to release an additional five security patches as part of the regular patch Tuesday cycle. Four of these fixes are aimed at the Windows operating system and, as always, the cumulative rating from Microsoft is Critical.

Hope you had a good Easter break because it's back to business.

Animated Cursor Vulnerbility? Not if you get your Security updates from MS via RSS!

2007-04-03T12:01:00.000+01:00

I sure that many of you are aware that later on today Microsoft will be releasing a patch for a zero-day security hole in Windows based operating systems that can be accessed by specially crafted .ANI files.

I subscribe to several security RSS feeds including EEyes zero day tracker and Microsoft's security response center blog

This morning when I checked the RSS feeds I was surprised to see that Microsoft's had not been updated. When I checked the website three news updates had been posted but the RSS feed itself had not been updated.

This vulnerability is easily the most critical zero day since Microsoft moved to monthly patching. The fact that an out of band patch is due to be released today only enforces that and therefore its insane that they haven't done any quality checking on something as simple as the RSS feed for Security Response Centers blog! I hope this is not a sign of the surprise with which Microsoft has been caught out.

I've emailed the security response center and hope to see the RSS feed fixed before the patch is released.

Time Recording

2007-03-23T12:01:00.000Z

Time is a precious commodity for both the individual and the company. Almost every company has a requirement to record time spent on tasks or working with clients yet too few companies actually use this data in a sensible way.

Time recording is something I consider an annoying yet essential task. It is useful to know how long it takes to build a server, to review a document, to configure a network switch and so on.

Proper project time planning is based on the knowledge of how long a task took previously, Anything else is a wild stab in the dark and this is why too many IT projects complete late. In many cases the project managers take wild guesses at how long a task will take to accomplish. Its easier to say "Deploy a server" then it is to accurately break it down into sub tasks of installation, configuration, application installation, application configuration, racking, cabling, firewall processes, change control processes, etc.
Breaking the task down like this allows time can be more accurately recorded and later analysed.
Crucially it will also provide a better estimate of how long those tasks will take to perform next time and thus allow better planning for future projects.

There is only one place that I know of where major tasks are broken down to individual components and where a task is practiced and recorded until the person can almost do it blindfolded in a set amount of time and that's NASA.

Obviously, no company can compete with the sort of resources that NASA has on tap but the ability to record time taken for tasks and use it for planning future projects is straight forward enough for a company to do so why are so many so poor at project time planning?

How urgent do you want that request?

2007-03-16T12:01:00.000Z

"I need this done ASAP, it's urgent"

Sounds familiar? How often do you hear those words or something similar?

It seems to me that all too often managers and users think that by adding the word 'urgent' to any request it will magically get done quicker. Well, here is a bit of news for you:

When everything is urgent, NOTHING can be urgent!

Urgent as a word starts to lose it's meaning. Tasks get completed in a normal time scale because everything is urgent and so prioritization becomes useless.
It almost seems that the word urgent is tacked on to the end of every request simply because people are scared that if they don't add the tag then the task will never get done - No doubt this is because of the high volume of urgent work flooding the team!

The problem with using the term urgent is similar to that of the boy who cried wolf. I'm sure that there have been occasions when genuinely urgent problems have been ignored because that person has cried urgent too many times. Demanding something urgently also has serious implications on quality. Often my response is "Do you want this right or right now?" The bemused look as people try to figure out what I mean is a picture.

In many ways the cry of 'it's urgent' comes from people who have real trouble understanding something I term 'real time' that is, the amount of time a job will take to complete. These are the people who can't understand why it takes half a day to deliver a fully secured, configured server and hence add in the 'it's urgent!' comment.
If these people really do have a valid urgent need EVERYTIME they ask for something then somewhere, something is terribly wrong and probably very badly planned to boot.

I know that right now there will be a bunch of people complaining that it's not their fault, they just pass on requests from above and that may well be true but somewhere an urgent need for something trivial has developed.

In many cases, the company has lived with the item/fix/software/whatever since it's inception so WHY is that very item suddenly so urgent? Most annoyingly the results of many urgent requests have a habit of dying a quiet death only to resurface years later as another urgent request.

Next time you need something urgently have a think. Is it REALLY urgent or is it just you don't want to have to wait?

Microsoft Release Windows 2003 Service Pack 2

2007-03-14T12:01:00.000Z

Microsoft have released Windows 2003 Service Pack 2. It's worth downloading and testing to ensure that you can integrate it into your server build process as soon as possible.

If this release follows Microsoft's standard practice of current minus one then Windows 2003 gold (i.e. no service pack) will no longer be supported for security updates.

For the really curious there is a tech note in the Microsoft Knowledge base that details all the fixes in this service pack.

The service pack will work with both Windows 2003 and Windows 2003 R2 releases.

Free Document Mangement System

2007-03-13T12:01:00.000Z

One of things that IT departments seem to be very good at is producing documentation, diagrams, PDF's and other assorted paperwork. Some of the material produced is actually very good but trying to keep track of it on today's huge sized hard drives in something of a challenge. What is needed is a good document management system.

One of the first document management systems I ever used was called soft solutions and it was by Novell. It integrated very easily into Wordperfect and made finding documents a very simple task. Since then I've been after something similar for personal usage. I have thousands of PDF's on various things along with thousands of documents and it's getting to the point where I'm bored of recreating the same document!

Microsoft and IBM both currently offer products to fit this market, Microsoft offer Sharepoint Portal Server and IBM has Document Manager. Both products will do the job but both are quite 'weighty' in terms of pre-requisites which is no surprise as both are designed to be used by enterprise sized companies.
What I was after was something lightweight. After some searching I found Knowledge Tree which comes in both a commercial and open source version. The open source copy is free for use.

In a future blog entry I will go through the process of setting up Knowledge Tree and importing documents.

A Novel Idea - Let's do what the business wants.

2007-03-09T12:01:00.000Z

I sat in a meeting the other day where a whole new project priority scheme was unveiled based around the unique idea 'Deliver what the business needs'.

The idea was greeted with thunderous silence. I'm sure most of the people in the meeting with me were thinking the same thing "As an IS department and as a SERVICE COMPONENT of the business-at-large weshould be following this model anyway?!".

Certainly there will be projects that IS needs to concentrate on that the business will not directly use and/or see no value in. Those projects are things like networking monitoring and infrastructure upgrades. The business-at-large do not benefit directly from the project but they benefit from the knock on effect of having the IS department respond to problems reported by a good monitoring system before the business feels the impact and they gain by the increased speed/benefits of a better infrastructure.

Ultimately, whatever projects an IS dept runs will need to be justified to the business, sometimes on a case-by-case basis and sometimes IS can lose out - For example, if a web monitoring system is delivered in place of an upgraded payroll system IS can be moaned at for choosing a system that hinders as the priority over a system that will help.

The solution here is to ensure that ALL projects you are running are fully visible to the business. Let them see what's going on. Let them see WHY the web monitoring system is more important than the new payroll system. SHOW THEM why infrastructure in one area needs to be upgraded to support the new payroll system.

The more IS communicates with the business and justifies actions the more the business will come to trust the IS dept as a bunch of people who know what they are doing.

Everything in IS seems to be a juggling act but there should always be room for clear, unambiguous English.

Migration of DHCP database

2007-03-08T12:01:00.000Z

On my home 'production' network I have a single Active Directory server that runs DNS and DHCP. Whilst not fault tolerant it does the job and for a network that can afford the downtime should the domain controller die its a workable solution.

Recently, this server has been running incredibly slowly. Its actually taking 8 minutes to boot.
As this server has been giving sterling service for a couple of years I decided it was probably time to replace the server with something a little faster and a lot cleaner.

Building the replacement domain controller was simple enough, An autobuild of Windows 2000 server then DCPROMO it to be a domain controller.
The FSMO roles transfered over no problems as did the DNS.

DHCP proved to be slightly more problematic.

All DHCP records are held in a database file under %systemroot%\system32\dhcp - Copying this database to the new server didn't work so it was time to hit Technet's knowledge base.

http://support.microsoft.com/?id=325473

The knowledge base pointed me in the direction of a tool called DHCPEXIM which despite the clunky interface is actually very easy to use. Just highlight the scope(s) you want migrate and click on Export.

On your new DHCP server run DHCPEXIM, select import and point it to the file you just exported. It will display a list of the scopes it knows about and bring them all into your DHCP server.

Note that your DHCP server can already have scopes configured but if you try to import a scope and that scope already exists on your server then the import will fail.

This was tested out on Windows 2000 server to Windows 2000 server but the docs say it should work on NT4 and Windows 2003 as well.

VMWare offer free P2V conversion tool

2007-03-07T12:01:00.001Z

I have a windows 2000 domain controller that I want to clone and put into my test network so I can rehearse an upgrade from Exchange 2000 to Exchange 2007 and to test out the Active Directory upgrade from 2000 to 2003. At first I looked at a couple of DR type options. The standard System State backup via NTBackup will restore into VMWare but because it restores a chuck of hardware related information as well the VMWare machine reboots then blue screens - not good. I'm sure there is a way round this with sysprep or with backing up that data separately but my experiments lead to a constantly rebooting server. I also found out that DCPROMO /ADV only works for Windows 2003.

Another option is to join the VM to the live network and DCPromo it as another domain controller, Snapshot, DCPROMO it down then restore the snapshot. This would work but I've heard of problems with errant entries in DNS when this approach is used.

After some head scratching I found a free tool on VMWare's site that does Physical to Virtual conversion (P2V) and it's free for single machines to VMWare workstation or VMWare server. If you want to convert to ESX server or convert a bunch of machines then you need a licence.
I've installed the software onto my Domain controller and I'll have a go and running the conversion this week and report back on how good or bad the process is.

Common Question - How do I move the hibernation file?

2007-03-06T12:01:00.000Z

Simple answer - You can't.

Longer answer - When the PC boots NTLDR knows where it's boot volume is. On that boot volume resides hiberfil.sys - The operating system will take a look at the file and if it's valid and active then the system will use restore the machine to it's hibernated state. If the hibernation file is not active then a normal boot process will occur.

Many people think that a registry hack will allow them to move the hibernation file to another volume but this is not possible because the registry is not loaded at the time NTLDR does the check for the location of the boot volume and the check for valid, active hiberfil.sys - the registry is not loaded and so the hibernation file must be located on the boot volume.

Be careful with security exceptions

2007-03-02T12:01:00.000Z

Last weekend I had an incident with my bank card and I found out that my card had been stopped by my bank 'because of a potential compromise of secured data'. After some digging I found out that somehow my card number had been leaked or the data had been compromised by my bank.
Now this alone is bad enough but what I was told next was worse. I was told that the bank would have contacted me to verify transactions against my account if they had my phone number.

This gave me pause for thought. The bank would have phoned me up and asked me to confirm transactions against my account.

"How would they have know it's me?" I asked, "Oh that's simple Sir. We would have asked you some security questions" he replied.

OK.

"How do I know that it's actually my BANK calling and not some stranger trying to compromise my data?" was my next question.

Silence.

I went on to explain "You ask me security questions to validate that I am who I say I am but I how do I validate that YOU are calling from the place you say you are?"

After some struggle he said that they could confirm they were from the bank by getting me to confirm some transactions but after more pushing he admitted that BEFORE they confirm any transactions they would need my security data.

The upshot of this is that I am expected to give out my confidential security data to a total stranger who may or may not be from my bank. I have no way to verify they are who they say they are.

There is one way the bank could prove it's them contacting me but the person on the other end of the phone never thought about it and neither did I until afterward. I'll leave the validation process as an exercise for the reader!

This whole exchange got me thinking about setting security standards then requiring exceptions that blow those standards out of the water. The classic is the faithful password. We are told time and again that passwords should NOT be revealed to anyone yet I know of one ISP that requests your password as a SECURITY CHECK, OK you have called them but it still promotes bad practice.

In any security configuration you are going to need exceptions, e.g. if your policy stats that all passwords expire once a month you will need exceptions for service accounts. The key to a good security policy and good security practice is to make sure those exceptions are well documented, well understood and sensible. To set a policy and then have a practice that routinely violates that policy is worse than not having one in the first place.

Windows Mobile 6 SDK Released

2007-03-01T12:01:00.000Z

With the announcement of Windows Mobile 6 at the Barcelona Smartphone and despite a false start on 12th February the Mobile 6 SDK is finally available for download from here.

This is the full SDK and requires Visual Studio 2005 to be installed before you can install it. Microsoft has promised a standalone emulator release although they have not given a date for the release.

They also announced that there will be an update around 1st May which should see the 'final' version of the SDK being released.

If you think you will be working with Windows Mobile 6 then you might want to check out Nathan Winters March MMMUG event. This event has Jason Langridge (Mr. Mobile in Microsoft) as a special guest. You can find out more about the event here.

DST Patch for Windows 2000 updated

2007-02-28T12:01:00.001Z

Just a quick update on the DST issue, I've tweaked the MSI I put together so that it can no longer be uninstalled. This is because the MSI overwrites the timezone values in the registry and uninstalling the patch was causing timezones to disappear.
There is no problem with the MSI if you don't uninstall it so I've created a v1.1 that does not allow uninstall.

The patch has been tested on Windows 2000 but not on Windows NT4. I will test on NT4 this week and post a blog article if it works.

The v1.1 patch is available here free of charge.