The Bit Bucket

Busy Project Time.........

2008-10-17T12:01:00.000+01:00

Just when you think it's all going to be quiet and maybe it will be a good time to get those niggling little tasks out of the way and to be able to sit down and write some decent blog articles someone comes up with the idea of decommissioning a server room to save on power. So now I'm involved in a project that requires the relocation of about 4TB of data to another filer, including updating and moving the servers that use the filer data....

Yes, it's going to be a busy few months.

And a project manager just asked me if I needed any help installing IIS...... Sometimes I'd rather be doing anything else than working in IT.

Some DNS Tips

2008-09-15T12:01:00.000+01:00

Several times in just the past week I've had to deal with DNS entries that have made things a touch more painful than they should have been so I thought it might be time for me to jot down a few notes on how DNS should be configured to save IS people's sanity!

First up the DNS servers themselves. You should always have a primary and secondary which generally, speaking are two different DNS servers at your ISP's location. If two are not available you should consider switching ISP's. Personally, I use three. Two from my ISP and one from OpenDNS. This way, should the ISP change for any reason and/or should access be denied to the ISP's DNS servers I've got a third, totally separate service available to me.

Next up, A records. These should always point to the IP address of the server in question and they should always use the hostname of the server. Sure, this can lead to some unfriendly names but it's really handy to know the proper hostname of the server. If you want to use something 'pretty' then use CNames. When you create the A record make sure the PTR record is also created in the reverse look up zone. This way, when you are trying to work out what physical server a CName is all you have to do is a reverse lookup against the IP address.

MX Records should also have two internal/DMZ based mail servers which they can deliver to and a third at the ISP which can retry delivery to your internal servers at a later date.

These are simple tips and they (or variants of them) can be found as best practice advice for standard DNS configurations.

Understanding your environment

2008-09-08T12:01:00.001+01:00

A practical demonstration of why understanding your environment is vital occurred a few evenings ago when some NetApp filer\domino work went wrong. A little bit of background first, domino data is stored on a NetApp filer which is shared using nfs. This is mounted by the domino server and it all (most of the time) works.

For some reason this particular server running Domino (let's call him Bob) was showing high i/o stats, although the server itself was responding fine. The filer (Nutkins) wasn't reporting any problems but it was deemed that Nutkins had to be at fault. There are a lot of connections to Nutkins after all and in fairness the mount point is living in an aggregate that is unbalanced in terms of i/o profile so the decision was made to create a new aggregate decided for Bob. Simple enough to do. For those not filer aware an aggregate is a collection of physical disks. In giving Bob his own aggregate it dedicated 8 spindles to the Domino data. More than enough to remove any i/o bottleneck.

Now, Nutkins itself has a very cool piece of technology called snapmirror. A snapmirror was duly setup and Nutkins began copying the data to its new home.

So, the big evening arrives. The paperwork is signed (in blood, naturally). The changes authorised, the servers poised....... A hush descends and the commands to stop Domino are typed into Bob......... and Domino promptly hangs.

Red flag 1 - when a manager says "oh, it always does that. Just issue kill -9 and everything will be fine, well except that a few databses might be corrupt" it's probably time to start worrying. However, the final snapmirror is initiated and the last 140mb of changes are copied (in 22 seconds no less, not even enough time to get a cup of tea). The snapmirror is then quiesed and broken. This makes the destination for the snapmirror writable. Over to the unix admin and a few key clicks later the export is mounted and Bob was started.........
Or not. Seems that a small fact was missed. Bob not only has data stored on Nutkins but also has a local directory for crash dump logs.

Red-flag 2 - when Bob's admin doesn't know the configuration of Bob's setup it is probably time to start panicking. Anyway, a tappety-tap of the keyboard and the directory is created. Oh, lets stop and start Bob hoping red flag 1 doesn't pop up. Mr. Unix issues the command and on the screen "server shutdown. Bob_stop not found". Ok, so did it shut down or not? Ps -ef | grep lotus and nope, nothing running. Red flag 1 avoided! So, start Bob and..... Nothing. Not happy. Hmmm. Time to fail back, something isn't understood\not working.. So Mr. Unix does his stuff and...... No Bob. Seems red flag 1 corrupted the data then the final snapmirror copied corrupt data. Also seems that the shutdown script has at least one bug in it which causes a loop to fail when the script is executed.

Anyway, to cut a long story short we backed out and made the change a few days later. There are several lessons learnt here mostly revolving around documentation, standarisation and knowing your environment. I'll leave it as an excercise to the reader to work out the rest!

AD Find

2008-08-21T12:01:00.001+01:00

AD Find is the second of the two tools I managed to find in the same week. This little tool weighs in at just 700K for the download and about 2mb for the actual file. This tool does exactly what it says, it finds things in Active Directory. The clever part about it is it's possible to say exactly what you want to get back and the format it should be in.
As an example, a few weeks back I had the issue with Bindview not liking non-ASCII characters.

Now, the version of Bindview that's being used where I work is a very old NT4 only aware application which means it will update the SAMAccountName attribute but not the display name.

This isn't a problem as there is a workflow from an HR application which deals with all of that, all bindivew should be doing is delegated group permissions (and yes, I know it's much easier in AD but thats a war story for another time).

Anyway, I was curious to know how many SAMAccountNames didn't match up with display names so I used ADFind to display the CN, Samaccountname, mail, firstname and lastname fields in a CSV format which could then be processed by a filer in Excel. Much quicker than messing around with the native Active Directory tools.

AD Explorer from Sysinternals

2008-08-21T12:01:00.000+01:00

Sometimes it's possible to stumble upon a tool and wonder just how you would have gotten a task accomplished without it. Last week I had the good fortune to stumble upon two such applications right at the time when I needed them most. I did consider buying a lottery ticket that evening!

The first one is AD Explorer and it's from sysinternals and it's exactly what it says, a explorer tool for Active Directory. It allows viewing, searching and editing of the AD in ways that are far superior to Active Directory Users and Computers. I suspect the only thing that AD users and computers can do (or do better) that this tool cannot are password changes, logon hour restrictions and limiting logon ID's to specific computers.

One very nice feature this tool has is the ability to take a snapshot of an Active Directory and compare it to another snapshot. Doing this shows just how many changes occur in the AD in just a few days. It's also a great way to see how many differences accumulate between your production and test active directory environments.

Overall this is a fantastic tool and one I'll be using when the MS technotes require delving into some obscure key via ADSIEdit. I'll also be using it in place of tools like Softerras LDAP browser unless I need to something LDAP specfic.

Why Total Cost of Ownership is a fallacy

2008-08-01T12:01:00.001+01:00

If I have one more potential supplier try and sell me something on the lie that it will "reduce TCO" I will not only scream but I will beat them to death with a CAT 5 cable.

Total Cost of Ownership (TCO) is one of those almost unmeasurable values that seems to have pride of place in the salespersons portfolio. How do they KNOW a new system (with it's associated equipment, licensing and training costs) will work out cheaper than the old one?
The idea is that newer systems have better support so rather than training someone in an older system and maybe having to buy in more expensive skills more legacy systems it works out cheaper to upgrade or replace with the latest model.

I don't disagree that for some systems which are truly legacy such the old DOS or OS/2 application may well work out cheaper in the long run but the one thing that will truly reduce TCO?

Understand your systems.

Take time to test and document the fixes.

Use your call logging system as a knowledge base.

These three tips alone will truly reduce TCO.

VMWare course

2008-07-15T12:01:00.001+01:00

For much of this week I'm on a VMWare course for the second half of my VMWare training. This part of the course is titled Deploy, Secure and Analyse. The course itself is to prepare me for a server consolidation project that the company I work for is kicking off.
The project invovles several VMWare clusters, a Hitachi SAN and blades. Lots of flashing lights and new technology to ~~break~~ support.

Legacy Systems and a very handy SQL comparrison Tool

2008-07-13T16:16:00.002+01:00

On Friday, I had the "pleasure" of having to get a legacy system up and running.
This system was originally introduced to allow users in the business to manage group membership for projects they had ownership of. The idea was that it would cut down user calls to the service desk by about 10% and allow the project managers themselves to get a speedier turn around for new starters.
Sounds fine in theory and in the world of NT4 it wasn't a problem. Move on to the world of Active Directory and things are a little different. The legacy system (Bindview v4.6) has been superceded about 5 times over but we can't just install the latest version. Trust me on this, the latest version is fine but there are many design decisions and compromises as well as several rejections for upgrading the system from a few years back that have all combined to lead to the current problem.

The actual problem was an interesting one. The system was complaining whenever anyone tried to edit a group. A restore of the back end SQL database fixed the problem until the next domain sync occurred when the database would corrupt itself again.

Obviously, the sync was pulling something from the domain that it didn't like.
For the first attempt at a fix I fired up SQL Trace which records every single SQL statement that goes to a selected database. The neat thing about Trace is that it's possible to point the trace results to a SQL database itself and then filter it to get rid of stuff you know isn't going to help - such as SQL agent tasks and so on.
Trace left me with a multi-variable SQL script spanning over 4,000 lines and quite difficult to read or even test so I decided that the next best thing was to restore the working database to new a database name and then find a tool to compare every object on the bindview user table to see what was different between the restore and the one that synced with the domain and promptly broke.

AdeptSQL was the third tool I tried and whilst it has a very simplistic point and click interface it's incredibly powerful for comparing two SQL databases. Once the comparison is done you get two side-by-side windows which represent the two databases. Changes are highlighted by colour - Red for deletions, Blue for new and black for no changes.
This left me with a 2,000 list of changes, deletions and amendments in the database.
AdeptSQL also lets you filter things out and by using these features I eventually tracked the problem down to the description field of two user accounts.
These accounts had spurious characters in them which Bindview being rather old and totally ASCII prompt fell over on. Removing these and waiting for a resync solved the problem.

Whilst AdeptSQL helped me solve that particular problem there is still the problem of this legacy system updating Active Directory whilst not being active directory aware which leads to some other fun and games with the display name versus the SAMAccount name but more on that in a later article.

Build your own NAS

2008-06-30T12:01:00.001+01:00

Things have really moved on in terms of storage. Not so long ago the largest hard drive you could buy for a home PC was a 200GB IDE. Today, 1TB SATA hard drives are available for less than £100 from my favourite hardware website AUT Direct.

I'll admit that I couldn't resist for long and as I've got a tower PC with 6 IDE hard disks in which are not doing anything at present it was just too much of a lure and I've ordered up 4 1TB disks.
The plan is to replace four of the IDE disks with these 1TB SATA drives and I've bought the necessary SATA drive bays to making swapping them out easier if needed.

As the motherboard is quite old I also purchased two SATA cards which will be able to handle the SATA disks.

The tower also has two IDE disks on an IDE expansion card. This was originally for the OS but I'm going to pull that
and put one of the SATA cards in it's place. The IDE disks are small (either 10 or 20GB) which I'm going to bin and replace with two 250GB IDE disks.

In total the box will have about 4.5TB raw storage capability. I need to configure the 4 SATA drives as RAID 5 in case of a failure. I also want to configure the two 250GB IDE's as RAID1 for the same reason but testing in in VMWare showed it wasn't quite that easy.

The operating system of choice will be OpenFiler. This OS supports all sorts of storage options including CIFS, NFS and iSCSI. It's free and actually supports more than some hardware solutions such as the Buaffalo terrastation I recently bought!

Even so, When finished and configured with the RAID arrays the box should be able to support an impressive 3.2 or so TB or usable storage.

A fun little project......!

Issues upgrading Domain Schema to 2003

2008-06-29T12:01:00.001+01:00

So I'm probably a little behind in upgrading my home networks domain schema to support Windows 2003 but better late than never!
The process itself was smooth enough once I'd corrected some problems on the machine but the upgrade logs were not the most helpful troubleshooting aid I've come across.
One particular error had me stumped for a few days:

"Error code: 0x57 Error message: The parameter is incorrect.."

No indication of which parameter it was but as it occurred when checking security descriptors and many blog articles refer to missing security ACL's on GPO's I had a look at those and sure enough, Enterprise admins was missing some rights so I fixed those up and....... the same problem. At this point I'd admit to a lot of head scratching. The event logs didn't shed much light until I realised that the security event logs were not accessible. Sure enough, somehow the ACL's on the security event logs had lost all their rights. Resetting these and then rebooting allowed the process to complete perfectly.

ITIL Overview Training

2008-06-20T10:23:00.005+01:00

The company I'm currently working at have decided that ITIL is the way forward. Yes, after several years of different ideas, options, tests and other madness they want to adopt the official ITIL framework over a period of 6-7 months.

Now, whilst I think that ITIL is a good idea and yes, I am something of a convert to the whole ITIL structure I think that the nature of the user/customer base here is simply one that won't tolerate the ITIL way of doing things because it will require them to become more proactive and less reactive. I really do believe that many IT departments are products of the greater company in which they find themselves. Have a company that's reactive and unstructured then your IT department will be as well because it fits in to the business model.

Still, the training was interesting if a little dry and I picked up a few things on Problem Management and Root Cause Analysis. Something I'm very interested in because of the way it deals with problems and provides permanent documented fixes. This is something I'll go into in more detail in a later blog.

As for ITIL here, well.... I really do hope it works but I can see it being a somewhat half-hearted implementation unless the business are prepared to be a little more structured.

The final thing I'll say on ITIL is that it's a nice framework with a focus on how IT should be run but it doesn't address any sort of approach for bringing it into the business. I know that ITIL practitioners will say that this is because each business is different but it would be nice to read some success stories and find out just how they implemented ITIL and what order they implemented it.

Posting update

2008-06-09T12:01:00.000+01:00

Yes I know I've not posted for a bit. No excuses and I promise I will try to be good for here on in!

Lot's of changes at work and enough material to fill the blog every day for a year but I do need to actually get on with writing some of it down!

One article a week from here on in. Not a new years resolution but a start of summer resolution.

InfoSec 2008

2008-04-28T12:01:00.000+01:00

Well, After some false starts involving problems with London Undergrounds District Line I made it to Olympia and to Infosec 2008. The event itself is a good one for picking up the latest trends in security and seeing a few demo's of various products and as always there was some good stuff to see there.

For example, Sophos have come on in leaps and bounds and I was most impressed with their new AV console. It can also do NAP (where a machine is quarantined until it means a specific criteria for patches and AV).

The Sophos solution also has a web based applet which can be deployed to guest machines (i.e. visitors). The classic here was the sales guy who was demonstrating it was telling me just how clean the solution was "It uninstall's without a trace so we don't change a THING on the users machine" he extolled. Hmm. But if it doesn't met the policy then the remediation servers will be the only ones the user can see. This allows the user to update AV definitions and patches. Now, if we can't touch a visitors machine then what's the point? It's a nice technology but worthless for that reason.
Guest machines should be in an isolated vlan with only net access. They should not only be isolated from the production network but from each other as well.

The Microsoft seminar was superficial but I did learn a few things about their NAT offering in Windows Server 2008 and it does look useful. Certainly on the "to test" list.

Overall, I came away from Infosec slightly underwhelmed. There didn't seem to be any new technologies or ideas that made me feel "yes, I like this. This is a good way forward". The last time I had that feeling was with Splunk and I still think that about the product. I do wonder if security is falling into something of a rut just waiting for the next big attack.......

nLite Automated builds

2008-04-14T12:01:00.002+01:00

I'm a big fan of unattended builds and I've been using them for over five years now. The process of creating an unattended build can be somewhat hit and miss so using something like VMWare to test the final build is often an essential.

nLite has been around for a while but the last time I used it I found that the resultant build could be flaky and often just not work.
These issues seems to have been fixed with current version as it's remarkably easy to create a custom build and to add service packs, drivers and patches.

Overall I'm very impressed with the tool and at price tag which is free I really cannot complain!

Mac OS

2008-02-26T12:01:00.000Z

I'm not much of a Mac fan. This is simply because I don't have a need to use Mac's. I have friends who use and love the Mac book pro and I've seen a few being used on the train when I travel into and back home from work. I'm still not a fan though so never looked into Mac OS until a few days ago when I was testing out a new security tool for some due dilligence work that was required and a copy of Mac OS would have been very useful for testing.

Could I just go to Apple's site and download a trial? Nope. Not allowed. It seems insane to me that Apple have no ability to allow the regular intel user the ability to try out Mac OS without having to buy the hardware. This policy must be causing Mac sales.

Mail for Exchange documentation woes

2008-02-22T12:01:00.000Z

Too many times now I've come across badly written documentation. That is documentation that leaves you hanging wondering "what next?" or "where do I go from here?".

An example of this is the Mail for Exchange application on my Nokia E61. Having spent no less than 4 hours trying to get it to work and still having no joy I realized just how painful the documentation is. I'll cover the fun and games with Mail for Exchange in a later article but for now I just want to highlight how badly written the documentation is.
When configuring my phone to connect to my Exchange server over wireless I get an error "Communication error, retry later". The documentation has a section that reads "Troubleshooting - Errors you may receive" and lists that error with no fix or reason why that error is occurring.

Thanks Nokia.

If you are going to present the user with an error you should at least give the user and idea of what to do with it.

Get ready for a bumper patch Tuesday

2008-02-12T12:01:00.000Z

With no less than 12 security updates coming out of Microsoft later on today and Vista SP1 slated for February 15 there will be a lot of update servers groaning under the weight of so many updates to download so it's probably a good idea to ensure your WSUS servers have plenty of free disk space and are as up to date as possible now to ensure they download the minimum necessary during the next couple of weeks.

Call for Eicar V2

2008-02-04T13:01:00.000Z

Many years ago it was recognised that there existed a need to test AV software without throwing live viruses around and so the EICAR test file was developed as a safe way of testing that AV software was indeed working.
This was fine but I think there is now a need for an EICAR v2. Something that is NOT recognised by AV software by default. Why would this be of use?

Well, A scenario I had last week involved a virus getting onto NetApp filers. Now, Netapp will send the file to an AV scanner and get one of three responses back: clean, infected or timed out.
Clean means the file gets added to the clean list and will not be rescanned until the file changes.
In other words, if the file has a virus that the definitions do not pick up that file is NOT rescanned even if new definitions are released. This means a virus-infected file can get onto a NetApp system.

Having an EICARv2 test file will enable testing of the automatic clean-list clearing type of scenario and be very useful to the IS industry in general.

First Patch Tuesday of 2008

2008-01-04T12:01:00.000Z

The first patch Tuesday of the year is rolling around somewhat early this year as the second Tuesday is on the 8th. Microsoft has decided to be kind this month though as only one critical and one important patch are being released.

It seems though that Microsoft's RSS feed is suffering from a New year hangover as it's not been updated with the information on the website.

Data lost by Revenue and Customs

2007-11-20T12:01:00.000Z

The news story linked above talks about the UK Government losing 25 million records containing names, addresses, national insurance numbers and bank details.

Apparently the data was password protected but not encrypted, Now depending on the application used there may be some encryption there. I'm hoping that the data is an encrypted database that also has a password on it which is where the confusion is coming from but why do I have a feeling that it's just a CSV file?

The thing is, this is NOT NEWS. It's happened before, there have been reviews and procedures created yet it KEEPS happening. It happens in pretty much all companies and yet no one seems to care.

I, for the life of me, cannot work out why security is second fiddle. With word terrorism, bank fraud, phishing and everything else why am I and other members of the IT security industry still fighting an uphill battle? What is it going to take to get security onto the agenda?

The state of IT

2007-11-01T12:01:00.000Z

I came across the above article earlier today and I know that examples of the above problems are not just endemic to development process but instead seem to be buried deep into the very psyche of the majority of IT projects today.

I honestly would not been surprised to see Matt Allwright of BBC's Rogue Traders pop up at some of the meetings and accused the attendees of doing a shabby job and, of course, they would be right.

The classic in the above linked article is the very last email complaining that 'I'd love to write a dev env setup guide, but I just don't have the time!'. Hang, Didn't that email exchange basically list most of the steps needed? If there is time for the email exchange and time to waste someones time in scrabbling around for this information then the setup guide could have been written ages ago!!

We, as IT professionals are constantly subjected to these shabby practices and yet we don't accept them from other professionals so why should we in our own industry?

Snooping on Facebook user profiles is a 'staff perk'

2007-10-29T12:01:00.000Z

I'm not a huge fan of Facebook as I really don't see the point of sites like these. Generally, If I've not spoken to anyone in a number of years then there is a reason for it so I really don't want to hook up with them again thanks to facebook.

At the end of June I wrote that facebook users provide far too many personal details and were at risk of identify fraud. Well it seems that others are just catching on to this idea with several horror stories of exactly that in the media and then today The Register has this l ittle gem of a story.

So it seems privacy settings on facebook are absolutely meaningless and staff consider snooping a 'perk'. If users privacy is treated in such a cavalier fashion by those that administer the site I can for see a risk that users will become more blasé about risks surrounding identity theft which will, in turn, create an entire identity theft industry around facebook.

Centralised Logging

2007-10-25T12:01:00.000+01:00

One of the essential features for even a small network is a centralised logging solution.
Having a centralised logging tool makes for much easier trouble shooting as it becomes possible to review logs and search for related events or even search for the same event on separate machines, traditionally this has required quite expensive software such as HP open view in order to implement but a fairly new company might be about to put an end to that.

Enter Splunk, The 'Google of IT data'. This application will happily collect all sorts of different logs once configured and the configuration is not too difficult.

Splunk needs to be installed onto a Linux, Mac or Solaris environment although a Windows version is promised soon. As a workaround Splunk recommend that SNARE is installed on Windows servers. This software will convert event logs into syslog format and send them to a named server.

Putting Splunk in the center of you logging infrastructure as a syslog server and pointing all your syslog capable devices at it and then using SNARE to roll up event logs as syslogs which also get sent to Splunk is very easy to do. Within a few hours you have a surprising amount of data available to be searched by splunk.

And the price for all this information?

SNARE is free, Splunk is free if the amount of data you send to the Splunk server is less than 500mb a day although some of the features are limited.

I will admit to being a fan of Splunk after playing with it in VMWare. Over the next few weeks I'm going to describe how to configure a simple splunk installation for Linux, Windows, NetApp filers and Cisco switches.

NT4 Emulator Key

2007-10-02T12:01:00.000+01:00

If you happen to run a large Windows environment you might be familiar with the in place upgrade method of upgrading your domain to Active Directory. If you run a large Windows environment that spans several sites over a variety of links then you will know that an in place upgrade can be a pain.

The main problem with an in place upgrade stems from the fact that client machines will always prefer to talk to the Active Directory server instead of the Windows NT4 Backup Domain Controller. This means you can end up in a situation where a remote sites clients are traversing a poor link to authenticate against the Active Directory server and ignoring the local NT4 Domain Controller.

To work around this issue Microsoft provide a registry hack called the Windows NT4 Emulation key. If a DWORD key called NT4Emulator is created in HKLM/System/CurrentControlSet/Netlogon/Parameters and given the value of 1 is created then the server will 'pretend' to be a Windows NT4 server thus the client machines do not see any Active Directory domain controllers on the network and so will be quite happy to authenticate locally.

I'll cover this key and some of it's drawbacks in some later articles.

If in doubt, reboot........ the train........

2007-09-24T12:01:00.000+01:00

My journey into work is normally quite uneventful. Since the move out to Kent it generally takes 20 minutes longer but the journey is actually fairly pleasant. Today was the exception.

About 20 minutes into the journey the trains brakes come on pretty hard slamming the train to a stop and we sat there for a couple of minutes before the guard come onto the tannoy to explain that there was a problem with the trains brakes (really?!) and that there were going to try a fix... This is the point that they REBOOTED the train. I kid you not, the annunciator at both ends of the coach went out, the air con died and the lights all went out......... A few minutes in the quiet and everything came back on but I would have loved to have seen a BIOS start up message scroll across the annunciators!

As a side note in this case the fix didn't work and the train was taken out of service at Orpington but I swear that's the first time I've been on a train that's needed a reboot!