Categories
Fixed Rugby RWC Uncategorized

Wales, Warburton, tip tackles and red cards. One day later

Whatever your view of the tackle that led to Sam Warburton\’s red card after 17 minutes in yesterday\’s first Rugby World Cup semi-final; there is one clear fact. It unambiguously changed the game.
Whether for better or worse, Wales played their socks off and the tackle count (France 126, Wales 56) clearly shows who was making the game.

But whether you agree with the red card or not, whether you think the tackle was dangerous or not; it is clear that this decision likely caused a change in the game\’s progress more than any try could have.

It is deeply ironic (to me that is) that when there is barely any room for doubt, and both touch judges have agreed with a referee that a try has been scored that the ref can still ask the TMO to adjudicate with a clear question \”any reason not to award a try\” or \”can you confirm that he was not in touch before scoring\” or \”can you confirm he was in control of the ball when it was grounded\” etc etc.

So why on earth can a referee pull out and flash his red card in a few seconds without reference to touch judges or TMO and set a team back by 1 player for the remainder of the match.
It is clearly unfair.

I propose that for all red card decisions the referee should be compelled to get the TMO\’s agreement that the alleged offence justifies the largest sanction on the field.

How could anyone object?

Wales were robbed of their chance, and an unimpressive French side will meet New Zealand and, I hope, get thoroughly thrashed. Although New Zealand getting a second world cup result without having to play hard in the final would devalue the trophy, and not clear the Kiwis of their \”chokers\” tag; I\’d much rather New Zealand won having had to play out of their skins – unlikely to be necessary on the evidence seen so far.

Yesterday I was Welsh, next Sunday I will be a Kiwi. I had hoped to be wearing and waving the cross of St George, but 30 uninspiring plonkers put paid to that…

Categories
Backdoor Fail Fixed Problem Succeed Uncategorized vmware

ESXi 4.1 Update 1 travail – lessons learned.

I’ve been biding my time over the last few months to migrate to ESXi.  Knowing that ESX4.1 is that last edition of the “full fat” VMware, I knew my next move would have to be to ESXi, so rather than make a bigger job whenever (cough) 5.0 is launched. I thought I’d change over the long weekend when I knew clients would be closed.
 
It was entertaining.
 
Building a boot and install USB stick rather than using a DVD burned with an ISO image was an important part of the test.
This is going to come in useful next month as I have some client work then, where the dirty nature of the computer room (a breeze block room in the corner of the warehouse) means that DVD drives become unusable within a few months – I dread to think (and am not responsible for!) the state of the servers and SAN…  So anyway I want to be able to boot and install from USB if necessary.
http://blog.vmpros.nl/2010/09/03/vmware-how-to-create-a-bootable-esxi-usb-stick/ didn’t really work for me, but http://www.ivobeerens.nl/?p=699 proved to be a good source of a procedure on how to do this.  However there are some caveats to the process:
  • Syslinux 4.0.4 (the latest) does not work (or at least did not for me) – stick to 4.0.3!!
  • When modifying the contents of the stick remember to do everything!
  • Whilst the storage in my instance is software iSCSI IT IS IMMENSELY PRUDENT TO DISCONNECT STORAGE.  As this install process initialises some storage, you do not want to accidentally wipe a LUN.  My recommendation is always to build ESX(i) hosts disconnected from storage.  It prevents an easily avoidable mistake.  Likewise I avoid “Boot from SAN” setup.
  • Make sure you follow all the steps. I managed to miss 1 or 2 a few times before I got it right.
  • Don’t forget that the KS.CFG is YOUR INSTALL SCRIPT.  It’s easy to forget this and take the content and run with it.  If you do, you’ll get an ESX box with 192.168.1.10 as its IP, VMware01 as the root password, and ESXi-01.beerens.local as its full name connected to a domain “beerens.local”.  I could be wrong, but I think this is unlikely to work in your world J
 
So once the stick is done:
  • Check for any Anti-Affinity rules in DRS, this will make sure your VM’s can have maximum mobility around the farm during the change.  You may want to weaken them
  • Move any non-running servers off local storage (if there is any) to SAN or other shared storage – cut and paste or storage migrate.  If you storage migrate you can change the host as well to unregister them from the server.
  • Storage migrate all running VM’s on local storage off the server to shared storage (no downtime here).
  • Put the ESX host in maintenance mode (and take the option to migrate all paused and stopped machines off the host).  All running guests will migrate off
This will leave you with a host doing no work, and having no VM’s stored in its local storage.
 
Now, and this is optional, but I highly recommend it.
  • Document the server setup – including network settings, iSCSI paths, vSwitch names and configs.  In fact everything you can!!!  If you are licenced for it, then consider Host Profiles as a means to the end.
  • Disconnect all external storage connections, and verify this by checking via vCentre.
 
 
Now you can start, insert the USB, boot the server, select boot from USB if required and watch it install.  If you have boot from USB as default, then at the end of the install you should remove the USB before it boots again.
Your KS.CFG will do the initial configuration and you have a new ESXi server.
 
This is where some of my fun started.  Now please bear with me – some of this was done late at night over a bank holiday, so I did not do my more normal thorough investigation, and I do not have answers to all the questions, but a list of issues encountered and some observations.
 
  1. vCentre
I thought my vCentre was up to date.  I was lazy, it was not.  I discovered on adding the new host to my network that there were some management issues from VC to ESX.  So I needed to upgrade vCentre.  I also discovered that some VM’s would not start when running on the new host – it seems they were mostly VM Version 4; but also (to make things harder) VMtools needs to be updated too!
 
  1. vCentre upgrade ISO
This is a 2.2GB download.  You do not want to do this on a 512KB ADSL connection.  I hoiked out my 3G MiFi unit, and downloaded it over the air instead to the laptop.  I achieved a 10 fold performance benefit by using this.  Fortunately I had 3.5GB left on the monthly allowance, so all was well.
 
  1. vCentre Upgrade action
Sadly this is a lengthy process, but by using full documentation from the installation (you do have this don’t you?) I was able to breeze through the dialog boxes and get everything up to date except Update Manager.  For some reason that part of the ISO is corrupt.  I am downloading it again as I type.
For prudence I snapshotted the VM that is the VC before starting.  At times later on, I would be tempted to restore to this, put ESX4.1 back on the host and give up.
Oh, and don’t forget to take the in place upgrade option – if you go for a new database your whole farm is screwed! (no, I didn’t)
 
  1. vCentre Client upgrade
On starting the vCentre Client, the new VC edition wants an upgrade before I can connect to it.  This install fails…
Now this was fun… My main management server (physical still – for good historical reasons), is where I do most of the work.   However this is now 6 years old and has a large number of VMware components go through it.  Unfortunately… some old MST file was hanging around and the VI Client upgrade failed.  By now it was late at night after a quick burst of investigation I decided on a more radical approach.  I stopped all VMware services, hacked out all the VMware stuff from the registry, killed VMware folders in Program Files, and rebooted the machine.  This did not completely fix the install, and found a few more VMware folders in the Documents and Settings tree, they went too.
 
  1. DNS and AD failure
Yes, you read that right.  When this box came back DNS was down, and AD was not working as a consequence.  Fearing I’d ripped something out I hadn’t meant to I was tempted to hit the backup tape (you do take backups don’t you?) but waited a bit…
This being more a test lab than a production network the primary physical box on which I was working is the original DC of the network.   The other DC’s are virtual, and it turned out that neither had started properly when I had restarted the ESX hosts a bit earlier.  We had had a power cut earlier in the day, and whilst the kit had all stayed up, it seemed (only with hindsight) that whilst I have UPS’s all round a slight barf on one UPS had impacted a network switch and the virtual world was not talking to the physical world properly.  Taking the IT Crowd “Turn it off and on again” philosophy to its logical limit… I shut down all the VM guests (you do have a PowerShell script for this don’t you?!) and shutdown the hosts.  I then power cycled the switches and waited for them to come back.  I then booted the ESX boxes, and the physical server and all was well.  A quick check round logs and events proved this was the case.
I’m not going to try to work out why, as this was now 1am…
 
  1. vCentre client now installed properly and I can connect to vCentre Server again.
A quick bit of configuration of vSwitches, and all seemed to be well except…
 
  1. iSCSI connections
One of the iSCSI connections relies on decent security from the SAN side – and with the new ESXi installation the IQN’s on the software iSCSI had changed, so the SAN had to be told it was allowed to connect!  A quick fix there, and the new ESXi box can see all storage, and works a treat.
 
  1. Finally all was well
 
  1. So I just need that good ISO for the Update Manager installation so that I can now manage updates across the VM’s (VM Version and VMTools for now).
 
 
Observations?
  • Well you can see from the above that Douglas Adams was right when he wrote “Don’t Panic” – I could have given up with the backups, snapshots and original ESX4.1 that I had and gone back to square one.
  • Document your setup, NOW.  You never know when it might come in useful
  • In ESXi the Service Console no longer exists – look for the Management Network in your ESXi networking setup
  • IQN’s can change
  • Check your VM version – some of your older VM’s may be 4 instead of 7.  In my experience, a VM version 4 had some issues starting and seeing network hardware on a new host.
  • Anti-affinity – keep an eye on it, and restore it when done
  • If you use ESXTOP on ESX, don’t forget – without the service console, you won’t get this on the host
  • ILO – if you have it, make sure you know the password – it saves a lot of hassle connecting to the host
  • Lastly NEVER FORGET you can use the VI Client directly to the host to work things.  If the VC goes down, it means you can still start stop guests, enter/exit maintenance mode, reboot and shutdown an ESX box.  This can be your friend.  A lot.
 
 
Oh, and very lastly – if you finish work at nearly 3am in the morning after some problems like this, then the early morning Radio4 news on the day Osama Bin Laden is killed makes for a pretty good wakeup call.
 

Categories
BitLocker Fixed Hyper-V Microsoft Problem Succeed twitter Uncategorized vmware Windows 7

Hurrah – a hibernating Hyper-V laptop!

Well, almost J

I got a new laptop last year and having bumped up the RAM and disk, I wanted to use for a virtualised lab on board whilst travelling or at clients.  Having experimented and asked around on Twitter there was no way (my preferred method) of having Windows 7 with ESXi running under VMware Workstation and then have 64bit guests in vCentre – the VT is not exposed to the ESX guests.  This would have given me the best of ESXi (and a VMware lab), and the VM’s I wanted for carrying a lab in the bag.  VMware workstation was not much use to me as without any memory management I would run out of headroom (although the tree cloned drives would be nice).
A non-trivial additional factor was that I insist on encrypted disks in my laptops.
I then experimented with getting a dual boot world going.  BitLocker and Boot from VHD work well, but not together.  I got a Bitlockered guest machine under Hyper-V as a VHD to boot, but the content was a bit flaky – device drivers).  I then tried getting a dual boot to work with the second boot from a VHD but BitLocker got in the way.  See: Am I really asking too much of Hyper-V  I learned a bit about BDCEDIT along the way.
Eventually after a couple of gotchas/glitches I gave up on the BitLocker VHD or alternate boot option as it was taking too much time (and I had read in a few places I was asking the impossible).    And besides: Word from the wise on BitLocker
Becoming impatient, I then restarted my thinking.  I continued with the Windows 2008 R2 build (Bitlockered drive), with the intent of then building the VM’s that I wanted.
The first bit was to get Windows Server 2008 R2 look more like Windows 7 so it could be my standard desktop-like working world along with some other bits and pieces – I added the following to the machine (some are dependencies):
  • Web Server (IIS)
  • .NET Framework 3.5.1 Features
  • BITS
  • Desktop Experience
  • Ink and Handwriting Services (it’s a tablet)
  • Remote Server Administration Tools
  • Telnet Client (I never usually remember this is off by default!)
  • PowerShell ISE
  • Windows Server Backup Features
  • Wireless LAN Service (it’s a laptop!)
  • BitLocker Drive Encryption
  • Group Policy Management
  • Windows Server Migration Tools (just in case)
I then installed all the usual productivity tools, Office, DropBox, the loathsome iTunes etc. etc.
However, Hyper-V cannot use a Wi-Fi network for external access.  My Lab network is behind a Threat Management Gateway 2010 Server, so only this needs true connectivity.  So a quick bit of research, and I came across the idea of a bridge between the Hyper-V network and the Wi-Fi here: Connecting Hyper-V over WiFi and it works a treat.
So the laptop was where I wanted it to be, the VM’s were being created.  BUT….  You cannot hibernate a Hyper-V machine.  This is clearly a sensible idea, but for the road warrior, it’s more than a nice to have.  To wait for a machine to fully shutdown can be embarrassingly long.
So over to the internet.
The first hit was “Create Dual Boot” solution.  This works by duplicating the boot entry (back to BCDEDIT), and then you choose to run with or without Hyper-V.  Without Hyper-V you can hibernate the machine and bring it back quickly.  But you need to reboot the machine to get Hyper-V back, and then you can start your VM’s.  After that you can run your productivity apps, but can no longer hibernate the machine.  This can be found here: Creating a no hypervisor boot entry on Windows Server 2008
And then I found this:
Hibernate and sleep with Windows Server
All you do is the following three steps:
  • Set Hyper-V to start on demand “SC CONFIG HVBOOT START= DEMAND” (note the space after the = sign); then reboot the machine
  • Enable Hibernation “POWERCFG -HIBERNATE ON”
  • Then when you want to run VM’s – “NET START HVBOOT”
Lo and behold.  I have a single boot machine.  Until I start HVBOOT then the machine will hibernate.  Once you have started HVBOOT, then you have to shut down the machine instead, but this is good enough for now.  I’m not certain what impact not running Hyper-V will have on the performance of the machine, but not much I guess.
What next?
Well I guess that I might put VMware Workstation on as well to get some VM’s running whilst still being able to hibernate – maybe just 1 or two so that I can PowerShell in Windows 7 as well…  If only Workstation could use VHD’s (or Hyper-V VMDK’s!!!!)
Oh, and if you try to start a VM without HVBOOT running?  You get this:

Categories
Exchange Exchange 2007 Fixed Problem Uncategorized

Exchange 2007 problem licked (I hope!)

I\’ve had a number of occasions when my test lab Exchange 2007 servers (a Mailbox and a HubCas box) start up and then give me errors on a number of service and fail to start. This is normally because I\’ve taken action to stop all my VM\’s copy some datastores and then start up again. It\’s usually a weekend activity, but due to power fluctuations and some hassle, it happened this afternoon.

Usually some brute force stop/start activity (it is a test lab!) fixes it, but over a period of time, and today was no exception. However trouble struck again tonight, and for some reason I decided enough was enough. The events (amongst others) are 2114 on DS Access, and a whole bunch on Topology checks

I hit the usual options (Google, Microsoft knowledgebase, and eventide.net) and came up with a bunch of links that I looked at, a couple interested me (for different reasons!)
EventID.net
Microsoft Knowledgebase  (note how this one is nicely for Exchange 2003/2000 and not 2007!)


Event id’s page came up with a whole host of options, but in this case the words
“A possible root cause is an additional DNS A record for a DC in the Exchange Servers Site, record that happens to be for an interface for which the Exchange Server has no connectivity.
In our case, all of our servers have a secondary NIC that is used for tape backup traffic. This interface has no routing to the real network that AD and Exchange live on. So here\’s what, even though DNS registration is disabled on this secondary NIC, it still registers itself. If a system has DNS installed, each time the DNS Server starts or a zone is reloaded, it registers all interfaces that are configured to answer DNS queries. To determine if the Exchange server (or any member server or client) has resolved an IP for a DC, use nltest /dsgetdc:\”domain\”. See M275554 for additional information”
Piqued my interest.

This was because the main DC in my network (originally the workhorse that ran my entire business 6 years ago) does have a bunch of IP’s on it:
• My main network’s IP
• A IP from a second subnet for connecting to other kit
• 2 NIC’s from VMware Server (which is hosting a single “off my ESX kit” VirtualCentre VM for management purposes*)


So I wondered – were the Exchange boxes selecting a bad IP to use for AD Topology Discovery and getting confused.

So, I:
• Disabled the VMware NIC’s (not strictly necessary in my setup)
• Removed the second subnet from the ‘proper’ NIC
• Removed the (now invalid) A records from my internal DNS servers
• IPCONFIG /FLUSHDNS on the exchange boxes (just in case)

On restart of the exchange services everything came up trumps!

Time for bed methinks.


*The reason for this is that if I hose the ESX setup, I can still manage it – VMware (and I) recommend in a proper production setup , self hosting the VirtualCentre box

Updated: a few minutes later…
As a full test of \”does this work\”, I\’ve restarted all the Exchange boxes, and it works a treat.  Services came up really fast this time (normally I just leave running and come back after a cuppa), but no, this time almost as soon as I had logged onto the servers, the services were running.  Nice!