Ping watchdog with Powershell

We have one unreliable virtual server which sometimes just loses its network connection. Until we have a permanent solution, the temporary fix is to have the computer reboot if it loses network connection.

Note: There is nothing as permanent as a temporary solution. With that warning, let's get to it.

Step 1 - Produce the following Powershell script into c:\bin\pingdog.ps1

$netup = new-object Test-Connection -quiet "goo.gl"
if( $netup -eq $False ) {
  Restart-Computer
}

Step 2 - Fix your security settings

Start Powershell. Enter Set-ExecutionPolicy RemoteSigned or whatever level you're comfortable with that'll still run your script.

Step 3 - Schedule it

If you're using Windows Server 2003 (like we, ungh) schedule a daily task to run %SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe c:\bin\pingdog.ps1

From the task's Properties, the Schedule tab, press the Advanced button. Tick Repeat Task, every 12 minutes (or whatever you fancy), and repeat for 24 hours.

If you're using Windows Server 2008 or later, you should be able to set the task to run evey x minutes right from the interface.

Step 4 - Check your logs

Keep an eye on the failing computer's system log to see if it's restarted when you looked away.

The spontaneously disappearing printer

In retrospect, it's all very obvious, like most things technical are. I got a case from a customer's site saying that every once in a while, they became unable to print anything and they needed to restart the printer to get output on paper. Needless to say, they were pretty annoyed about the situation. But with that information, it could have been pretty much anything. 

I called the site and asked for some details. The site in question is a small store with just a few computers and two printers. They said the problematic printer was a Brother and they use it from all the computers in the shop. I took a remote connection to their site and found through some iteration the computer providing the print queue. The shared printer turned out to be a network printer (as indicated by the Ports thing on the printer queue) so i did three things: 

  1. i started digging through any Windows event logs for errors (but couldn't find any),
  2. i had a look at the web UI of the printer, and
  3. i started a continuous stream of pings (ping -t) to the printer

There was one thing i didn't realize to do which i'll tell you in a while.

The UI was horrible to look at and had a copyright 2006 text on it. The network configuration (and other "dangerous") bits were protected by a password.

Suddenly the printer's web UI became unavailable. I looked at the ping thing but the address kept on answering. I thought that maybe the printer had faulty firmware and a flash would be in order. There was however something strange with the pings. The TTL value had suddenly changed. I checked out the ARP tables (arp -g) for the printer, flushed the ARP tables, and checked again. This could only mean one thing. There were two devices on the network with the same IP address.

The thing i had left out earlier was to check the ARP tables before i started pinging. I would then have had a MAC address for the printer as it still was available.

I called the store and asked them what networked stuff they really had there. It turned out that their credit card reader is on the LAN as well. A-ha! So keeping the ping -t up, i asked them to power off the printer -- the pings stayed up -- then unplug the credit card reader. Plop. Connections time out. Ask to power up printer. Ping response. Flush and check ARP table. And indeed, i now had two MAC addresses for that IP address. Plug in credit card terminal. TTL changes. Flush and check ARP. And the MAC address has changed. 

Target acquired. Cause confirmed.

Since i had no access to the credit card terminal, the only solution would be to change the IP address of the printer. Ping to find an unused address on the LAN. Google for the standard password of said printer (and sigh with relief that it was valid). Change the IP address of the printer. Change the IP address for the print queue's Port. Test print from all computers. Phew. Mission accomplished.

So what can be learned here? Static IP addresses are bad. DHCP is good. And documenting your network is even better. Had the TTL value not changed when the pings started to go to the credit card terminal, the debugging would have taken even longer.

Needless to say, the shop's boss was very happy (and i hope he mentions it to my boss :). On Monday i shall have a look at another similar store where they have a spontaneously disappearing server...

The do's and dohs of File and Settings Transfer Wizard

Your job: Install a new computer for a client and transfer all the documents. The old box is a Windows XP, the new one runs Windows 7.

The tool: Windows File and Settings Transfer Wizard.

The caveats: Many.

It's a well known story. Your client has a new computer to install. She's used the old one for quite a while and it's full of documents (in weird places), although you've suggested storing them on the server, "just in case". But old habits die hard.

Thankfully, Microsoft has a pretty good tool for this case, namely the Files and Settings Transfer Wizard. This baby does most of what you'd believe it show and it's been bundled with Windows since the XP times. 

Step one: get an external hard disk or a fairly large USB flash fob (thumb drive, "minnepinne"). While you could do this over the net, it's probably faster over the USB. Nothing will be deleted from the external device so don't worry about that bit. 

Step two: Start the process from the new computer. Plug in the external memory device and fire up the F&STW. While the old one probably has the equivalent software installed, the file format has most likely changed between your brand spanking Win7 and the old XP box. This was learned The Traditional Way.

You do not need to be logged in as the user whose data is to be transferred but you will need admin rights on both the source and the target box.

Now tell the Wiz that you are on your new computer and that you haven't done the transferring bit just yet. FSTW will create an installer on the external disk, after which it'll close (given a button-push or two). 

Step three: Eject the external disk and plug it into the old computer. Run the installer created above. Start it and press the appropriate Next buttons. The transfer will commence. Have tea, this'll take a while.

Step four: Again, eject the external hard disk, plug it into the new computer. Navigate to where you created the transfer files (which probably is where you left the installer a few steps back) and open up the .MIG file there. This will awaken the FSTW anew, to let you drop the files and settings on the target computer.

Let it churn. Have more tea.

Step five: Check trough the transfer logs. Save the log for transferred files. Resist the urge to check trough the list of applications missing, and even more so, resist the urge to install the missing programs on the target machine. This, too was learned The Traditional Way.

Reboot the target computer. This will not be evident until you continue stepping through the logs. This, also, was learned The Traditional Way.

Step six: Surprise surprise, FSTW has not transferred the Outlook .PST Data Files from the source computer. Eject the external disk from the new computer, plug it into the old. Open the Control Panel from the old computer, open the Mail applet, check which Data Files are in use, click each one and the button to show the actual folder in which the .pst file is in. Outlook must not be running while doing this. Exit it, completely. Old Outlooks will leave a thread hanging to check for new mail.

Manually copy the .pst files to the external medium. Eject disk. Plug it into new computer. Create a directory c:\Users\%username%\Outlook and manually copy the .pst files from the external disk into it.

Only now, start Outlook on the new computer.

At this stage, i have no idea if Outlook will have its settings transferred or incorporate the .pst files on the previous computer, as this too was learned The Traditional Way (or more so, Obscured in The Traditional Way) so you're on your own here. The only thing to add is that you can use the Mail applet from the Control Panel to add the .pst files to the new account if they aren't there from before.

Apart from that, Good Luck. You're a sysadmin, and you need it.

Ghetto application deployment with Zap files

I discovered a painfully simple (and only slightly inelegant) way of deploying software in a Windows Active Directory environment, namely Zap files. While you'd usually want to deploy an .msi file, you use Zap files when you want to deploy an .exe file.

Big fat caveat -- The installation will run on the user's rights, so s/he must have software installation privileges on the computer s/he's running, or the installer must have admin credentials baked in somehow. On a secure network, you don't let your users install stuff on their computers.

And with that said, here's how to do it. Windows Server 2008 recommended.

1. Create a file share if you don't already have one. Use Share and Storage Management from Administrative tools or Server management to do it The Right Way [0]. To be Really Swanky, use DFS to publish the share on a domain scope instead of on server scope. In this example, i'll be more ghetto and shall call the share \\fileserver\Install and i shall call the fictional package to install agent.exe

2. Put your agent.exe file somewhere within the share created above; for the sake of this example, in \\fileserver\Install\agent.exe

3. Create a text file agent.zap (you can create it as agent.txt and rename it to dot-zap later) and place it in another share, or the same if you don't believe in security by obscurity, or don't have a compulsive manner in keeping things in neat little boxes. Here's what you'll put in the agent.zap file

[Application]
FriendlyName = "The Agent"
SetupCommand = "\\Fileserver\Install\agent.exe /any /switches"

Wikipedia tells me there are loads of other commands, but this will do for the Ghetto Installation we're doing now. Anyway, the [Application] row must be written like that, in verbatim. The next row is what's going to be shown to the users when they want to install the file. And the SetupCommand shall point to the UNC path where the installer resides. Any command switches can be put after the executable name within.

4. Open Group Policy editor. Browse to your users' folder (or where-ever you want to apply the deployment). As i'm on Small Business Server, that would be around ...\My Business\Users. YMMV. Right-click to Create a GPO in this domain and link it here. Call it Published Software (since eventually you'll put more published software here).

5. Under User Configuration / Policies / Software Settings / Software Installation, right-click New / Package. Navigate to where your .zap file is, make sure the file type selector is .ZAP (and learn that .zap stands for ZAW down-level Application Package) and select it. Click OK. Select Published to force the installer down your users' throats(generally a bad call) or Advanced to modify the settings and under the Advanced tab on the next dialog box, unselect Auto-install this application not to force feed the app.

6. Showtime. Log on as a user on a workstation. Open Control Panel. If you're on Win7, find the option "Get Software". If you're on an earlier incarnation of the ubiquitous desktop operating environment, go Add/Remove Programs or the like and choose Install published software. You should now see The Agent listed there! Yay presto!

And that's about the size of it on a space like this. Experiment and write about your experiences in the comments below!

[0] Also known as The One Microsoft Way :)

Group policy preferences discovered

How do you map a network drive using Group Policy? You use Group Policy Preferences. No script required.

I got a pretty typical request from a client today. He'd hired a new employee, for whom i'd installed a computer a few days ago. One thing i hadn't done was to map to a network drive. Typical task, typical setting. I don't know why she (the new hire) didn't have the drive mapped but i promised him (my client) i'd fix it tonight. Which i did.

There are two ways to map a network drive:

  1. the Stupid way, which is to log in on a computer as the user and map the network drive and set it to re-map between sessions (/persistent:yes)
  2. the Ordinary way, which is to have a logon script run from the logon server, mandated by group policy.

There is also a New way offered on Windows Server 2008: Group Policy Preferences. Unlike Group Policy Settings, preferences are something that are suggested rather than mandated to the user, who may change the suggested preferences if so wanted. Another thing is that there are a bunch more preferences available than i'd found in GP Settings, and the one i was looking for was indeed the preference for drive maps.

For magic to happen, open the Group policy manager and create a new Group policy opject (GPO) where the users you want to target are. Call it Drive mappings. Go to User configuration -> Preferences -> Windows settings -> Drive maps. Right-click it and New -> Mapped drive. Set Action as Update (or Replace; see help file for info), fill in the UNC path (ie. \\server\sharename), give it a nifty Label and a Drive letter. And you're there. Repeat for other drive letters as necessary, creating other GPOs for other groups who have their own network drives. There's even variable substitution so you could probably map a drive for a group or a site or something equally local.

Given all this, drive letters are hopelessly outdated; it's just the fact that people are so used to them that it'll take a while for them to die out. And the same goes for home directories on the net. The Correct Way would be to have the venerable [My] Documents folder silently residing on the server and replicated for offline use (hint: use Folder Redirection), and any shared or common folders under the Libriaries meta-folder-thingy on the new and improved Windows 7 file explorer.

But that's for another time, when i've updated all their workstations to Windows 7.

If a tape backup system requires two hours of specialist time just to change the tapes, something is wrong

I just came back from a customer[0][1], having spent the AM there changing their backup tapes. The good: ArcServe support tweeted me back even before i had the chance to write this (good work @arentejaswi!). The bad: everything else.
 
The manoeuvre required to change the backup tapes includes copious amounts of arbitrary-length waiting time and split-second reaction times to when one waiting has ended. It requires living with a tape drive and its controlling software that both seem to have individual minds of their own, sometimes with conflicting goals. To perform this seemingly mundane task, i need to "move a tape" (from the drive to the magazine, but only if the tape is in the drive). I need to run inventories on the tapes which take 20 minutes or more a pop. Sometimes the backup software informs me that the "Unit is busy" (which unit?), and i'll have to wait for another 20 minutes. I manually need to inform which tapes are in the "save set" and which are in the "scratch set", which probably is backup-lingo for which tapes can be saved onto (that's the scratch set, mind you) and which tapes should be left untouched (the save set, which incidentally consists of tapes that even aren't in the bloody tape drive).
 
All in all, using the system requires that i have a system-level understanding of it. And i don't. To operate it, i don't even know if i should; a properly trained monkey should be able to change tapes.
 
One problem i had today was that the backup software claimed tape 11 was in slot number five when i knew it was tape 20 in there. Tape 11 was in fact in the tape box. It took an hour of convincing the system and i'm still not sure it approved.
 
Still, all of that is technicalia. A system should not be so complicated to maintain that it requires hours of specialist time to do the seemingly mundane task of changing the tapes. The system should take care of doing inventories. It should understand which tapes were removed and which were replaced. It could even suggest to me which tapes i should insert next. Or it should accept whatever tapes i feed it and be able to take it from there.
 
At the same time, it feels unethical to the customer that they're going to see a bill from us for that changing-the-tapes time. It's not like it's their fault that changing tapes on a backup copy system sucks. But i know that something in all of this must be wrong.
 
[0] A real one this time
[1] Taking a therapeutic detour through my favorite curry joint

How not to destroy your workstation

Dear client,
 
Once again i will have to bill your company for removing viruses/spyware/some-other-ware from one of your workers' machines. The job took me seven hours of which i only have the heart to bill you for four. After all, i want to keep you as a client in the future too. But for the price i should be billing you, you could get your employee a new computer.
 
Here's what you should do.
 
In short terms, educate your users that their workstations are for work only. That it will cost you the equivalent of one new computer each time i have to make it work again after the fun software they installed onto it brings it to a screeching halt. this money could be put into much more fun and/or productive use. Ask them to be very, very careful with the tool you've provided them with. A craftsman will take care of his or her tools even if they belong to their company and not themselves.
 
Ask them to get a personal computer for personal work. If you can, sponsor them into getting a personal computer. We can even work out something that is so easy to re-install that if it's broken again, it will be painless to get it back to wor... to play.
 
Or we could put all the work stuff on a terminal server. The users can bang their computers to bits for all i care, but the work is behind a remote connection.
 
Here's my favorite one, and it's not even expensive. We'll install a second environment for your people to play with. If they're at work, they boot into "work mode" and if they're at home, they boot into "play mode". I have the perfect suggestion for you.
 
So please, let's sit down and talk. This will only take a while and you'll save lots of extra money for it.
 
(fictional message to a customer)

Recovering from a bad Windows profile

Sometimes, Windows XP loses the user's profile and goes with a temporary profile instead which is just that, temporary. Any changes made to that profile -- like Outlook settings -- are lost with the next logout.

Here is a simple ten step process to get the profile back.

0. Reboot the computer. You'll see why in a minute.

1. Log in with Administrative priveleges (domain or local). If you're trying to recover your own profile and your own logon has admin priveleges, you need to take the longer route. You need to log in as somebody else than you're trying to restore. [0]

2. With Windows Explorer, navigate to C:\Documents and Settings. From View » Options » Advanced [1], set the appropriate option to show hidden files and folders.

3. Make a backup copy the Problematic user's directory under Documents and Settings -- for this discussion, we shall call it C:\Documents and Settings\Problematic or Problematic for short -- just in case. This is why you needed to reboot; if the user has been logged in since the last boot, there will be some files locked inside the Problematic directory.

4. Tricky time. Rename the hidden (and now made-visible) directory Default User into Default User Original. Rename the Problematic directory to Default User. [2]

5. Log out Administrator and ask Problematic to log in. Since Problematic does not have a profile, a new one is created using the data from Default User. This is not just magical, but doubly so, as the bad data isn't copied verbatim but used as profile fodder to create a new and altogether less Problematic user profile!

6. Log out Problematic (who know for the discussion really should be called something else :) and log in as an administrator.

7. Delete the "Problematic" Default User directory. Rename the Default User Original into Default User.

8. Log out. Feel smug.

OK, that was only nine steps so keep one in store for your next sysadmin magick. We both know you will both use and need it.

[0] In short, the longer router involves creating a new user and granting that user admin privs.
[1] OK, that isn't the exact path, but you'll find it. It's the second rightmost menu. I don't have an XP handy at the moment.
[2] You could probably achieve the same thing right clicking My computer > Properties > ... > User profiles and removing the offending profile, but this method includes recovering the b0rken profile itself. Do this for extra karma.

How not to receive mail

I just had a very Douglas Adamsy moment. There's a passage i love in one of his books where there's a big hole in the space ship but the hole in question also knocked out the sensor that would have told the monitor computer that there's a piece of the space ship missing. Which causes the maintenance robot to fall through the hole in the space ship because it does not know it exists.

The moral of the story? Monitor your monitors.

The case in question? Email. I have two email accounts that i use, one with the gmail domain name and one on my vanity domain. I actively use the Gmail account because i got it first, but i use the vanity email address for all outgoing mail regardless from which email address i send it. This means that when i send mail from Gmail, the replies get sent to the vanity domain which are automagically forwarded back to my Gmail account. It's a bit of a hack, but it has worked so far.


But oh.

At some stage in December apparently, i managed to mistype the forwarding settings so that all my email to the vanity domain got forwarded to another user at Gmail. Of course, i was not aware of this. I just stopped receiving some of my email. I just never attributed that to my own stupidity. But i should have.

When i today checked my vanity domain, i had a bunch of non delivery reports from that other account. Not very cool. I suppose i could go and send an apology mail to that other user but i fear that too is going to be bounced.

So if you haven't received a response from me to a mail you sent to my vanity domain, i am sorry (this, unfortunately also goes for billing statements sent from my ISP). My fault. Mails should be delivered now. And i shall cluebat myself just to improve my memory.