The magic email

This is just me ranting, if you’d rather skip it feel free, I won’t hold it against you.

Like any other Tuesday, I’m sitting at my desk, working. I get a call from one of my “VIP” users, telling me about another user having issues with his email. New Guy started on Monday. I have set up an email (let’s call it new.guy@company.com), but have not given him the password, or the link to OWA, or configured Outlook, or his phone to access the email.

Apparently the “VIP” user sent New Guy a test email at 10:20am, which he received on his laptop (say what?), and now he is not getting email sent to him. Well no shit he’s not getting them, no one has asked me to set up his email yet. New Guy is kind of a special case, in that he is going to be a marketing rep for us, but he is a retained consultant, so he isn’t actually an employee. As such, he will not be using any company equipment, he just needs me to configure his @company.com email address on his personal laptop.

So I ask the “VIP” user how he could have gotten the test email since he doesn’t have anything other than the actual email address? She doesn’t know. Ok, I’ll be over to configure his email in just a minute.

In retrospect what I should have done was gotten in my car and went fishing.

Like an idiot, I go over to the office where New Guy is working, and let them both know that, barring any unannounced changes in the laws of physics, there is no way that any email sent to his @company.com email address could have possibly made it’s way onto New Guy’s computer, with nothing more than a name to go on.

The “VIP” user just won’t let it go. Swearing that the email was there! It was really for reals on his computer!

I ask to see the email on his computer.

“It disappeared.”

Alllllllllllrighty then! In the words of The Master (Frank, if by some weird coincidence you’re reading this, you’re still the man); PFM is the only explanation for this. Pure. Fucking. Magic.

So, I tell Ms. VIP that I will call the hosting company and have them trace the message, after I set up New Guy’s email.

So I sit down to set up his @company.com email account, and temporarily forgot that I wasn’t dealing with Office 2010 (what everything in our company runs), but Office 2007. For about five minutes I think I’ve just blitzed his personal email account in Outlook. Shit. I knew I should have backed this machine up before touching it.

I ask him who hosts his email (he had previously told me that his “company” didn’t have a mail server).

He responds “I get my email from Microsoft. By way of iPage, and it’s delivered to Outlook. But it comes from Microsoft.

Dammit.

I know how to solve this, lemme just grab my helpdesk uniform real quick:

Sadly I don’t get to play Il Duce at the office.

Suddenly the whole magic email thing becomes clear. The “VIP” user told him “I just sent you a test email,” to which he replied “ok got it.” What she heard was “I am reading your email right now,” but what he actually said was “I am acknowledging what you just said to me.”, and by the time I got there, New Guy was too embarrassed to admit the confusion, and had to stick with his “I had the email in my personal email account, but it magically disappeared!” story.

Can someone just admit that they made a mistake without trying to blame the technology, just one fucking time? Please?

I can prove that your story is complete bullshit. I will prove that your story is complete bullshit.

So while contemplating the violent murder of New Guy, I flash on the fact that this is Outlook 2007, the profile is there, I just have to set it to ask on launch which profile to use. Fixed. Email working.

I’m getting out while the getting is good, and then New Guy says “hey you know while I’ve got you here, I can’t change my password, it won’t let me.”

He’s talking about his user account password, not his email (which he still doesn’t have a password for). Ok, no big deal, whoever last worked on his computer set the ‘user cannot change password’ flag on the account, let me just uncheck that…

The account isn’t listed in Local Users and Groups. Weird. Win+R > CMD > {ENTER} > net localgroup administrators (clickity).

“Hmmm, it appears that you’re logged on with a domain account from the XYZ domain. Is this a client you worked for at one point?”

“Oh yeah, I bought the laptop from them when I finished the job 9 months ago.”

Clickity, clickity.

“It doesn’t look like there are any other local accounts on this machine. Do you still remote into their network for something?”

“No I used to have a remote access thingy, but I gave that back when I finished the job 9 months ago. I was on their network about a month ago though when I had them fix my email. They’re in Texas.”

“Well that’s why you can’t change the password on your account. It’s a domain account, so you need a connection to their DC to change the password that is stored in AD. If you don’t work for them anymore you’re going to want to get that fixed, eventually the cached credentials will expire, and you won’t be able to log into the account anymore.”

“Oh it’ll be fine.”

“Ok, call me if you need anything else while you’re here.”

Bye bye now.

Which led to a completely absurd call to our email host that started out with “I know this is going to sound completely preposterous, but hear me out…”

Of course, as I knew would happen, the host traced the email, and noted that it was not delivered to any device until 4:45pm (which is when it was finally delivered to the new outlook profile I set up for New Guy).

I really don’t like it when I have to waste not only my time, but the time of someone else because a “VIP” user will not accept that I have been doing this for 15+ years and when I say something is not possible that is not a guess.

End of rant.

Advertisements

Assigning an IP Address to an APC Network Management Card without having the APC software.

I work for a company that does Electrical construction and design. When I took this job I though that would translate into having little things like plenty of power in the server room, well documented wiring, and maybe having everything in the building(s) on UPS units. I should have known better.

Much like most IT Professionals home networks are kinda messy, working for a company full of electricians translates to wiring… issues.

We do have a generator, and we do have a building UPS in the main building. That UPS is 15 years old. The UPS battery had never been replaced when I got here (it basically had a runtime measured in microseconds). Only some outlets are on the UPS circuits. The server room is in another building. That building is on the generator, but there is only one circuit that is on the UPS. That circuit was for the 17 year old Toshiba PBX that was mounted (I kid you not) in a cupboard in the bathroom in that building.

The servers were on UPS units. APC SmartUPS 1500 units. Four of them. That were 6 years old. With batteries that had never been replaced. Desk units that were housed on shelves in the rack. That were at approximately 150% of draw capacity (when all servers were at peak draw). Unconnected in any way to any of the servers.

This meant that if the power failed (which is does, because this is an older part of town), most of the workstations would go down, and the generator would kick on. Approximately 1/1000000000 of a second before the UPS batteries on the servers and the few workstations on building UPS circuits ran out. Most of the time.

So clearly my first priority was to get the servers on a UPS that was correctly sized for the draw, with fresh batteries. Enter the APC SmartUPS RT 5000 with two battery arrays. This unit will handle all of our current servers, with about 30% extra capacity for expansion. I bought this unit with a Network Management Card so that I could remotely monitor the unit. And then never configured it. I’m not going to make excuses, it was just one of those things where I got busy and forgot.

Until I needed to get access to a UPS in the other building. This got me thinking about the UPS on the servers. So I plugged the NMC into one of the switches, and… nothing.

As it turns out, the NMC in this particular unit will not pull a DHCP lease without intervention through some APC software that ships with the unit. Which I cannot find. Not an insurmountable issue.

Enter the ARP command.

Address resolution protocol (ARP) can be used to configure the NMC. All we need is the MAC address of the NMC.  The MAC address is located on the quality assurance slip that shipped with the NMC, and is also located on a white sticker on the NMC itself.Unfortunately the QA slip is long gone, and the sticker on the NMC with the MAC printed on it is on once of the actual circuit boards of the NMC, now safely concealed inside the UPS.

Now I DO NOT recommend that you do this, but it is possible to remove the NMC with the UP running. If you feel like taking your life in your own hands, you could just unscrew the NMC and pull it out of the UPS without taking the UPS offline. Which is what I did.

Now we just need a computer on the same subnet.

Open up a command prompt and type the following (MAC Address format: xx-xx-xx-xx-xx-xx):

arp -s [IP_ADDRESS_FOR_THE_NMC] [MAC_ADDRESS_OF_THE_NMC]

Hit Enter.

Next, use Ping with a size of 113 bytes to assign the IP address defined by the ARP command.

Use one of the following Ping commands (To clarify, the -l option is the letter L but must be lowercase when executing the command):

Windows command format: ping [IP_ADDRESS_ASSIGNED_ABOVE] -l 113

*NIX command format: ping [IP_ADDRESS_ASSIGNED_ABOVE] -s 113

Now, you can Telnet to the card by typing: telnet [IP_ADDRESS_OF_THE_NMC]

Use “apc” for username and password.

Configure/apply any additional changes.

Log out to save changes.

That’s it, now you can use the web interface of the NMC to make configuration changes or retrieve information from the UPS unit.

All of this could have been avoided had I simply taken 30 minutes to configure the UPS when I installed it, or even taken 5 minutes to document the MAC and put all of the paperwork that came with the UPS in a folder in my file cabinet. Learn from my mistakes, DOCUMENT EVERYTHING.

 

Kung Fu for SysAdmins: Using Version Control Systems for scripts

Like nearly every administrator I know, I have a few scripts that I use to help me do my job easier and faster. Well ok, more than a few. Alright, alright, I have more scripts than most people have hair.

I have scripts that were written for an NT4 Alpha Cluster. I never get rid of them. I’ve lost more scripts than most people will ever have. I’ve forgotten more about most scripting languages than most people will ever know.

I find that the challenge is keeping track of them. If I could turn my computers, network storage, USB drives, and email upside down and shake them, enough scripts would fall out to fill the library of congress. Damned if I know where they’re all stored, but I know I have them.

Unfortunately, most of them are variants on the same script, or actually are the same script. Many times if I can’t find the exact script I am looking for, I’ll slap one together, do some quick testing, give it a unique name (you know something that makes it easy to know what it does, like “test74.vbs”), use it once or twice, and then forget what it was for.

Yes I’ll admit it, I have a problem. If there were an AA for scripters, I’d probably be the president of the local chapter.

Or at least I would have been before I started using Version Control software. About a month ago I was editing a script that I was having troubles with, accidentally overwrote something that broke it, and closed the editor. Not a big deal on small scripts, but this one was at about 1500 lines. It took me HOURS to figure out exactly where the code was that I’d overwritten, but I never did figure out exactly what I did that broke it (though I did get it working well enough to do what I needed it to do).

I’ve known about Version Control software for years (I worked at a software company once upon a time), I’ve even used Subversion before. I’d been meaning to set it up for myself for years, and this incident was the catalyst I needed to actually do it.

Keeping the LAOAE principle in mind, I wanted my repository to be available to me in all the places that I’d be likely to be working on a script, mostly at work and at home.

But I already have enough servers to admin, and I don’t really want to have to care for and update yet another thing on my work network, so I started looking for hosted subversion offerings. I’m not a software company, so I had to weed out the ones that wanted absurd monthly fees (and offered absurd feature sets). I just wanted hosted Subversion, I didn’t need team collaboration, or project management features. I thought about going Open Source with it (like github, or Google code) but decided that since I would be using it to also host code that belongs to my employer, it would probably be better to go with a commercial solution.

Did I mention that I didn’t really want to spend any money on this? Yeah, free is king in the land of the Sysadmin. There were several services that fit the bill, in the end I decided to use ProjectLocker. Their free offering gives you three users and 300MB of storage (even for my bloated script collection this is plenty) three Repositories, and three Projects (each project can have an unlimited number of files and folders), though for only $19/month you can move up to 15 users and 10GB of storage (full details of their offerings can be found here).

Oh and all of their plans offer both Git and Subversion, so whichever you are more comfortable with is available. I have used subversion in the past, so that’s what I chose to go with.

Now before I get any further in this post, I am going to write this with the assumption that you have at least a conceptual knowledge of how Version Control Systems (VCS) work. If you don’t and would like to, this free ebook is a great place to start (and really relevant, as it’s also the official documentation for Subversion).

Setting up a VCS for use with scripts

ProjectLocker (Subversion)

Initial setup of a ProjectLocker account is quick and easy:

  1. Select your Service Level (Free is fine for me).
  2. Select you term (Free is Always Free).
  3. Enter a promotional code (optional).
  4. Enter a Referral Source email address (if you have one, the referrer gets free storage space in small increments).
  5. Click “Next Step”.
  6. Agree to the Terms of Service (if you do), and select the “I am ready to setup my account” button.
  7. The information on this page is pretty self explanatory, click the “Place My Order” button at the bottom when done

Save the Login URL on the resulting page! This is the URL you will use to access your repository (https://portal.projectlocker.com).

Log in to your repository and fill out the requested information (it helps keep the free offering available).

That’s pretty much it on the Subversion side, though you can set up users and additional projects if you like. Since I am using this to host my personal code, as well as code that belongs to my employer, I set up two projects: Personal_code and Work_code (I know, original right?).

Once you have a repository running, ProjectLocker will give you a URL to the repository, it’ll look something like this: https://pl3.projectlocker.com/TestCompany77/Personal_code/svn

This is what you’ll need to access the repository from a client, so write this down (bookmark it, whatever you have to do, you will need this)! (obviously use the one in your account, the one listed above won’t work for you)

I also set up a user account for the main IT email account at the office so that it has access to the Work project only. This way when or if I leave my current job, I can just hand off the login to the next person (or leave it with someone here) and they will be able to log in and access all of the scripts that are owned by the company, including being able to see all the changes I’ve made, and any comments I’ve made during the commit process.

TortiseSVN (subversion client)

So now that we have a repository, we need a subversion client. On Windows that leads us to TortiseSVN. There are others, and you are certainly welcome to use a different subversion client,  but TortiseSVN is the hands down leader of the pack for features and maturity. It’s also really easy to install and configure.

TortiseSVN is a command line interface to subversion, but it integrates with the Windows Shell. This means that there is no “program” window to access for settings and such, you access everything via context menus (right click).

Once you have it installed on a Windows computer, you’ll need to link it to your Repository. The easiest way I’ve found to do this is to use the TortiseSVN Repo-browser (you’ll want to get familiar with this tool anyway, as it is installed with TortiseSVN by default and you can use it to… well browse your repository).

Just right click on any file or folder, highlight the TortiseSVN entry, and select Repo-browser from the resulting menu:

TortiseSVN context menu

In the Repo-browser window enter the URL for your subversion repository and click the OK button:

When the Repo-browser attempts to connect to your repository, it will ask you for a username and password, and if you want to store that credential in a file on your computer (that’s up to you):

Once you have entered the credentials, it will show you the contents of your repository:

Now TortiseSVN is ready to use! The Repo-browser interface is fully drag and drop, so you can just drop your script files and folders on the right pane, and it will upload them to subversion. You’ll need to enter a commit message, and once the upload completes, you have your first version of your files in subversion.

NotePad++ (text editor)

Now all we need is an editor that can make use of this, and we’re in luck. Perennial favorite NotePad++ has an extension for TortiseSVN. First, you’ll need to install NotePad++ (or open up the portable version on your USB drive). Once you have NotePad++ open, Select the Plugin Manager from the Plugins menu as shown here:

One of the reasons that I’ve been such a big fan of NotePad++ is it’s extensive selection of available plugins. In the Plugin Manager, just select the Tortise SVN plugin and click Install as shown in the image below.

This plugin requires two supporting plugins, which will automatically be selected for install, as shown here:

Once the installation is complete, NotePad++ will need to be restarted (it will tell you this and prompt you for the restart of the program). After NotePad++ is restarted, we are ready to start actually using our version control system!

If you’ve never used a VCS before, you’ll need to understand the concept of Working Sets and the Checkin/Checkout relationship. These topics are far outside the scope of this post, but if you are new to this, I’d really suggest reading the official Subversion documentation (or at least skimming it). You can find the official Subversion book here.

In order to work on any files stored in the repository, we first need to create a local copy of the repository, or a “Working Set”:

  1. Create a folder where you want to store your Working Set. You can name it anything but “svn”, as this name is reserved for use by TortiseSVN. I usually choose something like C:\SOURCE, so it’s easy to remember.
  2. Right click on the folder you just created and choose SVN Checkout from the context menu.

At this point, TortiseSVN is going to prompt you for some information about how you want the Working Set created, in the form of this window:

The only thing you really need to be concerned with here is the Revision section. You only want to change this if you don’t want the latest revision of the files in your repository (which is what HEAD revision means). If you need an earlier revision, select the Revision radio button, and then use the Show Log button at the right to search for the desired revision. If this is the first time you’ve set this up, or you want the latest revision, just click the OK button. After all the files from your repository are copied, you’re ready to start editing!

Up to this point, everything we’ve discussed has been pure configuration, something that you’ll do one time per computer, and that’s it.

Using the Version Control System

Here is an example of the typical workflow of editing a script without using a VCS:

  1. Find the script (this is usually the hardest part).
  2. Edit the script.
  3. Save the file.
  4. Pull your hair out because the 2500+ line script you just changed isn’t working right (hopefully you’ve never experienced this step).

Now here is an example of the typical workflow of editing a script using the VCS we just set up:

  1. Open the file you want to edit from the Working Set in NotePad++. For this example I’ll use the file C:\Admin\Scripts\SOURCE\Shell\reboot.cmd from my local Working Set.
  2. Make your changes in NotePad++, and save the file. If you do not save the file, there is nothing to commit to the repository.
  3. From the NotePad++ Plugins menu, select Tortise SVN > TSVN – File commit, as shown in this image:

This tells TortiseSVN that you want to write the changes you’ve made to your repository, and it will prompt you to do so using the Commit window:

As you can see there are several options before you actually commit the file. In practice you’ll usually only need the Message area and the OK button. The Message area is basically a comment area for what these changes represent. The more verbose you are here, the easier it will be to understand the changes that were made in the revision, and it will also make it much easier to find a revision where a particular change was made. Once you click the OK button, TortiseSVN will commit your changes to the repository, as shown in this window:

Clearly you can see this is a slightly more complicated process, but the pay off totally worth it if something goes wrong.

How to figure out where you broke something after you’ve been using this for a while

The exact workflow that any given person uses will probably be slightly different than that used by any other person, so this is going to be a description of my particular workflow, and how I track things.

I’m writing a PowerShell Module (sshhh), and I have a function in it called Get-Sysinternals. When I first wrote this function, it would default to downloading all of the Sysinternals utilities. I’ve edited it and altered it several times since then, and I wanted the default behavior to be that it would only download updates to the tools if they were already installed on the local computer in a specific folder. I then later added some switches to change the default behavior, and somewhere along the line, I broke the default behavior.

To figure out what changed, I needed to see what the code looked like when the default behavior was changed. To do this I opened the file (AdminsArsenal.psm1) in NotePad++, and from the NotePad++ Plugins menu, I selected Tortise SVN > TSVN – File log.

This brings up a viewer for all of the Commit activity for the file, which looks something like this:

Scrolling through the commit messages, I find that at revision 76 I changed the default behavior.

What you’ll do at this point depends on how you want to handle this. You could right click the file in explorer, and select TortiseSVN > Update to revision…, if you just wanted to revert the file to a point where you know it worked. Personally I just wanted to see the code in revision 76, so I right clicked the revision I had highlighted, and selected Compare with working copy from the resulting context menu:

Now this particular example was not a great one to use, as I completely rearranged the functions in that module in a later revision so the compare is pretty sloppy, but you get the point.

You’ll also notice that in the Log Messages window, it shows the Author which makes it really easy to determine who made what changes to a given file if you have more than one person working on your scripts.

Kung Fu for SysAdmins: Windows 7 made easy

Back when I was in IT trade school I was taught a really valuable lesson in how to make your life a bit easier as an administrator; one of my instructors walked up to my computer, unplugged my mouse, and said “now do what you were supposed to, just without the mouse.”

At the time I was like “what a dick”, but since then, I’ve really come to appreciate the lesson he was getting at: shit happens, and sometimes it is just faster to do it without a mouse.

So I keep hearing about how people making the transition directly from Windows XP to Windows 7 are going to have trouble (granted, the issues mentioned are minor) adjusting to some of the changes Microsoft has made to where settings are located. I would argue that only people who haven’t put in the effort to learn the correct way to access these settings are going to have issues.

Let me elaborate. One of the common complaints I hear from fellow administrators about Windows 7 is that the setting to change folder options is now more difficult to find. It’s been changed from Windows Explorer>Tools>Folder Options>View Tab, to Windows Explorer>Organize>Folder and Search Options. Sure you could pop open google and find instructions in relatively little time, or you could do it the correct way from the start (which works on all versions of Windows since XP): Run>Control Folders>View Tab (in XP).

This is not the only thing that can be accessed using Windows Control Commands either. See if like me, you have been using this method since Windows 2000 (or there abouts), this was never an issue (this particular setting actually changed in Vista, but many people have opted to skip Vista and go directly to Windows 7).

Here is a list of some of the things you can access in Windows without the mouse (type these at a Run Prompt):

  • control = Opens the Control Panel Window
  • control admintools = Opens the Administrative Tools
  • control keyboard = Opens the Keyboard Properties Window
  • control color = Opens the Display Properties (at the Appearance Tab in Windows 7)
  • control folders = Opens the Folder Options Window
  • control fonts = Opens the Font Policy Management Window
  • control international (or intl.cpl) = Opens Regional and Language Options
  • control mouse (or main.cpl) Opens mouse properties
  • control userpasswords = Opens the User Accounts Editor
  • control userpasswords2 (or netplwiz) = Opens User Account Access Restrictions
  • control printers = Opens the Printers and Faxes Window
  • control desktop (Windows Vista/7 only) = Opens Control Panel>Personalization
  • appwiz.cpl = Opens the Add or Remove Programs Utility
  • optionalfeatures = opens the Add or Remove Windows Component utility
  • desk.cpl = Opens the Display Properties (Themes Tab)
  • hdwwiz.cpl = Opens the Add Hardware Wizard
  • irprops.cpl = Opens the Infrared utility (does nothing if no IR devices are installed)
  • joy.cpl = Opens  Game Controller Settings
  • mmsys.cpl = Opens the Sound and Audio device properties window (Volume Tab)
  • sysdm.cpl = Opens the System Properties window
  • telephon.cpl = Opens the Phone and Modem options window
  • timedate.cpl = Opens the Date and Time Properties window
  • wscui.cpl = Opens the Windows Security Center in XP (opens the Action Center in Windows Vista/7)
  • access.cpl = Opens the Accessibility Options Window (does not work in Windows 7)
  • wuaucpl.cpl = Opens Automatic Updates
  • powercfg.cpl = Opens the Power Options Properties window
  • ncpa.cpl = Opens the Network Connections window
  • bthprops.cpl = Opens the Bluetooth Control window (does nothing if no bluetooth devices are installed)
  • certmgr.msc = Opens the Certificate Management MMC
  • compmgmt.msc = Opens the Computer Management
  • comexp.msc (or dcomcnfg) = Opens the Computer Services MMC
  • devmgmt.msc = Opens Device Manager
  • diskmgmt.msc = Opens Disk Management
  • eventvwr.msc (or eventvwr) = Opens the Event Viewer
  • fsmgmt.msc = Opens Shared Folders
  • napclcfg.msc = Opens the NAP client configuration tool
  • services.msc = Opens Service Manager
  • taskschd.msc (or control schedtasks) = Opens the Task Scheduler
  • gpedit.msc = Opens the Group Policy MMC
  • lusrmgr.msc = Opens Local Users and Groups
  • secpol.msc = Opens the Local Security Settings window
  • ciadv.msc = Opens the Indexing Service Window
  • ntmsmgr.msc = Opens the Removable Storage Manager
  • ntmsoprq.msc = Opens the Removable Storage Operator Requests
  • wmimgmt.msc = Opens the WMI (Windows Management Instrumentation) window
  • perfmon.msc (or perfmon) = Opens the Performance Monitor
  • mmc = Opens a blank Microsoft Management Console
  • mdsched = Opens the Memory Diagnostics tools
  • dxdiag = Opens DirectX diagnostics tools
  • odbcad32 = Opens the ODBC Data Source Administration window
  • regedit (or regedt32) = Opens the Registry Editor (these commands actually open different Registry editors, google for the differences)
  • drwtsn32 = Opens Dr. Watson
  • verifier = Opens the Driver Verification Manager
  • cliconfg = Opens the SQL Server Client Network Utility
  • utilman = Opens the Utility Manager (in Windows 7 this opens the Ease Of Access Center)
  • msconfig = Opens the System Configuration Utility
  • sysedit = Opens the System Configuration Editor
  • syskey = Opens the Windows Account Database Security Manager
  • explorer = Opens Windows Explorer
  • iexplorer = Opens Internet Explorer
  • wab = Opens the Windows Address Book
  • charmap = Opens the Character Map
  • write = Opens Wordpad

Now these are not the only things you can type at a run command to get results, but I find that those more than cover most activities you will likely need to do on a day to day basis.

I would say that I find myself using only 10-15 of those commands on any kind of a regular basis. You may only need 4-5 regularly, or none at all. Everyone does things differently, however I have found these commands to be the only thing that stays constant over several versions of the Windows OS. So for me, it takes the Least Amount Of Administrative Effort to simply use these commands.

%d bloggers like this: