Build Status Notification Lamp - TFS2010

Web Mail Notifier Sillyness

We have a dedicated build wall which is pretty efficient in displaying the build information. It really is!

BUT, ever since I moved my desk away from the big screen TV's that display the build notifications, it makes it difficult to tell what's the status of the builds at our morning stand-up.

This was until a couple of days ago a friend got me hooked on to the "USB Web Mail Notifier" available here. (Although, I bought mine off some random dude on EBay (from the UK).

All I really needed was to know if our Nightly Builds that run our Unit and Integrations Tests failed.

Now I know that there are heaps of other ways to get this information but the one I went with appealed to my inner nerd :)

By using a simple powershell script I am able to query our Team Foundation Server (TFS-2010) to see if the latest Nightly Build has been successful.

The powershell script and the binaries required to use the USB light are available here at http://tiny.cc/BuildNotification.

You need to customise the powershell script by entering your details in the  "Set user variables" section of the script:

This is what it looks like.

When developers have been naughty (and not run integration tests locally before checking in) ...

But after you fix it, all is good ..

Find Overlapping Date Ranges

I recently had to implement a "DateRange" validator to check for any overlaps in the date ranges.

Here is my approach to it :

I started with a custom DateRange class.

 
public class DateRange
{
    public int SortOrder { get; set; }
    public DateTime Start { get; set; }
    public DateTime End { get; set; }

    public DateRange() {}

    public DateRange(DateTime sTime, DateTime eTime)
    {
        Start = sTime;
        End = eTime;
    }
} 

Then I call the "HasOverlap" method on my validator to check for overlaps.

 
/*
* LOGIC:
* 
* 1. Take a List of date ranges. Example:
            [Tuple A]         --------         [Sort Order 1]
            [Tuple B]--------                  [Sort Order 2]
            [Tuple C]     --------             [Sort Order 3]
            [Tuple D]     --------             [Sort Order 4]
            [Tuple E]             --------     [Sort Order 5]
            [Tuple F]     ----------           [Sort Order 6]
         
* 2. Now sort the range by start + End Dates. This results in:
            [Tuple B]--------                  [Sort Order 1]
            [Tuple C]     --------             [Sort Order 2]
            [Tuple D]     --------             [Sort Order 3]
            [Tuple F]     ----------           [Sort Order 4]
            [Tuple A]         --------         [Sort Order 5]
            [Tuple E]             --------     [Sort Order 6]    
         
* 3. The Logic is that there will be an overlap IF EVEN ONE of 
*    the 2 base conditions are met:
*      (a) After sorting the list, a given tuple (TUPLE X) 
*          will be deemed overlapping 
*              if the END-DATE of (TUPLE X) is GREATER THAN the START-DATE 
*              of ANY tuple WHERE the sort order is GREATER THAN that of (TUPLE X)
*          
*      --- OR ---
*      
*      (b) After sorting the list, a given tuple (TUPLE X) 
*          will be deemed overlapping 
*              if the START-DATE and END-DATE of (TUPLE X) MATCHES 
*              the START-DATE and END-DATE of ANY tuple 
*              WHERE the sort order is GREATER THAN that of (TUPLE X)
*/
private bool HasOverlap(IList<DateRange> ranges)
{
    // 1. Sort Dates based on Start & End Dates
    var sortedRange = ranges
                        .OrderBy(p => p.Start)
                        .ThenBy(p => p.End)
                        .ToList();

    var sortCounter = 0;
    sortedRange.ForEach(e =>
    {
        e.SortOrder = sortCounter;
        sortCounter++;
    });


    // 2. Check if the end dates are > any start dates except the same one 
    return sortedRange
        .Any(tuple => (from innerLoop in sortedRange
                        where (
                                tuple.End > innerLoop.Start &&
                                innerLoop.SortOrder > tuple.SortOrder
                                ) ||
                                (
                                tuple.End == innerLoop.End &&
                                tuple.Start == innerLoop.Start &&
                                innerLoop.SortOrder > tuple.SortOrder
                                )
                        select innerLoop).Any());
}

Additionally I also pass my DateRanges through a sanity check to ensure that the start < end dates.

Powershell Fun (Key - Value Pairs)

Using Powershell to Import Key-Value Pairs from a file

Given a file named ConfigurationKeyValuePairs.conf with Key Value pairs that looks like: 

Key^Value
WebsiteHostName^localhost
QueryServiceHostName^localhost
ApplicationServiceHostName^localhost
ReportServerHostName^localhost
ReportServerPort^8080
IncludeExceptionDetailInFaults^true
ICacheServicePort^8050
Compilation.Debug^true   

You can extract these into a powershell variable with the following command:

Get-Content .\ConfigurationKeyValuePairs.conf | %{$configurationSettings = @{}} { if ($_ -match "(.*)\^(.*)") { $configurationSettings[$matches[1]]=$matches[2]; }}

The keys can now be referred to using 

$configurationSettings.Key

---

References:  http://blogs.msdn.com/b/mwilbur/archive/2007/02/23/powershell-1-liner-key-values-pairs-from-text-file-to-hashtable.aspx

---

Windows Server 2008 Command-Line Tools Summary

	Command	Description
	Arp	Display and modify the IP to physical address translation tables used by the Address Resolution Protocol (ARP).
	Assoc	Display and modify file extension associations.
	Attrib	Display and change file attributes.
	Break	Configure extended Ctrl-C checking.
	Bcdedit	Configure properties in name database to control boot loading.
	Cacls	Display or modify access control lists of files.
	Call	Call a script or script label as a procedure.
	CD/Chdir	Display the name of or changes the current directory.
	Chcp	Display or set the active code page number.
	Chkdsk	Check a disk for errors and display a report.
	Chkntfs	Display the status of volumes. Set or exclude volumes from automatic system checking during system boot.
	Choice	Create a selection list from which users can select a choice in batch scripts.
	Cls	Clear the console window.
	Cmd	Start a new instance of the Windows command shell.
	Color	Set the colors of the command-shell window.
	Comp	Compare the contents of two files or sets of files.
	Compact	Display or modify the compression of files or sets of files.
	Convert	Convert FAT volumes to NTFS.
	Copy	Copy or combine files.
	Date	Display or set the system date.
	Del	Delete one or more files.
	Dir	Display a list of files and subdirectories within a directory.
	Diskcomp	Compare the contents of two floppy disks.
	Diskcopy	Copy the contents of one floppy disk to another.
	Diskpart	Invoke a text-mode command interpreter so that you can manage disks, partitions, and volumes using a separate command prompt and commands that are internal to Diskpart.
	Doskey	Edit command lines, recall Windows commands, and create macros.
	Driverquery	Display the current device driver properties and status.
	Echo	Display messages, or turns command echoing on or off.
	Endlocal	End localization of environment changes in a batch file.
	Erase	See Del.
	Exit	Exit the command interpreter.
	Expand	Uncompress files.
	FC	Compare two files and display the differences between them.
	Find/Findstr	Search for a text string in files.
	For	Run a specified command for each file in a set of files.
	Format	Format a floppy disk or hard drive.
	Fsutil	File system utility - displays and configures file system properties.
	Ftp	Transfer files.
	Ftype	Display or modify file types used in file extension associations
	Goto	Direct the Windows command interpreter to a labeled line in a script.
	Gpresult	Display Group Policy information for a machine or user.
	Graftabl	Enable Windows to display extended character sets in graphics mode.
	Help	Display Help information for Windows commands.
	Hostname	Display the computer name.
	ICACLS	Display, modify, backup, and restore ACLs for files and directories.
	IF	Perform conditional processing in batch programs.
	Ipconfig	Display TCP/IP configuration.
	Label	Create, change, or delete the volume label of a disk.
	Md/Mkdir	Create a directory or subdirectory.
	Mklink	Create symbolic and hard links.
	Mode	Configure a system device.
	More	Display output one screen at a time.
	Mountvol	Manage a volume mount point.
	Move	Move files from one directory to another directory on the same drive.
	Openfiles	Display files opened by remote users for a file share.
	Nbtstat	Display status of NetBIOS.
	Net Accounts	Manage user account and password policies.
	Net Computer	Add or remove computers from a domain.
	Net Config Server	Display or modify configuration of Server service.
	Net Config Workstation	Display or modify configuration of Workstation service.
	Net Continue	Resume a paused service.
	Net File	Display or manage open files on a server.
	Net Group	Display or manage global groups.
	Net Localgroup	Display or manage local group accounts.
	Net Pause	Suspend a service.
	Net Print	Display or manage print jobs and shared queues.
	Net Session	List or disconnect sessions.
	Net Share	Display or manage shared printers and directories.
	Net Start	List or start network services.
	Net Statistics	Display workstation and server statistics.
	Net Stop	Stop services.
	Net Time	Display or synchronize network time.
	Net Use	Display or manage remote connections.
	Net User	Display or manage local user accounts.
	Net View	Display network resources or computers.
	Netsh	Invoke a separate command prompt that allows you to manage the configuration of various network services on local and remote computers.
	Netstat	Display status of network connections.
	Path	Display or set a search path for executable files in the current command window.
	Pathping	Trace routes and provides packet loss information.
	Pause	Suspend processing of a script and wait for keyboard input.
	Ping	Determine if a network connection can be established.
	Popd	Change to the directory stored by Pushd.
	Print	Print a text file.
	Prompt	Change the Windows command prompt.
	Pushd	Save the current directory then changes to a new directory.
	Rd/Rmdir	Remove a directory.
	Recover	Recover readable information from a bad or defective disk.
	Reg Add	Add a new subkey or entry to the Registry.
	Reg Compare	Compare Registry subkeys or entries.
	Reg Copy	Copy a Registry entry to a specified key path on a local or remote system.
	Reg Delete	Delete a subkey or entries from the Registry.
	Reg Query	List the entries under a key and the names of subkeys (if any).
	Reg Restore	Write saved subkeys and entries back to the Registry.
	Reg Save	Save a copy of specified subkeys, entries, and values to a file.
	Regsvr32	Register and unregister DLLs.
	Rem	Add comments to scripts.
	Ren	Rename a file.
	Replace	Replace a file.
	Route	Manage network routing tables.
	Rmdir	Remove a directory.
	Set	Display or modify Windows environment variables. Also used to evaluate numeric expressions at the command line.
	Setlocal	Begin localization of environment changes in a batch file.
	Sc	Display and configure background processes (services).
	Schtasks	Schedule commands and programs to run on a system.
	Sfc	Scans and verifies protected operating system files.
	Shift	Shifts the position of replaceable parameters in scripts.
	Shutdown	Perform system shutdown.
	Sort	Sort input.
	Start	Start a new command-shell window to run a specified program or command.
	Subst	Maps a path to a drive letter.
	Systeminfo	Display machine properties and configuration.
	Tasklist	Display currently running tasks and services.
	Taskkill	Kill or stop a running process or application.
	Time	Display or sets the system time.
	Title	Sets the title for the command-shell window.
	Tracert	Display the path between computers.
	Tree	Graphically displays the directory structure of a drive or path.
	Type	Display the contents of a text file.
	Ver	Display the Windows version.
	Verify	Tells Windows whether to verify that your files are written correctly to a disk.
	Vol	Display a disk volume label and serial number.
	Xcopy	Copy files and directories.
	WMI	Display WMI information.

Scraping HTML Table data into a Google Docs Spreadsheet

Tried something interesting today.

I needed to scrape the Root Zone Database data from the Internet Assigned Numbers Authority (IANA) website at [http://www.iana.org/domains/root/db] .So I imported the data to a google docs spreadsheet and then exported it to a csv as follows:

1. Create a new Google Docs spreadsheet
2. In the first row, import the data using the ImportHtml() function with the following syntax [=ImportHtml("http://www.iana.org/domains/root/db","table",0) ] 
3. This should now populate the rest of the table with the data from the web page.
4. I can now download the spreadsheet as a csv ( File ⇒ Download as ⇒ Comma Separated Values )

Expanding TFS 2010 Database post Lab Management setup

Monday Morning:

I usually love Mondays. A brand new week, so much to look forward to, UNLESS you get an angry email from your System Administrator & you know you have a plethora of other development tasks you need to finish this week.

The Email:

Findings:

After some digging around, here is what I had discovered:

• Our Team Foundation Server database [it uses Microsoft SQL Server 2008 (RTM) - 10.0.1600.22 (X64)]
• The highest recorded growth was in the following few tables

Post our initial investigation, it became clear that TFS had decided to store binary build objects in the attachment tables after we had set-up Lab Management & decided to use Test Driven Development (Unit Tests run on Gated Check Ins, Unit + Integration Tests run on Nightly Builds & Unit + Integration + UI Tests run on Staged & Release Builds.) I wrote the following scripts:

It was interesting to see that on every test run (triggered by a Gated Check In or a Release/Staged build) we were adding > 250 files to the Attachment Content table. Yikes!! This was looking more and more like an episode of  "When good servers go bad!"

Test Attachment Cleaner:

A couple of interesting google searches later I discovered the Test Attachment Cleaner [http://visualstudiogallery.msdn.microsoft.com/3d37ce86-05f1-4165-957c-26aaa5ea1010/]


The MSI installs itself to "C:\Program Files (x86)\Microsoft\Test Attachment Cleaner". The readme file explained it all ...

In Dev10, with the introduction of Visual Studio Test Professional 2010 & Visual Studio Premium/Ultimate 2010, testers can author manual and automated Test cases, configure the different diagnostic data collectors (as part of Test Settings), associate the Test Settings with Test Plan/Suites and then execute these test cases as part of Test Runs. The execution of a Test Run (whether automated or manual) generates a bunch of diagnostic data, which may be captured either automatically by the system or manually by the tester. This diagnostic data is critical in eliminating the “no repro” bug scenarios between the testers and developers. However, the downside of this rich diagnostic data captures is that the system/user generated diagnostic data, over a period of time, can grow at a rapid pace and start taking up database space.

With Dev10, the database admin has little or no control over what data gets attached as part of Test Runs – i.e., there are no policy settings he can control to limit the size of the data capture OR no retention policy to determine how long to hold this data before initiating a cleanup. In such scenarios, the Admin has no mechanism to:

a.       Determine which set of diagnostic captures is taking up how much space AND
b.      Reclaim the space for runs which are no longer relevant from business perspective.

The “Test Attachment Cleaner” tool fills this void by serving both the above points. 

I ran the following commands on each project:

tcmpt.exe attachmentcleanup /collection:http://localhost:8080/tfs/TFS2008_Migrated_Projects /teamProject: ProjectName  /settingsFile:.\SampleSettings\CustomScenario.xml /mode:Preview

tcmpt.exe attachmentcleanup /collection:http://localhost:8080/tfs/TFS2008_Migrated_Projects /teamProject:ProjectName /settingsFile:.\SampleSettings\CustomScenario.xml /mode:Delete

The CustomScenario xml file I used was as follows:

Additional Patches & Test Settings:

Additionally, it turns out that we needed the following patches/updates/hotfixes applied to TFS:

1. Microsoft Team Foundation Server 2010 Service Pack 1: http://support.microsoft.com/kb/2182621
2. Cumulative update package 2 for Visual Studio Team Foundation Server 2010 Service Pack 1: http://support.microsoft.com/kb/2643415#appliesto
3. A hotfix that can reduce the size of the test data saved to the TFS database is available for Team Foundation Server 2010 Service Pack 1: http://support.microsoft.com/kb/2608743

Also, we need to ensure that the test settings file referenced in the build has the "Enable Deployment" option unchecked.

Post these updates, our test runs showed a significant reduction in the attachments saved to TFS as shown by the following query: