I like to fill my test code with TONS of logging. I log damn near everything because I hate it when the team has to waste time answering, “Can you get me a repro?” when we can just peruse the logs to see what went wrong. I also like to watch the traces get logged in real time using DebugView as my tests execute. So after I made some recent changes in my code to programmatically manage our AppFabric cache, I was super bummed to see my real time logging completely disappear.
The following code shows a snippet of what my test code is doing to repro the issue:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
The traces simply no longer get output to the default trace listener. Sure, if I setup a text file listener in my app.config, the traces are properly logged to the file. But what fun is that? I need this stuff in real time now, not after my 100 or so tests run.
So I posed this question to @drub0y, Mimeo’s Chief Software Architect. Sure enough minutes later he responded with:
Well damn. JustDecompile to the rescue. Let’s see how we could have done this ourselves.
Fire up JustDecompile, and then load %PROGRAMFILES%\AppFabric 1.1 for Windows Server\Microsoft.ApplicationServer.Caching.Core.dll to the assembly list.
Click on the Search button
Go to the Full Text tab and enter: “Listeners”
You should then get a single hit:
And sure enough, when you double click on that, you get the code that @drub0y sent me in the mail above.
The fix? Easy. I just had to add a simple Trace.Listeners.Add(“DefaultTraceListener”) in my code after invoking the powershell pipeline.
I spent a bunch of time helping out one of my devs yesterday with a problem that arose from an absolutely trivial change. Isn’t that how all debugging sessions start? “It was a simple change, I don’t understand why these tests don’t work!” In his defense, the change really was trivial (adding a bool property to a service contract). So what happened? Let’s start with NuGet.
At Mimeo, we have a large infrastructure of backend WCF services to power our Photo business. The services are generally very well componentized, allowing us to push out new builds of components without (or with very little) impact to other dependent services. All of this is managed with NuGet. We use NuGet extensively to create and consume packages for our service contract dependencies. Stable components have packages built out of their Main branch and are considered normal “Release” packages. When certain sprints require breaking changes across several components, we create “pre-release” packages for the other components within that branch to consume. NuGet works pretty damn well. And when it works, it’s magical. When it doesn’t, we have problems.
Yesterday, my developer made a trivial change to one of our components. He simply added a new bool to the service’s data contract. Let’s call this service OIS. The change was checked in, the continuous integration system built and published a new pre-release NuGet package and all looked good. He then updated the NuGet dependency in an upstream service to pick up this newly built NuGet package with his data contract change. It was a simple NuGet package update (it was already using a pre-release version of OIS from last week). The update succeeded, the local build was successful. Like a good dev, he attempted to run the smoke and BVT tests on his box to verify nothing broke. And it broke. The tests simply would not run.
The tests bombed out with the error, “Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly ‘OIS, Version=1.1.0.0, Culture=en-US, PublicKey=null’ or one of its dependencies.” The reference was there in the project, and the solution built successfully, but the DLL could not be loaded? So we looked in the TestResults folder to see what binaries got deployed with the tests. Sure enough, the assembly was not there. Back to the References – Copy Local was set to true. WTF?
Let’s Start Over
Step 1 in debugging: go back to the baseline and make sure the original results are expected. We undid all of the changes and reverted back to last week’s OIS NuGet package. Build succeeded, tests…passed! Interesting. Now we know there’s a problem with the new NuGet package. We did a side by side of the old and new OIS NuGet packages in the NuGet Package Explorer. Everything looked *exactly* the same except for the timestamps, of course. We checked the ‘packages’ folder in the solution to make sure it expanded correctly, and it did. The package was fine?! The assembly is there to build against, but for some reason it’s simply not getting copied as a dependency during test execution.
Isolate the Problem
Step 2 in debugging: isolate the problem. I then created a new console app solution, pulled in the newly built NuGet package, and then just instantiated one of the data contract classes. Build succeeded, execution…failed! Nice, now we’re getting somewhere. The failure was exactly the same – FileNotFoundException. The assembly was not there in the bin\debug folder for the console app at all. For shits and giggles, we said screw it – let’s copy the damn assembly there and see what happens. We ran the console app and execution…failed! WTF? The assembly is right there, but it can’t find it? We then verified that the assembly version and public key of the assembly matched the exception message. Of course it did. This is clearly some sort of assembly binding issue. Time for the big guns.
Reach Into Your Tool Belt
The .NET Assembly Binding Log Viewer (FusLogVw) is a tool that allows one to trace assembly loads for .NET applications. It shows rich debugging information for where it searches for a particular dependent assembly, where it finally loads it from, and what errors it encounters along the way. This can also be accomplished by flipping a regkey, which is what the tool really does. But the tool shows the events in simple Winforms app so that you don’t have to look at it in Explorer. So anyway, back to the issue – why isn’t OIS.dll loading? I turned on the option to Log All Binds and executed our test console app. Here’s what we saw for the attempted load of OIS.dll:
Well that’s interesting, why is it trying to pick it up from en-US? Why does it think it’s a satellite resource assembly? Again, for shits and giggles, we said screw it – let’s copy the damn assembly to an ‘en-us’ subfolder and see what happens. We ran the console app and execution…passed!! Whoa.
Slam Head Firmly Against Wall. Repeat.
How did adding a simple bool cause the assembly to be considered a satellite assembly? Let’s check this out in ILDasm and compare the manifest metadata between last week’s working OIS.dll and today’s new OIS.dll. ILDasm showed us the following new addition to the latest OIS.dll that wasn’t there in the old one:
So the assembly *is* being built differently now. This made no sense. Something must have changed outside of my dev’s simple bool. After looking at OIS’s change history, we detected another seemingly minor addition to its AssemblyInfo.cs made the Friday evening by another developer:
[assembly: AssemblyCulture(“en-US”)]
Blargh!! This innocent addition completely changes everything. The AssemblyCulture attribute marks the assembly as a satellite assembly. These are assemblies that should only contain localized resources that the main assembly would read from during program execution. For more information consult the documentation on MSDN. The fix was simple – remove the attribute from OIS’s AssemblyInfo.cs, rebuild the NuGet package, consume, enjoy.
It’s been really tough lately to find the time to read. I am not usually an avid reader. When I do read, it takes me for fricken ever to finish a book. With 2 small kids at home (one is an infant) and a long commute into the city, it’s been hard to find the desire (and time) to sit and just read something. You should see the stack of unread books on my nightstand. Turns out that people at work had the same sort of problem – they want to read, but don’t they don’t want to read. So how can I satisfy my want for learning without having the time and motivation? My answer: force it to happen with a deadline and make it a group effort.
This is why I started a book club at work. [And those close to me are probably ROFL right now reading that. That’s ok, I’ll give you a sec to let it out of your system.] There’s a bunch of technical and managementy books that I’ve seen or heard about, but just couldn’t ever get my ass in gear to read them. I really needed a forcing function to help me consume these books. And I figured if I had other people involved, then I would be compelled to participate. So a book club seemed like the perfect answer. And after 4 books in, I can say that it’s working fairly well.
Our book club is pretty simple and follows these guidelines:
One book will be selected by a different member of the group each time.
Try to pick books that people can finish in 4-6 weeks, respecting people’s deadlines and workload (so no War and Peace type epics)
Try to pick books that are somewhat related to work. These can be development books (e.g. Pragmatic Programmer), test books (e.g. Explore It), or business books (i.e. Switch).
Discussions will be scheduled in a Google Hangout every 4-6 weeks. Anyone can participate regardless of whether you read the book or not.
Admittedly, the book club hasn’t been a huge success, but it hasn’t been a total failure. It’s been…OK. The biggest hurdle so far has been participation. When first approaching the team with the idea of starting a book club, there was a ton of enthusiasm from about 20 or so people. Lots of people replied with, “Great idea!” or, “Sure, I’ll totally do it!” But so far, there’s only been 3 people, including myself, who read the book and participate in the discussion (well, 1 person participates in the discussion, but doesn’t read the book. Sigh.). Several people say that they will read and join the meeting, but don’t end up doing either. And the sad truth is that it still comes down to two main “problems” – time and motivation.
Some of the folks just don’t want to be bothered with reading a book on a deadline. When they come home from work, it’s their time. Time to spend with the family. Time to spend relaxing and not dealing with work. Time to read what they want to read on their own deadline. For other folks, they don’t have the motivation. Enrichment isn’t something they care to do. They come to work, do their work, go home. Usually, it’s these same folks who don’t participate on Stack Overflow or Github, or read technical blogs, or don’t do anything outside of work that’s remotely related to work (i.e. “techy things”). And for a small number of folks, the content of the books just aren’t interesting to them and they would rather not expand outside their interests.
It’s tough to argue with time. It’s a tough nut to crack and I can empathize – I’ve been there, too, quite often. But I’m a big fan of enrichment and learning new stuff even if it’s not directly related to my immediate tasks. It’s part of filling my tool belt since you never know when you will need to pull out any of this information.
We’ll see how long this book club lasts. I for one enjoy reading a new book every month or two. I’m achieving my goal, so I’m happy in continuing this a long as my one other cohort participates. And if this book club does end up fizzling out, who knows, I may still stick with this reading thing.
Here’s the list of books we’ve read so far. If anyone has suggestions for other books or for tips on running this book club better, let me know in the comments.
There are those random times where you find a bug and think, “Damn, I was born to test!” The other day I found a bug where this was the case – quite literally…
At Mimeo.com, we have a set of internal web apps written in ASP.NET MVC that are used by folks on the production floor to manage load and capacity for our presses. There were some new features added to the tool, and the devs asked me to take a look at the user stories and make sure things were in order. Now, I don’t normally test this tool. The devs have a really good battery of unit tests and are extremely diligent of testing the tool themselves. So I figured that I’ll just spend an hour or two doing some exploratory testing to dot the t’s and cross the i’s. And then I found a bug.
In one particular area of the app, the View grabs your domain account and passes it off to a Controller. So when I started to test out one of the newly added features, I kept getting a partially rendered page. This happened across all browsers I tested against. Now I’m thinking to myself, with all the unit tests and manual/exploratory tests the devs do themselves, how the hell did they let this through? Now I had the IE Dev Tools open already (as I usually do when exploratory testing a site), so I figured I should take a network trace and see if anything pops up. Here’s what I found:
It’s interesting that I’m getting a 400 response. Very odd. So, let’s talk to the devs. Neither of the two devs, Chris and Phil, could repro this issue. “Works fine for me!” Of course it does. L We took a closer look. My domain account is DOMAIN\nshenoy. But this is coming through as DOMAIN%0Ashenoy. When the developers looked at their resource request, Chris got “DOMAINchris” and Phil got “DOMAINphil”. Here’s the JavaScript code in the Index.cshtml that grabs the domain account:
var initParameters = {
...
userName: “@User.Identity.Name”
...
};
And here’s the script that actually uses this value:
Hmm, so there’s a call to encodeURIComponent(). If we type “encodeURIComponent(DOMAIN\nshenoy)” into the console, sure enough we get DOMAIN%0Ashenoy. So it looks like it’s encoding the “\n” into an ASCII newline. And what’s happening for Chris and Phil? Seems that encodeURIComponent() is simply stripping out that backslash. Whoa.
Lesson 1: when dealing with domain accounts, be sure to replace to “\” with “\\” to properly escape it.
Cool, so I went ahead and just did a quick and dirty fix to the code like the following:
And this worked. The call to encodeURIComponent() now returned “DOMAIN%5Cnshenoy”. Yay? No. Because NOW I started to get a 404. What the what?! Here’s what the request looks like now in the network profiler:
This looks right to me. The username is correctly escaped, so we should be good to go right? Well, no. Obviously. So why isn’t this route resolving? The backslash is a special character and needs to be treated differently. To work around this issue of query strings containing special characters, we need to supply the “id=”. Changing the function like so made things work:
This could have been solved other ways (i.e. make a custom resolver in the RouteConfig.cs), but this works too.
Lesson 2: when naming your children, try to make their names interesting for testing purposes.
If my name wasn’t Nithin, I wouldn’t I have found this bug (thanks mom and dad!). Though I guess if I was a \timothy or \robert, it may have also resulted in interesting test cases. 😉
But seriously, pay attention to strings containing special characters. And also use the tools available to you. Simply watching network traffic in the dev tools while testing gave a wealth of information for why this strange behavior was happening in the first place.
Special thanks to @drub0y for reviewing this post.
Several months ago, my sister gave my 4 year old son this simple little dragonfly nightlight. It has an RGB LED running off of two coin cell batteries that just slowly cycles through all the colors. Simple and effective – my kid loves it. Two things we learned from having this light:
I’ll be damned if my boy could remember to shut the damn light off in the morning.
Coin cell batteries can get expensive (see #1).
Frustrated one day, I mentioned to my boy half-jokingly, “We should build something to make the light automatically turn off.” To which he replied, “Yes daddy, and I want it to shut off after 20min.”
Which brings me to lesson 3: if you mention an idea to my son, he expects it to come to fruition (see Halloween Robot Costume). Clearly he’s on the management track.
I wasn’t quite sure how to proceed. The first reaction was to use a microcontroller. It would be pretty simple to use a Netduino to write code to just turn a pin off after 20 minutes. A few problems with the initial approach. First, the Netduino needs a 3V source and the night light requires 6V; so I would need effectively 2 power supplies and have the Netduino switch the power off to the night light via some sort of transistor. Not to mention that I would be throwing away $30 on a microcontroller for something so small and trivial.
After Googling with Bing for a while, I came across several explanations on using a 555 timer as a monostable multivibrator. And who doesn’t like vibrators? Am I right? A monostable vibrator generates a pulse for a certain amount of time when a trigger is applied. In my case, I want this one-shot to last about 20min to keep the night light on.
I began by designing this on my Windows Phone and Surface RT. Yes, I’m a former ‘Softfie and I drank the Kool-Aid hard. And yes, I’m still listening to music on my Zune, so suck it. I used the iCircuit app, which is available for both devices (and apparently the iPad, too, if you’re into that sort of thing). The app is great for playing with a design, with many simple components to choose from. I loved that the Windows Phone version had sample circuits available to play with. And of course one of the samples was, you guessed it, a monostable multivibrator using a 555. The one disappointing thing is that pin numbers aren’t given in iCircuit. I loved how you can put a scope at any spot in the design and see exactly what’s going on. Very informational and educational.
Good, I got a sample working in iCircuit. But now how do I get this to give a 20min pulse and then shut off? From looking at the datasheet for a 555, we see that the formula is approximately: t = 1.1 * R * C (where t is in seconds). Well damn, that’s easy enough. I know that I have a bunch of 100u capacitors. So using a 100u cap, let’s see what resistor I would need to yield 20 minutes:
20min * 60 sec/min = 1.1 * R * 100u
1200 = 1.1 * R * 0.0001
1200 / 1.1 / 0.0001 = R
10.9M ohms = R
Yikes! That’s big. And I don’t seem to have any in my supply. But I did have 5 1M ohm resistors. Let’s see how close this comes:
t = 1.1 * 5M ohm * 0.0001F
t = 550 sec
Hmm, 9 minutes – close enough. I wired this up in iCircuit and it worked like a champ. I also added a simple SPST switch to act as the trigger. Here’s the result:
Next step – wire it up on a solderless breadboard. The biggest challenge was to figure out the pin out of the 555. Again, I referred back to the 555 datasheet. Initially I used a small resistor of 150K and an LED for the output just to make sure things would work without having to wait the entire time. Everything was wired up. I put the batteries in. The LED turned on. Yay! I hit the switch to trigger it again. And…
Nothing.
Nada.
I double checked my diagram, resistor and capacitor values, swapped out my test LED. Nothing. Sad face. But then I triple checked the datasheet. Something smelled fishy.
If you look at the iCircuit diagram above, we see that the trigger is negated. But according to the datasheet for my NE555 component, the trigger on pin 2 is not inverted!
Facepalm!
Simply wiring the switch to connect to GND completely resolved the issue. Here’s a picture of the breadboard with the completed working circuit:
The next step is to solder this altogether and nicely package this up with the dragonfly light attached. We’ll see how this pans out.
In the previous post, I gave a solution for automating the dreaded “Windows Security” dialog in IE using UIAutomation. Reader Ven left this comment:
“Hey, This is exactly what I’ve been looking for, but when I seem to run it on IE, the code breaks when it reaches
AutomationElement userList = windowsSecurityDialog.FindFirst(TreeScope.Children, userListCondition);
and gives me a null value, I’m not entirely sure why, but I was also wondering where did you obtain these parameters from?”
Turns out that my code works great on Windows 8, but fails on Windows 7 (Ven’s config). Originally I mentioned that I would fix up my code and update the gist. But it got me thinking: I should probably explain how I even figured out the code to begin with. And what better way to do it, than to show how I used UISpy.exe
The key to figuring out the code was to use a tool called UISpy. This tool is very similar to Spy++ in that both navigate the UI elements within a window. Here’s a good explanation from the MSDN forums:
1) Is there any difference between UISpy.exe and Spy++ ?
The first problem is acquiring UISpy.exe. You can try to find it in one of the older Windows SDKs (supposedly), or you can download from my SkyDrive: http://sdrv.ms/13Ky16o. It’s just an executable and doesn’t require any installation.
There are lots of really good tutorials on using UISpy, so I’ll just be presenting highlights in this post. I highly recommend visiting the docs on MSDN or just Bing it. This post is going to show how I used UISpy on Win8 to solve the original problem, and then augment my previous code to work with Windows 7. So let’s start by navigating IE to my page under test and fire up UISpy on Win8:
The UI is quite straightforward. The Content View shows a tree control where the root node is the Desktop, and the children are all the child windows from the Desktop. The Properties window shows all of the UIAutomation properties of the item selected in the Content View tree.
Now, let try to find the “Windows Security” dialog in the UISpy content tree. To do this, make sure the “Focus Tracking Mode” button (the icon looks like a keyboard) is selected. This allows you to click on any control in any window and have the tree automatically select the item that has focus. When I select the User Name edit control in the Windows Security dialog, UISpy now looks like this:
As you can see, the Content View now has “edit” “User name” selected and there’s a red outline around the User name control in the security dialog. If you click around to different controls (or windows) you’ll see the same behavior where it gets a red outline and is selected in UISpy. So what are we really looking at?
UISpy is showing us the element hierarchy in the window. We see that Windows Internet Explorer is a “pane” that has a child “dialog” called “Windows Security”, that has a child “list”, that has a child “list item” called “Use another account”, which has a child “edit” control called “User name”.
At this point, I’d like to refer to the excellent tutorial on http://www.mathpirate.net/log/2009/09/27/swa-straight-outta-redmond/. He walks thru using UISpy to automate calc with UIAutomation. Go take a few minutes and read that, then come back here. Go on, I’ll wait patiently…
Great, you’re back! Good read,eh? Let’s see how that relates to my Gist.
We need to set up the Condition objects in order to create the queries for navigating the elements. As described above, we need to find IE –> Windows Security –> List –> ListItem called “Use another account” –> edit controls –> OK button. Once again, here’s the gist:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Now let’s take a look at Ven’s comment from the previous post again:
“…the code breaks when it reaches
AutomationElement userList = windowsSecurityDialog.FindFirst(TreeScope.Children, userListCondition);
and gives me a null value”
Interesting. This implies that there’s no user list.
So, let’s start my Win7 VM and fire up UISpy.exe. Once I navigate my IE to my website under test, UISpy looks like the following:
Ok, this is basically the same as Windows 8.
Now, let try to find the “Windows Security” dialog in the UISpy content tree. When I select the User Name edit control in the Windows Security dialog, UISpy now looks like this:
We see that now Windows Internet Explorer has a child Dialog called “Windows Security”, that has a child “list item” called “Use another account”, which has a child “edit” control called “User name”. And how does this compare to Windows 8?
Win7: IE “pane” –> Windows Security “dialog” –> ListItem called “Use another account” –> edit controls –> OK button
Win8: IE “pane” –> Windows Security “dialog” –> List –> ListItem called “Use another account” –> edit controls –> OK button
Doh! The Windows 8 version of the dialog has the “User another account” listitem as a child of a “List” element, whereas on Windows 7 the listitem is a direct child of the dialog itself. If we dig in a little more, there are also a few other differences:
The class name for the “Use another account” listitem is different
The class name and AutomationId for the username edit box is different
The class name and AutomationId for the password edit box is different
Armed with this info, let’s fix up the code to work on Windows 8 and Windows 7. The new code is as follows:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
So with a little investigation, we were able to make the code work across Windows 8 and Windows 7. I’ll leave Vista and XP as an exercise to the reader. 🙂
I decided to finally learn some Selenium in order to test an internal Line of Business (LOB) web application. After a quick crash course on Selenium automation, I got a prototype for my test initialization working, but quickly hit this:
I was a bit miffed to learn that Selenium does not natively handle the authentication dialog. Bummer. After fighting with UIAutomation for several hours, here’s a solution I came up with. Hopefully this will save some other people time.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Note that this code only works with IE. WebKit browsers implement the authentication dialog in their own funky ways. But for my purposes, our LOB app is IE based only. #winning
One of my favorite phone screen questions goes something like this:
You’ve just joined a team that has a large test automation suite. Say something like 3000 tests. These tests are run every night against the latest build. You’ve noticed that over the past month the pass rate for the runs vary anywhere between 70% and 90%. In other words, one day the pass rate will be 72%, the next day 84%, the next day 75%, the next day 71%, etc, different everyday. How do you go about analyzing the test suite stability to get the pass rate up?
Depending on the experiences of the tester, this question can go in several directions. Naive testers will just start digging in and debug the first failure without any context, without prioritization, without understanding what they are doing. This is the most common interviewee, unfortunately. Excellent “Senior” well established testers, or those who are relatively active currently in the test community may question why we even care about pass rate; and then we’ll have a discussion about what actually is a test suite and why are so many tests run every night (e.g. why not utilize code coverage to determine which tests to run based on what code changed). In the 2 years that I’ve asked this question, I’ve never had this discussion. 😦 Good testers will start by asking lots of questions to understand the test suite better, what is getting run, whether the builds are changing each night or is the same code, are the tests prioritized, etc. Good testers will dissect the problem down to get data and methodically analyze the data in order to make sense of the chaos.
I ask this interview question because I keep running into this scenario – I keep joining teams that have large automation suites with a “some amount” of test instability. I say “some amount” in quotes because, too often, the people who own the tests don’t take the time (or don’t know how) to understand what specifically is failing and why. Sometimes, the test suites are huge (10,000 tests). But even with a suite of 500 tests, when you see about 20% of your tests failing with every run, it’s human nature to throw up your hands and move on to something else more exciting, because you’ve got other shit piling up that needs to get done for this sprint. Sure, one mitigation would be to schedule time in the sprint to address the tests. But often, people don’t know where to start in order to figure out how much time needs to be pre-allocated. This gives me sad face.
Let’s assume for purposes of this blog, that we have a test suite of regression scenarios, all are of equivalent priority (e.g. Priority 1 tests) and, for whatever reason (lack of sophistication, tooling, end of sprint full regression test, etc) we need to run all of these tests.
So what do you do? You know you have a big test suite. You know that some number of tests are always failing, some number are sometimes failing, some are always passing (and some are inconclusive/timing out). Where do you begin with your analysis? How do you prioritize your work in order to budget time for fixing stuff?
Step 1: Get the Data
If you don’t have data, you don’t have shit. If you’re not using a test runner or automation system that automatically captures test result data and stores them somewhere (SQL is a fine choice), then build one. On my current team, we use SpecFlow and run the tests via MSTest in Visual Studio. We parse the TRX files and import the data into a custom SQL database that captures some very simple data. Here’s a quick and dirty schema we whipped up to capture the basic data we need (yes, this could be greatly improved, but we wanted something simple and fast):
Step 2: Analyze Data
Now you get to figure out where you want to begin. All things being equal, attack the worst tests first. If you have tests that are higher priority for acceptance, attack those first. Figure out the criteria that you need in order to get the biggest bang for the buck. In many cases, the Pareto principle applies: 80% of the failures come from 20% of the tests. Here’s a [sanitized] graph of the data we’re currently seeing in a certain test suite:
This isn’t the first time I’ve seen this graph. I’ve seen this on literally every single team I’ve worked on in my 15year career.
Step 3: Get Off Your Ass and Fix It
You have the data. You have the graphs. You know what tests suck. So do something about it! The graph above pretty clearly shows that about 20% of the tests in this dataset suck, with about 5% sucking hard (i.e. failing all the time). And then there’s a long tail of tests that fail about 20% of the time. This may require more analysis time.
Tips for Analysis:
1. Collect meta data about the test runs. Any sort of meta data about the test run, tests, environment, configuration can help in your analysis. For example, I was on a team where we were down to analyzing failures in the ‘long tail’ of the graph. A particular set of tests would always fail on the 2nd Tuesday of the month. After digging into the test run properties, we determined that the tests were run on a German build of Windows 2008 R2 only the 2nd Tuesday of the month. Bingo! Sure enough, when we manually ran the test on German Win2K8R2, it would fail. Every. Time.
2. Make the time to fix the tests. If your team is spending the time to automate regression tests, then having them sit around constantly failing for unknown reasons is a complete waste of the team’s time. That’s time that could have been spent doing more exploratory testing, perf testing, scalability testing, etc. Stop wasting time and money.
Now the uber question is why does this all matter? What do test pass rates really tell you anyway? Well, it depends on how your team tests. You could fix all of the automated tests and make sure they pass 100% of the time, making the techniques above moot. Great. But what scenarios are you missing? What code isn’t getting covered by your tests? How much time are you spending exploring the codebase for issues? How’s the perf? The automated tests by themselves are completely pointless if the team isn’t doing testing. The regression tests are just another weapon in the testing arsenal. Make them work and work well so that the team can spend time doing some actual testing.
We ran into a nasty problem yesterday at work where we couldn’t reliably debug into code when debugging a test in Visual Studio 2012. Breakpoints weren’t getting hit, lines were jumping around as if the code was optimized, all around a whole lot of nonsense. We finally came across this forum post, “Debugging unit tests in VS 2012 doesn’t work,” pinpointing the problem: when code coverage is enabled in your testsettings file, debugging tests may not work.
Sure enough, disabling code coverage from the test settings file resolved the issue.
Nithin's Blog 