AppFabric v1.1 Installer Hates You

[Blogging this because we’ve hit this a few times at work in the past 3 months]

If all attempts for installing AppFabric v1.1 fail with the ever helpful 1603, look in your %TEMP%\AppServerSetup1_1_CustomActions(datestring).log file [where datestring is the last time you invoked the installer]. You may see a message like this:

6/23/2014 11:32:45 AM EXEPATH=C:\Windows\system32\\net.exe PARAMS=localgroup AS_Observers /comment:"Members of this group can observe AppFabric." /add LOGFILE=C:\Users\me\AppData\Local\Temp\AppServerSetup1_1_CustomActions(2014-06-23 11-32-31).log
Error: System error 1379 has occurred.
Error: The specified local group already exists.

Answer: Visit this StackOverflow answer: Basically, blow away the AS_Observers and AS_Administrators local groups. Then try to run the installer again.

Then go grab some ice and Tylenol. That past hour of slamming your head on your desk probably left a bruise.

Posted in Work | Tagged , , | Leave a comment

Better Workaround for AppFabric Cache’s Tracing Hatred

As a follow up from my last post, here’s a simple extension method to forcibly restore the default trace listener every time you invoke an AppFabric Caching Administration Management cmdlet. It’s not the prettiest code, but it’s functional:

Posted in Work | Tagged , , , , | Leave a comment

AppFabric DataCacheServerLogManager Hates Your Trace.WriteLines

I like to fill my test code with TONS of logging. I log damn near everything because I hate it when the team has to waste time answering, “Can you get me a repro?” when we can just peruse the logs to see what went wrong. I also like to watch the traces get logged in real time using DebugView as my tests execute. So after I made some recent changes in my code to programmatically manage our AppFabric cache, I was super bummed to see my real time logging completely disappear.

The following code shows a snippet of what my test code is doing to repro the issue:

The traces simply no longer get output to the default trace listener. Sure, if I setup a text file listener in my app.config, the traces are properly logged to the file. But what fun is that? I need this stuff in real time now, not after my 100 or so tests run.

So I posed this question to @drub0y, Mimeo’s Chief Software Architect. Sure enough minutes later he responded with:


Well damn. JustDecompile to the rescue. Let’s see how we could have done this ourselves.

  1. Fire up JustDecompile, and then load %PROGRAMFILES%\AppFabric 1.1 for Windows Server\Microsoft.ApplicationServer.Caching.Core.dll to the assembly list.
  2. Click on the Search button
  3. Go to the Full Text tab and enter: “Listeners”

You should then get a single hit:


And sure enough, when you double click on that, you get the code that @drub0y sent me in the mail above.

The fix? Easy. I just had to add a simple Trace.Listeners.Add(“DefaultTraceListener”) in my code after invoking the powershell pipeline.

Posted in testing, Work | Tagged , , , , | 1 Comment

Migrating TFS shelvesets to another branch

I keep forgetting how to do this, so I’m posting it so I don’t have to keep Googling this with Bing every time:

tfpt unshelve "[shelveset name]" /migrate /source:"$/serverpath/branch1" /target:"$/serverpath/branch2"

Posted in testing, Work | Tagged , | Leave a comment

FusLogVw and ILDasm to the Rescue

I spent a bunch of time helping out one of my devs yesterday with a problem that arose from an absolutely trivial change. Isn’t that how all debugging sessions start? “It was a simple change, I don’t understand why these tests don’t work!” In his defense, the change really was trivial (adding a bool property to a service contract). So what happened? Let’s start with NuGet.

At Mimeo, we have a large infrastructure of backend WCF services to power our Photo business. The services are generally very well componentized, allowing us to push out new builds of components without (or with very little) impact to other dependent services. All of this is managed with NuGet. We use NuGet extensively to create and consume packages for our service contract dependencies. Stable components have packages built out of their Main branch and are considered normal “Release” packages. When certain sprints require breaking changes across several components, we create “pre-release” packages for the other components within that branch to consume. NuGet works pretty damn well. And when it works, it’s magical. When it doesn’t, we have problems.

Yesterday, my developer made a trivial change to one of our components. He simply added a new bool to the service’s data contract. Let’s call this service OIS. The change was checked in, the continuous integration system built and published a new pre-release NuGet package and all looked good. He then updated the NuGet dependency in an upstream service to pick up this newly built NuGet package with his data contract change. It was a simple NuGet package update (it was already using a pre-release version of OIS from last week). The update succeeded, the local build was successful. Like a good dev, he attempted to run the smoke and BVT tests on his box to verify nothing broke. And it broke. The tests simply would not run.

The tests bombed out with the error, “Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly ‘OIS, Version=, Culture=en-US, PublicKey=null’ or one of its dependencies.” The reference was there in the project, and the solution built successfully, but the DLL could not be loaded? So we looked in the TestResults folder to see what binaries got deployed with the tests. Sure enough, the assembly was not there. Back to the References – Copy Local was set to true. WTF?


Let’s Start Over

Step 1 in debugging: go back to the baseline and make sure the original results are expected. We undid all of the changes and reverted back to last week’s OIS NuGet package. Build succeeded, tests…passed! Interesting. Now we know there’s a problem with the new NuGet package. We did a side by side of the old and new OIS NuGet packages in the NuGet Package Explorer. Everything looked *exactly* the same except for the timestamps, of course. We checked the ‘packages’ folder in the solution to make sure it expanded correctly, and it did. The package was fine?! The assembly is there to build against, but for some reason it’s simply not getting copied as a dependency during test execution.


Isolate the Problem

Step 2 in debugging: isolate the problem. I then created a new console app solution, pulled in the newly built NuGet package, and then just instantiated one of the data contract classes. Build succeeded, execution…failed! Nice, now we’re getting somewhere. The failure was exactly the same – FileNotFoundException. The assembly was not there in the bin\debug folder for the console app at all. For shits and giggles, we said screw it – let’s copy the damn assembly there and see what happens. We ran the console app and execution…failed! WTF? The assembly is right there, but it can’t find it? We then verified that the assembly version and public key of the assembly matched the exception message. Of course it did. This is clearly some sort of assembly binding issue. Time for the big guns.


Reach Into Your Tool Belt

The .NET Assembly Binding Log Viewer (FusLogVw) is a tool that allows one to trace assembly loads for .NET applications. It shows rich debugging information for where it searches for a particular dependent assembly, where it finally loads it from, and what errors it encounters along the way. This can also be accomplished by flipping a regkey, which is what the tool really does. But the tool shows the events in simple Winforms app so that you don’t have to look at it in Explorer. So anyway, back to the issue – why isn’t OIS.dll loading? I turned on the option to Log All Binds and executed our test console app. Here’s what we saw for the attempted load of OIS.dll:

Well that’s interesting, why is it trying to pick it up from en-US? Why does it think it’s a satellite resource assembly? Again, for shits and giggles, we said screw it – let’s copy the damn assembly to an ‘en-us’ subfolder and see what happens. We ran the console app and execution…passed!! Whoa.


Slam Head Firmly Against Wall. Repeat.

How did adding a simple bool cause the assembly to be considered a satellite assembly? Let’s check this out in ILDasm and compare the manifest metadata between last week’s working OIS.dll and today’s new OIS.dll. ILDasm showed us the following new addition to the latest OIS.dll that wasn’t there in the old one:

.hash algorithm 0x00008004

.ver 1:1:0:0

.locale = (65 00 6E 00 2D 00 55 00 53 00 00 00)    // e.n.-.U.S…


So the assembly *is* being built differently now. This made no sense. Something must have changed outside of my dev’s simple bool. After looking at OIS’s change history, we detected another seemingly minor addition to its AssemblyInfo.cs made the Friday evening by another developer:

[assembly: AssemblyCulture(“en-US”)]

Blargh!! This innocent addition completely changes everything. The AssemblyCulture attribute marks the assembly as a satellite assembly. These are assemblies that should only contain localized resources that the main assembly would read from during program execution. For more information consult the documentation on MSDN. The fix was simple – remove the attribute from OIS’s AssemblyInfo.cs, rebuild the NuGet package, consume, enjoy.

Posted in testing, Work | Tagged , , , , , | Leave a comment

No time to read? Then start a book club! O_o

It’s been really tough lately to find the time to read. I am not usually an avid reader. When I do read, it takes me for fricken ever to finish a book. With 2 small kids at home (one is an infant) and a long commute into the city, it’s been hard to find the desire (and time) to sit and just read something. You should see the stack of unread books on my nightstand. Turns out that people at work had the same sort of problem – they want to read, but don’t they don’t want to read. So how can I satisfy my want for learning without having the time and motivation? My answer: force it to happen with a deadline and make it a group effort.

This is why I started a book club at work. [And those close to me are probably ROFL right now reading that. That’s ok, I’ll give you a sec to let it out of your system.] There’s a bunch of technical and managementy books that I’ve seen or heard about, but just couldn’t ever get my ass in gear to read them. I really needed a forcing function to help me consume these books. And I figured if I had other people involved, then I would be compelled to participate. So a book club seemed like the perfect answer. And after 4 books in, I can say that it’s working fairly well.

Our book club is pretty simple and follows these guidelines:

  1. One book will be selected by a different member of the group each time.
  2. Try to pick books that people can finish in 4-6 weeks, respecting people’s deadlines and workload (so no War and Peace type epics)
  3. Try to pick books that are somewhat related to work. These can be development books (e.g. Pragmatic Programmer), test books (e.g. Explore It), or business books (i.e. Switch).
  4. Discussions will be scheduled in a Google Hangout every 4-6 weeks. Anyone can participate regardless of whether you read the book or not.

Admittedly, the book club hasn’t been a huge success, but it hasn’t been a total failure. It’s been…OK. The biggest hurdle so far has been participation. When first approaching the team with the idea of starting a book club, there was a ton of enthusiasm from about 20 or so people. Lots of people replied with, “Great idea!” or, “Sure, I’ll totally do it!” But so far, there’s only been 3 people, including myself, who read the book and participate in the discussion (well, 1 person participates in the discussion, but doesn’t read the book. Sigh.). Several people say that they will read and join the meeting, but don’t end up doing either. And the sad truth is that it still comes down to two main “problems” – time and motivation.

Some of the folks just don’t want to be bothered with reading a book on a deadline. When they come home from work, it’s their time. Time to spend with the family. Time to spend relaxing and not dealing with work. Time to read what they want to read on their own deadline. For other folks, they don’t have the motivation. Enrichment isn’t something they care to do. They come to work, do their work, go home. Usually, it’s these same folks who don’t participate on Stack Overflow or Github, or read technical blogs, or don’t do anything outside of work that’s remotely related to work (i.e. “techy things”). And for a small number of folks, the content of the books just aren’t interesting to them and they would rather not expand outside their interests.

It’s tough to argue with time. It’s a tough nut to crack and I can empathize – I’ve been there, too, quite often. But I’m a big fan of enrichment and learning new stuff even if it’s not directly related to my immediate tasks. It’s part of filling my tool belt since you never know when you will need to pull out any of this information.

We’ll see how long this book club lasts. I for one enjoy reading a new book every month or two. I’m achieving my goal, so I’m happy in continuing this a long as my one other cohort participates. And if this book club does end up fizzling out, who knows, I may still stick with this reading thing.

Here’s the list of books we’ve read so far. If anyone has suggestions for other books or for tips on running this book club better, let me know in the comments.

February: Secrets of a Buccaneer Scholar, by James Bach

March: The Icarus Deception, by Seth Godin

April: Perfect Software and Other Illusions about Software Testing, by Gerald Weinberg

July: Blink: The Power of Thinking Without Thinking, by Malcolm Gladwell

September: The Tipping Point, by Malcolm Gladwell

Posted in Work | Tagged | 1 Comment

Born to Test?

There are those random times where you find a bug and think, “Damn, I was born to test!” The other day I found a bug where this was the case – quite literally…

At, we have a set of internal web apps written in ASP.NET MVC that are used by folks on the production floor to manage load and capacity for our presses. There were some new features added to the tool, and the devs asked me to take a look at the user stories and make sure things were in order. Now, I don’t normally test this tool. The devs have a really good battery of unit tests and are extremely diligent of testing the tool themselves. So I figured that I’ll just spend an hour or two doing some exploratory testing to dot the t’s and cross the i’s. And then I found a bug.

In one particular area of the app, the View grabs your domain account and passes it off to a Controller. So when I started to test out one of the newly added features, I kept getting a partially rendered page. This happened across all browsers I tested against. Now I’m thinking to myself, with all the unit tests and manual/exploratory tests the devs do themselves, how the hell did they let this through? Now I had the IE Dev Tools open already (as I usually do when exploratory testing a site), so I figured I should take a network trace and see if anything pops up. Here’s what I found:


It’s interesting that I’m getting a 400 response. Very odd. So, let’s talk to the devs. Neither of the two devs, Chris and Phil, could repro this issue. “Works fine for me!” Of course it does. L We took a closer look. My domain account is DOMAIN\nshenoy. But this is coming through as DOMAIN%0Ashenoy. When the developers looked at their resource request, Chris got “DOMAINchris” and Phil got “DOMAINphil”. Here’s the JavaScript code in the Index.cshtml that grabs the domain account:

var initParameters = {
    userName: “@User.Identity.Name” 

And here’s the script that actually uses this value:

my.getPressesUrl = function(userName) {
    return my.pressesUrl + “/” + encodeURIComponent(userName);

Hmm, so there’s a call to encodeURIComponent(). If we type “encodeURIComponent(DOMAIN\nshenoy)” into the console, sure enough we get DOMAIN%0Ashenoy. So it looks like it’s encoding the “\n” into an ASCII newline. And what’s happening for Chris and Phil? Seems that encodeURIComponent() is simply stripping out that backslash. Whoa.

Lesson 1: when dealing with domain accounts, be sure to replace to “\” with “\\” to properly escape it.

Cool, so I went ahead and just did a quick and dirty fix to the code like the following:

userName: ‘@User.Identity.Name.Replace(“\\”, “\\\\”)’

And this worked. The call to encodeURIComponent() now returned “DOMAIN%5Cnshenoy”. Yay? No. Because NOW I started to get a 404. What the what?! Here’s what the request looks like now in the network profiler:


This looks right to me. The username is correctly escaped, so we should be good to go right? Well, no. Obviously. So why isn’t this route resolving? The backslash is a special character and needs to be treated differently. To work around this issue of query strings containing special characters, we need to supply the “id=”. Changing the function like so made things work:

my.getPressesUrl = function(userName) {
    return my.pressesUrl + “/?id=” + encodeURIComponent(userName);

This could have been solved other ways (i.e. make a custom resolver in the RouteConfig.cs), but this works too.

Lesson 2: when naming your children, try to make their names interesting for testing purposes.

If my name wasn’t Nithin, I wouldn’t I have found this bug (thanks mom and dad!). Though I guess if I was a \timothy or \robert, it may have also resulted in interesting test cases. 😉

But seriously, pay attention to strings containing special characters. And also use the tools available to you. Simply watching network traffic in the dev tools while testing gave a wealth of information for why this strange behavior was happening in the first place.

Special thanks to @drub0y for reviewing this post.

Posted in testing, Work | Tagged , , , , | Leave a comment