Using SpecFlow for Testing

Some time ago, I was asked to put together a talk on how we’ve embraced BDD with SpecFlow here at Mimeo. The original deck has been shared out on SkyDrive. Here’s a sort of transcript of the talk I prepared for.

 


clip_image001Before joining Mimeo, my only experience was working on teams who practiced big up front planning. Releases would typically be 1 – 1 1/2 years (at best) with milestones usually lasting 4-6 months, if not longer. We would spend weeks writing dev specs and test plans, getting them reviewed, bought off, updated, etc. These were supposed to be “living documents.” But in practice, once the document was signed off, they were seldom, if ever, updated at all. So when I found out that Mimeo did releases every 4-6 weeks, I was crapping my pants wondering how the hell am I supposed to plan my testing so quickly?

clip_image002I fell back to what I knew – write a quick test plan. I created a simple “one pager” (literally 1 page, not 10 pages as many “one-pagers” at Microsoft were) to help me layout the test methodology and high level test cases. Since we were agile, I felt that this could truly be a living doc and could be updated from sprint to sprint. Yeah, no. Again, this was dead. It was hard to get folks to read it, review it, or understand the tests.

clip_image003

The other problem: Tests are hard. Tests are hard to design; hard to develop; hard for people to read and understand what they were really intending to test. When reviewing the one pager with dev and project managers, it was hard to relate the value of these atomic functional tests to the stories.

Then we learned about the concept of Behavior Driven Design (BDD). The fundamental concept is to “write software that matters.” Write only that which pertains to the user stories for a particular sprint that satisfy the scenarios of the stakeholders. BDD breaks down a user story into the high level tests scenarios needed for acceptance. With BDD, test scenarios (behaviors) should be written that map closely to the user stories and written in a plain ubiquitous language that everyone can understand.

“’Behaviour’ is a more useful word than ‘test’” [Dan North]

At Mimeo, we adopted SpecFlow. SpecFlow is essentially the .NET version of Cucumber, which has been popular in the Ruby world. Like Cucumber, SpecFlow utilizes the Gherkin syntax for writing scenarios. At the core of Gherkin are the Given/When/Then step definitions. For those familiar with TDD, this maps directly to the 3-A’s pattern of Arrange/Act/Assert. The Given is your Arrange, where you setup your test. The When is your Act, where you perform an action under test. Finally, Then is where you Assert or validate your output.

Here’s an example feature (user story) and scenario directly taken from SpecFlow’s template:

Feature: Addition In order to avoid silly mistakes As a lazy mathematician with a calculator I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calculator And I have entered 70 into the calculator When I press add Then the result should be 120 on the screen

What you see here is an actual working executing scenario. The magic lies in the SpecFlow runtime, code generation, and for the test developer to “wire up” these steps into automation (visit specflow.org for details). But this example has some powerful characteristics:

1. Scenarios are easy to understand by anyone. Developers, testers, project planners, business owners, customers – almost anyone can look at the above scenario and determine exactly what it is doing. There is an immediate appreciation for the value of this test to the user story.

2. Pushes quality upstream. If teams can take user stories in their sprint planning meetings and start breaking down scenarios using the Gherkin syntax, then the team has just gone thru and bootstrapped the testing effort before the planning was even done.

3. Promotes team collaboration. When the team is communicating using the same language that is easy to understand, it’s so much easier for test, dev, PM to have discussions for whether a particular set of scenarios makes sense.

4. Gets the test team involved early.

The last point has been an important one for me. When joining Mimeo, testers were routinely lagging behind the devs in story acceptance. In a typical 4 week sprint, this meant that tests that should have been automated were instead manually tested, with the hopes of automating them getting pushed to sometime in the future. Teams also didn’t have a good understanding of the tests that were performed (automated or manual), thus making confidence ambiguous and increasing risk and uncertainty. Since moving to SpecFlow, testers have been in lock step with development, sometimes even ahead of the game. Teams generally have better certainty and confidence when accepting a story because they are on the same page for what is actually being verified.

Lessons Learned

It’s been about a year since we’ve been using SpecFlow, so naturally there were a few lessons that we’ve learned the hard way.

1. Think about the scenarios and plan them out a little before you start developing them. Or at least try.

clip_image004

I was lucky in that the areas that I began with had no tests at all. I started to just get in there a little and tried writing my step definition and kept hitting brick walls in the Given. There was a significant amount of work that I needed to do for just setting up the test environment and components to get them into a state for testing. Try to think about the things you need to do to arrange or setup yourself for the scenarios to work. Do you need databases? Mocks? Test data? How will get these? How will you utilize these?

2. Scenario development is highly iterative. Have patience. clip_image005

As you develop new scenarios, get new user stories, get feedback about debugging, you will find the need to refactor your scenarios and step definitions. I’ve refactored my tests several times so far, each time making it easier for new scenarios to be developed.

3. You may not succeed the first time. And that’s ok.clip_image006

There’s an initial learning curve to get started with BDD, so you’ll start “somewhere.” You may quickly see that the first approach sucked and it’s time for a different approach. Again, have patience.

Advertisements
This entry was posted in SpecFlow, testing, Work. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s