Lessons Learned From My First ThoughtWorks Project

Hey Guys,

It has been a while since my last post here. Lots of things changed in 2011. The big change happend in March when I joined ThoughtWorks Brazil and moved to Porto Alegre. From March to November (8 months) I worked in a distributed team between Brazil and EUA. During the project, we as a team had the opportunity to learn lots of good practices together and this post has the objective to share them with the community. All the topics below came up in team retrospective at the end of the project and correspond to team’s lessons learn from that time.

P.S.: Because I didn’t write down the whole discussion at the time of the retrospective, some arguments may not represent team’s opinion, but my personal one.

1. Visit your client

Because our team were distributed, people were frequently flying back and forth between our office and the client. Some people may think this strategy to expensive to afford (flights, hotels, meals) and it is in fact. However, the success of a project is tightly  related to team members relationship. In co-located teams it is easier to build trust relationships between team members because they see each other all the time, go lunch together, hang out in the weekends. That is not true in a distributed team. It’s crucial that team members had some face-to-face meetings in a while, preferably at the beginning of the project. Believe me, this approach will actually same some money. So every distributed project must have a budget intended for members travel.

2. Remote pairing is hard

ThoughtWorkers love pairing. I would say that our team in Brazil paired in 90% of the tasks. We do pairing to share knowledge between team members, rump up new team members, write better code. We had a strategy of dedicating two brazilian developers to pair with two devs in the US through Skype and LogMeIn. The idea behind that was to balance expertise between both locations. For example, we had specific parts of the application that only on side of the team knew about and we want to eliminate such risk. Unfortunately, we found out that remote pairing is hard. The main reason? It slows development down. In our experience, the challenges in remote pairing were (1) the connection dropped frequently breaking development flow (2) typing in a remote box is slow which resulted in devs having to express their solution through conversations instead of code (3) explain how to code a solution is much harder than implement it yourself. Due to all these challenges we decided to do remote pairing only in very specific scenarios.

3. Use real data/users before going live

During eight months we had four main releases. The frist one was a success which translates to happy users, client and team. Unfortunately, over time users started complaining about application slowness, unpredictable results and data security faults. So what we did wrong? Just after the release the system had almost no data so pages loaded fast, bad data didn’t exist, users didn’t have time to break the system yet. After 2 months we had tons of data inserted and as a consequence application’s report started to take a while to load. In addition, users posted all kinds of data and found out new flows that were never tested. The problem: we didn’t stress the application with real data/users. To solve these problems we basically scheduled user sessions meetings frequently so we could learn how they use the system and dumped production like data in our staging environment. After that we could find problems before they appear in production and solve it as part of our normal process flow.

4. Write acceptance tests

I use to write lots of unit tests before joining ThoughtWorks (I already had TDD mentality). However, I’ve never written acceptance tests. So every release was painful: manual testing took too long and team didn’t know for sure what could break. In my first ThoughtWorks project a user story is only Dev Done after all acceptance scenarios were implemented. Basically, before the story is Ready to Dev, a business analyst sits down with PO and QAs to write scenarios in a Given/When/Then format. That would drive our development so all scenarios were implemented. After that, we were sure that the story was really done. In addition, those scenarios were added to our build pipeline, working as a safety net for developers and preventing us to run tons of manual testing when realeasing. With good acceptance tests coverage, we were able to release fast and with confidence.

5. Keep your build green

In our team we had a continuous integration server responsible for checking-out the source code, compiling the solution, running static code analysis, running automated tests and building deployment artifacts. All these tasks were triggered after every check-in. The continuous integration server helped us by integrating devs work into the final solution and checking for possible errors. A clear example is when someone forgets to check-in source code files or breaks existing automate tests. When that happens the build becomes red indicating the application is in a bad state. We as a team defined a rule that whenever the build is red, a pair would stop whatever they were doing to fix it. Meanwhile no other dev should check-in. Because our team was quite big if others keep checking-in code into the repository, fixing the build becomes harder. This rule helped us keeping our build green as much time as possible which means the application were most of time in a good state, ready to be deployed to staging for example.

6. Insert data for each test

Although acceptance tests are important when developing large software applications, they introduce a big problem: maintenance. Our team learned how expensive is maintaining acceptance test due to their non-determinism. One of the reasons for that was lack of isolation. Right in the beginning, the acceptance test suite was built to insert all necessary data before running all tests and new test data was added to a SQL script file as needed. After a while tests started to share data like users, users group and other sensitive information. If any other test modifies them, other tests that rely on that data would fail unexpectedly. One solution we found was inserting data before each test. Instead of having a single script with all test data, we programmatically added only the necessary data for a single test to run. Although it increased test running time, this change brought more confidence in the build: when the build failed the team knew that some feature was broken.

7. Group communication channel

I have already said that we were a distributed team and communication was a big challenge. One strategy that really helped our team communication was the creation of an open communication channel were all team members were connected all the time. In our case, we had a Skype group conversation for that and we used it with no restrictions to inform what is going on in the project. Instead of privates and parallels conversations we really recommended using the chat to inform when a user story is ready for QA or blocked, the build broke, the meeting was cancelled and even discuss technical issues. In that sense all team members were in the same page about project’s event.

Conclusion

I am pretty sure that we as team learned a bunch of other lessons. However, those were the ones that we highlighted in our retrospective. I hope that this list will be useful to you someday. If you have any other topic to add in that list, please fell free to do it as a comment.

Cheers,
Fernando

LINQ for JavaScript

LINQ is one of my favorite features in the C# language so far. There is no such a thing as filtering, ordering and grouping data using an easy and fluent API. In the last weeks I found myself writing lot of JavaScript code. More than just ajax calls, DOM manipulation and fancy animations, my team and I maintain a huge js model which includes entities, controllers and repositories. Because of that, many of the operations that we regularly implement in server-side are implemented in client-side as well such as finding, sorting and grouping elements in data collections. On the server we use LINQ to perform such operations, but how to implement them in JavaScript?

Luckly, there is a JavaScript library that provides all LINQ operations with a very similar sintaxe. Just as C#, LINQ for JavaScript operates under IEnumerable objects. Therefore, the first thing to do is create an IEnumarable object with the FROM function. To illustrate the library usage, I will start with an array of people. The code below includes the Person object definition and the creation of an enumerable of people from an array. Very straightforward.

Since the enumerable is created, we are free to perform all LINQ operation on it. Let’s start with a simple search: I want to find the person whose name is ‘Fernando’. The code below illustrates that. Notice the sintaxt used by the library. While in C# we write lambda expressions (x => x.Name), here we use the symbol $.

Next, we filter the collection with the people with age greater then 25 and select their names.

Next, we order the collection by age and select their names.

Finally, we group the collection by sex. In addition, each group is ordered by name.

As you can see, there is no secret. The sintaxe is really really similiar. I could show others the operations here, but they are all avaible in the library’s documentation. So, if you got interested by LINQ for JavaScript, take a look in the project page on codeplex. It helped me a lot and hope it can bu useful for you as well.

See you,
Fernando

JavaScript Immetiate Object Initialization

Hello everyone,

During this weekend I watched a couple of sessions in channel 9 recorded during the MIX 11 event. I really liked Elijah Manor’s talk whose title is Good JavaScript Habits for C# Developers. In addition to simple tips on JavaScript programming, such as False-y Values and Comparison Operators, Elijah showed some techniques that not every JavaScript programmer is aware of. One of them is using Immediate Functions for creating objects in modularized applications. Today, I will share how to do that.

First of all, let’s understand what the term Immediate Function refers to. It is a sintax that enables you to execute a function as soon as it is defined. This pattern is used to wrap an amount of work without leaving any global variables behind. Here is an example:

example1.js

Using this pattern we are able to instantiate JavaScript objects without polluting global space with temporary variables. Moreover, the pattern supports the creation of modularized apps by extending existing objects on demand. Let me illustrate it with an example. Imagine we are creating a bookstore. For now, we just need add, remove and getAll operations. Here is the first version of the bookstore object:

bookstore.js

Notice the bookstore parameter in line 1. We use this variable thoughout the Immediate Function to append new operations to it. In line 25 we pass a new object to the bookstore and keep it in a global variable window.bookstore, which enables us to access bookstore anytime in our application. In addition, after the function is executed the variable books is discarted maintaining the global space clear.

Now, imagine later on the project we need additional operations such as sortByTitle and filterByAuhtor. This can be easily done by creating a new file and appending new operations to the bookstore object, such as the example below:

bookstore-extensions.js

This piece of code is similiar to the last one, except by the parameter passed to the Immediate Function in line 11, which uses the existing bookstore instance. That way, new operations are added to the bookstore object and all temporary variables are discarted after the function is executed. Later, we can combine all the JavaScript files and deploy it as a single bookstore.

This pattern is also useful to avoid variable overlapping between different libraries. A very common example is the dollar sign ($) which is used by different existing JavaScript libraries. To make sure the sign refes to jQuery library, for example, we make it a parameter in the Immediate Function and pass the jQuery instance in the initizalization. See the example below:

bookstore-ajax.js

That’s it guys.
For more tips on JavaScript programming, take a look at Elijah Manor’s talk, there are lots of others stuff there. Also, if you are looking for a good JavaScript reference, I strongly recommend the JavaScript Patterns book by Stoyan Stefanov which includes lots of samples of how to create JavaScript code for large ans scalable applications.

See you,
Fernando

Binding JS array to C# list in ASP.NET MVC

Hey guys,

This will be a very short post about a problem that I faced this week while writing some JavaScript code.

At first sight, the acticity should be a very simple one: send an array of data from client to server via AJAX. My pair and I tried a simple POST request similiar to the code below:

Unfortunately, the parameter is always null in the server.

After some time wasted reading the code looking for erros, we decided to verify the post data sent by the browser. That is what Firebug showed to us:

We then realized that the serialization of the array is wrong. But why? Let’s take a look at jQuery’s documentation, more specifically about jquery.Ajax() data setting.

data: Data to be sent to the server. It is converted to a query string, if not already a string. It’s appended to the url for GET-requests. See processData option to prevent this automatic processing. Object must be Key/Value pairs. If value is an Array, jQuery serializes multiple values with same key based on the value of the traditional setting.

Traditional option regards the parameter serialization style. That is a clue. Going a little deeper in our search we found this:

“As of jQuery 1.4, the $.param() method serializes deep objects recursively to accommodate modern scripting languages and frameworks such as PHP and Ruby on Rails. You can disable this functionality globally by setting jQuery.ajaxSettings.traditional = true;.”

Now, we know the reason. Due to the way other web frameworks handle JSON parameters, jQuery serialization changed the way arrays are serialized. Therefore, we as .NET developers need to use the traditional = true setting.

And Boommmm… it works perfectly. Let’s take a look at the request’s paremeters.

That is what ASP.NET MVC needs

That’s it guys.

See you,
Fernando

Extending ASP.NET MVC Model Binding Mechanism

Hey guys,

In the last posts I’ve talked about the use of Moq framework when writing ASP.NET MVC unit tests. Today, I will be talking about ASP.NET MVC Model Binding mechanism and how to extend it to make your life easier when writing testable web applications. [I have written a similar post on SharpShooters blog, but it is in portuguese]

Basically, Model Binding is the mechanism that ASP.NET MVC framework uses for mapping HTTP Request variables into Actions’ parameters. In other words, every time you submit a form to a Controller, the Model Binding mechanism passes through all variables submitted trying to match them with Actions’ parameters using a very specific standard. The standard is “parameterName.PropertyName”. To illustrate how things happen, let’s use a small example. Imagine that we want to implement a blog platform which allows the creation of posts with title and content. Model, View and Controller for creating it are illustrated below:

Post.cs

Create.aspx

PostController.cs

As you already know, the Create Action receives newPost parameter with all the information submitted in the form. However, if the ASP.NET MVC Team hadn’t implemented Model Binding, we would need to type something like this:

Create.aspx

PostController.cs

In short, Model Binding is responsible for binding values passed in HTTP Requests to Actions’ parameters.

Now that we already know what Model Binding is about, we can extend it to bind any parameter we want. Imagine that we want to add a User to a Post. Class and Controller would have to change to something similar to this:

Post.cs

Controller.cs

Although this code works, it is very difficult to test since I need to mock User.Identity.Name. I can mock it using a technique presented by Michael Feathers in his excellent book Working Effectively With Legacy Code which basically involves two steps:

1. Extract the piece of code in a protected virtual function;

2. Create a mock which inherit from PostController and overrides the protected function by returning a specific username;

PostController class would result in the code below.

PostController.cs

And here is the test for the controller class.

MockPostController.cs

MockControllerTest.cs

Now, the test pass

However, every time that a User is referenced by other class, this approach is needed. Imagine we have another entity, for example Comments, which has a reference to User as well. The same solution would be used to test CommentsController class. It would be great if we could pass the User as a parameter in PostController.Create action, wouldn’t be? Something like this:

PostController.cs

To do that, we need to extend ModelBinding mechanism so it can bind the value of a User type. This is really simple. All we have to do is implement IModelBinder interface and register it in the Global.asax. The code below shows the extension.

UserModelBinder.cs

Global.asax

Everything is now configured and we don’t need to worry about retrieving the logged user anymore.

In adittion to that, it’s much more easy to test PostController class. We can create a new user and pass it as parameter, as shown below.

PostControllerTest.cs

To conclude, Model Binding is a very importante feature of ASP.NET MVC framework, saving us a lot of work. It is also extensible and allows us to create our own bindings and testable applications.

Hope it’s usefull for you someday

See you,
Fernando

TDD in ASP.NET MVC Applications with Moq Framework

Last post, I introduced the use of Moq framework to code unit tests in .NET applications. Today, I’ll use Moq to test my ASP.NET MVC controllers.

One of the key benefits of ASP.NET MVC is the facility to set up automated tests and use methodologies such as TDD (Test-Driven Development). Using the Red-Green-Refactor mantra, we can create testable ASP.NET MVC controllers and benefit from the use of TDD. So, let’s start.

In this post, we will develop part of a sport store web site responsible for managing products through CRUD operations. First of all, I will create a new ASP.NET MVC Empty Application and a Test Project in Visual Studio 2010. Next, I will create ProductController.cs, responsible for handling requests from all CRUD operations that we are going to provide. Right now, our controller looks like the code below.

The first thing we nedd is a List view where adminitrators can see all existing products, manage products from the list and add new products to the application. For now, we know that we need to pass all products to the view. It is time to write our first test case which demonstrates that behaviour. After some work, here is my test for List operation.

Notice that after writing my test, two new classes appeared: Product and IProductRepository. In addition to that, we added a dependency between ProductController and IProductRepository. What we are doing here is designing our code while writing tests, which means that we are creating testable code. In the test above, we are saying that FindAll method will Returns the list we’ve created with four products and verifying the those products where passed to the view. That simple!

Now that we have a test, let’s run it to see if it pass. Unfortunately, it failed because we did not wrote the implementation for it yet. That is how our controller looks like until now.

After coding the right List method behavior, shown below, our test pass successfully. Notice that we didn’t write any data access layer code for Product yet. By using the Moq framework, we could fake repository’s behaviour and forget about those boring database scripts and mapping files.

The next step is create the Save operation. Let’s use the same approach, starting with a test case. The test implementation is shown below.

The test above says that when saving a product, the method SaveOrUpdate from IProductRepository is called once and the user is redirected to List action. Again, this test fail because we haven’t implemented that behaviour in ProductController. After writing the code below, our test pass with success.

We are going just fine. However, our tests are starting to present some code duplications. Now, it’s clear that we need to refactor it (Red-Green-Refactor, rememeber?). After some work on it, our test class look like that:

Until now, we tested just the base case. What happens if a user tries to save an invalid model? We have to deal with that in our controller. Let’s write a new test case for it.

This time, we verify that SaveOrUpdate method is never called and the user continues in the same view. Again, the test fail. Here is the new Save implementation that makes the above test pass.

To complete our task of creating a full CRUD for our controller, we still need to implement Delete and Update. However, I will leave it for you as homework

In short, I have shown how to use TDD while creating an ASP.NET MVC application and Moq Framework to isolate presentation layer tests from any other layer of the system.