Some Numbers

[00:04] (external link, opens in new tab) About numbers. Here is an example of of whether it's actually worth it. And I did this study. It was an experiment (external link, opens in new tab) that I did in a big company a few good years ago. And we had two teams that were taking the same project and adapting it to different customers. So it was more like adaptation. It was a billing system. And in this case, we had two teams that were adapting pretty much the same features to do two different customers. And I was teaching there was doing consulting and the company agreed that we do an experiment and we have one team that went through the TDD (external link, opens in new tab)

[00:38] (external link, opens in new tab) And we'll compare it to the results of the other team that didn't do TDD. So they did so we had the two teams and the two teams did the two features in parallel. And the team that did test driven development is on the right. And what happened is that the team that did TDD took double the amount of time, 14 days instead of seven to implement the same set of features.

[01:09] (external link, opens in new tab) But then at the end of the iteration, the team took 22 days instead of 19 to release. Not seven days more, but three days more. And obviously they that's because they had more tests, so it was easier for them to debug. They had a testing and fixing cycle. with QA it was and the testing took longer and bug fixing took longer for the other team because they did not have tests. So when they fixed bugs they also broke other code. The difference is that the other team when they broke the other code they saw it and they fixed it right there at the same time.

[01:42] (external link, opens in new tab) So it was easier for them to release the same product three days later. Now the big question is what did they get for the extra three days? So here's what they got. The amount of bugs found in production was 90% less. 90% less bugs. So the customer was happier and the developers were happy because they like the APIs. They like the fact that they have this safety net for the future for refactoring code.

[02:12] (external link, opens in new tab) So everyone was happier and that took three days more. I don't know a single manager that would not agree with that strategy. The problem is that it takes a long time to see that. In this case, it was a mini iteration, mini experiment. Now, that's not necessarily true in big organizations. So, let me be absolutely truthful. I lied. This is not exactly how it worked. This is what actually happened. Between the implementation and testing, they had integration phases.

[02:44] (external link, opens in new tab) Integration phase is when the product has to integrate with all the other modules in the company because it's a huge billing system and usually in that company integration was called hell week because everything sucked in that week. Nothing was working. Everyone was you know just working their asses off into the night debugging things. And surprisingly, the amount of bugs found was less, but but also integration took less time for the team with test-driven development.

[03:23] (external link, opens in new tab) And what we found was that when there were integration problems for the team that did TDD, the unit tests (external link, opens in new tab) actually helped them to know where the the problem was not. So imagine that you have a problem integrating with something and you don't know where the bug is. So you have to start debugging all the way through the entire system stack to see whether the bug is in your logic or in integration or the configuration. But for these guys because they had some unit test for some for for the logic

[03:57] (external link, opens in new tab) And when they found a bug in integration and they saw that they had a unit test for that logic and they said oh the unit test for that is passing then the bug must be in the integration between the modules. It solved the amount of time it took them to debug where the problem was which was much faster was much much less time. So the integration phase took much less. So in reality what ended up is that they released in 24 days instead of 26 days for the other team. So that means that the team that did TDD took double the amount of time to implement because when you write tests you do write more code sometimes double

[04:36] (external link, opens in new tab) But they actually released sooner. So the investment was repaid. This was a team with about 3 months of experience. So with more experience, it would have taken them even less to write the test. So it wouldn't have been double. It would have been less. For experienced teams, it could be 50% more. But the point was that the release was actually faster.

[05:06] (external link, opens in new tab) And they still got all these things, the 90% less bugs in production. So now the big question is why isn't everyone doing this? Because not everyone can do an experiment of the 24 days. Most companies can only experiment with a few months in between at least and they're afraid of investing all that money and not seeing the ret the return. So, it's hard to see the bottom line when all you see is the top line, which is double.