--- title: "Some Numbers" url: "https://courses.osherove.com/courses/the-art-of-unit-testing-and-tdd-in-java/some-numbers.md" canonical: "https://courses.osherove.com/courses/the-art-of-unit-testing-and-tdd-in-java/some-numbers/" tags: [] author: "Roy Osherove" description: "Lesson 7 of 21 in The Art of Unit Testing and TDD in Java (Day 1), 5:37. Free video by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Some Numbers 1. [Courses](https://courses.osherove.com/#courses) 2. [The Art of Unit Testing and TDD in Java](https://courses.osherove.com/courses/the-art-of-unit-testing-and-tdd-in-java/) 3. Day 1 Lesson 7 of 21 Duration: 5:37 [Watch on YouTube](https://www.youtube.com/watch?v=-uKdh5b8J5I) [How much time to you lose](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-11 "How much time will unit testing add to the current process? (The Art of Unit Testing, 3rd ed., ch. 11)") -- and save-- by using [TDD](https://livebook.manning.com/book/the-art-of-unit-testing-second-edition/chapter-1#ch01lev1sec6 "Test-driven development (The Art of Unit Testing, 2nd ed., ch. 1)")? here are some numbers from my own experience. ## In this lesson Roy Osherove describes [an experiment](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-11 "Is there proof that unit testing helps? (The Art of Unit Testing, 3rd ed., ch. 11)") at a big company where two teams adapted the same features of a billing system, one with [TDD](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-1#sigil_toc_id_27 "Test-driven development (The Art of Unit Testing, 3rd ed., ch. 1)") and one without. The TDD team took 14 days instead of seven to implement the features but released in 24 days instead of 26, partly because its [unit tests](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-1#sigil_toc_id_17 "Defining unit testing, step by step (The Art of Unit Testing, 3rd ed., ch. 1)") showed where integration bugs were not, and it had 90% fewer bugs in production. He says many companies don't try this because they fear investing without seeing the return for months. ## Transcript 894 words [\[00:04\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=4s) About numbers. Here is an example of of whether it's actually worth it. And I did this study. It was [an experiment](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-11#sigil_toc_id_199 "Experiments as door openers (The Art of Unit Testing, 3rd ed., ch. 11)") that I did in a big company a few good years ago. And we had two teams that were taking the same project and adapting it to different customers. So it was more like adaptation. It was a billing system. And in this case, we had two teams that were adapting pretty much the same features to do two different customers. And I was teaching there was doing consulting and the company agreed that we do an experiment and we have one team that went through the [TDD](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-1#sigil_toc_id_29 "Three core skills needed for successful TDD (The Art of Unit Testing, 3rd ed., ch. 1)") [\[00:38\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=38s) And we'll compare it to the results of the other team that didn't do TDD. So they did so we had the two teams and the two teams did the two features in parallel. And the team that did test driven development is on the right. And what happened is that the team that did TDD took double the amount of time, 14 days instead of seven to implement the same set of features. [\[01:09\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=69s) But then at the end of the iteration, the team took 22 days instead of 19 to release. Not seven days more, but three days more. And obviously they that's because they had more tests, so it was easier for them to debug. They had a testing and fixing cycle. with QA it was and the testing took longer and bug fixing took longer for the other team because they did not have tests. So when they fixed bugs they also broke other code. The difference is that the other team when they broke the other code they saw it and they fixed it right there at the same time. [\[01:42\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=102s) So it was easier for them to release the same product three days later. Now the big question is what did they get for the extra three days? So here's what they got. The amount of bugs found in production was 90% less. 90% less bugs. So the customer was happier and the developers were happy because they like the APIs. They like the fact that they have this safety net for the future for refactoring code. [\[02:12\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=132s) So everyone was happier and that took three days more. I don't know a single manager that would not agree with that strategy. The problem is that it takes a long time to see that. In this case, it was a mini iteration, mini experiment. Now, that's not necessarily true in big organizations. So, let me be absolutely truthful. I lied. This is not exactly how it worked. This is what actually happened. Between the implementation and testing, they had integration phases. [\[02:44\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=164s) Integration phase is when the product has to integrate with all the other modules in the company because it's a huge billing system and usually in that company integration was called hell week because everything sucked in that week. Nothing was working. Everyone was you know just working their asses off into the night debugging things. And surprisingly, the amount of bugs found was less, but but also integration took less time for the team with test-driven development. [\[03:23\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=203s) And what we found was that when there were integration problems for the team that did TDD, the [unit tests](https://livebook.manning.com/book/the-art-of-unit-testing-second-edition/chapter-1#ch01lev1sec1 "Defining unit testing, step by step (The Art of Unit Testing, 2nd ed., ch. 1)") actually helped them to know where the the problem was not. So imagine that you have a problem integrating with something and you don't know where the bug is. So you have to start debugging all the way through the entire system stack to see whether the bug is in your logic or in integration or the configuration. But for these guys because they had some unit test for some for for the logic [\[03:57\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=237s) And when they found a bug in integration and they saw that they had a unit test for that logic and they said oh the unit test for that is passing then the bug must be in the integration between the modules. It solved the amount of time it took them to debug where the problem was which was much faster was much much less time. So the integration phase took much less. So in reality what ended up is that they released in 24 days instead of 26 days for the other team. So that means that the team that did TDD took double the amount of time to implement because when you write tests you do write more code sometimes double [\[04:36\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=276s) But they actually released sooner. So the investment was repaid. This was a team with about 3 months of experience. So with more experience, it would have taken them even less to write the test. So it wouldn't have been double. It would have been less. For experienced teams, it could be 50% more. But the point was that the release was actually faster. [\[05:06\]](https://www.youtube.com/watch?v=-uKdh5b8J5I&t=306s) And they still got all these things, the 90% less bugs in production. So now the big question is why isn't everyone doing this? Because not everyone can do an experiment of the 24 days. Most companies can only experiment with a few months in between at least and they're afraid of investing all that money and not seeing the ret the return. So, it's hard to see the bottom line when all you see is the top line, which is double. ## Related - [How much time will unit testing add to the current process?](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-11) · The Art of Unit Testing, 3rd ed., ch. 11 The book's answer to how much time unit testing adds to the process, the trade-off the experiment measures. - [Experiments as door openers](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-11#sigil_toc_id_199) · The Art of Unit Testing, 3rd ed., ch. 11 Section on experiments as door openers, like the two-team TDD experiment described in this lesson. - [How Test Driven Development proved itself in a big way!](https://osherove.com/blog/2004/9/9/how-test-driven-development-proved-itself-in-a-big-way.html) · osherove.com Roy's account of TDD proving itself on a real project, another field report like this lesson's numbers. ### Also linked in this lesson #### The Art of Unit Testing, 3rd ed. - [Test-driven development](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-1#sigil_toc_id_27), ch. 1 - [Defining unit testing, step by step](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-1#sigil_toc_id_17), ch. 1 - [Three core skills needed for successful TDD](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-1#sigil_toc_id_29), ch. 1 #### The Art of Unit Testing, 2nd ed. - [Test-driven development](https://livebook.manning.com/book/the-art-of-unit-testing-second-edition/chapter-1#ch01lev1sec6), ch. 1 - [Defining unit testing, step by step](https://livebook.manning.com/book/the-art-of-unit-testing-second-edition/chapter-1#ch01lev1sec1), ch. 1 [Previous lesson Common TDD Questions and Answers](https://courses.osherove.com/courses/the-art-of-unit-testing-and-tdd-in-java/common-tdd-questions-and-answers/) [Next lesson Introduction to Mocks and Stubs](https://courses.osherove.com/courses/the-art-of-unit-testing-and-tdd-in-java/introduction-to-mocks-and-stubs/) --- ## About the author **Roy Osherove**: Senior Solutions Architect at AWS working on AI for production and AI-driven development; author of The Art of Unit Testing and Elastic Leadership, with about 20 years of consulting and training on unit testing, TDD, team leadership and continuous delivery. ### Sites - [RobotPaper](https://robotpaper.ai/): This blog: AI patterns and anti-patterns, AI and coding agents, and engineering leadership. - [osherove.com](https://osherove.com): Roy's personal site. Author of The Art of Unit Testing and Elastic Leadership; consulting and training on unit testing, TDD, team leadership and continuous delivery, plus about 20 years of blog posts. - [The Art of Unit Testing](https://www.artofunittesting.com): The Art of Unit Testing, 3rd Edition (Manning): readable, maintainable, trustworthy unit tests, plus TDD workshops in C#, JavaScript, Java and Kotlin. - [5whys.com](https://www.5whys.com): Roy's writing on software team leadership: self-organizing teams, surviving chaos, learning mode and the Elastic Leadership model. - [Pipeline Driven](https://pipelinedriven.org): Continuous delivery and DevOps practices: build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps. - [Elastic Leadership](https://www.elasticleadership.com): Elastic Leadership, Roy's framework for software team leaders (survival, learning and self-organizing modes). ### Social - X: https://x.com/royosherove - LinkedIn: https://www.linkedin.com/in/osherove/ ### Contact - Contact form: https://forms.gle/jQSx5KoXvrVA1JCK7 ### License Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page). License: https://creativecommons.org/licenses/by/4.0/