--- title: "Tipping Point" url: "https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/tipping-point.md" canonical: "https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/tipping-point/" tags: [] author: "Roy Osherove" description: "Lesson 7 of 17 in Beautiful Builds and Continuous Delivery Patterns (Team Collaboration Patterns for CI), 9:34. Free video by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Tipping Point 1. [Courses](https://courses.osherove.com/#courses) 2. [Beautiful Builds and Continuous Delivery Patterns](https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/) 3. Team Collaboration Patterns for CI Lesson 7 of 17 Duration: 9:34 [Watch on YouTube](https://www.youtube.com/watch?v=fy9kPT0qENk) ## In this lesson When an application is made of components developed in parallel, often by separate teams, Roy Osherove gives each component its own small pipeline of CI, unit test and integration test builds. Because pipelines differ between components, he adds a convention he calls the tipping point: a last build configuration named tip, which has a snapshot dependency on the steps before it and publishes the binaries that have gone through the whole process. The system integration and deployment chain then takes every component from its tip through snapshot dependencies, so it always gets the latest deployable versions, the component builds can run in parallel on several build agents, and a component can also be pinned to a specific version. ## Transcript 1,472 words [\[00:03\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=3s) Hi everyone, Roy Osherove again and this time we're going to talk about [continuous delivery](https://pipelinedriven.org/article/continuous-delivery-values "Continuous Delivery Values (pipelinedriven.org)") with multiple teams or components working in parallel and what do you do when you have an application that you want to deliver that is made up of all of these components. Now in the previous videos I talked about versioning problems between components and their dependencies. For example, a component might depend on other components and then we would have snapshot dependencies between components to check to check their versions to make sure that the versions are all the same. [\[00:39\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=39s) We're going to use kind of the same thing, but this time we're going to take a bigger view. For example, in this case, let's imagine that because we have all these components and each component is actually developed by a separate team. In this case, what we can do and we can also do this if if it's all done by a single team, but if all these components are developed separately in parallel and they have their own schedules of when they get released and when the and the team fixes bugs on a different in different times or they work in different time zones, whatever. [\[01:17\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=77s) When we work in these kind of separate areas, it's very important that each component actually has its own little pipeline. So instead of just having a single CI unit test and integration steps in our build, we can actually have a small parallel universe for each component or team that is working and for each component there is actually a set of steps that it's going through. So each component will have its own CI bill and its own unit testing and its own integration test [\[01:50\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=110s) So that each component can be developed and maybe actually released as a separate small product and then later we might have a bigger build that does integration etc with all these components. But then there is a different problem when we have all of these components. The question is where do you get the artifacts? If I wanted to get the artifacts from the components and into my build process where I want to deploy the application, where do I get the artifacts from? [\[02:24\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=144s) Should I get them? I can get them directly from the CI. I can get them after they have passed integration testing. And in here based on this diagram it kind of makes sense that we actually get artifacts and publish artifacts from the integration test configuration for each component. But that doesn't necessarily ring true because from my experience different components can have actually different stages and different processes. [\[02:55\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=175s) For example, some components might have integration and they might even even have acceptance tests or they might have various other stages like manual not manual but automated installer tests or various configuration management generation, code generation, things like that. Publication to various locations. So actually the the pipelines might have for some components more than three steps and for some of them less than three steps. [\[03:28\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=208s) Some components might not have unit tests. Some of them might not have integration tests. So then when we want to have when we want to know what what to publish from each component there is no one single way of knowing which which location in the flow should we get the binaries from. It kind of makes sense that it's the last one but sometimes it's not the last one. So it's important to create something that [\[03:58\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=238s) A convention where we always know where to get the latest version of the binaries after having been tested and gone through the entire important flow so we can deploy them as part of the application. And I call this pattern the tipping point because we're going to actually create an a tip for each component that contains the binary. So instead of a component having just CI unit test integration now each component is going to have a convention it's going to have one more [\[04:30\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=270s) Configuration and we call this tip or this just one name you can name it whatever you want. But the idea is that all components have a tip and then we know that tip is the thing that contains the binaries after they've gone through the entire process and we know that they're deployable. they're good to go. So a tip h in terms of dependencies you would have probably would take the artifacts from directly from the CI after compilation. [\[05:04\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=304s) But a tip configuration would also make sure that the the binaries that it publishes are actually the ones that have gone through the entire process. So we might have multiple snapshot dependencies inside a component. A tip will have a snapshot dependency on integration test. So that if I trigger the tip build, it will actually trigger integration test first if the code has changed since the last build. And then there's a snapshot between integration and unit test. [\[05:35\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=335s) So that integration test will actually not run before a unit test has run if the code has changed. And the same for unit test versus CI. That means that if I click and I try to run the tip configuration it might trigger CI unit test and integration tests before tip can actually publish anything. And that's exactly what we want. We want tip to actually represent the best version the best deployable version of those binaries. So now that we have that tip location, [\[06:08\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=368s) Imagine that we have these four components and we and each one of them is developed in parallel and they have their own little pipelines. Now we can see flow and now let's imagine that we have this deployment chain configurations right we have system integration deploying all the components to test and staging and production. Now we can actually have the that convention work for us. So we can have those snapshot dependencies taken directly from tip. So I have a a single convention where if I want to do a system integration test with all the components in their best versions from the latest versions, I just take the things from tip and I actually have a snapshot dependency. [\[06:53\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=413s) Now snapshot dependency is is kind of a choice here. I have a snapshot dependency because I want to make sure that when I test system integration I actually am testing it with the latest deployable version of all components and that means that if one of the components actually fails like component one I won't actually be able to run the system integration now that is either good or bad for you when I was working on one of those systems we had we had taken artifacts but we didn't have snapshot dependency between components. [\[07:29\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=449s) And so what we ended up is with is that we had components in different versions deployed into system integration and then things seems to be okay at the component level but then when they came into the actual integration system integration things just started to fail because of versioning issues where things not were not in the latest version sometimes for and we I explained this in the previous videos. So snapshot dependencies from the component tips is actually a very good way to make sure that we have the latest version and because each one of them does not have a dependency on the others in this case. [\[08:07\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=487s) This can actually run four par parallel build agents that will trigger and run in parallel if we wanted to so that we don't have to wait linearly for each component to build. So only we will only have to wait for the slowest component but not for all of them one after the other. And this way when we get to system integration we have the full versioning support and we get the latest deployable binaries so that when we finally get to 40 in the deploy chain [\[08:44\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=524s) We know that what we have is something that is the latest version of all the components. we can't actually get there before all components have a verifiable runnable tip because we have snapshot dependencies. If we wanted to, we could just pin to a specific version of a component to make sure that we always deploy to system integration with version 101 of the tip of that component. But we can also choose to have the latest version via snapshot dependency on the absolute latest version. [\[09:16\]](https://www.youtube.com/watch?v=fy9kPT0qENk&t=556s) But at least we have a choice now. And we can also again see the flow and make sure that we have fully integrated systems. The system has integrity. And with that, I would like to thank you and I'll see you in the next ## Related - [Managing delivery pipelines](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-10) · The Art of Unit Testing, 3rd ed., ch. 10 The 3rd edition's section on managing delivery pipelines, the per-component pipelines this lesson organizes. - [Using TFS Integrator to manage complex project dependencies](https://osherove.com/blog/2007/1/28/using-tfs-integrator-to-manage-complex-project-dependencies.html) · osherove.com Roy on projects that depend on another team's binaries and may keep building against an older version, like pinning a component. ### Also linked in this lesson #### pipelinedriven.org - [Continuous Delivery Values](https://pipelinedriven.org/article/continuous-delivery-values) [Previous lesson Version by Snapshot](https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/version-by-snapshot/) [Next lesson APi Artifacts](https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/api-artifacts/) --- ## About the author **Roy Osherove**: Senior Solutions Architect at AWS working on AI for production and AI-driven development; author of The Art of Unit Testing and Elastic Leadership, with about 20 years of consulting and training on unit testing, TDD, team leadership and continuous delivery. ### Sites - [RobotPaper](https://robotpaper.ai/): This blog: AI patterns and anti-patterns, AI and coding agents, and engineering leadership. - [osherove.com](https://osherove.com): Roy's personal site. Author of The Art of Unit Testing and Elastic Leadership; consulting and training on unit testing, TDD, team leadership and continuous delivery, plus about 20 years of blog posts. - [The Art of Unit Testing](https://www.artofunittesting.com): The Art of Unit Testing, 3rd Edition (Manning): readable, maintainable, trustworthy unit tests, plus TDD workshops in C#, JavaScript, Java and Kotlin. - [5whys.com](https://www.5whys.com): Roy's writing on software team leadership: self-organizing teams, surviving chaos, learning mode and the Elastic Leadership model. - [Pipeline Driven](https://pipelinedriven.org): Continuous delivery and DevOps practices: build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps. - [Elastic Leadership](https://www.elasticleadership.com): Elastic Leadership, Roy's framework for software team leaders (survival, learning and self-organizing modes). ### Social - X: https://x.com/royosherove - LinkedIn: https://www.linkedin.com/in/osherove/ ### Contact - Contact form: https://forms.gle/jQSx5KoXvrVA1JCK7 ### License Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page). License: https://creativecommons.org/licenses/by/4.0/