--- title: "Shipping Skeleton" url: "https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/shipping-skeleton.md" canonical: "https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/shipping-skeleton/" tags: [] author: "Roy Osherove" description: "Lesson 3 of 17 in Beautiful Builds and Continuous Delivery Patterns (Productivity Patterns), 8:26. Free video by Roy Osherove." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page)." license: "https://creativecommons.org/licenses/by/4.0/" --- # Shipping Skeleton 1. [Courses](https://courses.osherove.com/#courses) 2. [Beautiful Builds and Continuous Delivery Patterns](https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/) 3. Productivity Patterns Lesson 3 of 17 Duration: 8:26 [Watch on YouTube](https://www.youtube.com/watch?v=X4Pkqu2bYoc) ## In this lesson Roy Osherove introduces the [shipping skeleton](https://pipelinedriven.org/article/2014/3/2/pattern-shipping-skeleton "Pattern: Shipping Skeleton (pipelinedriven.org)"), a walking skeleton for the [delivery pipeline](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-10 "Managing delivery pipelines (The Art of Unit Testing, 3rd ed., ch. 10)"): a tiny application that only says hello world, plus an automated shipping system that takes it as close to production as possible. Both then evolve together in small steps, and his advice is to [start a project by shipping it](https://pipelinedriven.org/article/2014/3/2/start-products-with-an-empty-build-and-deploy-pipeline "Start Products with an Empty Build and Deploy Pipeline (pipelinedriven.org)"), so every change goes all the way to production. He lists the machines needed, a build machine with agents and test, staging and production machines, possibly in the cloud on [Amazon EC2](https://osherove.com/blog/2010/5/31/running-teamcity-from-amazon-ec2-cloud-based-scalable-build.html "Running TeamCity from Amazon EC2 - Cloud based scalable build and continuous Integration (osherove.com)"), and shows a TeamCity project with four numbered build configurations, CI and deploys to test, staging and production, each triggered by the previous one. ## Transcript 1,392 words [\[00:04\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=4s) Okay, welcome back. My name is Roy Osherove. We're going to talk about where do you start with [continuous delivery](https://pipelinedriven.org/article/continuous-delivery-values "Continuous Delivery Values (pipelinedriven.org)"). Now, one common problem that a lot of teams have is that the build is unruly, very hard to maintain or even get started with. So, I'm going to introduce a pattern which I call the [shipping skeleton](https://pipelinedriven.org/article/2014/3/2/pattern-shipping-skeleton "Pattern: Shipping Skeleton (pipelinedriven.org)"). Now the shipping skeleton is a pattern that's built on a different pattern. And that different pattern is called a walking skeleton. Now the walking skeleton was created by Alistair Alistair Cockburn and [\[00:40\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=40s) It's it's a pretty old pattern. Basically what it says is that this is a walking skeleton is a tiny implementation of the system that performs small end-to-end function. And then he talks a little bit the fact that it doesn't have to have the full architecture but it can connect the main architectural components and then evolve. So I took the same idea and I kind of said what if we had a walking skeleton for the [delivery pipeline](https://leanpub.com/pipelinedriven "Pipelines (pipelinedriven.org)"). And I call that a shipping skeleton. So a shipping skeleton is a combination of two elements. [\[01:13\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=73s) You have the skeleton of the application but this time I'm not talking about architecture at all. It's just a tiny implementation of the system. As far as we're concerned, it can just say hello world. It does nothing but say show hello world to the user. And then you have a shipping system for the skeleton. And the shipping system is automated and comes as close as possible to the production servers or where the customer can see it and use it. And it should require no human contact as much as possible. [\[01:46\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=106s) So you shouldn't need to merge things to be able to use the shipping system for example. So once you have the shipping system and the skeleton in place they should you should be able to evolve both in parallel. So they can be very very simplistic things but once you get things going into that template you can just start evolving things as time goes by. And to me it it solves a lot of the maintainability problems that we have with the build systems. And I kind of think of it like a hotel room. [\[02:18\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=138s) If you stayed in a hotel for two weeks, it would be really bad if they had cleaned your room just once a week, right? After a week, your room would just look horrible. At least my room just look horrible. But because they clean your room every day, it's just a little messy. So it's very very easy to clean. So it's a very incremental way of evolving and growing the build system as well as the code. So as the code needs new things, the build system gets new fe features. [\[02:49\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=169s) So by default, it might not need all the complicated stuff, but when the complicated stuff is needed, you add it as time goes by. So it's kind of like one of those, I don't know if they're called bonsai trees or whatever that is, where you trim very little trees and the whole point is to grow very very beautifully into specific shapes. That's that. So here's an example. You can just have a shipping skeleton code which would just say console write line hello world and then you would then take that and run it through an [automated build](https://livebook.manning.com/book/the-art-of-unit-testing-second-edition/chapter-7#ch07lev1sec1 "Automated builds running automated tests (The Art of Unit Testing, 2nd ed., ch. 7)") system continuous delivery system that compiles it, tests it and deploys it and it could deploy it to test servers, production servers, staging servers, hopefully all of the all of the above. [\[03:39\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=219s) Now important tactic if the best way to you the best way and the best time to use a shipping skeleton is at the beginning of your project. So my best advice to you is to [start a project by shipping it](https://pipelinedriven.org/article/2014/3/2/start-products-with-an-empty-build-and-deploy-pipeline "Start Products with an Empty Build and Deploy Pipeline (pipelinedriven.org)"). Okay. The first few weeks you end up with a shipping skeleton where you can actually show a demo that says hello world on the production server and then maybe after that all it takes is just reversing a DNS entry so that production server becomes live. [\[04:09\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=249s) But you ship to production and you start by shipping and then every little change you make goes all the way into production. And this is very very nice because you start getting feedback and you no no longer live in a bubble as we talked about in previous videos when we talked about feedback from the customer. Now in terms of hardware you might just need a build machine. I personally like to use Team CD and agent machines. And agents are the things that actually listen to the continuous delivery or continuous integration server like TeamCity and execute the scripts that it tells them to to run. [\[04:49\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=289s) So you would need a build machine or something that runs the CI server something that is a test m machine a staging machine and a production machine. And of course this can all be in the cloud and I've perfectly I've used [Amazon EC2](https://osherove.com/blog/2010/5/31/running-teamcity-from-amazon-ec2-cloud-based-scalable-build.html "Running TeamCity from Amazon EC2 - Cloud based scalable build and continuous Integration (osherove.com)") and it's very very usable. I've heard also good things about Azure, but I've personally never used it. So, I don't have a good opinion or a bad opinion of it. Once I do, I'm happy to to to add any recommendations. [\[05:19\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=319s) Now it's important to u to also understand that staging looks exactly like production. So that the same problems that you have to deploy to production, you should have the same problems to deploy to staging. So staging is actually used to test your deployment scripts. In terms of build configurations, build configurations are the steps or the the the stages that your code go through goes through be as it goes closer and closer to the customer. [\[05:54\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=354s) So you might have a CI continuous integration build and that is the quick feedback that just runs the unit test and then you might have a slow build. Some people call it nightly build, but I like to run slow builds even during the day. You might have a build or can build configuration that deploys to test servers, to staging servers, and production. And you'll notice that the last three are all about deploying, but the thing that changes is the environment. So all these things are part of the CI server as configurations. [\[06:27\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=387s) And they run scripts which get parameters from the CI system and they basically just say okay now you deploy to this and you deploy to that. But that's the basic build system that I would have as maybe I wouldn't even have staging where maybe I just deploy to test and then directly to production until such point where I can no longer just deploy to production because people are using it and then I would have staging servers. Here's an example from TeamCity where I have the most basic this is maybe after 3 weeks of having web application and continuous delivery on that web application and what you see is basically a single project and under that project we have four build configurations [\[07:16\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=436s) And the names also have numbers in them I put the numbers as in the name titles so that we have sorting so that they're sorted by flow and and and when something happens and each one of them is happening one after the other. So we have CI and and that only runs like 13 tests right now. And then we automatically deploy to test and we automatically deploy to staging and to production. And we can set things up so that each one gets triggered by the previous one finishing so that when we when we check in and CI gets run [\[07:53\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=473s) We end up with deployment to production maybe a few minutes later. And that's one of the nicest things that continuous delivery can happen and you your code will actually end up in production if you want it to as quickly as possible. If you don't want it to, you can put it in a branch or in a [feature flag](https://pipelinedriven.org/feature-toggle-frameworks-list "Feature Toggles (pipelinedriven.org)") which we will talk about in a separate video. If you if you'd like more information about the this and other things, just go to beautifulbuilds.com or just contact me on Twitter at Roy. [\[08:24\]](https://www.youtube.com/watch?v=X4Pkqu2bYoc&t=504s) Thank you. Cheers. ## Related - [Pattern: Shipping Skeleton](https://pipelinedriven.org/article/2014/3/2/pattern-shipping-skeleton) · pipelinedriven.org Roy's written Shipping Skeleton pattern: a hello world with the automated build and deploy cycle working before any features. - [Start Products with an Empty Build and Deploy Pipeline](https://pipelinedriven.org/article/2014/3/2/start-products-with-an-empty-build-and-deploy-pipeline) · pipelinedriven.org Step-by-step recipe for an empty build and deploy pipeline on day one, with CI and deploy configurations in TeamCity or Jenkins. - [Running TeamCity from Amazon EC2 - Cloud based scalable build and continuous Integration](https://osherove.com/blog/2010/5/31/running-teamcity-from-amazon-ec2-cloud-based-scalable-build.html) · osherove.com Roy's walkthrough of running a TeamCity server and agents on Amazon EC2, the cloud option mentioned for the build machines. ### Also linked in this lesson #### The Art of Unit Testing, 3rd ed. - [Managing delivery pipelines](https://livebook.manning.com/book/the-art-of-unit-testing-third-edition/chapter-10), ch. 10 #### The Art of Unit Testing, 2nd ed. - [Automated builds running automated tests](https://livebook.manning.com/book/the-art-of-unit-testing-second-edition/chapter-7#ch07lev1sec1), ch. 7 #### pipelinedriven.org - [Continuous Delivery Values](https://pipelinedriven.org/article/continuous-delivery-values) - [Pipelines](https://leanpub.com/pipelinedriven) - [Feature Toggles](https://pipelinedriven.org/feature-toggle-frameworks-list) [Previous lesson What's the point of Continuous Delivery?](https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/what-s-the-point-of-continuous-delivery/) [Next lesson Confidence Waves](https://courses.osherove.com/courses/beautiful-builds-and-continuous-delivery-patterns/confidence-waves/) --- ## About the author **Roy Osherove**: Senior Solutions Architect at AWS working on AI for production and AI-driven development; author of The Art of Unit Testing and Elastic Leadership, with about 20 years of consulting and training on unit testing, TDD, team leadership and continuous delivery. ### Sites - [RobotPaper](https://robotpaper.ai/): This blog: AI patterns and anti-patterns, AI and coding agents, and engineering leadership. - [osherove.com](https://osherove.com): Roy's personal site. Author of The Art of Unit Testing and Elastic Leadership; consulting and training on unit testing, TDD, team leadership and continuous delivery, plus about 20 years of blog posts. - [The Art of Unit Testing](https://www.artofunittesting.com): The Art of Unit Testing, 3rd Edition (Manning): readable, maintainable, trustworthy unit tests, plus TDD workshops in C#, JavaScript, Java and Kotlin. - [5whys.com](https://www.5whys.com): Roy's writing on software team leadership: self-organizing teams, surviving chaos, learning mode and the Elastic Leadership model. - [Pipeline Driven](https://pipelinedriven.org): Continuous delivery and DevOps practices: build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps. - [Elastic Leadership](https://www.elasticleadership.com): Elastic Leadership, Roy's framework for software team leaders (survival, learning and self-organizing modes). ### Social - X: https://x.com/royosherove - LinkedIn: https://www.linkedin.com/in/osherove/ ### Contact - Contact form: https://forms.gle/jQSx5KoXvrVA1JCK7 ### License Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original page). License: https://creativecommons.org/licenses/by/4.0/