Continuous Delivery

[00:02] (external link, opens in new tab) I want to finish up with the the big picture of of the build process (external link, opens in new tab). I'm actually writing a book called Beautiful Build (external link, opens in new tab) right now and this is this is part of that book and I want to talk to you about how I approach the delivery process. So the first thing I want to talk about is the idea of compilation. Can you imagine compiling your code once a week?

[00:34] (external link, opens in new tab) You can write code the whole week, but you only get to compile it once a week. Can you imagine what life would be like? Pretty great, right? Life would be hell on that day, but on the other days, it's just going to be one big party because you you know you're going to fail. So we have compilers to help us continuously to know that the code we write actually compiles actually

[01:05] (external link, opens in new tab) Actually actually fits the compiler that requires it. So we can do that continuously. What about tests? Unit tests (external link, opens in new tab). Can you imagine running tests only once a week? You could do that. A lot of people actually do and that creates a lot of problems. But a lot of people have stopped doing running tests once a week or once every two weeks.

[01:36] (external link, opens in new tab) A lot of them actually run the test continuously as well. So they have the test in a in a continuous integration (external link, opens in new tab) cycle. Things like what are you using? Jenkins. Things like TeamCity and then every time they check in the code the unit tests run and they give a quick very quick feedback and you can also run the test locally on your machine and get quick feedback and that's nice because you know that your code not only compiles but it also matches the reality of the requirements or at least the requirements that you know.

[02:16] (external link, opens in new tab) Now let's look at the next level. What about can you imagine releasing every two months? Of course you can. That's what we do. Can you imagine releasing every two weeks? Some people do that. Can you imagine releasing every day? Not many people do that. What about releasing many times a day? That's what we're talking about. Why do we want to release multiple times a day? continuously because whenever we release we find out new problems that we didn't think about.

[02:50] (external link, opens in new tab) For example, does our code actually match the system where it's supposed to be deployed? How long does it take you to know when you wrote a piece of code that it actually works on the end machine where it's supposed to run? For example, on the actual computer of the user or if it's a website that it runs with the actual browser that the user is using. Not only that, how do you know that the user who's using it is actually using what they want? Even if it works perfectly with the browser, is it what they actually wanted?

[03:21] (external link, opens in new tab) And releases give us the ability to get feedback from the users to know we are on the right track or not. Without frequent releases, we don't have feedback that our code matches the reality of what's actually requested. So in a way, releases are the best form of compilation (external link, opens in new tab) for our code. We have compilers to make sure that it compiles for the computer. We have unit test to make sure it compiles for the requirements we think we have. We have release to make sure that our code can actually be released.

[03:51] (external link, opens in new tab) So release is actually the best test for our code. To deploy as much as possible is to test our code as much as possible against what we think the reality is and what it actually is. If we release every two months, we might be working two months on the wrong thing. Or if we release every two months or we release at the end of the year, we might find out at the end of the year that the requirements from the hardware or the security or anything else like that were that we assumed were true were not and now we have to go back and maybe change some of the product or our estimation or the deployment and everything based on the end result where we're supposed to deploy.

[04:36] (external link, opens in new tab) So deploying is actually a compilation of our code in the best way possible. So what I want us to do is to think about it in the culture of continuous release as continuous compilation. So my recommendations that we're going to talk about here is that we going to have we're going to talk about build automation. And I know that you already have some of it, but I want to talk about the vision of how build automation could work and how to solve common problems when we we do automated builds and how to choose good tools for continuous integration and automated builds.

[05:18] (external link, opens in new tab) First I want to talk about the three levels of things that happen during the build cycle. We have at the lowest level we have build scripts that those are the scripts like Ant or batch files or script files like shell files shell scripts that are the things that actually do all the hard work. On the top level, we have servers that manage builds like TeamCity, like Jenkins.

[05:50] (external link, opens in new tab) Those are the things that are supposed to trigger automated builds whenever we check in or every hour. And on the middle, we have the things that execute the commands from the high level. we have build agents (external link, opens in new tab) or sometimes we don't have the high level, we just have the mid level of a service that automatically runs or we have a cron job that that that waits for something to happen and then imports something. But they all execute at the end the build script.

[06:21] (external link, opens in new tab) Now why is it important to know about the build actors? Because the build actors should have different types of information. And a lot of times when we do automated builds, we share the information between all these actors when we actually shouldn't. Each actor should have different types of information and that actually makes our builds more maintainable. As a simple example, the build script shouldn't know the password to the FTP server when deploying. Who should know that? The high level, the build controllers.

[06:53] (external link, opens in new tab) TeamCity should know the password because that's a systemwide knowledge that keeps changing. We don't want that in source control. On the other hand, the team or Jenkins doesn't need to know the exact places in our code path in the source control where the unit tests are or where the zip files are or where the jar files should be. It should just know the root location because if it knows about the structure of our code, the structure of our code can change from version to version and so we would need to keep changing the high level controllers and the configuration.

[07:30] (external link, opens in new tab) Who should know about the structure of our code? The build script because the build script is part of source control. So it's versioned. So if you get the build script from three months ago, it will match exactly the structure of the code from three months ago. If you try to run and the knowledge is up there in the build controller and you try to run a build from three months ago, it might not work because it assumes a specific code structure if it thinks it knows it and it's going to be wrong. Another example,

[08:01] (external link, opens in new tab) I just had this at another project is that we had the we had a TeamCity instance that we had to reinstall and everyone was freaking out because they were afraid that all the build scripts, all the build actions and the build steps in TeamCity will be deleted. We'll have to reconfigure them and they were not versioned. It's in a database, but we have to restore a database and all that stuff. That's complicated. But we didn't have all that problem, all those problems because TeamCity was just a very simple trigger to invoke a script file that was in source control.

[08:37] (external link, opens in new tab) So to reinstall it and to put one simple step that invokes a build a build script took about 10 minutes because we were not afraid because all the knowledge of what to do was mostly here. Okay, so that's how I think about them. I forgot to show you this nice image. So the the build controllers, they control the environment. They know about where are the servers to deploy to.

[09:08] (external link, opens in new tab) They don't know the password. They know the passwords to the place to to various servers. They know which route from source control to take. The mid-level they just execute. they have a command from the build controller to say run me the script that's located in this piece of source. So they get the source and they get triggered and then they execute this and this script doesn't know anything. It just executes. It doesn't know where it came from. It doesn't know where it's coming to. It's relative. It doesn't know the location.

[09:40] (external link, opens in new tab) It can run in any computer. It should be in its own little world just running and running. Poor little cat. Now before we understand the build we have to understand the difference between continuous integration and automated builds. Continuous integration is when to trigger the builds. That's the build controllers. Automated builds are the build scripts. Those are the scripts that determine what should happen. So I should have a just trigger for example when someone checks in.

[10:11] (external link, opens in new tab) So a continuous integration will trigger an automated build. That's how you think about it. Now I want to talk about some patterns specifically things that keep happening. I've been working with builds for more than 15 years and I've had all of these problems and that's why I'm actually writing that book. So I want to start with with the simplest force is that we want to start having feedback as quickly as possible.

[10:43] (external link, opens in new tab) But to set up an automated build is very very difficult because our project is already running. We have too many things going on and so we're kind of screwed. And this is where I would say that we should start with a shipping skeleton (external link, opens in new tab). A shipping skeleton is hopefully you started as the first iteration of your project. Now if you start your project on an enforced iteration, you start by releasing the product and then you have a spine, you have a chain of builds that will trigger from continuous integration all the way to release.

[11:22] (external link, opens in new tab) If you can automate that on the first iteration that you have in the first couple of weeks, then you have a spine to work on and then just add more and more features. So the skeleton is basically a simple hello world application. It doesn't do anything. It just says hello world as far as we're concerned either on a web page or a console or something something that we can ship and then show on the end machine of a customer or some or someone and then we take this little piece of code three lines of code and then we say okay let's make this go through all the release cycle that we're going to have during this project.

[11:57] (external link, opens in new tab) That means it's going to go through running unit test which don't exist yet. We'll go through running integration tests (external link, opens in new tab) that don't exist yet. We'll go through deployment to test, deployment to staging, deployment to release, and I'll show you the structure of that in a second. But the nice thing about having a shipping skeleton is that you don't focus on the functionality first. You focus on shipping first. And that means that you might need to get the hardware to deploy to. If you don't have a production server, you should have one on the first iteration.

[12:30] (external link, opens in new tab) If you have to wait for the production server to to happen at the end of the project, what's going to happen is that at the end of the project, especially when you're the most stressed, you're going to have to stop and start configuring the server and then realize all the constraints that you had that you didn't think about when you started building your product. But if you get all that out of the way on the first iteration, what's going to happen is that you're going to figure out all those problems in the beginning and you're going to solve all that crap in the beginning. And what's going to happen is that you're going to fix it and you're just going to forget about it.

[13:04] (external link, opens in new tab) From that moment on, you're always going to deploy and deploy and deploy. So what happens in the first iteration, it's all being recorded. You should know that. So if you hear in the recording, okay, so we want to have a shipping skeleton. And what's going to happen is that okay what is the simplest build chain that I could think of well we have in terms of hardware or virtual machines what I like to have is that to have first we have need a build machine we need a server or something that runs a continuous integration server could be Jenkins could be TeamCity could be something else I personally like TeamCity because the the I think it's much more productive and I don't have to edit XML files and I hate editing XML files And one thing I know about Java,

[13:56] (external link, opens in new tab) Lots of XML files. It's just I don't know. It's the thing. It's like you program XML and sometimes you do a little Java and XML is just horrible for maintainability by people because it's case sensitive and it's space sensitive and it's a lot it's just very sensitive and that means that it's not very humanly friendly. So I like to avoid XML as much as possible. So I will have TeamCity and TeamCity also has a build agent built in.

[14:26] (external link, opens in new tab) That means that already I can start running my build with a single agent for free. I get two more agents if I want to and I can install them on two more different build machines. But I would also want a test environment machine something to deploy to to test so that either testers or anyone else can go and see demos of the product in its latest version. I would want a staging machine as well as a production machine. A staging machine is a production is a production machine which sits next to me and looks exactly like a production machine.

[14:59] (external link, opens in new tab) So it has all the constraints in terms of security, active directory or anything like that. It has the same exact constraints as production. For example, if I if I have to deploy to production via FTP, I also have to deploy to staging via FTP. Okay, that's important because by being able to deploy to staging, I'm actually testing my deploy script and then I know that if I can deploy to staging, I can deploy to production.

[15:29] (external link, opens in new tab) You don't want to deploy to production continuously five times a day. You want to deploy to production by a click of a button maybe, but not automatically. That's the last place. But everything else can happen automatically. So we want to test our build scripts by having a staging machine. Now I've done this with the Amazon EC2 and had six different virtual machines. In the current project we're working on we have seven build agents and 20 build machines. And it's a wonderful thing because we have many people checking in when we keep seeing the statuses of the builds whether they broke something or not.

[16:06] (external link, opens in new tab) And we have automated configuration and automated deployments. In terms of the build configurations, the build configurations are the confi are basically the stages that my code goes through. The first one is continuous integration. That's the one that should run the fastest. Usually it runs unit tests and it just compiles the code. That shouldn't take more than a couple of minutes. It gets the source control and it runs the unit tests.

[16:39] (external link, opens in new tab) Then I might have if I already have integration test I might create a slow build configuration. Now I usually call it nightly but as far as I'm concerned this can run multiple times a day as well. If nobody checks in what the hell the CPU is not being used I might as well run the slow builds as well. And one thing I like to do is to actually trigger the slow build automatically once the continuous integration build runs and passes. The third thing that I do is that I have a deploy. At the minimum, I need a deploy to test if I don't have staging yet.

[17:13] (external link, opens in new tab) But I need somewhere to show a demo. I don't want to show it on the developer machine. Now, the the problem with test is that the test machine is usually not clean. It might have debugging tools or even compilers, IDEs. So, it's not as good for demos because what you would really want to show is a demo on something that looks as close as possible to the customer machine because you want to find out as many problems as as pos as you can possibly get

[17:44] (external link, opens in new tab) Another build configuration to deploy to staging and deploy to production. Now, what usually happens is that the same deploy script for staging and production is being used only with different parameters between those last two configurations. But deploy to test usually is a separate is a different deploy script because the because the requirements for deploying to test are usually less severe than the deploys to staging and production. That would be so four out of these would be the ones I would start with.

[18:18] (external link, opens in new tab) The the one I would not necessarily start with is the slow build because I might not have integration tests just yet. And then as time goes by, I add more features to the build. Yes. >> Where would you run the demos then? >> Where would I run the demos? If I if I have all of these, I would run them on production if I could. In one of the projects that I did, which was I'm not going to say, but in one of the projects that I did, we had a production server running.

[18:49] (external link, opens in new tab) It just wasn't live. The DNS was not set up yet. But we could go to the IP of the machine and show a demo on the production machine, which was the best demo because it was publicly available. We could send the IP to our customers and they would see it to our internal customers. They would see it. They would just need to enter a password to be able to see the website. So for us that was perfect because if I can if I can go the whole way and deploy a new feature and say, "Okay, give me five minutes." And it took five minutes to go through.

[19:22] (external link, opens in new tab) Give me five minutes I'll show you and you can just go to the IP and login and you can see the feature that would be best because then I wouldn't have any fear that it works on that machine but doesn't work on that machine so staging is great but production is always better the only time when production wouldn't be better is if people are actually actively working on it and you're going going to screw up their day by deploying a new version maybe delete something or something like that. So deploy to staging is the is hopefully the minimum that you can get.

[19:52] (external link, opens in new tab) But you know realistically is re in reality you cannot always argue with the facts and the facts are that you might only have a test server. So a test server is good for demos as well as far away from developer machines that that's better for demos. That's kind of the rule. Okay. Closer to production is better for demos because it shows the application in a real situation. How many you have a database? How many databases? >> If I have a database, then I probably would have a test database, a staging database, and a production database.

[20:27] (external link, opens in new tab) Yeah. And then that would force me to have a deploy script that also merges the databases and scripts. But that would force me to do that, which I would have done anyway. I just start with it when it's simple. And then adding small features to the script is much easier than having the whole thing in the middle of the project doing it all at once. Sometimes there's also a developer server, so you might end up a developer database. So, it really depends. Sometimes each developer has their own database.

[20:58] (external link, opens in new tab) Sometimes there's a shared database for developers. Either way, you have to be able to script very automatically setting up and removing the databases if you need to. This is what it looks like when I was using TeamCity. Now the the the there's a project that you don't see the name here but but on the left you can see the numbers. The numbers are actually just strings. I put the strings in the names of the of the build configurations so that you can see the flow the order.

[21:29] (external link, opens in new tab) That's important for a build to be beautiful. I should be able to see the flow of where the code comes in and where the code gets out. So I don't want I want things sorted very very nicely. So and I also gave it numbers of hundreds so that I can fit in other build configurations in the middle if I need to to put them in the middle. So if I if I decide that tomorrow I need between CI and deploy to test. If I decide that I need a a slow build, a nightly build, I just call it 050

[22:02] (external link, opens in new tab) And then slow build so that it matches right between the first two. So readability is very important. This is the beginning of a very early project. You can see we only had 13 tests. Now, here's another problem that keeps happening is that when you have a lot of builds, what's going to happen is that you're going to have a

[22:33] (external link, opens in new tab) You're going to have this problem. The more the build becomes more important, the more you want the build to do. For example, CI would run the unit tests. But if you want to deploy to test, you would want to run the unit test first, right? You would want to make sure it compiles and runs the unit test just like CI did before you deploy. Now obviously if you had a slower build you would want before deploying to test you would want to run CI or you would want to run the unit test and the integration tests and to compile first before doing that.

[23:10] (external link, opens in new tab) So the situation that happens is that each build becomes slower and slower because they do more and more things. So the sol solution to this is what I call commulative build that each build adds on top of the other build. So instead of having each build do one more thing than the other builds, what we want to do is to use a feature called artifacts so that each build does only one thing. Now artifacts are something that not all continuous integration servers support.

[23:44] (external link, opens in new tab) I'm not sure if Jenkins supports it or not. It could. I'm not sure. I definitely know that TeamCity uses it. And the the nice thing about it is that if you learn to use it, you can you can have very effective builds. So in this what you see here what I've set up is that I have continuous CI build having doing compilation and running tests and then I publish all the output all the binaries that were compiled and were passing the test. I publish them as artifacts

[24:15] (external link, opens in new tab) And as I publish them as artifacts, they are available for all the other builds to consume. So I'm actually sharing the binaries to the other builds. That means that the other builds don't have to repeat the same steps. They can just take the binaries that already passed tests and already been compiled and just add one more thing on those binaries. For example, if I had a slow build, I don't need to get things from source control. I don't need to compile and I don't need to run the unit tests. I just get the binaries from the artifacts of the CI and I just run the slow builds.

[24:50] (external link, opens in new tab) That means that the integration build saves time. It doesn't need to get source control compile and all that stuff. The same for test. Before I run deploy to test, I don't need to do compile and run unit test. I just need to get the artifacts from the CI and just deploy them. So that just takes a few seconds. the same for staging and prod. And there's something even more interesting here. I can I can say that the thing that I deploy to production is the actual thing that was deployed to staging that was deployed to test that was run that was passing the test in CI.

[25:27] (external link, opens in new tab) The actual binaries, not just a copy that was recompiled, the actual binaries. That's a powerful feature. And one way to make sure that happens is by having what's called a snapshot dependency between the builds. What we have now is artifacts dependencies that the builds take the artifacts from each other. But we can also set what's called a snapshot dependency. So that TeamCity will make sure that if I get the artifacts from this build from deployment of to test to deployment to staging then it will make sure that the snapshot of the code is the same version.

[26:06] (external link, opens in new tab) So if before I deploy to staging someone committed a new piece of code then test will actually run again before deployment to staging happens. So to make sure that if I if I trigger deploy to production and there's a new version of source code that hasn't been gone through all the steps it will automatically trigger CI deploy to test staging and only then prod will actually happen and that's what I want. I want the same version happening in all the builds.

[26:36] (external link, opens in new tab) One of the biggest problems that I've had in one of the projects is that we had a lot of teams delivering many modules and the modules had dependencies on one on the other. And what happened is that they they would keep getting the wrong versions of different modules because it was a big project. keep people kept committing things into source control and so the builds were getting were sharing the binaries but the viruses were not updating fast enough. So other builds that were triggered were getting an old version of the binaries and suddenly compilation failures

[27:10] (external link, opens in new tab) When when all they had to do was to start doing snapshot dependencies and that would have made sure that that could never happen again. So this makes our builds really really fast. from continuous integration to deployment four minutes. That's what you want. That's continuous delivery (external link, opens in new tab). Of course, the more tests you have, the slower it gets. But being able to deliver to production even in 30 minutes, that's something that you can be proud of.

[27:48] (external link, opens in new tab) Another thing that I talked about is what happens with versioning. you want to run a version of your product and deliver something from a month ago. This is where having the scripts in source control and injecting them into the build controller to execute is an important skill. It's important technique. So I call it build script injection (external link, opens in new tab) and the basic idea is that you take the build script you put it in source control at the root and then when the thing when the CI controller such as TeamCity gets the latest version the agent gets the latest version you just tell it look get latest version and invoke the build file at the root with this name that's all you have to know

[28:29] (external link, opens in new tab) And so if I wanted to add more features to my build script to my build I just change the build script and check it into source control that gets injected automatically and gets executed by TeamCity or Jenkins or whatever it is that's very powerful because I only make the build based on the current structure of the source. So if the build from three months ago had a different structure of the source and I try to run the build with the source code from three months ago, it will just work because the structure has changed but this the build script has also changed.

[29:06] (external link, opens in new tab) So the build scripts the code knows how to build itself and the CI controller only triggers the build. That's the difference between the knowledge of the structure or knowledge of the of the systems around it. Another common thing that happens is that you keep having the same script run over and over and you keep creating copies of scripts. You want to deploy to test, you want to deploy to staging, to deploy to production and then you have three different build scripts.

[29:38] (external link, opens in new tab) What you should have is to try to don't repeat yourself also in build script just like programming. If you could have one script which just accepts parameter for deployment that's a much cleaner way but it only does deploys. So a script should do one thing but it can do the same one thing for different for different locations. So if I want to deploy to staging, if if I need to upload to an FTP and then restart something on a remote server for staging and I need to do the exact same things for production, there is no reason not to do it as a as a script that takes parameters.

[30:14] (external link, opens in new tab) So I call this pattern fill in the blanks. And this is an example from a built file in a application called FinalBuilder. It actually costs money, but I find that it's very maintainable way because I don't have to deal with XML again. And what's cool about FinalBuilder is that and I I'm not getting any money from them. I'm just a happy customer. What's cool about this is that you can see the structure of the build. You can even put break points. You can debug the build the the build script and it has hundreds of built-in tasks

[30:49] (external link, opens in new tab) From restarting services to copying things to unzipping whatever you want. It's you it's mostly Windows-based. So that's the only caveat that we have. But one thing that all script scripts should be allowed to do is to have these parameters right here. You see text to find and test page. This is a script that's called you see the top check site. I call the scripts from many different deploys.

[31:19] (external link, opens in new tab) For example, after I deploy to test, I call this script. I say tell me that the site that I just deployed actually is actually working. Give me a sanity test. So it goes to the site and it has a loop and then it looks for a specific either 404 or whatever it is and it downloads the file and tells you and it breaks the build if the deploy didn't work. But then I use the same file for staging and and production as well. I just change the variables either as an environment variable or as a command line variable that allows me to reuse it.

[31:53] (external link, opens in new tab) So if tomorrow checking a page changes or I need to change how I check it for example test a different page I either change the parameter or I just change the script but usually I just change the parameter in things like TeamCity and that's it. So I can change the text defined without changing the filing source control because that's knowledge that keeps changing about the system.

[32:24] (external link, opens in new tab) And of course we want the builds to be run everywhere. So I don't I don't I shouldn't care what is the machine where the build is running on. So I always want to have a relative path. I never want to know the exact location. So all build scripts should be relative to the location where they are. They shouldn't know where they're located. This is actually a very big antiattern that I see in a lot of places.

[32:55] (external link, opens in new tab) One thing that I keep doing is that I have these scripts and then I extract just like I have extract method I have extract script which I usually do. So I would have one script call another script. This is a script that changes a lot of XML settings. So it configures a website and then at the end after it deploys it calls check site. Now what what you'll notice is that there is hierarchy here.

[33:25] (external link, opens in new tab) So I can take something like a FinalBuilder script file and I can introduce a new developer into my script and teach them how the build works usually in a 10 minutes which is something that a lot of people cannot do because the build files are huge XML files of 500 lines which nobody can edit by but but one guy and that one guy or girl they're now the bus factor (external link, opens in new tab). Have you heard of the bus factor? The bus factor is a very important factor. It's how many people have to get hit by a bus before your project stops working.

[34:00] (external link, opens in new tab) So a bus factor of one is the worst. Okay? It means that there's only one person who knows how to work with a specific technology. And that person, if they're sick, you're stuck. They get hit by a bus, you're stuck. There's always one or two of them in each project. And the idea is to make the knowledge sharable as quickly as possible, having them pair as quickly as possible. By making the build easy to understand, you can remove the bus factor and not have that one person who's in charge of the build process. You want the whole team to be able to be in charge of the build process and change their own structure.

[34:41] (external link, opens in new tab) Another thing that happens is because we have all these builds, we have CI, we have integration, you might have acceptance test builds. When do we start coding? After we check in the code, right? A lot of people would wait an hour before they start changing the code again because the builds take such a long time. Well, one thing that I like to do is to have build waves. So my confidence grows better and better the more builds pass after I check in the code. So whenever I check in the code, I make sure that CI runs automatically and once that finishes, I make sure that either slow build or deploy to test runs automatically.

[35:18] (external link, opens in new tab) So I have the first two or three run automatically. But I start coding after CI has passed and that gives me the feed quick feedback says everything is okay. Now let's do the integration test but you can start coding again. That's why I don't recommend combining into a single build the CI and the integration because that forces the developers to wait until the end of all the tests instead of wait until the end of the unit tests

[35:51] (external link, opens in new tab) And that takes a lot of time from the project. Now there's another antiattern that keeps happening and you probably heard it in a lot of agile circles. don't break the build. Or when I was a just a beginning agile coach, I used to tell people that if they break the build, they have to wear a shirt that says I broke the build or they have to bring cookies or something stupid like that. Well, let me please ask you not to do that.

[36:24] (external link, opens in new tab) You do not want to encourage people not to break the build. In fact, breaking the build is what the build is for. What's the point of a build? The point of a build is that it it breaks when you break it. It it's supposed to tell you something. It's just like a compiler. You're happy when a compiler tells you that you did something wrong. You don't want to wait too much. So that's why you want to have the builds. You want people to commit and to break the build. The only thing you don't want to do is to wait when the build is broken. So the antiattern should be that people wait too much when the build is broken and they assume that just one person has to fix it instead of the whole team.

[37:03] (external link, opens in new tab) That should be a shared responsibility. If you encourage people not to break the build, one of two things can happen. First the is that they are afraid to check in code and so the the code check has become longer and longer, more complicated and actually the more more more chances that the build will break and when it does break it's going to have you're going to have a much harder time to fix it. The second thing that could happen is that people as they wait

[37:35] (external link, opens in new tab) They will not ask for help because they they will feel ashamed that they broke the build. And so that actually encourages less social behavior because you don't you broke the bill so it's your fault. You should feel bad about it and so you should work alone to fix it. Or I don't want to break the bill but I'm afraid I will but I'm not sure and so I'm not sure if I should ask someone to work with me on that.

[38:06] (external link, opens in new tab) It causes a lot of stress for people. Instead of asking for help, people might be just shy and just work alone even more and more and more. So it gets them to be a bit even more secluded and work in their own bubble. The point of the build is to share information. When a build fails, you should be happy. You should be like, "Dude, the build failed. Let's be happy. Let's do something about it." Never let anyone know that you feel bad about the build failing.

[38:39] (external link, opens in new tab) You should feel happy that the build that the build fails because it means that you learned something. And yeah, back to the XML. If you if your builds do have a lot of XML editing in there, you've already lost. Not today, but maybe in three months or maybe in a year, you're going to get to the point where the XML has to be edited and your bus factor is gone or you have no idea what what space you've added or what uppercase letter you missed to make the whole thing not work.

[39:09] (external link, opens in new tab) And if you will ever want to refactor your build, you're going to have a hard time. So, just to summarize, a beautiful build is when you can see the flow. You can see where the code goes in and where it goes out. You can easily change it. You can easily understand it. You can understand what each phase means and you can easily know when to continue coding again and when to get feedback from the customer. So everything should be easy. And if a build is truly beautiful, it's boring.

[39:41] (external link, opens in new tab) Okay? A beautiful build feels boring. Okay? If your build is like very risky and very oh let's see what happens now when I check in everything becomes then then that's an antiattern because a release should be very boring. A release should be just clicking a button and say okay take a look now. That's what a release should feel like. It shouldn't feel like okay let's do a release. Okay let's plan a week where we do the release. So well you work on the configuration files and you make sure the servers are up to date and you tell the client you have to wait and you take down the system.

[40:18] (external link, opens in new tab) That shouldn't it shouldn't be like that. It should just be a simple boring piece of clicking somewhere and that's it. You should be able to see all the parts of a build. So I should be able to drill down and see each part very very simply. But if I zoom out, I should be able to see the flow no matter what level I am. So I can always see the flow from the outside, but if I look in, it's just like functions in code where you can drill in, but the functions have good names.

[40:53] (external link, opens in new tab) They're readable and they call sub functions inside. Or another way to think about it is like a newspaper. A newspaper has a title, it has a subtitle, it has a summary, and then it has the text. as for for each level you can get a different understanding of the system but you can choose at which level to look at it. Okay. So we have high level and low low-level patterns. The high levels are what you want the builds to be version aware. That means that you want the build scripts to be in source control and you don't want the you don't want the build controller to know anything about the structure of the code.

[41:32] (external link, opens in new tab) You want the build to be commulative. That means that each build does one thing and it gets and it gets the assets from the other builds before it does it. We have build waves which are means that the bills get triggered one after the other. So that makes our confidence higher and higher as time goes by. But we can start coding very early. And we have a shipping skeleton which is the beginning of a project. But if you're in the middle of a project, you can still choose to do a shipping skeleton.

[42:03] (external link, opens in new tab) You can say, "Okay, let's do a hello world, make it go through, and then we'll adapt our product and and and and replace hello world with our product and see what we have to add to make everything go." But you have to start somewhere, even if you're in the middle of a project. Low level, we we make sure that the scripts are readable. So, we extract scripts to make them readable. We we make sure that the scripts have parameters. They're not hard-coded. We make sure that the scripts don't care where they run from. They're always relative.