Continuous Delivery with TeamCity

[00:03] (external link, opens in new tab) Okay. So, what we're going to talk about now is a short overview of the way I view the build process. Some of it you might have already heard at the beginning of the first day. We will cover it a bit more officially. So, my motto is automate all the things or basically automate the things that you hate. If you hate it, automate it. That's what I say. And I'm actually writing a book about this called Beautiful Builds (external link, opens in new tab). It's actually in progress. If you Google for it, you can find like an early version of it.

[00:35] (external link, opens in new tab) Now the first thing I want to talk about is something we started talking about but I want to talk about the idea of a release compilation (external link, opens in new tab). So just like we had code compilation or running tests continuously that we want to know that we didn't break anything. The same thing for release. So you want to know that the code that works also works for the current reality for the servers that we have for the customers that we have. And so what we want to set up is continuous integration (external link, opens in new tab) and configuration cycle

[01:08] (external link, opens in new tab) Just like we have continuous test running. So we have automated build so we can compile code run it to test it functionally deploy it and get feedback. But we want the feedback not every couple of weeks. We want the feedback multiple times a day if we could. So what I'm going to talk about is patterns on how to solve common problems in the build process and things that help make a good build process. First of all, I want to talk about the actors, the different actors that go into a build system.

[01:42] (external link, opens in new tab) So what I've noticed in the past 15 years I've been working in the software industry is that when we're talking about build automation we have three levels of actors or three levels of tools or techniques that we use. The high level are things that control the builds and trigger them. We usually call them continuous integration servers but they are the build controllers (external link, opens in new tab). there the the place where you would go to trigger a new build or to configure when a new build gets triggered.

[02:14] (external link, opens in new tab) There is a a middle level where where we actually have the workers, the things that execute the build. Those could be the build agents. We could have scripts that listen to something. We don't have to have build agents to run builds. We could just have shell scripts if we wanted to. So those shell scripts would be considered the midlevel. Those are the things that do the the the ugly work, the copying, the the running the command lines and and the lowest level are the build scripts.

[02:44] (external link, opens in new tab) Those are the things that get executed after all the copying has been done and the system has been set up and those are the things that would run specifically the tests. So when we usually talk about automating things, we talk about build scripts, but we don't talk about the high level actors when those build scripts get executed and who executes those build scripts, which is the things in the middle tier. What I'm going to talk about first is the high level concepts.

[03:16] (external link, opens in new tab) U so the high level are the things that make make basically know everything about the environment. They know where the servers are. they have a different type of knowledge than the things in the middle. They know where things are. They know, for example, the passwords to the FTPs in the system. The mid-level actors, they know about the current machine they're in, but they're basically templated. They they are given a location, they know how to figure out and run the test in that location. And the build scripts, they are just stupid.

[03:47] (external link, opens in new tab) They just get get they have an entry point. They work with relatives paths but they don't know anything. They get the passwords that they need from the environment variables from parameters. They're basically just little engines doing blind work. So the only the so there are two types of knowledge here. The build script at the lower part is the thing that knows where the files are relative to its location. For example, if I want to run tests, I need to know that there's a spec folder and some specs I want to run, some specs I want to run nightly, some of them I want to run on a on a daily basis.

[04:25] (external link, opens in new tab) So, I just know their location, but I but I don't know how I got them. I'm the build scripts are in source control, the lowest level. The high level is the thing that sends in the parameters that where you configure all those parameters and the mid level is basically how you what you pass the parameters to that that's the actual executable. that runs the build scripts or you might have a manual trigger where you just run it manually and you edit the file manually. So there's two differences between there there's a big difference between what I call continuous integration and what I call automated builds.

[05:02] (external link, opens in new tab) There's the difference is that automated builds are what you do and continuous integration is determination of when to do it. If you separate it into that, you also get two different pieces of knowledge. And that's important because once you have two different pieces of knowledge of when to do something and what parameters to send to it, that's a something that you keeping in one place. But the knowledge of what to do or where the files are is knowledge that should be versioned. That's knowledge that should be in source control.

[05:38] (external link, opens in new tab) And that means that we don't want the things on the top to know the location of files. We just want them to know as little as possible about the file system and the way the source control tree looks. The reason is because we want to be able to run a build from two months ago very very easily if we wanted to. But if it's configured for a for a tree hierarchy that's not true for two months ago, then we're out of luck. We have to change the top. But if the bottom is in source control, it's versioned.

[06:11] (external link, opens in new tab) So it's two months ago was already fitting this the style from two months ago. And I know that sounds a bit weird. We're going to take a look at that in a second. But let's talk about some patterns if you look interested. So the first I'm going to talk about the patterns in three forms force, problem and solution. We'll talk about the basic force why we're doing this talk anyway is we want to start getting feedback from the customers as quickly as possible. One of the reasons that we actually want to automate things.

[06:42] (external link, opens in new tab) But the problem is that there are too many details specs and there's so much things that we have to take care of to get everything working. So the solution is to start with a shipping skeleton (external link, opens in new tab). I discussed this, but let me just be clear what that means. A shipping skeleton means that you define a release iteration where you ship the least amount of code that you can. Basically a hello world and but you make it go through all the the the steps of a build chain up all the way up to compiling and deploying if you need to all the way to deploying to a production server.

[07:20] (external link, opens in new tab) And then once that works then you fill out all the details. Now you can say the thing that I keep deploying now has a new feature. Now it has that feature it has that feature and by starting small and having a big focus on a specific thing of delivering the least amount of value at the beginning but just delivering it you can then start adding on features incrementally. You can't just say ah this build there's so many things I have to do. You have to start with a shipping skeleton and then start adding things to it.

[07:50] (external link, opens in new tab) Your job will be much easier, especially if you're in the middle of a project. So you want to sh you want to have a shipping skeleton, but you have no idea where to start. So we have to start with what I call the simplest build chain. So the my idea of a simplest build chain is to first of all get some hardware. We we need a hardware or a virtual hardware to run the builds. So, we need a virtual machine or a real machine that runs a TeamCity or any kind of continuous integration server.

[08:21] (external link, opens in new tab) But I personally like TeamCity because it's free for up to 25 people and it has some features that other continuous integration servers do not have. We need a test machine and when I say machine, I mean either hardware or virtual but we need a test machine, a staging machine and the production machine. Production machine is the thing that people will actually use, the physical thing if possible. The staging machine is a thing that looks exactly like the production machine but we don't mind breaking it but it has the exact same problems and constraints.

[09:00] (external link, opens in new tab) So if the production machine has a limitation on let's say permissions or it's open only on specific ports can only talk to it through FTP then the staging machine will look and have exactly these constraints what's the point the point is that when we are deploying to staging we are literally testing our deployment mechanism to production so we should be feeling Great if we are able to deploy to staging because it means that we're able to deploy to production.

[09:35] (external link, opens in new tab) Okay, if at any point we feel that we deploy to staging but it doesn't mean we can deploy to production. It means it's not a staging machine anymore. We have to get back. That means that staging machine doesn't have any extra stuff unless there's absolutely no other way to do it. But it needs to be in the exact same problem space. Now, as far as I'm I'm concerned, I can have all of these in the cloud except the production machine. Unless the production machine is a website that's in the cloud as well.

[10:08] (external link, opens in new tab) If it's in the same network, the production machine, then the staging machine has to be in the same network. So, what build configurations do we have? So, build configurations are the types of builds or the categories of builds that we run. When I say build, I mean a collection of automated actions that we do. Okay. The first one is a continuous integration build. A continuous integration build is a collection of actions that if you put your code through it, you know, you didn't break someone else's code, you didn't break your code, and that the most basic stuff works.

[10:46] (external link, opens in new tab) But it also has to be fast. If it's not fast, the feedback loop is too long. And the point is you want to check in source control have the continuous integration build get triggered automatically by that and within less than a minute be able to move on to on on happily coding new features. The slow build is for everything that that takes too much time. That's where you would have very lengthy integration tests (external link, opens in new tab), cucumber spec that use the browser, everything that takes seconds to run and then you have to multiply it by thousands if you have thousands of tests.

[11:24] (external link, opens in new tab) So slows can take a long time. So you usually run them after continuous integration tests, but I might trigger them automatically every time I check in as well and get the feedback within two hours. especially get feedback in two hours than tomorrow. So as far as I'm concerned, I might have 20 build machines if I have a lot of build configurations. I would run all of them continuously if I needed to, but I would only wait until the continuous integration passed to be able to continue to to start coding.

[11:57] (external link, opens in new tab) I also have a deployment configuration. A deployment gets triggered only after we have slow builds passing and continuous integration builds passing. But I can automate the deployment so that it gets triggered automatically whenever the previous builds have worked and passed. I can also automate the deployment to staging. That would be good for me because if I deploy to staging automatically that I don't that I can show demos on staging as if it was a customer's machine

[12:33] (external link, opens in new tab) And if I'm a and the only thing that I I would refrain from fully automating the deployment and triggering of is production. That's the only place where I would say you know what you probably want a human being to push a single button for that one to happen. But again based on the structure of the organization you might have people saying you can only deploy to test servers between specific hours in a day because otherwise testers are working on the machine. So that's more about organization but the first three should pro you could probably automate most you could probably have them running almost as a single piece

[13:11] (external link, opens in new tab) But you look at them as three separate pieces because you need the first one to pass to start coding. Here's what that looks like. looks like in TeamCity. So we have in here I this is an example from a previous project I've been to. It's just the beginning but this is the f the the first iteration. We only have 13 tests as you can see. And I and these are separate build configurations in a single project which is named after the project that I worked on.

[13:44] (external link, opens in new tab) And I also gave them in the name I gave them the letters so that they will sort correctly because what I want to be able to see in the build is the flow. What does the code go through almost like a graph. So this is the CI build. It runs all the tests. I only had unit tests (external link, opens in new tab) here. I automatically deploy to test and staging and deployment to production is just by a click of a button.

[14:18] (external link, opens in new tab) Now I want to get the feedback as quickly as possible from all the builds. But what happens in real life is that I've seen and I've done as well is that the more important a build is like the closer it is to production the more actions you want the build to do right you don't want deploy to production something that didn't go through testing right so you don't want staging to go through before you actually ran on the night go so what I find a lot of people do is that the continuous integration build, it runs the unit tests, but then the nightly build, it runs the unit tests and it runs the long slow test.

[14:58] (external link, opens in new tab) And then the deployment to test runs the unit tests, runs the 90 tests and deploys. So actually the closer you get to production, the builds take longer and longer and that becomes quickly very very problematic. So, one of the things that we can do with tools like TeamCity is to avoid that and create what I call a cumulative build (external link, opens in new tab). A build that accumulates things, but it doesn't repeat them. It just adds layers of confidence on top of the existing builds.

[15:32] (external link, opens in new tab) This is what it looks like. I would have the continuous integration build have be the only build that uses source control. That's the only build that actually checks out things from let's say git repository and runs the tests. And then I would have something called artifacts. Build artifacts are things that you can set up between build configurations so that each configuration can pass output from the things that pass the build like all the DLLs or the Ruby files and move those files

[16:09] (external link, opens in new tab) To the next configuration. And so I can say to this configuration, you don't have to run the tests anymore because the files in your folder are the actual files that have gone through and passed the test on the build before. So that means I can save the time of checking out and running the tests and only focus on running the slow tests here. And then I would have this build share artifacts with this build.

[16:39] (external link, opens in new tab) And so in deployment to staging, I would only have things that have actually been literally have been through the previous builds. And here I don't have a slow build because I didn't need one yet. But I know that what I'm deploying to staging are the actual files that passed tests, the actual files that went through the test computer and will be now in staging and the safer production. And that means that the builds become much much smaller and much much faster.

[17:12] (external link, opens in new tab) So they so the the deployment builds usually take maybe a few minutes. Worst case scenario. How fast does this does it become? This is an example of of the build situation here. 34 seconds to run the the unit tests. One minute, one minute, one minute. So that means that if I wanted to, I could click the run button here and within three and a half minutes I would have a new version in production.

[17:48] (external link, opens in new tab) Okay, that's an example of release continuously continuous release process. So if I wanted to deploy a new feature, I just deploy new feature. If I want to fix something, I just deploy the fix. And for small teams, this is great. For big teams, we can go a bit more complicated. But for now, that's that's the basic rule is that the builds do one thing each.

[18:18] (external link, opens in new tab) If I do want to build a version from one month ago because I don't know we found the bug then I cannot do it with the the way some people are working with integration today because today today what you can do and team also allows this which I think is an anti-pattern is that it allows you to to have build steps in here that run the test directly. instead of having the build scripts triggered

[18:49] (external link, opens in new tab) And being source control. So you can have the the build scripts like running directly from TeamCity and not not having them in source control and then you would have no versioning of the build scripts. So what we do want is to inject the build script from source control and have it executed by the team agent. That means that the team agent gets the source control and then executes whatever is the rake file (external link, opens in new tab) in our case in the root.

[19:21] (external link, opens in new tab) But the rake file in the root is under our possession. The version is part of source control. So the rake file is always true to the current structure of the code and team is basically a blind trigger that executes the rake file. So if I wanted to run the build from three months ago, I would basically check out the code from three months ago and the rake file will match perfectly because the rake file from three months ago matches the structure of the files and the features and the tests and all that stuff.

[19:54] (external link, opens in new tab) Okay. What if we want to repeat the same scripts with different inputs or targets? Scripts are parameterized but we want the the knowledge of those parameters to be in the not directly in the scripts. We don't want the scripts to be too smart. So this is an example of a script in a tool called FinalBuilder. Now this is not available for Linux or Mac.

[20:26] (external link, opens in new tab) This is only running this is written in Delphi. So it runs usually on Windows machines. But it's a great tool. I've been using it in the Windows world for a long time. But every build script including rake would have the ability to just use the environment environment variables here. So if I wanted to to to check that the site exists, I would have a generic script that says I want you to get me the URL.

[20:56] (external link, opens in new tab) I don't know what the URL is. could change tomorrow because the location on the server or the port and all that stuff that changes where I where I deploy to, right? And what text should I find? The text to find to know that the site is up keeps changing as well. So that knowledge should be at a high level, a level where an administrator says, "Oh, now it's on that server." But they that administrator can just change one parameter in the team configuration. No XML needed.

[21:27] (external link, opens in new tab) And the builds will just start working against a different server and that's how it should be. Another pattern we want to be able to run the build anywhere. That means that our scripts can never know the actual path. They can always only be relative. This is an example from a project I did at BU. And this is basically the location of the project file. this file here slash solution file that I want to compile and that means that I can run this build script anywhere.

[22:00] (external link, opens in new tab) I I don't know what drives we have I just know it's the current location slash something. So relativity is very important in build scripts. What about readability and maintainability? We want to be able to have scripts that are readable. So we can extract scripts. We can have scripts that call other scripts if you want to, just like functions calling other function. This is an example of a script that edits XML files and changes parameters, but then at the end it calls a check site script.

[22:33] (external link, opens in new tab) It doesn't check the site itself. It reuses the scripts. We don't want to repeat ourselves. So the dry rule exists for scripts just as much as it does for code. Questions so far. Now what about feedback? I said that I want to start coding as quickly as possible after continuous integration. So I I use what I call build waves. I can have things triggered automatically. Continuous integration triggered by the check-in deployment triggered by continuous integration.

[23:09] (external link, opens in new tab) So I want to have these two triggered and each one that passes I feel more and more confident about my code. I can start coding here but I might have two or three levels of more builds that run. So I get confident within a minute but after an hour I'm even more confident because more builds have passed in that hour that do more complicated things. So I want to be able to do that and for that I would set triggers. You can configure the triggers in TeamCity very easily.

[23:44] (external link, opens in new tab) Now there is an anti-pattern. You probably heard this in a lot of agile circles. Don't break the build or I even did this did this where I was in a team and we had a a special shirt that says I broke the build and the person who broke the build last had to wear the shirt. But that's an anti pattern because we do want people to break the build. What happens if we give people shirts when they break the build? They stop breaking the bill.

[24:14] (external link, opens in new tab) Doesn't mean the build won't be broken. We just know about it too late. What's the point of the build? When the build fails, the last thing you want to do is to get back and not do anything. What you want to do is to jump in and just say, "Dude, builds and tests are made to be broken, so you should be happy." Just like a compiler error, the compiler tells you it's okay. You we found it early. So that's why we have the build. So if we don't use the build for what it's intended for,

[24:47] (external link, opens in new tab) Then we're just making everyone just hide behind their desks. And then people will check in source control less and less until they're absolutely sure they won't break the build. That's not what we want. We want feedback that the code works. So we don't want to have that anti-pattern. Okay, here's another anti-pattern. If you have to edit XML to run your builds, you have lost. Okay, because XML is I think almost the COBOL of today.

[25:20] (external link, opens in new tab) Okay. It's it's uppercase. It's case sensitive. It's space sensitive. You get one character wrong and the world hates you basically. And XML for any build that has is older than two weeks, it becomes too long to maintain. It's unreadable. It's unmaintainable. So XML is the wrong way to do it. What do we what do we want? We want our builds to be continuously beautiful.

[25:51] (external link, opens in new tab) We want to be able to see the flow. We want to be able to see how the code works, what's going on right now. That's why I always advocate to have a build screen, something that tells you what's going on right now. Something hanging on the wall telling you what's the status of the application right now of all the servers, of the current source code, which stage, which stage it is, etc. It needs to be coherent. That means that if something breaks, I need to be able to easily see where it's coming from.

[26:21] (external link, opens in new tab) So, I can drill down as much as possible, but I can look back and I can see just the structure. I don't have to see the details unless I choose to. Just like code, you know, Uncle Bob has this nice way of saying that when you read code, you want to read it like a newspaper. You have a title, then you have a subtitle that gives you a bit more details. And if you want you can drill down and read all the dirty details about it. So it has levels of abstraction. The same here everything works but you choose at which level to look at and it still is readable.

[26:58] (external link, opens in new tab) So those are the basic patterns. We want our builds to be version aware. We want to do the builds cumulatively incrementally. We want the same code to go through all of the builds instead of having each build compile or run tests on its own version of the code. that saves us time and we start with the shipping skeleton and then we use it like a spine and we add more features to it. If we find that the build becomes a bit less maintainable, we extract subscripts and we execute them just like functions. We make sure that the scripts can run anywhere.

[27:31] (external link, opens in new tab) So we just use parameters in the scripts and the scripts are always relative to where they are. And with that, I want to show you how I would with TeamCity. So, this is a TeamCity server, and I've already set up a one build in a project called test one, but it would be the name of your project. H and in this build what's going on is that every time I check in

[28:06] (external link, opens in new tab) I can see the different files that were checked in. I can see the diffs between them. For example, the gem file here changed. It's very easy to see the differences. I can see the tests that were passed. So this is an example of feature tests. These are all the tests that passed. This is running RSpec and Cucumber (external link, opens in new tab). And what we're going to do, we're going to create a continuous integration of our own. And this is what it looks like. If I go to administration

[28:39] (external link, opens in new tab) Under projects, in the root project, I can create either a sub project or build configuration. So I'm going to create a build configuration here. I'm going to call it CI 2. But it would be the first one. I would just call it CI continuous integration. And I don't need to use the other stuff here for now. But when I want to start sharing artifacts to other builds, this location here is where I would share them.

[29:11] (external link, opens in new tab) And I would say share everything under the root to to other builds and then other builds can pick and choose or I would say just give me everything under lib which is also possible. But for now, we don't have other builds, so we don't need that. So, I'm going to create a build configuration. CI2

[29:41] (external link, opens in new tab) And VCS settings is where I decide what source control to use. I've created a source control called local git, but I'll show you what it's like to to have source control that's connected to git. So, I have git here. I can connect to team system, to anything. I can give it a name remote git and I can go to to GitHub and say okay let me get

[30:13] (external link, opens in new tab) Let me get this. So I'll use the TDD demo for the source control. And what we'll do now is I need the fetch URL. I can say GitHub verb demo.git. And I think that should be enough. And I can test the connection and see what happens. Connection successful. And I can set some other things. For example,

[30:43] (external link, opens in new tab) By default, it would look and check in check trigger stuff every 60 seconds. It will if multiple people check in, it will gather their their projects. But I can say for demo purposes just just listen every 10 seconds. I can choose to clean all the files. I can have a lot of different branches that I want to get here. By default, I'm just getting the master branch and getting everything. So, I'm attaching to the remote git

[31:16] (external link, opens in new tab) Automatically on the server, which is what I want. I can label the branch if I want to. And now I want to add a build step. So, a build step, this is the stuff that either triggers our build script. This is the stuff that happens after checkout or after get fetch. In our case, I can just as an example I can just say that we can run the script. Can we just say echo in Unix?

[31:47] (external link, opens in new tab) Just say echo. Hello from the build step. Okay. If we did that we can by default give it say a hello world. We can save that and we're ready to run our builds. So this is the CI2. If I click here, I can run it.

[32:18] (external link, opens in new tab) It's getting the changes. It's updating the sources and success. Obviously, we haven't done anything yet, but I can look at the build log. I can look just at the tail if it's still running. I can see a live view of the build log. I see step one, hello world, right here. I can look at important messages if there are any errors. You can look at the preview which is readable.

[32:48] (external link, opens in new tab) Okay, so far so good. That's the first build. Let's trigger this build. Edit settings. I want it to trigger automatically. So I go to the build triggers in the configuration settings. I add a new trigger. And I can have many different triggers. For example, this yes trigger is a trigger that makes sure that every once every time someone checks in, build gets triggered automatically. That's what we want. I can also choose that it's a build that happens every night at 12 or every day at 3:00 a.m.

[33:23] (external link, opens in new tab) I can have this build automatically start when a different build configuration finishes. That's important for when I want to have the build wave. So I would like to have this integration and then slow builds automatically run. I can automatically run builds on branches that people have made to make sure that that branch passes. And that's a bit more complicated for now. We won't touch that too much.

[33:53] (external link, opens in new tab) I can have a retry. A retry is good if your build fails, but it if it passes one of three builds, then you're good. That's definitely not for a CI build. CI build should be passing immediately. Okay. By default, we can just use the default rules. But if I wanted to only check in on specific folders that changed, I can add trigger rules here and say only only trigger this build if something that's not a config file change.

[34:26] (external link, opens in new tab) But I want to trigger on everything. Okay, so I saved it. Now, if I go to my source control and I just try to push. Okay, let's do a touch file. Change then something.xt

[35:01] (external link, opens in new tab) Check something in I'll push it to the folder. Okay, hopefully that's okay. Now, now if I just look here and I don't do anything, hopefully this should be detected. I don't know what's happening now.

[35:48] (external link, opens in new tab) Okay. So, this was running 3 minutes ago, but now I should see pending here because this build was also triggered. I al already have a trigger for the previous build that I have with tests and this one triggers as well. You can see that it triggers does nothing and Since I only have a single build agent installed on this machine, I can see it here. Then this build went into the queue and then when that one finished, this one started. Okay. So now let's say that we want to run

[36:22] (external link, opens in new tab) A rake file. And this is important because we want a rake file to be our build script. So in this case, my rake file only has two only has two tasks. Now if you don't know what a rake file is, it's the Ruby version of make. It's the way to automate whatever we want. So by default, I would create just a structure of tasks.

[36:54] (external link, opens in new tab) I have a default task which is what happens when you just say rake and you have a rake file and but here I say default is depending on a task called spec and a task called features this is the spec task it's based on a rake task from rspec so I have rspec core rake task and then I configure it to say just run string string calculator (external link, opens in new tab) specs for now.

[37:25] (external link, opens in new tab) I can rename it whatever I want. Of course, I can say spec or string spec and I give it the options of formatting things for documentation and the same for cucumber. I give it a name features and I say the options are to run every just the Sinatra features and I want it to format in a nice 3D way. And then I just say that the default task depends on spec and features. So if I just say right here directly,

[37:58] (external link, opens in new tab) I should see this. Okay. And that's good for me because now I know how to run the test and where they are. I don't want the TeamCity to work to know where the tests are. I could have had TeamCity run a command line that runs RSpec and that's not what I want. I want TeamCity to to have a a blind trigger that just triggers rake. So, if I go back to TeamCity and I go to the build steps and I I'm going to remove this, disable it, add the build step,

[38:33] (external link, opens in new tab) And say I want a rake file, not Gradle rake file. Can call it just rake and the path to the rake file I don't have to give it by default it will use the default rake file name I can I can enter the rake file content here but remember not a good idea because this is not versioned okay so if I the rake file that that is good for this version might not be good for the previous versions

[39:08] (external link, opens in new tab) Anything else I need to do here do I want to use bundler or not I have a I have a gem file so I'm going to use bundler but if you don't you don't have to use bundle exec I don't need Test::Unit as dependencies I just want rspec and cucumber and I'm going to save and I can I guess just delete that now I just have the rake task and I can go to projects and

[39:46] (external link, opens in new tab) So now what we should be good seeing is that it checks out and it ran the test and I can see the tests passing just like I did in the command line. Now how do I move the dependencies from u one build to the other from CI to 90 bill? We're going to see that soon. But first, I want you to experience what it's like to to build a integration with TeamCity on your own.

[40:20] (external link, opens in new tab) Okay, questions before we start. Okay, let's go.