Lesson 9 of 17
Deploy by Proxy
In this lesson
Roy Osherove describes a governmental project that had to do continuous delivery (external link, opens in new tab) while the security people did not allow access from staging into the production environment. The solution was an isolated deploy machine in the DMZ, reachable from development and with access to production, running a TeamCity instance that every couple of minutes picked up files uploaded to its FTP folder and deployed them to production. To test it, they moved staging into the same environment as production and deployed to it the same way, through an FTP folder and a script, so the production deployment scripts could fail on staging without harm.
Transcript
[00:03] (external link, opens in new tab) Now next I want to talk about a way to try to fix things in the real world. You see up until now I've been talking about some kind of an imaginary universe in which, you know, there's a test environment and you just deploy to a staging environment and then you just deploy to your production environment and you can do it all continuously, you know, and there are unicorns flying and pigs, I don't know, mm you know, dancing with cats, etc. But really what's really going on in the real world is that you really can't do that.
[00:38] (external link, opens in new tab) Unfortunately, here's what really happens in the real world. The real world some security person comes to you and says, "Well, in a in a really great real world, security person actually comes to you and talks to you and you can see their face. But really in the real world, a security person sends you an email 2 weeks too late after it's too late to do anything about it telling you, actually you cannot get access to the production environment. That's just wrong. So I don't want you to see any connection between staging and environment
[01:12] (external link, opens in new tab) And production. So suddenly this whole terrific idea of just deploying to production just dissipates into a river of tears and we have to figure it out. So here's something that I actually got to do in one of those in governmental projects. We had to do continuous delivery (external link, opens in new tab), of course. You know how the customer always says that. You continuous delivery, yes, great. Woohoo. Unfortunately
[01:42] (external link, opens in new tab) We also had that security permission. But we came up with this interesting solution. You see, in the middle between production and staging right between the green zone and the bad zone, we had put up a special machine in a zone called the demilitarized zone, the DMZ. And this machine would have access would have access from the development environment, from staging, from test, but it also has access to the production environment.
[02:13] (external link, opens in new tab) So, the security people agreed that as long as a person doesn't have direct access from staging here into production here. If it can be done in some automated way, then it's okay. So, we actually installed a TeamCity instance here in the isolated deploy machine and we uploaded a a bunch of files to an FTP folder located on this machine. And here there would be a TeamCity instance
[02:45] (external link, opens in new tab) That would listen that would trigger a build script that would run and listen to this FTP server. And if there were files there, it would automatically copy them and deploy them to production environment and reset whatever is needed to reset. So, this would run every couple of minutes etc. So, by doing that we were able to overcome this really really big problem of actually having access to the production. Now, one thing that we figured out is that hey, you know, this whole thing in the isolated deploy machine can be kind of tricky.
[03:18] (external link, opens in new tab) How do we debug it? How do we actually test it? And the way we tested it is that we actually pushed the boundary left. So, that now we would actually put the staging environment in the same environment as production because that's what you're supposed to do. And now because we have a staging environment we can actually test our deployment scripts to production by deploying things into staging. So, when when we actually wanted to deploy to staging, we actually had an FTP to deploy to staging and a script that would deploy to the staging machine and
[03:54] (external link, opens in new tab) Then we would have access to a different isolated deploy machine that would deploy to production. Now you can simplify this by having just a single machine that has access to both staging and production or if staging is on the production machine then a single machine has access to both but you have to figure out the scripts but by doing this you're actually able to test the deployment script to production and if the deployment script to production fails on staging nothing bad really happens.
[04:25] (external link, opens in new tab) And that's how you get over this human factor of security by allowing a proxy automated machine that does things basically automatically.