Continuous Integration and Deployment
Continuous integration and continuous deployment, an FP Complete comprehensive, easy to understand blog designed to help you learn why certain DevOps processes collectively create an environment where high-quality software can be developed quicker and more efficiently than ever before.
Aaron Contorer, is the Founder and Chairman of FP Complete, where we help companies use state-of-the-art devops and blockchain tools and techniques to produce secure, lightning-fast, feature-rich software, faster and more often.
Before founding FP Complete, Aaron was an executive at Microsoft, where he served as program manager for distributed systems, and general manager of Visual C++, the leading software development tool at that time. Also, Aaron architect-ed MSN’s move to Internet-based server software, served as the full-time technology adviser to Bill Gates, and he founded and ran the company's Productivity Tools Team for complex software engineering projects.
Let’s begin this presentation by stating the obvious:
Software development has become a complicated process
As information technology and software people, it’s easy to recognize how things are changing at an astonishing speed. To keep pace, we need tools and processes that allow us to rapidly deploy better code more frequently with fewer errors. Is that a high bar to reach? Yes, of course, it is. But it absolutely must be met—that is if you want your company to survive.
Inefficiencies are everywhere
In most companies, we would argue that the information technology team and the software engineering team are misunderstood by the rest of the company.
Of course, we are not implying they’re not trusted as in they’re not good, smart people. What's meant is that they often find it difficult to meet their deadlines, leading to repeated rollbacks and sprints becoming marathons and longer than initially expected timelines, ultimately causing everyone to feel rushed and end results lacking in quality supporting further technical debt.
IT battles for management’s trust
Type 1 versus Type 2.
When management begins to question the processes of engineering and IT, a bad dynamic often develops. No longer does the team get to focus on building great things for their end-users. Deadlines begin and continue to be missed. Meetings in the war room multiply. Rolling back again and again. Instead, they’re forced to focus on solving their struggles and dealing with interpersonal friction.
Believe it or not, many of the problems organizations are having aren’t people-problems. It’s not that they lack good intentions or brainpower.
Instead, the problem is this:
Modern software development can’t be performed using ancient technologies applied within simplistic workflows
We like the statement…
"The best crafts-person with a handsaw cannot do woodworking as efficiently as a robotic cutting tool."
When we automate our work, it becomes faster and easier to replicate. We don’t build in lots of mistakes. As a result, we get to move on with our lives instead of rolling back and reworking things over and over again. Rolling back again? Why roll back? Let's roll forward!
When we automate with good tools and better tested processes programmed in, and we repeat this same process every time, everyone can trust that our work will be performed with quality, and our systems will be more safe and secure.
Sounds ideal, doesn’t it? Of course, it does.
But how do you do it? Where does your team begin? What questions do you ask yourself to define the correct pathway to take and continue to follow? How do you evolve from the environment you’re operating in today to the utopia DevOps strategies will allow you to live and work within well into the future?
EXPLAINED: 3 Rapid DevOps success strategies designed to execute quick, efficient software builds
What follows is the first blog in a series of 3 blogs that explain the three rapid DevOps success strategies everyone here at FP Complete strongly suggests your company embrace, each explained in a manner that’s easy to comprehend and will undoubtedly satisfy the curiosities of beginners and experts alike.
Continuous integration and continuous deployment in DevOps
- What’s continuous integration and deployment?
- How is it implemented?
- What are all the benefits you’ll inevitably enjoy as a result of embracing the system?
For answers to those questions and so much more, read...
FP Complete’s Definitive Guide to Continuous Integration & Deployment In DevOps
Continuous integration and continuous deployment (CICD)
Continuous Integration and Continuous Deployment (CICD)—FP Complete’s founder and chairman, Aaron Contorer’s comprehensive, easy to understand guide designed to help you understand how these two DevOps practices will help you develop more high-quality software, quicker than ever before.
What is continuous integration really all about?
It's the notion that we should be able to integrate work from multiple software engineers, frequently, whether it's every few minutes or every evening. It needs to happen at a reasonably frequent speed so there's always an automated build that's running the software that was just written, automatically checking it for errors and automatically producing a version of our software—If the build succeeds at least—that shows what our latest code does. To some that may sound like a fairly obvious idea.
A lot of people are starting to do continuous integration, but a lot of others still are not. If you look at a company like Amazon, Google, or Facebook, you might see soft releases coming out every five seconds or every hour. But if you look in an older organization, software releases will come out every nine months, and a build only happening every week or every few weeks.
How software engineering projects traditionally unfold
Traditionally, we do big merges where we tell everybody there's going to be a release at a certain date—maybe it's the end of the sprint, maybe it's halfway through the sprint, maybe it's even several sprints on. And so releases happen infrequently, and they’re a big deal when they happen. Developers feel free to move forward quickly on their code, which is great. But then there are these really painful things called merges that happen when people say they are ready to put all of their work together, and the result is that bugs exist on developer’s desktops—or in small groups and teams—for a long time before those bugs are merged. Then you find out that you broke someone or they broke you because the software engineering is uncoordinated. Without the information flow that comes from having a way to continuously combine our code and continuously check that combined version for errors, merges get really hard. And so what happens is people start cracking down on how much testing they do. Everyone is running a separate set of tests on their desktop or on their laptop or their virtual machine, and we find bugs much much later and check-ins are just these great big train wrecks where all this code crashes together.
And the result?
We lose the trust of people outside the engineering organization because projects take a long time to complete, releases come out late, and they have bugs in them.
Why manual work isn’t reliable, reproducible, or scalable:
- Somebody will change a component and they won't realize that somebody else was dependent on how the old version worked.
- Somebody doesn't realize that there was a use case that was going on somewhere else in their team.
- People will write manual tasks that they'll run on their own machine, but they don't really represent how the rest of the system works—it just represents their own private thinking.
Since testing is done separately from the team, we really never know how well-tested the system is. Things fall out of repair when a developer moves on from working on one component to another.
Realistically, every time we make progress, we create new bugs—new integration problems, new oversights, new ways of hurting the guy down the hall. It's clear things can't go on that way.
Signs of schedule risk
Here are some ways to tell if you need to up your game in continuous integration; and then, we'll get to how to do it.
- First of all, you can find people who are trying not to merge their software, thinking they'll be embarrassed. They won't want people to see their code yet.
- You can see managers and project managers saying... You know, let's revise the launch date because this non-new feature overhead work is taking a long time. And when we look at our use of expensive personnel time and talent, we realize they're not spending it actually building the features that our users are waiting for, they're spending it banging their heads into each other.
Those are signs that lack of continuous integration is actually hurting your schedule in your business right now.
Signs of quality risk
As we know in engineering, there's also quality, which trades off against schedule. And these are some signs that you're not integrating your software properly yet, that there's room for improvement. Integration errors show up later—not only when I finally merged my code but even when the code is running in production. Then users make weird bug reports that surprised us, and we realized that some developer made a change that didn't integrate right. Sprints don't get done on time. We can't divide our project into small periods of time that have predictable ending dates. And so that hundred-meter run turns into a 400, and then a 1,000-meter run, and everybody's exhausted at the end and was never prepared to run that far. The quality of bugs, if you will, gets more severe, so the fixing of them gets more expensive. So if we're not integrating our software frequently enough, the quality of our work actually begins to suffer.
Signs of integration risk
We also see that there's a real risk that the code won't integrate at all, meaning the build doesn't even complete.
A sign of this would be a lack of communication between different people on the team. Each person saying..."Hey, I've done my job. So it's not really my problem if the software is not building correctly. "
That's a sign that engineering management and engineering process has to be more sophisticated.
Continuous integration allows faster iterations, consistent builds
But as we start doing builds more frequently and putting in place tools that will help us do that, we find that our life starts to get a little bit better. Code can come out more often. Even if sometimes integrating the code that somebody wrote today doesn't work too well, we often get a working build-out of our system because the time that's passed between the last build and the amount of code that people have had to add to the system since then, or edit since then, is fairly small. So the result is we get working versions of our app, or our whole distributed system released much more often. That allows us then to be testing all the time. So we can build an important part of continuous integration is building a scripted automated set of tests that will run every time to build appears to succeed. So it's one thing that the build tool says... "I'm done building, everything looks fine." But it's another thing to say,"I ran a suite of all the tests that all my engineers have written to date in an automated way", and all or most of them came back saying... "Yeah, I'm fine. This is a working build. "
Continuous integration provides faster feedback on business decisions
By getting builds out quickly, and by finding out which tests pass and failed, like clockwork, we can start to understand the real state of our projects, getting feedback about how we're doing as engineers, and getting to a place where we can actually have a releasable version of the software that built and passed its test, very, very often.
It seems apparent and maybe we all realize that when we go to a large software as a service site—let's just pick on let's say amazon.com for a minute—we don't go there and say... "Oh, wow, this year's new version of Amazon just came out with 7,500 new features". We just go to the site and sometimes there's a little bit of new software there. That's what they can do because they're using continuous integration, then continuously deploying those features that also allows the engineering team and IT team to get feedback from users much much faster about whether the improvements being made, or what they wanted is being achieved. And it allows them to demonstrate progress and build trust between the people who are doing the tech work, and the end-users for whom those changes are invisible if they haven't been integrated and deployed yet.
So, we're going to get a continuous flow of data that tells us...
- Is our relationship with our users working?
- How is our technical work working?
- Are any new coding standards or tools that we've introduced working correctly?
- That new person that we hired—are they writing better code now?
- Is the teamwork coordinated?
- Opens opportunities to review technical and cultural debt.
We can start monitoring our customer satisfaction level as a continuous thing. Instead of saying... "Well, they didn't like the last release, maybe they'll like the next release" ...we can ask every day... "How do customers like today's release of the system?"
Continuous integration provides a better pipeline for engineering workflow
How do we get to this place? By selecting the right tools and using these using tools correctly. The first step in continuous integration is usually to build an automated build of the system so that every person in the team—because they have the same software for building on their desktops and on the servers—can launch a build of the system anytime they want, meaning compile and link their latest code with all the rest of the team's latest code, and all of the imported components and systems that are being used by your team's code. Building an automated build that can run in an unattended fashion, and just tell you whether it succeeded or failed is mandatory for continuous integration. We then usually put that build on a server, but we know that that build is running the same software that the engineers have.
How do we do that?
By having a standardized image of the software that's being used to compile and build our system. Once we have that we know that when we build the software, it's the same result the same executable bit for bit as if you built the software on your machine, or it's simple we've built on our teams build server. We then can move on to build an automated suite of tests, which really consists of having the engineers and the team each build some tests to see if their components are working, as well as having some quality engineers who will build integration tests, which are checking whether the app as a whole is working.
Then the next step will be automated deployment to make sure that those new versions of the software will be put on the server as soon as they're approved, or blessed if you will. We'll come back to continuous deployment in just a moment.
Continuous integration validates correctness, saves time
By continuously integrating, we get these top-level benefits. We're not just talking about small, technical preferences. Continuous integration changes what it's like to work in the software team for the better. It saves time and alleviates cultural and technical debt simultaneously. It is a best practice. It makes sure we're being honest by ensuring that our software, our latest code, is really being looked at by everyone. It makes sure that nobody made a bad assumption. That means continuous integration doesn't just check if somebody typed a comment in the wrong place, it also checks, for example... Did we misunderstand the user’s requirements? ...because the time delay between when somebody asked us to implement a new feature and when they can see that feature running, either in a test build or deployed system is reduced. That means faster feedback loops, means faster engineering and faster business improvement, which means more trust, and, frankly, if you're a business, earning more money.
Why roll backwards? Let's roll forward!
Ultimately, we're preventing fire drills and continuously meeting in the war room; we're preventing chaos and continuously rolling back; we're preventing friction; we're preventing wasted time, all by simply automating to speed the cycle time. No amount of great craftsmanship, no amount of when I give you the software at the end of the year you're really going to like it is a substitute for rapid turnaround time that the automated builds and automated testing will give us. In fact we pride the idea of celebrating the end of the day. Go home and spend quality time on what you'd like.
Once we have builds coming out frequently, continuous deployment is the next step. What that means is there's somebody, or some bot, that's going to look at the result of every automated build and decide... Does this belong on the production server? Have we tested this enough? And you may go directly from your engineering team to production or you may have a fake production system known as a staging system that your software goes to next, depending on how complex your quality is. But the key, again, is automation. We'll look at some tools in the future, like, chef puppet answerable salt stack. This is a high-level presentation so we won't be going into how each tool works, but we can use tools like that to automatically deploy the latest version of an app to a host or to a cloud.
Complement continuous integration with continuous deployment
What's very important to understand in the discipline and the management of continuous integration and continuous deployment is this: The notion of an "IT wizard" should be a thing of the past. This is an idea that we want to be done with as much as possible. We're trying to turn the production of software into a normal event, one that happens every hour, every day. That means we need to turn the deployment of software onto servers into a normal event. It's not enough to say that I have one genius who can make the server work. If that's what people are saying in your company today—let's get that person here, she knows how to make that server work—that's a symptom that you do not yet have proper continuous integration and continuous deployment. If you feel like that and you don't know what to do, that would be a good time to go to the fpcomplete.com/contactus and let us know. And because we can bring you some best practices from other companies that are already there, and you won't have to reinvent the wheel.
Here we offer a couple of very specific tips about doing continuous deployment. One is you never want your production server to be the first time that a server has ever been in that state. So you want to make sure that you have automated tests running on a full version of your cloud system, other than your production system. And to do that, there's a jargon term we want to introduce here called immutable servers. It sounds so very science-fiction-y, but it's a very simple concept. The idea of immutable servers is you don't mutate or modify a server that you deployed yesterday or last week, you delete it and make a new server that has the new software on it. (Actually, you do it in the other order first: you bring up the new server, make sure it looks healthy, and then you can take down the old server if everything looks good.)
In the old world, where servers were extremely expensive pieces of hardware, we would modify and modify and modify, we'd install a patch, we'd install today's build, we'd try not to take anything down. That world is largely in the past when it comes to application servers. Long-running databases that hold our enterprise database of record may be things that have to stay running for very long periods of time. But application servers—we can bring them up, take them down anytime we want. They're just virtual machines with software images on them. So if you have people in your team whose job is to be surgeons of servers, whose job is to go and modify servers to get the new build on them, that's a sign that you need improvements in continuous deployment. Because what you want to have is an automated tool that creates new cloud servers, maybe even a whole new virtual private clouds, and puts your software on them, checks with automated testing that everything looks good, move the traffic over from the old servers for the new servers, and then literally deletes the old servers, because they're virtual machines, they just go away. And if that doesn't pass those tests, you don't delete those old servers. You just leave the traffic, you roll back your deploy, leave the traffic on the old servers and go back to the engineering team and say... "Whew, our automated test found a problem or—worst case—our users found a problem just after that latest build, so let's go back and fix it. Everything's okay, nobody loses their job."
How to get started with continuous integration and continuous deployment
How do you get to a world that looks like that, though? There are a lot of things that you can do to move forward with continuous integration and continuous deployment. These are some quick ideas to get you started. Of course, you're welcome to contact us, and we'll help you if you want. You want to start by using ready-to-use continuous integration tools. We've listed three of the most popular ones here, but there are a lot more that are easy to find too.
A leading tip for you is do not invent your own system for doing the gathering of code the integration builds. These are complex solved problems. When you start inventing your own tools, you're taking your good engineers—often your best engineers, frankly, because builds are hard—off of building features for users. So use off-the-shelf tools and, not only that, expect that those tools won't work on day one. They have to be configured and set up and integrated with how your software is getting built. So, find out how other organizations have been using these tools. At FP complete we've been doing continuous integration and continuous deployment for over seven years—and there's a long learning curve. One of the things that it's our pleasure to do is to work with dozens of different companies and see how everyone's engineering environment is set up and then spread the good word to spread useful practices between companies who are willing to share their insights. Don't try to invent continuous integration yourself. It is a solved problem, it works great. Get expertise from others.
Once you've got that running, be sure to encourage your team to merge their software into the team server as frequently as possible, and to write automated tests as much as possible that can become part of the ever-growing knowledge of your server of what constitutes a working build.
Stay tuned for next week's DevOps blog episode which focuses on Monitoring and Logging!
Upcoming GovCloud Webinar!
Save a seat at FP Complete's upcoming webinar, DevOps, FedRAMP Compliance, and Making Your Migration to Govcloud Successful, presented by Senior FP Complete Engineer, Jason Boyer which airs Wednesday, November 20th, at 10am PST!
Learn many successful compliant strategies involved in the confusing and difficult process of migrating your DevOps to the government designated cloud.