The client was building an FDA regulated medical device provided as a “Software as a Service” (SaaS) solution. The client had created a solution based on various scripting technologies which required a large amount of manual system configuration, which was difficult to replicate reliably on different developer systems. There was no automated build server validating the source code and running test suites.
Some standardization had begun by using heavyweight virtualization technologies (like Virtualbox) which helped, but the overall system cost (in terms of CPU and memory usage) negatively impacted developers, and the solution was expensive to maintain. The technology and methodology being used also limited the number of developers that could work on the project which further limited the scope of the project. Additionally, the versions of system libraries were rarely, if ever, constrained or tracked in the build system, making it difficult to demonstrate traceability of the provenance of the generated executables which would be a show-stopper for the FDA regulatory process.
In order for FP Complete to make this project a success, the project needed to be functional, scalable and accountable in order to meet FDA standards. To meet these objectives, the project requirements had to fulfill the following parameters:
FP Complete used lightweight containerization, which leverages some aspects of the host OS (like the kernel), which allowed for faster initialization, lower memory and CPU usage, simplified disk management, and easier upgrades.
This allowed FP Complete to ship a complete filesystem for a working Linux operating system to each developer, build server, and production machine which neatly codified all system library versions, build tool versions, and command line utilities creating several benefits:
Unfortunately, Continuous Integration systems provide much weaker tooling around tracking build plans themselves . As part of the effort to have reproducible builds and also meet FDA auditability requirements, the build scripts were moved into the source control repository for the device software. These scripts included records of the container version and build tool versions required for completing the build successfully.
The result of this was that each revision of the source code fully identified all tooling dependencies and build plans needed to produce it. Finally, the build process was configured to inject extra metadata about the source code revision into the generated device software which enhanced the auditability of the code.
The upshot of all of this was:
Since introducing the build system, the client’s software team has been able to grow and has gone from around half a dozen developers to nearly three dozen developers, QA engineers, and system administrators working on the code base and reliably building on a daily basis. The continuous integration solution regularly builds the device software, and provides the foundation on which the pull request and code review process ensures that the master branch of the software remains stable, decreasing integration friction amongst the team immensely.
Leveraging add-on technologies like Docker Compose, we have set up a simple addition to the build process which allows engineers to launch fully configured devices, including database servers, message queues, and the device software itself. This has allowed the development and QA teams to have a simple, reproducible environment for reporting and fixing bugs. FDA compliance has been assured.
FP Complete continues to maintain this build system, adding features as desired. The structure of the solution makes it easy to extend the system, test within our Continuous Integration environment, and deploy to all engineers with a simple push to the master branch.