Up to now, Stack has been using a number of CI solutions:
There are a number of problems we’re running into:
Travis’s lack of resources and limited build times lead to
unreliability of tests, which are frequently hitting Travis’s 50
minute time limit even with all dependencies cached. I don’t think
we’ve ever gotten the macOS build to complete. We’ve also seen a
lot of network instability which leads to spurious build
Running integration tests with Gitlab CI gets us past the
resourcing/timeout problems, but Gitlab CI doesn’t integrate well
with Github (e.g., showing test status for pull requests), which
rules out moving all tests to it.
For Gitlab CI, we have to manage a Gitlab runner on an EC2
instance, and pay costs of the EC2 instance that hosts it.
Using multiple CI services increases complexity and maintenance
Back in September, Microsoft announced Azure Pipelines, a new CI/CD service
that integrates with Github, provides unlimited free build minutes,
a 20 hour time limit for jobs, and supports all three major
platforms (Linux, macOS, and Windows). This looked very intriguing,
and we’ve been trying it out with success.
Most builds complete in less than 15 minutes.
All three platforms are building reliably (when there are test
failures, they are legitimate).
Unit tests are run on every push.
Integration tests are run nightly on the
stable branches, and nightly binaries are published as
There is now
one place to see the status of all automated
Converting all the existing pipelines was not trivial, but also
not overly difficult. UPDATE We've since moved this CI over to Github Actions, and have removed the files referenced. See the YAML configurations for unit tests,
nightly integration tests, and emplates
The Azure Pipelines UI is not very intuitive. In particular,
adding a second pipeline (for nightlies) was confusing, and it’s
hard to find settings and options.While we are generally happy with
Azure Pipelines, that is not to say everything is perfect:
Parts of the CI configuration (such as scheduled builds) are not
controlled by the YAML files and must be configured through the
Azure Pipelines does not have built-in support for caching. For
Stack, this isn’t a big deal since built-in caching solutions are
rarely a great fit for large Haskell projects, so we use the
cache-s3 tool to
manage a cache in an S3 bucket.
We’ve found bugs, such as an incompatibility between bash and powershell
This ties us a bit more closely to Microsoft, and it’s unknown
whether they’ll offer all the features we use for free
indefinitely. That said, we’re already tied to Microsoft by using
Microsoft has acquired), and moving to a different CI platform
in the future isn’t terribly difficult should it turn out to be
We’ve added a page to the Stack documentation with details
on how to enable Azure Pipelines for your own Haskell projects.
Any contributions to move other commercialhaskell or fpco repos over to
Azure Pipelines would be welcome (we don’t necessarily plan to move
them all over immediately if Travis is working well enough, but
happy for help any time).
Do you like this blog post and need help with DevOps, Rust or functional programming? Contact us.