NOTE This blog post made the rounds last week before the
branch was actually merged and the post was still on a review
server. I'm officially publishing it as the pull request
is now merged.
There is a collection of features in Stack that have been added
in bit by bit, as opposed to being designed into a cohesive whole
from the start. The features work, but could be a bit better. We've
known for a while that, instead of putting in place strategic
fixes, a more general refactoring of the core dependency management
logic was in order. I'm happy to announce that these changes have
landed in the master branch, and will be part of the next major
release of Stack.
I'd like to motivate the limitations in Stack that encouraged
this change, discuss the new system, mention some potential future
changes, and share a few thoughts on the (very pleasant) Haskell
refactoring process itself.
NOTE These features have not currently been
released, so don't try using them in a stable Stack executable.
If you'd like to test them out (and I'd certainly appreciate the
extra testing), you can run
stack upgrade --git to
build a Stack executable from the master branch.
Consider this fairly standard snippet of a stack.yaml file:
This is leveraging a number of features of Stack right off the
- Using an LTS Haskell snapshot to capture a consistent set of
- Specify multiple project packages, in this case
- Specify an extra dependency from a Git repository.
- Specify multiple subdirectories for a Git repository to
directly support "megarepos".
- Specify an extra dependency from the upstream package index
(which is Hackage unless you've done something weird).
This is great, but there's a bit of pain involved in this:
- My personal pet peeve: that
extra-dep: true for
the Git repo. Because of how features were added, we included Git
repos together with "project packages," and then added a hack to
explicitly state that they should be treated as dependencies (so
stack test, for example, won't build their
dependencies). This feels weird.
- We also have this
extra-deps stanza, which
accepts package name/version combos, but doesn't accept Git
repos (or HTTP(S) tarballs, which are also supported in the
- As Hackage cabal file revisions become more common, it's taking
away a level of reproducibility that
specify the exact revision of a package we'd like to use.
subdirs feature is nice, but it's weird that
it's not more directly connected to the Git repo information
(notice how it's a level up).
- I've specified
yesod-bin as an extra-dep, which
provides an executable named
yesod. Ideally, if one of
my packages specified a dependency on that executable, Stack would
automatically build the
yesod-bin package. While this
logic works for LTS Haskell and Stackage Nightly snapshots, it
doesn't work for these extra-deps.
- Specifying dependencies like this inside the
stack.yaml file makes them local to just the project
I'm working on. There are advantages to this approach regarding
disk space (which I won't get into here), but there's a big
downside: I can't share precompiled libraries between projects that
are defined this way. I'd like to be able to recapture that sharing
- And more generally: why can't I just define this customized
lts-8.12 and share it among multiple
projects, possibly even from an immutable URL?
As you can see, the problems aren't insurmountable, but they are
annoyances, and they seem to overlap quite a bit.
Let's rewrite that
stack.yaml file to be a little
bit more straightforward:
- git: https://github.com/yesodweb/yesod
The first thing to notice is that the
value is now just a list of the actual code in our project, not the
Next, we still have
from Hackage. But we have this funny
at the end. This is a hash of the cabal file contents we want to
use. This gives us much stronger guarantees of reproducibility than
we had previously. Instead of getting whatever most recent version
happens to be available, you'll get an exact cabal file. This
feature has been present for a while in Stackage snapshots, but
hasn't been accessible for local dependencies.
Next, we've moved the Git repo information out of
packages and into
extra-deps where it
logically belongs. We also no longer need that extra
location key. We had that so that we could also define
subdirs keys. We now
instead put the
subdirs key next to the
commit keys, and don't need
extra-dep at all (since it's implied by being within
Behind the scenes, the code managing these things has changed
drastically. Most importantly for our discussion here, Stack now
uses the same code paths for loading up snapshots and loading up
package information within the
stack.yaml file. In
addition to just being a good practice for keeping us sane, this
means that build tool detection now works for project packages and
This answers a good deal of our points above (hold off for the
last two when we get to custom snapshots).
Four package locations
That probably seemed like a bit of a jumble, so let's start
over. Every package has a location, which tells Stack where
to get it from. Stack supports for different package locations:
- Local file path, like
- Package index, specified version a package name, version
number, and optional cabal file hash. You're probably wondering
"why not just call it Hackage?" The reason is that you can extend
your list of package indices to augment or override Hackage (such
as to use a corporate package repository). That's why we use the
- Git or Mercurial repo, specified by a repo URL, a
commit, and an optional list of subdirectories within the repo to
look for packages. If omitted, Stack will look in the root of the
- HTTP(S) URL, which isn't demonstrated above, but which
is just a standard
http(s):// URL pointing to a
All four of these have been supported in Stack since (almost)
its inception. The differences now are that:
extra-deps now supports all four forms
packages still supports local file paths, Git
repos, and HTTP(S) URLs, but for the latter two requires you to
extra-dep: as either
false. We'll discuss this a bit
more below. There are two reasons package index location isn't
- It wasn't supported previously, and is an illogical thing to
do: you would never have a situation of working on a package pulled
from the index. If you want to do that, you should probably clone
the source of that package (such as with a Git submodule).
- It could introduce ambiguities in parsing between a package
index and a filepath (imagine you have a local directory
foo-1.2.3). We've allowed this ambiguity to exist in
extra-deps; if you have such a filepath, you can
always preface it with
This is all well and good, but isn't much more than a cosmetic
improvement (though, in my opinion, it's a very nice cosmetic
improvement). But this gets much cooler with custom snapshots.
Stack has had some support for custom snapshots for a while, but
it's never been fully implemented, since we've been waiting for
this extensible snapshot concept to land. Since most people aren't
very familiar with custom snapshots today, I'm not going to compare
and contrast, but instead just jump in to explaining how they work
Stack configurations always discuss a resolver, which
specifies a GHC version, a set of additional packages, build flags,
and some other pieces of metadata. You've probably seen a few kinds
of resolvers until now:
lts-8.12, using LTS Haskell
nightly-2017-07-01, using Stackage Nightly
ghc-8.0.2, using a specific GHC version without
any extra packages available
Custom snapshots answer a simple question: what if I want to
define my own snapshot which isn't LTS Haskell or Stackage Nightly?
And that's really all they are: a format for defining your own
snapshots like Stackage does. However, they've got a number of cool
features that Stackage snapshots don't:
- They are extensible (thus the whole blog post name): you
can define a parent snapshot for any snapshot and inherit its
- You can use all four types of package locations when defining a
custom snapshot. Say you have a package that isn't ready to be
released to Hackage, or is only for your internal team to use. You
can stick it in a tarball on a webserver, or in a Git repo, and
refer to it just as you would from
extra-deps in your
- Unlike defining packages in
built in a custom snapshot can be shared in the package cache and
reused between projects.
Let's see how this would modify our
above. First, I'm going to define a
name: my-snapshot # For user display only
- git: https://github.com/yesodweb/yesod
Notice how I've kept the same
resolver value here.
What I'm stating is that I'd like my snapshot to start off with the
same GHC version and package set defined in
and then add new packages. Next, I've copied my entire
extra-deps section in here, and called it
packages instead (since these are the packages that
actually make up the snapshot, not some extra dependencies added on
Note that, because a custom snapshot is intended to contain
immutable package data, it does not support local filepaths
as package location, as these are expected to change over time.
Instead of a Stackage snapshot or compiler version, my
resolver now gives the path to the snapshot config
file. This can be a file path, or an HTTP(S) URL. The
packages section stayed the same, but my
extra-deps is no longer necessary: all of my
dependencies are now defined within the custom snapshot.
And in case anyone wants to get cheeky: yes, a custom snapshot
can put another custom snapshot in its
You can layer these things up as many layers as you'd like. Have
Just to summarize, Stack supports three different kinds of
- A compiler name
- A Stackage snapshot name (LTS or Nightly)
- A custom snapshot
Where's the global
Global packages are those packages which are shipped with GHC
itself (or at least end up in its global database). A funny
question I bet many people never thought about is: where are global
packages defined? Are they in the snapshot, or does Stack look them
up from GHC itself? There are advantages both way:
- If it comes from the snapshot, it may be wrong. For example, do
you have the
- If it comes from GHC itself, then Stack can't do a
init without first installing every GHC version it wants to
test compatibility with.
Previously, global information came from the Stackage snapshots.
But both because of the "possibly incorrect" reason, and because it
would be a royal pain to define all of the global packages in each
custom snapshot file, Stack now does something different:
- When you use
stack init or
(which implicitly calls
stack init), Stack will rely
on global hints present in a snapshot as a good guess about
which packages GHC provides.
- When you're ready to start building your code against a
specific GHC version, Stack will query GHC's global database.
between stack.yaml and custom snapshot
You may be conflicted about whether you should add extra
dependencies into a stack.yaml file (as you've probably done until
now) or define a custom snapshot. My answers may change over time
with experience, but here are some good guesses:
- If you're making lots of rapid iterations over the dependencies
(e.g., testing five different versions of a package), use
extra-deps. You will avoid Stack having to create separate snapshot
databases and do a bunch of "copying precompiled package"
- If you're using the same modified sets in many projects, use a
- If you're changing a package deep in the dependency hierarchy
stm), make a custom snapshot
to try and save work (though I'm not sure if my logic is sound
Things we've lost
Besides the potential for some kind of breaking change in
behavior to have crept in (NOTE please help me by testing
against your projects!), the only lost features I'm aware of
- The custom snapshot configuration has changed significantly.
Since very few people are using it, and it was always an
experimental feature, this is probably an acceptable
- We no longer have support for subdirs on tarballs and index
packages. This is because the feature just doesn't make sense, and
was only accidentally present. (We do have support for
filepath locations, which is equally silly, but was also trivial to
add support for and made one of the integration tests happy.)
Here's my biggest feature for consideration: making the project
packages only support filepaths. I can think of no logical
case where we'd want to support HTTP(S) or Git repos as "project
packages" (meaning things that run tests, for instance). In my
packages key would accept a list of
extra-deps key would accept exactly what it
- Everyone could transition from their current
true syntax over the next few versions before we remove the
I'm not normally in favor of breaking backwards compatibility in
Stack, but miscategorized extra deps has resulted in much
confusion, so I'd be happy to see it go, even if it requires
stack.yaml files over time.
Do you like this blog post and need help with DevOps, Rust or functional programming? Contact us.