13 Jul 2017
NOTE This blog post made the rounds last week before the branch was actually merged and the post was still on a review server. I'm officially publishing it as the pull request is now merged.
There is a collection of features in Stack that have been added in bit by bit, as opposed to being designed into a cohesive whole from the start. The features work, but could be a bit better. We've known for a while that, instead of putting in place strategic fixes, a more general refactoring of the core dependency management logic was in order. I'm happy to announce that these changes have landed in the master branch, and will be part of the next major release of Stack.
I'd like to motivate the limitations in Stack that encouraged this change, discuss the new system, mention some potential future changes, and share a few thoughts on the (very pleasant) Haskell refactoring process itself.
NOTE These features have not currently been released, so don't
try using them in a stable Stack executable. If you'd like to test
them out (and I'd certainly appreciate the extra testing), you can run
stack upgrade --git to build a Stack executable from the master
Consider this fairly standard snippet of a stack.yaml file:
resolver: lts-8.12 packages: - ./site1 - ./site2 - location: git: https://github.com/yesodweb/yesod commit: 7038ae6317cb3fe4853597633ba7a40804ca9a46 extra-dep: true subdirs: - yesod-core - yesod-bin extra-deps: - html-conduit-184.108.40.206
This is leveraging a number of features of Stack right off the bat:
- Using an LTS Haskell snapshot to capture a consistent set of dependencies
- Specify multiple project packages, in this case
- Specify an extra dependency from a Git repository.
- Specify multiple subdirectories for a Git repository to directly support "megarepos".
- Specify an extra dependency from the upstream package index (which is Hackage unless you've done something weird).
This is great, but there's a bit of pain involved in this:
- My personal pet peeve: that
extra-dep: truefor the Git repo. Because of how features were added, we included Git repos together with "project packages," and then added a hack to explicitly state that they should be treated as dependencies (so that
stack test, for example, won't build their dependencies). This feels weird.
- We also have this
extra-depsstanza, which accepts package name/version combos, but doesn't accept Git repos (or HTTP(S) tarballs, which are also supported in the
- As Hackage cabal file revisions become more common, it's taking away
a level of reproducibility that
extra-depscannot specify the exact revision of a package we'd like to use.
subdirsfeature is nice, but it's weird that it's not more directly connected to the Git repo information (notice how it's a level up).
- I've specified
yesod-binas an extra-dep, which provides an executable named
yesod. Ideally, if one of my packages specified a dependency on that executable, Stack would automatically build the
yesod-binpackage. While this logic works for LTS Haskell and Stackage Nightly snapshots, it doesn't work for these extra-deps.
- Specifying dependencies like this inside the
stack.yamlfile makes them local to just the project I'm working on. There are advantages to this approach regarding disk space (which I won't get into here), but there's a big downside: I can't share precompiled libraries between projects that are defined this way. I'd like to be able to recapture that sharing ability.
- And more generally: why can't I just define this customized version
lts-8.12and share it among multiple projects, possibly even from an immutable URL?
As you can see, the problems aren't insurmountable, but they are annoyances, and they seem to overlap quite a bit.
Let's rewrite that
stack.yaml file to be a little bit more
resolver: lts-8.12 packages: - ./site1 - ./site2 extra-deps: - html-conduit-220.127.116.11@sha256:de32ca4d6df94a7c027a11db1b2e32ef1a7ccfe0565923f24528613ade821343 - git: https://github.com/yesodweb/yesod commit: 7038ae6317cb3fe4853597633ba7a40804ca9a46 subdirs: - yesod-core - yesod-bin
The first thing to notice is that the
packages value is now just a
list of the actual code in our project, not the dependencies.
Next, we still have
html-conduit-18.104.22.168 coming from Hackage. But we
have this funny
@sha256:... bit at the end. This is a hash of the
cabal file contents we want to use. This gives us much stronger
guarantees of reproducibility than we had previously. Instead of
getting whatever most recent version happens to be available, you'll
get an exact cabal file. This feature has been present for a while in
Stackage snapshots, but hasn't been accessible for local dependencies.
Next, we've moved the Git repo information out of
packages and into
extra-deps where it logically belongs. We also no longer need that
location key. We had that so that we could also define
subdirs keys. We now instead put the
next to the
commit keys, and don't need
extra-dep at all
(since it's implied by being within
Behind the scenes, the code managing these things has changed
drastically. Most importantly for our discussion here, Stack now uses
the same code paths for loading up snapshots and loading up package
information within the
stack.yaml file. In addition to just being a
good practice for keeping us sane, this means that build tool
detection now works for project packages and dependencies too.
This answers a good deal of our points above (hold off for the last two when we get to custom snapshots).
Four package locations
That probably seemed like a bit of a jumble, so let's start over. Every package has a location, which tells Stack where to get it from. Stack supports for different package locations:
- Local file path, like
- Package index, specified version a package name, version number, and optional cabal file hash. You're probably wondering "why not just call it Hackage?" The reason is that you can extend your list of package indices to augment or override Hackage (such as to use a corporate package repository). That's why we use the general term.
- Git or Mercurial repo, specified by a repo URL, a commit, and an optional list of subdirectories within the repo to look for packages. If omitted, Stack will look in the root of the repo.
- HTTP(S) URL, which isn't demonstrated above, but which is just a
http(s)://URL pointing to a tarball.
All four of these have been supported in Stack since (almost) its inception. The differences now are that:
extra-depsnow supports all four forms
packagesstill supports local file paths, Git repos, and HTTP(S) URLs, but for the latter two requires you to explicitly state
false. We'll discuss this a bit more below. There are two reasons package index location isn't supported here:
- It wasn't supported previously, and is an illogical thing to do: you would never have a situation of working on a package pulled from the index. If you want to do that, you should probably clone the source of that package (such as with a Git submodule).
- It could introduce ambiguities in parsing between a package
index and a filepath (imagine you have a local directory
foo-1.2.3). We've allowed this ambiguity to exist in
extra-deps; if you have such a filepath, you can always preface it with
This is all well and good, but isn't much more than a cosmetic improvement (though, in my opinion, it's a very nice cosmetic improvement). But this gets much cooler with custom snapshots.
Stack has had some support for custom snapshots for a while, but it's never been fully implemented, since we've been waiting for this extensible snapshot concept to land. Since most people aren't very familiar with custom snapshots today, I'm not going to compare and contrast, but instead just jump in to explaining how they work now.
Stack configurations always discuss a resolver, which specifies a GHC version, a set of additional packages, build flags, and some other pieces of metadata. You've probably seen a few kinds of resolvers until now:
lts-8.12, using LTS Haskell
nightly-2017-07-01, using Stackage Nightly
ghc-8.0.2, using a specific GHC version without any extra packages available
Custom snapshots answer a simple question: what if I want to define my own snapshot which isn't LTS Haskell or Stackage Nightly? And that's really all they are: a format for defining your own snapshots like Stackage does. However, they've got a number of cool features that Stackage snapshots don't:
- They are extensible (thus the whole blog post name): you can define a parent snapshot for any snapshot and inherit its settings.
- You can use all four types of package locations when defining a
custom snapshot. Say you have a package that isn't ready to be
released to Hackage, or is only for your internal team to use. You
can stick it in a tarball on a webserver, or in a Git repo, and
refer to it just as you would from
- Unlike defining packages in
stack.yaml, packages built in a custom snapshot can be shared in the package cache and reused between projects.
Let's see how this would modify our
stack.yaml from above. First,
I'm going to define a
resolver: lts-8.12 name: my-snapshot # For user display only packages: - html-conduit-22.214.171.124@sha256:de32ca4d6df94a7c027a11db1b2e32ef1a7ccfe0565923f24528613ade821343 - git: https://github.com/yesodweb/yesod commit: 7038ae6317cb3fe4853597633ba7a40804ca9a46 subdirs: - yesod-core - yesod-bin
Notice how I've kept the same
resolver value here. What I'm stating
is that I'd like my snapshot to start off with the same GHC version
and package set defined in
lts-8.12, and then add new
packages. Next, I've copied my entire
extra-deps section in here,
and called it
packages instead (since these are the packages that
actually make up the snapshot, not some extra dependencies added on
Note that, because a custom snapshot is intended to contain immutable package data, it does not support local filepaths as package location, as these are expected to change over time.
resolver: my-snapshot.yaml packages: - ./site1 - ./site2
Instead of a Stackage snapshot or compiler version, my
gives the path to the snapshot config file. This can be a file path,
or an HTTP(S) URL. The
packages section stayed the same, but my
extra-deps is no longer necessary: all of my dependencies are now
defined within the custom snapshot.
And in case anyone wants to get cheeky: yes, a custom snapshot can put
another custom snapshot in its
resolver field. You can layer these
things up as many layers as you'd like. Have fun!
Just to summarize, Stack supports three different kinds of resolver values:
- A compiler name
- A Stackage snapshot name (LTS or Nightly)
- A custom snapshot
Where's the global package information?
Global packages are those packages which are shipped with GHC itself (or at least end up in its global database). A funny question I bet many people never thought about is: where are global packages defined? Are they in the snapshot, or does Stack look them up from GHC itself? There are advantages both way:
- If it comes from the snapshot, it may be wrong. For example, do you
- If it comes from GHC itself, then Stack can't do a
stack initwithout first installing every GHC version it wants to test compatibility with.
Previously, global information came from the Stackage snapshots. But both because of the "possibly incorrect" reason, and because it would be a royal pain to define all of the global packages in each custom snapshot file, Stack now does something different:
- When you use
stack new(which implicitly calls
stack init), Stack will rely on global hints present in a snapshot as a good guess about which packages GHC provides.
- When you're ready to start building your code against a specific GHC version, Stack will query GHC's global database.
Choosing between stack.yaml and custom snapshot
You may be conflicted about whether you should add extra dependencies into a stack.yaml file (as you've probably done until now) or define a custom snapshot. My answers may change over time with experience, but here are some good guesses:
- If you're making lots of rapid iterations over the dependencies (e.g., testing five different versions of a package), use extra-deps. You will avoid Stack having to create separate snapshot databases and do a bunch of "copying precompiled package" stuff.
- If you're using the same modified sets in many projects, use a custom snapshot
- If you're changing a package deep in the dependency hierarchy (like
stm), make a custom snapshot to try and save work (though I'm not sure if my logic is sound here)
Things we've lost
Besides the potential for some kind of breaking change in behavior to have crept in (NOTE please help me by testing against your projects!), the only lost features I'm aware of are:
- The custom snapshot configuration has changed significantly. Since very few people are using it, and it was always an experimental feature, this is probably an acceptable trade-off.
- We no longer have support for subdirs on tarballs and index packages. This is because the feature just doesn't make sense, and was only accidentally present. (We do have support for filepath locations, which is equally silly, but was also trivial to add support for and made one of the integration tests happy.)
Here's my biggest feature for consideration: making the project packages only support filepaths. I can think of no logical case where we'd want to support HTTP(S) or Git repos as "project packages" (meaning things that run tests, for instance). In my ideal world:
packageskey would accept a list of filepaths
extra-depskey would accept exactly what it does now
- Everyone could transition from their current
extra-dep: truesyntax over the next few versions before we remove the old support
I'm not normally in favor of breaking backwards compatibility in
Stack, but miscategorized extra deps has resulted in much confusion,
so I'd be happy to see it go, even if it requires rewriting
stack.yaml files over time.