NOTE This blog post made the rounds last week before the branch was actually merged and the post was still on a review server. I'm officially publishing it as the pull request is now merged.
There is a collection of features in Stack that have been added in bit by bit, as opposed to being designed into a cohesive whole from the start. The features work, but could be a bit better. We've known for a while that, instead of putting in place strategic fixes, a more general refactoring of the core dependency management logic was in order. I'm happy to announce that these changes have landed in the master branch, and will be part of the next major release of Stack.
I'd like to motivate the limitations in Stack that encouraged this change, discuss the new system, mention some potential future changes, and share a few thoughts on the (very pleasant) Haskell refactoring process itself.
NOTE These features have not currently been released, so don't
try using them in a stable Stack executable. If you'd like to test
them out (and I'd certainly appreciate the extra testing), you can run
stack upgrade --git to build a Stack executable from the master
Consider this fairly standard snippet of a stack.yaml file:
resolver: lts-8.12 packages: - ./site1 - ./site2 - location: git: https://github.com/yesodweb/yesod commit: 7038ae6317cb3fe4853597633ba7a40804ca9a46 extra-dep: true subdirs: - yesod-core - yesod-bin extra-deps: - html-conduit-22.214.171.124
This is leveraging a number of features of Stack right off the bat:
This is great, but there's a bit of pain involved in this:
extra-dep: truefor the Git repo. Because of how features were added, we included Git repos together with "project packages," and then added a hack to explicitly state that they should be treated as dependencies (so that
stack test, for example, won't build their dependencies). This feels weird.
extra-depsstanza, which accepts package name/version combos, but doesn't accept Git repos (or HTTP(S) tarballs, which are also supported in the
extra-depscannot specify the exact revision of a package we'd like to use.
subdirsfeature is nice, but it's weird that it's not more directly connected to the Git repo information (notice how it's a level up).
yesod-binas an extra-dep, which provides an executable named
yesod. Ideally, if one of my packages specified a dependency on that executable, Stack would automatically build the
yesod-binpackage. While this logic works for LTS Haskell and Stackage Nightly snapshots, it doesn't work for these extra-deps.
stack.yamlfile makes them local to just the project I'm working on. There are advantages to this approach regarding disk space (which I won't get into here), but there's a big downside: I can't share precompiled libraries between projects that are defined this way. I'd like to be able to recapture that sharing ability.
lts-8.12and share it among multiple projects, possibly even from an immutable URL?
As you can see, the problems aren't insurmountable, but they are annoyances, and they seem to overlap quite a bit.
Let's rewrite that
stack.yaml file to be a little bit more
resolver: lts-8.12 packages: - ./site1 - ./site2 extra-deps: - html-conduit-126.96.36.199@sha256:de32ca4d6df94a7c027a11db1b2e32ef1a7ccfe0565923f24528613ade821343 - git: https://github.com/yesodweb/yesod commit: 7038ae6317cb3fe4853597633ba7a40804ca9a46 subdirs: - yesod-core - yesod-bin
The first thing to notice is that the
packages value is now just a
list of the actual code in our project, not the dependencies.
Next, we still have
html-conduit-188.8.131.52 coming from Hackage. But we
have this funny
@sha256:... bit at the end. This is a hash of the
cabal file contents we want to use. This gives us much stronger
guarantees of reproducibility than we had previously. Instead of
getting whatever most recent version happens to be available, you'll
get an exact cabal file. This feature has been present for a while in
Stackage snapshots, but hasn't been accessible for local dependencies.
Next, we've moved the Git repo information out of
packages and into
extra-deps where it logically belongs. We also no longer need that
location key. We had that so that we could also define
subdirs keys. We now instead put the
next to the
commit keys, and don't need
extra-dep at all
(since it's implied by being within
Behind the scenes, the code managing these things has changed
drastically. Most importantly for our discussion here, Stack now uses
the same code paths for loading up snapshots and loading up package
information within the
stack.yaml file. In addition to just being a
good practice for keeping us sane, this means that build tool
detection now works for project packages and dependencies too.
This answers a good deal of our points above (hold off for the last two when we get to custom snapshots).
That probably seemed like a bit of a jumble, so let's start over. Every package has a location, which tells Stack where to get it from. Stack supports for different package locations:
http(s)://URL pointing to a tarball.
All four of these have been supported in Stack since (almost) its inception. The differences now are that:
extra-depsnow supports all four forms
packages still supports local file paths, Git repos, and HTTP(S)
URLs, but for the latter two requires you to explicitly state
extra-dep: as either
false. We'll discuss this a bit
more below. There are two reasons package index location isn't
foo-1.2.3). We've allowed this ambiguity to exist in
extra-deps; if you have such a filepath, you can always preface it with
This is all well and good, but isn't much more than a cosmetic improvement (though, in my opinion, it's a very nice cosmetic improvement). But this gets much cooler with custom snapshots.
Stack has had some support for custom snapshots for a while, but it's never been fully implemented, since we've been waiting for this extensible snapshot concept to land. Since most people aren't very familiar with custom snapshots today, I'm not going to compare and contrast, but instead just jump in to explaining how they work now.
Stack configurations always discuss a resolver, which specifies a GHC version, a set of additional packages, build flags, and some other pieces of metadata. You've probably seen a few kinds of resolvers until now:
lts-8.12, using LTS Haskell
nightly-2017-07-01, using Stackage Nightly
ghc-8.0.2, using a specific GHC version without any extra packages available
Custom snapshots answer a simple question: what if I want to define my own snapshot which isn't LTS Haskell or Stackage Nightly? And that's really all they are: a format for defining your own snapshots like Stackage does. However, they've got a number of cool features that Stackage snapshots don't:
stack.yaml, packages built in a custom snapshot can be shared in the package cache and reused between projects.
Let's see how this would modify our
stack.yaml from above. First,
I'm going to define a
resolver: lts-8.12 name: my-snapshot # For user display only packages: - html-conduit-184.108.40.206@sha256:de32ca4d6df94a7c027a11db1b2e32ef1a7ccfe0565923f24528613ade821343 - git: https://github.com/yesodweb/yesod commit: 7038ae6317cb3fe4853597633ba7a40804ca9a46 subdirs: - yesod-core - yesod-bin
Notice how I've kept the same
resolver value here. What I'm stating
is that I'd like my snapshot to start off with the same GHC version
and package set defined in
lts-8.12, and then add new
packages. Next, I've copied my entire
extra-deps section in here,
and called it
packages instead (since these are the packages that
actually make up the snapshot, not some extra dependencies added on
Note that, because a custom snapshot is intended to contain immutable package data, it does not support local filepaths as package location, as these are expected to change over time.
resolver: my-snapshot.yaml packages: - ./site1 - ./site2
Instead of a Stackage snapshot or compiler version, my
gives the path to the snapshot config file. This can be a file path,
or an HTTP(S) URL. The
packages section stayed the same, but my
extra-deps is no longer necessary: all of my dependencies are now
defined within the custom snapshot.
And in case anyone wants to get cheeky: yes, a custom snapshot can put
another custom snapshot in its
resolver field. You can layer these
things up as many layers as you'd like. Have fun!
Just to summarize, Stack supports three different kinds of resolver values:
Global packages are those packages which are shipped with GHC itself (or at least end up in its global database). A funny question I bet many people never thought about is: where are global packages defined? Are they in the snapshot, or does Stack look them up from GHC itself? There are advantages both way:
stack initwithout first installing every GHC version it wants to test compatibility with.
Previously, global information came from the Stackage snapshots. But both because of the "possibly incorrect" reason, and because it would be a royal pain to define all of the global packages in each custom snapshot file, Stack now does something different:
stack new(which implicitly calls
stack init), Stack will rely on global hints present in a snapshot as a good guess about which packages GHC provides.
You may be conflicted about whether you should add extra dependencies into a stack.yaml file (as you've probably done until now) or define a custom snapshot. My answers may change over time with experience, but here are some good guesses:
stm), make a custom snapshot to try and save work (though I'm not sure if my logic is sound here)
Besides the potential for some kind of breaking change in behavior to have crept in (NOTE please help me by testing against your projects!), the only lost features I'm aware of are:
Here's my biggest feature for consideration: making the project packages only support filepaths. I can think of no logical case where we'd want to support HTTP(S) or Git repos as "project packages" (meaning things that run tests, for instance). In my ideal world:
packageskey would accept a list of filepaths
extra-depskey would accept exactly what it does now
extra-dep: truesyntax over the next few versions before we remove the old support
I'm not normally in favor of breaking backwards compatibility in
Stack, but miscategorized extra deps has resulted in much confusion,
so I'd be happy to see it go, even if it requires rewriting
stack.yaml files over time.