Collect in Rust, traverse in Haskell and Scala

By Michael Snoyman Published October 6, 2020
❮❮ Page 1 of 13 ❯❯
Rust: Of course it compiles, right?
Where Rust fits in your organization
Avoiding duplicating strings in Rust
Using Rust for DevOps tooling
HTTP status codes with async Rust
DevOps: Unifying Dev, Ops, and QA
Different levels of async in Rust
DevOps for (Skeptical) Developers
Our history with containerization
Our history with containerization
By FP Complete Team
FP Complete Ranked #1547 Among Inc 5000 Companies!
Cloud Deployment Models: Advantages and Disadvantages
Understanding cloud auth
Understanding cloud auth
By Michael Snoyman
Understanding DevOps Roles and Responsibilities
Preparing for Upcoming Cloud Computing Trends
Cloud preparation checklist
Cloud preparation checklist
By Sibi Prabakaran
Rust at FP Complete, 2020 update
DevOps Security and Privacy Strategies
Webinar Review: Learn Rapid DevOps Success
How To Successfully Onboard A Remote Workforce
16 Smart Project-Management Strategies Every Tech Leader Can Use
Reducing Maintenance Costs With Functional Programming
❮❮ Page 1 of 13 ❯❯
  
  [
  {
    "name": "devops",
    "slug": "devops",
    "permalink": "https://www.fpcomplete.com/categories/devops/",
    "pages": [
      {
        "relative_path": "blog/rust-for-devops-tooling.md",
        "content": "<p>A beginner's guide to writing your DevOps tools in Rust.</p>\n<h2 id=\"introduction\">Introduction</h2>\n<p>In this blog post we'll cover some basic DevOps use cases for Rust and why \nyou would want to use it.\nAs part of this, we'll also cover a few common libraries you will likely use\nin a Rust-based DevOps tool for AWS.</p>\n<p>If you're already familiar with writing DevOps tools in other languages,\nthis post will explain why you should try Rust.</p>\n<p>We'll cover why Rust is a particularly good choice of language to write your DevOps\ntooling and critical cloud infrastructure software in.\nAnd we'll also walk through a small demo DevOps tool written in Rust. \nThis project will be geared towards helping someone new to the language ecosystem \nget familiar with the Rust project structure.</p>\n<p>If you're brand new to Rust, and are interested in learning the language, you may want to start off with our <a href=\"https://www.fpcomplete.com/rust/crash-course/\">Rust Crash Course eBook</a>.</p>\n<h2 id=\"what-makes-the-rust-language-unique\">What Makes the Rust Language Unique</h2>\n<blockquote>\n<p>Rust is a systems programming language focused on three goals: safety, speed, \nand concurrency. It maintains these goals without having a garbage collector, \nmaking it a useful language for a number of use cases other languages aren’t \ngood at: embedding in other languages, programs with specific space and time \nrequirements, and writing low-level code, like device drivers and operating systems. </p>\n</blockquote>\n<p><em>The Rust Book (first edition)</em></p>\n<p>Rust was initially created by Mozilla and has since gained widespread adoption and\nsupport. As the quote from the Rust book alludes to, it was designed to fill the \nsame space that C++ or C would (in that it doesn’t have a garbage collector or a runtime).\nBut Rust also incorporates zero-cost abstractions and many concepts that you would\nexpect in a higher level language (like Go or Haskell).\nFor that, and many other reasons, Rust's uses have expanded well beyond that\noriginal space as low level safe systems language.</p>\n<p>Rust's ownership system is extremely useful in efforts to write correct and \nresource efficient code. Ownership is one of the killer features of the Rust \nlanguage and helps programmers catch classes of resource errors at compile time \nthat other languages miss or ignore.</p>\n<p>Rust is an extremely performant and efficient language, comparable to the speeds \nyou see with idiomatic everyday C or C++.\nAnd since there isn’t a garbage collector in Rust, it’s a lot easier to get \npredictable deterministic performance.</p>\n<h2 id=\"rust-and-devops\">Rust and DevOps</h2>\n<p>What makes Rust unique also makes it very useful for areas stemming from robots \nto rocketry, but are those qualities relevant for DevOps?\nDo we care if we have efficient executables or fine grained control over \nresources, or is Rust a bit overkill for what we typically need in DevOps?</p>\n<p><em>Yes and no</em></p>\n<p>Rust is clearly useful for situations where performance is crucial and actions \nneed to occur in a deterministic and consistent way. That obviously translates to \nlow-level places where previously C and C++ were the only game in town. \nIn those situations, before Rust, people simply had to accept the inherent risk and \nadditional development costs of working on a large code base in those languages.\nRust now allows us to operate in those areas but without the risk that C and C++\ncan add.</p>\n<p>But with DevOps and infrastructure programming we aren't constrained by those \nrequirements. For DevOps we've been able to choose from languages like Go, Python, \nor Haskell because we're not strictly limited by the use case to languages without \ngarbage collectors. Since we can reach for other languages you might argue \nthat using Rust is a bit overkill, but let's go over a few points to counter this.</p>\n<h3 id=\"why-you-would-want-to-write-your-devops-tools-in-rust\">Why you would want to write your DevOps tools in Rust</h3>\n<ul>\n<li>Small executables relative to other options like Go or Java</li>\n<li>Easy to port across different OS targets</li>\n<li>Efficient with resources (which helps cut down on your AWS bill) </li>\n<li>One of the fastest languages (even when compared to C)</li>\n<li>Zero cost abstractions - Rust is a low level performant language which also\ngives the us benefits of a high level language with its generics and abstractions.</li>\n</ul>\n<p>To elaborate on some of these points a bit further:</p>\n<h4 id=\"os-targets-and-cross-compiling-rust-for-different-architectures\">OS targets and Cross Compiling Rust for different architectures</h4>\n<p>For DevOps it's also worth mentioning the (relative) ease with which you can \nport your Rust code across different architectures and different OS's. </p>\n<p>Using the official Rust toolchain installer <code>rustup</code>, it's easy to get the \nstandard library for your target platform.\nRust <a href=\"https://doc.rust-lang.org/nightly/rustc/platform-support.html\">supports a great number of platforms</a>\nwith different tiers of support.\nThe docs for the <code>rustup</code> tool has <a href=\"https://rust-lang.github.io/rustup/cross-compilation.html\">a section</a>\ncovering how you can access pre-compiled artifacts for various architectures.\nTo install the target platform for an architecture (other than the host platform which is installed by default)\nyou simply need to run <code>rustup target add</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">$ rustup target add x86_64-pc-windows-msvc \ninfo: downloading component &#39;rust-std&#39; for &#39;x86_64-pc-windows-msvc&#39;\ninfo: installing component &#39;rust-std&#39; for &#39;x86_64-pc-windows-msvc&#39;\n</span></code></pre>\n<p>Cross compilation is already built into the Rust compiler by default. \nOnce the <code>x86_64-pc-windows-msvc</code> target is installed you can build for Windows \nwith the <code>cargo</code> build tool using the <code>--target</code> flag:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">cargo build --target=x86_64-pc-windows-msvc\n</span></code></pre>\n<p>(the default target is always the host architecture)</p>\n<p>If one of your dependencies links to a native (i.e. non-Rust) library, you will\nneed to make sure that those cross compile as well. Doing <code>rustup target add</code>\nonly installs the Rust standard library for that target. However for the other \ntools that are often needed when cross-compiling, there is the handy\n<a href=\"https://github.com/rust-embedded/cross\">github.com/rust-embedded/cross</a> tool.\nThis is essentially a wrapper around cargo which does all cross compilation in \ndocker images that have all the necessary bits (linkers) and pieces installed.</p>\n<h4 id=\"small-executables\">Small Executables</h4>\n<p>A key unique feature of Rust is that it doesn't need a runtime or a garbage collector.\nCompare this to languages like Python or Haskell: with Rust the lack of any runtime\ndependencies (Python), or system libraries (as with Haskell) is a huge advantage \nfor portability.</p>\n<p>For practical purposes, as far as DevOps is concerned, this portability means \nthat Rust executables are much easier to deploy than scripts.\nWith Rust, compared to Python or Bash, we don't need to set up the environment for \nour code ahead of time. This frees us up from having to worry if the runtime \ndependencies for the language are set up.</p>\n<p>In addition to that, with Rust you're able to produce 100% static executables for \nLinux using the MUSL libc (and by default Rust will statically link all Rust code). \nThis means that you can deploy your Rust DevOps tool's binaries across your Linux \nservers without having to worry if the correct <code>libc</code> or other libraries were \ninstalled beforehand.</p>\n<p>Creating static executables for Rust is simple. As we discussed before, when discussing\ndifferent OS targets, it's easy with Rust to switch the target you're building against.\nTo compile static executables for the Linux MUSL target all you need to do is add \nthe <code>musl</code> target with:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">$ rustup target add x86_64-unknown-linux-musl\n</span></code></pre>\n<p>Then you can using this new target to build your Rust project as a fully static \nexecutable with:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">$ cargo build --target x86_64-unknown-linux-musl\n</span></code></pre>\n<p>As a result of not having a runtime or a garbage collector, Rust executables \ncan be extremely small. For example, there is a common DevOps tool called \nCredStash that was originally written in Python but has since been \nported to Go (GCredStash) and now Rust (RuCredStash).</p>\n<p>Comparing the executable sizes of the Rust versus Go implementations of CredStash,\nthe Rust executable is nearly a quarter of the size of the Go variant. </p>\n<table><thead><tr><th>Implementation</th><th>Executable Size</th></tr></thead><tbody>\n<tr><td>Rust CredStash: (RuCredStash Linux amd64)</td><td>3.3 MB</td></tr>\n<tr><td>Go CredStash: (GCredStash Linux amd64 v0.3.5)</td><td>11.7 MB</td></tr>\n</tbody></table>\n<p>Project links:</p>\n<ul>\n<li><a href=\"https://github.com/psibi/rucredstash\">github.com/psibi/rucredstash</a></li>\n<li><a href=\"https://github.com/winebarrel/gcredstash\">github.com/winebarrel/gcredstash</a></li>\n</ul>\n<p>This is by no means a perfect comparison, and 8 MB may not seem like a lot, but\nconsider the advantage automatically of having executables that are a quarter of the \nsize you would typically expect. </p>\n<p>This cuts down on the size your Docker images, AWS AMI's, or Azure VM images need\nto be - and that helps speed up the time it takes to spin up new deployments.</p>\n<p>With a tool of this size, having an executable that is 75% smaller than it \nwould be otherwise is not immediately apparent. On this scale the difference, 8 MB,\nis still quite cheap.\nBut with larger tools (or collections of tools and Rust based software) the benefits\nadd up and the difference begins to be a practical and worthwhile consideration.</p>\n<p>The Rust implementation was also not strictly written with the resulting size of \nthe executable in mind. So if executable size was even more important of a \nfactor other changes could be made - but that's beyond the scope of this post.</p>\n<h4 id=\"rust-is-fast\">Rust is fast</h4>\n<p>Rust is very fast even for common idiomatic everyday Rust code. And not only that\nit's arguably easier to work with than with C and C++ and catch errors in your \ncode.</p>\n<p>For the Fortunes benchmark (which exercises the ORM, \ndatabase connectivity, dynamic-size collections, sorting, server-side templates, \nXSS countermeasures, and character encoding) Rust is second and third, only lagging \nbehind the first place C++ based framework by 4 percent. </p>\n<img src=\"/images/blog/techempower-benchmarks-round-19-fortunes.png\" style=\"max-width:95%\">\n<p>In the benchmark for database access for a single query Rust is first and second:</p>\n<img src=\"/images/blog/techempower-benchmarks-round-19-single-query.png\" style=\"max-width:95%\">\n<p>And in a composite of all the benchmarks Rust based frameworks are second and third place.</p>\n<img src=\"/images/blog/techempower-benchmarks-round-19-composite.png\" style=\"max-width:95%\">\n<p>Of course language and framework benchmarks are not real life, however this is \nstill a fair comparison of the languages as they relate to others (within the context \nand the focus of the benchmark).</p>\n<p>Source: <a href=\"https://www.techempower.com/benchmarks/\">https://www.techempower.com/benchmarks</a></p>\n<h3 id=\"why-would-you-not-want-to-write-your-devops-tools-in-rust\">Why would you not want to write your DevOps tools in Rust?</h3>\n<p>For medium to large projects, it’s important to have a type system and compile \ntime checks like those in Rust versus what you would find in something like Python\nor Bash.\nThe latter languages let you get away with things far more readily. This makes \ndevelopment much &quot;faster&quot; in one sense.</p>\n<p>Certain situations, especially those with involving small project codebases, would \nbenefit more from using an interpreted language. In these cases, being able to quickly \nchange pieces of the code without needing to re-compile and re-deploy the project\noutweighs the benefits (in terms of safety, execution speed, and portability)\nthat languages like Rust bring. </p>\n<p>Working with and iterating on a Rust codebase in those circumstances, with frequent\nbut small codebases changes, would be needlessly time-consuming\nIf you have a small codebase with few or no runtime dependencies, then it wouldn't\nbe worth it to use Rust.</p>\n<h2 id=\"demo-devops-project-for-aws\">Demo DevOps Project for AWS</h2>\n<p>We'll briefly cover some of the libraries typically used for an AWS focused \nDevOps tool in a walk-through of a small demo Rust project here. \nThis aims to provide a small example that uses some of the libraries you'll likely\nwant if you’re writing a CLI based DevOps tool in Rust. Specifically for this \nexample we'll show a tool that does some basic operations against AWS S3 \n(creating new buckets, adding files to buckets, listing the contents of buckets).</p>\n<h3 id=\"project-structure\">Project structure</h3>\n<p>For AWS integration we're going to utilize the <a href=\"https://www.rusoto.org/\">Rusoto</a> library.\nSpecifically for our modest demo Rust DevOps tools we're going to pull in the \n<a href=\"https://docs.rs/rusoto_core/0.45.0/rusoto_core/\">rusoto_core</a> and the \n<a href=\"https://docs.rs/rusoto_s3/0.45.0/rusoto_s3/\">rusoto_s3</a> crates (in Rust a <em>crate</em>\nis akin to a library or package).</p>\n<p>We're also going to use the <a href=\"https://docs.rs/structopt/0.3.16/structopt/\">structopt</a> crate\nfor our CLI options. This is a handy, batteries included CLI library that makes \nit easy to create a CLI interface around a Rust struct. </p>\n<p>The tool operates by matching the CLI option and arguments the user passes in \nwith a <a href=\"https://github.com/fpco/rust-aws-devops/blob/54d6cfa4bb7a9a15c2db52976f2b7057431e0c5e/src/main.rs#L211\"><code>match</code> expression</a>.</p>\n<p>We can then use this to match on that part of the CLI option struct we've defined \nand call the appropriate functions for that option.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">match</span><span style=\"color:#657b83;\"> opt {\n    Opt::Create { bucket: bucket_name } </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n        </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Attempting to create a bucket called: </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, bucket_name);\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> demo = S3Demo::new(bucket_name);\n        </span><span style=\"color:#859900;\">create_demo_bucket</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">demo);\n    },\n</span></code></pre>\n<p>This matches on the <a href=\"https://github.com/fpco/rust-aws-devops/blob/54d6cfa4bb7a9a15c2db52976f2b7057431e0c5e/src/main.rs#L182\"><code>Create</code></a>\nvariant of the <code>Opt</code> enum. </p>\n<p>We then use <code>S3Demo::new(bucket_name)</code> to create a new <code>S3Client</code> which we can\nuse in the standalone <code>create_demo_bucket</code> function that we've defined \nwhich will create a new S3 bucket.</p>\n<p>The tool is fairly simple with most of the code located in \n<a href=\"https://github.com/fpco/rust-aws-devops/blob/54d6cfa4bb7a9a15c2db52976f2b7057431e0c5e/src/main.rs\">src/main.rs</a></p>\n<h3 id=\"building-the-rust-project\">Building the Rust project</h3>\n<p>Before you build the code in this project, you will need to install Rust. \nPlease follow <a href=\"https://www.rust-lang.org/tools/install\">the official install instructions here</a>.</p>\n<p>The default build tool for Rust is called Cargo. It's worth getting familiar \nwith <a href=\"https://doc.rust-lang.org/cargo/guide/\">the docs for Cargo</a>\nbut here's a quick overview for building the project.</p>\n<p>To build the project run the following from the root of the \n<a href=\"https://github.com/fpco/rust-aws-devops\">git repo</a>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">cargo build\n</span></code></pre>\n<p>You can then use <code>cargo run</code> to run the code or execute the code directly\nwith <code>./target/debug/rust-aws-devops</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">$ ./target/debug/rust-aws-devops \n\nRunning tool\nRustAWSDevops 0.1.0\nMike McGirr &lt;mike@fpcomplete.com&gt;\n\nUSAGE:\n    rust-aws-devops &lt;SUBCOMMAND&gt;\n\nFLAGS:\n    -h, --help       Prints help information\n    -V, --version    Prints version information\n\nSUBCOMMANDS:\n    add-object       Add the specified file to the bucket\n    create           Create a new bucket with the given name\n    delete           Try to delete the bucket with the given name\n    delete-object    Remove the specified object from the bucket\n    help             Prints this message or the help of the given subcommand(s)\n    list             Try to find the bucket with the given name and list its objects``\n</span></code></pre>\n<p>Which will output the nice CLI help output automatically created for us \nby <code>structopt</code>.</p>\n<p>If you're ready to build a release version (with optimizations turn on which \nwill make compilation take slightly longer) run the following:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">cargo build --release\n</span></code></pre><h2 id=\"conclusion\">Conclusion</h2>\n<p>As this small demo showed, it's not difficult to get started using Rust to write\nDevOps tools. And even then we didn't need to make a trade-off between ease of\ndevelopment and performant fast code. </p>\n<p>Hopefully the next time you're writing a new piece of DevOps software, \nanything from a simple CLI tool for a specific DevOps operation or you're writing \nthe next Kubernetes, you'll consider reaching for Rust.\nAnd if you have further questions about Rust, or need help implementing your Rust \nproject, please feel free to reach out to FP Complete for Rust engineering \nand training!</p>\n<p>Want to learn more Rust? Check out our <a href=\"https://www.fpcomplete.com/rust/crash-course/\">Rust Crash Course eBook</a>. And for more information, check out our <a href=\"https://www.fpcomplete.com/rust/\">Rust homepage</a>.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/",
        "slug": "rust-for-devops-tooling",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Using Rust for DevOps tooling",
        "description": "A beginner's guide to writing your DevOps tools in Rust.",
        "updated": null,
        "date": "2020-09-09",
        "year": 2020,
        "month": 9,
        "day": 9,
        "taxonomies": {
          "categories": [
            "functional programming",
            "devops"
          ],
          "tags": [
            "devops",
            "rust",
            "insights"
          ]
        },
        "extra": {
          "author": "Mike McGirr",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/rust-for-devops-tooling/",
        "components": [
          "blog",
          "rust-for-devops-tooling"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "introduction",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#introduction",
            "title": "Introduction",
            "children": []
          },
          {
            "level": 2,
            "id": "what-makes-the-rust-language-unique",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#what-makes-the-rust-language-unique",
            "title": "What Makes the Rust Language Unique",
            "children": []
          },
          {
            "level": 2,
            "id": "rust-and-devops",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#rust-and-devops",
            "title": "Rust and DevOps",
            "children": [
              {
                "level": 3,
                "id": "why-you-would-want-to-write-your-devops-tools-in-rust",
                "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#why-you-would-want-to-write-your-devops-tools-in-rust",
                "title": "Why you would want to write your DevOps tools in Rust",
                "children": [
                  {
                    "level": 4,
                    "id": "os-targets-and-cross-compiling-rust-for-different-architectures",
                    "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#os-targets-and-cross-compiling-rust-for-different-architectures",
                    "title": "OS targets and Cross Compiling Rust for different architectures",
                    "children": []
                  },
                  {
                    "level": 4,
                    "id": "small-executables",
                    "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#small-executables",
                    "title": "Small Executables",
                    "children": []
                  },
                  {
                    "level": 4,
                    "id": "rust-is-fast",
                    "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#rust-is-fast",
                    "title": "Rust is fast",
                    "children": []
                  }
                ]
              },
              {
                "level": 3,
                "id": "why-would-you-not-want-to-write-your-devops-tools-in-rust",
                "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#why-would-you-not-want-to-write-your-devops-tools-in-rust",
                "title": "Why would you not want to write your DevOps tools in Rust?",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "demo-devops-project-for-aws",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#demo-devops-project-for-aws",
            "title": "Demo DevOps Project for AWS",
            "children": [
              {
                "level": 3,
                "id": "project-structure",
                "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#project-structure",
                "title": "Project structure",
                "children": []
              },
              {
                "level": 3,
                "id": "building-the-rust-project",
                "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#building-the-rust-project",
                "title": "Building the Rust project",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 2540,
        "reading_time": 13,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/devops-unifying-dev-ops-qa.md",
        "content": "<p>The term DevOps has been around for many years. Small and big companies adopt DevOps concepts for different purposes, e.g. to increase the quality of software. In this blog post, we define DevOps, present its pros and cons, highlight a few concepts and see how these can impact the entire organization.</p>\n<h2 id=\"what-is-devops\">What is DevOps?</h2>\n<p>At a high level, DevOps is understood as a technical, organizational and cultural shift in a company to run software more efficiently, reliably, and securely. From this first definition, we can see that DevOps is much more than &quot;use tool X&quot; or &quot;move to the cloud&quot;. DevOps starts with the understanding that development (Dev), operations (Ops) and quality assurance (QA) are not treated as siloed disciplines anymore. Instead, they all come together in shared processes and responsibilities across collaborating teams. DevOps achieves this through various techniques. In the section &quot;Implementation&quot;, we present a few of these concepts.</p>\n<h2 id=\"benefits\">Benefits</h2>\n<p>Benefits of applying DevOps include:</p>\n<ul>\n<li>Cost savings through higher efficiency.</li>\n<li>Faster software iteration cycles, where updates take less time from development to running in production.</li>\n<li>More security, reliability, and fault tolerance when running software.</li>\n<li>Stronger bonds between different stakeholders in the organization including non-technical staff.</li>\n<li>Enable more data-driven decisions.</li>\n</ul>\n<p>Let's have a look <em>how</em> these benefits can be achieved by applying DevOps ideas:</p>\n<h2 id=\"how-to-implement-devops\">How to implement DevOps</h2>\n<h3 id=\"automation-and-continuous-integration-ci-continuous-delivery-cd\">Automation and Continuous Integration (CI) / Continuous Delivery (CD)</h3>\n<p>Automation refers to a key aspect of the engineering-driven part of DevOps. With automation, we aim to reduce the need for human action, and thus the possibility of human error, as far as possible by sending your software through an automated and well-understood pipeline of actions. These automated actions can build your software, run unit tests, integrate it with existing systems, run system tests, deploy it, and provide feedback on each step. What we are\ndescribing here is usually referred to as <strong>Continuous Integration (CI)</strong> and <strong>Continuous Delivery (CD)</strong>. Adopting CI/CD invests in a low-risk and low-cost way of crossing the chasm between &quot;software that is working on an engineer's laptop&quot; and &quot;software that running securely and reliably on production servers&quot;.</p>\n<p>CI/CD is usually tied to a platform on top of which the automated actions are run, e.g., Gitlab. The platform accepts software that should be passed through the pipeline, executes the automated actions on servers which are usually abstracted away, and provides feedback to the engineering team. These actions can be highly customized and tied together in different ways. For example, one action only compiles the source code and provides the build artifacts to subsequent actions. Another action can be responsible for running a test-suite, another one can deploy software. Such actions can be defined for different types of software: A website can be automatically deployed to a server, or a Desktop application can be made available to your customers without human interaction.</p>\n<p>Besides the fact that CI/CD can be used for all kinds of software, there are other advantages to consider:</p>\n<ol>\n<li><strong>The CI/CD pipeline is well-understood and maintained by the teams</strong>: the actions that are run in a pipeline can be flexibly updated, extended, etc. <a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#infra-as-code\">Infrastructure as Code</a> can be a powerful concept here.</li>\n<li><strong>Run in standardized environments</strong>: Version conflicts between tools and configuration or dependency mismatches only have to be fixed once when the pipeline is built. Once a pipeline is working, it will continue to work as the underlying servers and their software versions don't change. No more conflicts between operating systems, tools, versions of tools across different engineers. Pipelines are highly reproducible. Containerization can be a game-changer here.</li>\n<li><strong>Feedback</strong>: Actions sometimes fail, e.g. because a unit test does not pass. The CI/CD platform usually allows different reporting mechanisms: E-mail someone, update the project status on your repository overview page, block subsequent actions or cancel other pipelines.</li>\n</ol>\n<p>The next sections cover more DevOps concepts that benefit from automation.</p>\n<h3 id=\"multiple-environments\">Multiple Environments</h3>\n<p>The CI/CD can be extended by deploying software to different environments. These deployments can happen in individual actions defined in your pipeline. Besides the production environment, which runs user-facing software, staging and testing environments can be defined where software is deployed to. For example, a testing environment can be used by the engineering team for peer-reviewing and validating software changes. Once the team agreed on new software, it can be deployed to a staging environment. A usual purpose of the staging environment is to mimic the production environment as closely as possible. Further tests can be run in a staging environment to make sure the software is ready to be used by real users. Finally, the software reaches production-readiness and is deployed to a production environment. Such a production deployment can be designed using a gradual rollout, i.e. canary deployments.</p>\n<p>Different environments not only realize different semantics and confidence levels of running software, e.g. as described in the previous paragraph, but also serve as an agreed-upon view on software in the entire organization. Multi-environment deployments make your software and quality thereof easier to understand. This is because of the gained insights when running software, in particular on infrastructure that is close to a production setting. Generally, running software gives much more insights into the performance, reliability, security, production-readiness and overall quality. Different teams, e.g. security experts or a dedicated QA-team (if your organization follows this practice) can be consulted at different software quality stages, i.e. different environments in which software runs. Additionally, non-technical staff can use environments, e.g. specialized ones for demo purposes.</p>\n<p>Ultimately, integrating multiple environments structures QA and smoothens the interactions between different teams.</p>\n<h3 id=\"fail-early\">Fail early</h3>\n<p>No matter how well things are working in an organization that builds software, bugs happen and bugs are expensive. The cost of bugs can be projected to the manpower invested into fixing the bug, the loss of reputation due to angry customers, and generally negative business impact. Since we can't fully avoid bugs, there exist concepts to reduce both the frequency and impact of bugs. &quot;Fail early&quot; is one of these concepts.</p>\n<p>The basic idea is to catch bugs and other flaws in your software as early in the development process as possible. When software is developed, unit tests, compiler errors and peer reviews count towards the early and cheap mechanisms to detect and fix flaws. Ideally, a unit test tells the developer that the software is not correct, or, a second pair of eyes reveals a potential performance issue during a code review. In both cases, not much time and effort is lost and the flaw can be easily fixed. However, other bugs might make it through these initial checks and land in testing or staging environments. Other types of tests and QA should be in place to check the software quality. Worst case, the bug outlives all checks and is in production. There, bugs have much higher impact and require more effort by many stakeholders, e.g. the bug fix by the engineering team and the apology to the customers.</p>\n<p>To save costs, cheap checks such as running a test suite in an automated pipeline should be executed early. This will save costs as flaws discovered later in the process result in higher costs. Thus, failing early increases cost efficiency.</p>\n<h3 id=\"rollbacks\">Rollbacks</h3>\n<p>DevOps can also help to react quickly to changes. One example of a sudden change is a bug, as described in the last section, which is discovered in the production environment. Rollbacks, for example as manually triggered pipelines, can recover the well-functioning of a production service in a timely manner. This can be useful when the bug is a hard one and needs hours to be identified and fixed. These hours of degraded customer experience or even downtime makes paying customers unhappy. A faster mechanism is desired, which minimizes the gap between a faulty system and a recovered system. A rollback can be a fast and effective way to recover system state without exposing customers to company failure much.</p>\n<h3 id=\"policies\">Policies</h3>\n<p>DevOps concepts impose a challenge to security and permission management as these span the entire organization. Policies can help to formulate authorizations and rules during operations. For example, implementing the following security requirements may be required:</p>\n<ul>\n<li>A deployment or rollback in production should not be triggered by anyone but a well-defined set of people in authority.</li>\n<li>Some actions in a CI/CD pipeline should always be run while other actions are intended to be triggered manually or only run under certain conditions.</li>\n<li>The developers might require slightly different permissions than a dedicated QA team to perform their day-to-day work.</li>\n<li>Humans and machine users can have different capabilities but should always have the least privileges assigned to them.</li>\n</ul>\n<p>The authentication and authorization tools provided by CI/CD providers or cloud vendors can help to design such policies according to your organizational needs.</p>\n<h3 id=\"observability\">Observability</h3>\n<p>As software is running and users are interacting with your applications, insights such as error rates, performance statistics, resource usages, etc. can help to identify bottlenecks, mitigate future issues, and drive business decisions through data. There exist two major ways to establish different forms of observability:</p>\n<ul>\n<li><strong>Logging</strong>: Events in text form that software outputs to inform about the application's status and health. Different types of logging messages, e.g. indicating the severity of an error event, can help to aggregate and display log messages in a central place, where it can be used by engineering teams for debugging purposes.</li>\n<li><strong>Metrics</strong>: Information about the running software that is not generated by the application itself. For example, the CPU or memory usage of the underlying machine that runs the software, network statistics, HTTP error rates, etc. As with logging, metrics can help to spot bottlenecks and mitigate them before they have a business impact. Visualizing aggregated metrics data facilitates communication across technical and non-technical teams and leverages data-driven decisions. Metrics dashboards can strengthen the shared ownership of software across teams.</li>\n</ul>\n<p>Logging and metrics can help to define goals and to align a development team with a QA team for example.</p>\n<h2 id=\"disadvantages\">Disadvantages</h2>\n<p>So far, we only looked at the benefits and characteristics of DevOps. Let's have a brief look at the other side of the coin by commenting on the possible negative side effects and disadvantages of adopting DevOps concepts.</p>\n<ul>\n<li>\n<p>The investment into DevOps can be huge as it is a company-wide, multi-discipline, and multi-team transformation that not only requires technical implementation effort but also training for people, re-structuring and aligning teams.</p>\n</li>\n<li>\n<p>This goes along with the first point but it's worth emphasizing it: The cultural impact on your organization can be challenging due to human factors. While a new automation mechanism can be estimated and implemented reasonably well, tracking the progress of changing people's way of communication, feeling of ownership, aligning to new processes can be hard and might lead to no gained efficiencies, which DevOps promises, short-term. Due to the high impact of DevOps, it is a long-term investment.</p>\n</li>\n<li>\n<p>The technical backbone of DevOps, e.g. CI/CD pipelines, cloud vendors, integration of authorization and authentication, likely results in increased expenses through new contracts and licenses with new players. However, through the dominance of open source in modern DevOps tooling, e.g. through using Kubernetes, vendor lock-in can be avoided.</p>\n</li>\n</ul>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>In this blog post, we explored the definition of DevOps and presented several DevOps concepts and use-cases. Furthermore, we evaluated benefits and disadvantages. Adopting DevOps is an investment into a low-friction and automated way of developing, testing, and running software. Technical improvements, e.g. automation, as well as increased collaboration between teams of different disciplines ultimately improve the efficiency in your organization long-term.</p>\n<p>However, DevOps represents not only technical effort but also impacts the entire company, e.g. how teams communicate with each other, how issues are resolved, and what teams feel responsible for. Finding the right balance and choosing the best concepts and tools for your teams represents a challenge. We can help you with identifying and solving the DevOps transformation in your organization.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/",
        "slug": "devops-unifying-dev-ops-qa",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "DevOps: Unifying Dev, Ops, and QA",
        "description": "The term DevOps has been around for many years. Small and big companies adopt DevOps concepts for different purposes, e.g. to increase the quality of software. In this blog post, we define DevOps, present its pros and cons, highlight a few concepts and see how these can impact the entire organization.",
        "updated": null,
        "date": "2020-08-24",
        "year": 2020,
        "month": 8,
        "day": 24,
        "taxonomies": {
          "tags": [
            "devops",
            "insights"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Moritz Hoffmann",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/devops-unifying-dev-ops-qa/",
        "components": [
          "blog",
          "devops-unifying-dev-ops-qa"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "what-is-devops",
            "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#what-is-devops",
            "title": "What is DevOps?",
            "children": []
          },
          {
            "level": 2,
            "id": "benefits",
            "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#benefits",
            "title": "Benefits",
            "children": []
          },
          {
            "level": 2,
            "id": "how-to-implement-devops",
            "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#how-to-implement-devops",
            "title": "How to implement DevOps",
            "children": [
              {
                "level": 3,
                "id": "automation-and-continuous-integration-ci-continuous-delivery-cd",
                "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#automation-and-continuous-integration-ci-continuous-delivery-cd",
                "title": "Automation and Continuous Integration (CI) / Continuous Delivery (CD)",
                "children": []
              },
              {
                "level": 3,
                "id": "multiple-environments",
                "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#multiple-environments",
                "title": "Multiple Environments",
                "children": []
              },
              {
                "level": 3,
                "id": "fail-early",
                "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#fail-early",
                "title": "Fail early",
                "children": []
              },
              {
                "level": 3,
                "id": "rollbacks",
                "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#rollbacks",
                "title": "Rollbacks",
                "children": []
              },
              {
                "level": 3,
                "id": "policies",
                "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#policies",
                "title": "Policies",
                "children": []
              },
              {
                "level": 3,
                "id": "observability",
                "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#observability",
                "title": "Observability",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "disadvantages",
            "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#disadvantages",
            "title": "Disadvantages",
            "children": []
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/devops-unifying-dev-ops-qa/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 2023,
        "reading_time": 11,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/devops-for-developers.md",
        "content": "<p>In this post, I describe my personal journey as a developer skeptical\nof the seemingly ever-growing, ever more complex, array of &quot;ops&quot;\ntools. I move towards adopting some of these practices, ideas and\ntools. I write about how this journey helps me to write software\nbetter and understand discussions with the ops team at work.</p>\n<div style=\"border:1px solid black;background-color:#f8f8f8;margin-bottom:1em;padding: 0.5em 0.5em 0 0.5em;\">\n<p><strong>Table of Contents</strong></p>\n<ul>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#on-being-skeptical\">On being skeptical</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#the-humble-app\">The humble app</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#disk-failures-arent-that-common\">Disk failures aren't that common</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#auto-deployment-is-better-than-manual\">Auto-deployment is better than manual</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#backups-become-worth-it\">Backups become worth it</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#deployment-staging\">Deployment staging</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#packaging-with-docker-is-good\">Packaging with Docker is good</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#custodiansmultiple-processes-are-useful\">Custodians/multiple processes are useful</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#kubernetes-provides-exactly-that\">Kubernetes provides exactly that</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#declarative-is-good-vendor-lock-in-is-bad\">Declarative is good, vendor lock-in is bad</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#more-advanced-rollout\">More advanced rollout</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#relationship-between-code-and-deployed-state\">Relationship between code and deployed state</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#argocd\">ArgoCD</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#infra-as-code\">Infra-as-code</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#where-the-dev-meets-the-ops\">Where the dev meets the ops</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#what-we-do\">What we do</a></li>\n</ul>\n</div>\n<h2 id=\"on-being-skeptical\">On being skeptical</h2>\n<p>I would characterise my attitudes to adopting technology in two\nstages:</p>\n<ul>\n<li>Firstly, I am conservative and dismissive, in that I will usually\ndisregard any popular new technology as a bandwagon or trend. I'm a\nslow adopter.</li>\n<li>Secondly, when I actually encounter a situation where I've suffered,\nI'll then circle back to that technology and give it a try, and if I\ncan really find the nugget of technical truth in there, then I'll\nadopt it.</li>\n</ul>\n<p>Here are some things that I disregarded for a year or more before\ntrying: Emacs, Haskell, Git, Docker, Kubernetes, Kafka. The whole\nNoSQL trend came, wrecked havoc, and went, while I had my back turned,\nbut I am considering using Redis for a cache at the moment.</p>\n<h2 id=\"the-humble-app\">The humble app</h2>\n<p>If you’re a developer like me, you’re probably used to writing your\nsoftware, spending most of your time developing, and then finally\ndeploying your software by simply creating a machine, either a\ndedicated machine or a virtual machine, and then uploading a binary of\nyour software (or source code if it’s interpreted), and then running\nit with the copy pasted config of systemd or simply running the\nsoftware inside GNU screen. It's a secret shame that I've done this,\nbut it's the reality.</p>\n<p>You might use nginx to reverse-proxy to the service. Maybe you set up\na PostgreSQL database or MySQL database on that machine. And then you\nwalk away and test out the system, and later you realise you need some\nslight changes to the system configuration. So you SSH into the system\nand makes the small tweaks necessary, such as port settings, encoding\nsettings, or an additional package you forgot to add. Sound familiar?</p>\n<p>But on the whole, your work here is done and for most services this is\npretty much fine. There are plenty of services running that you have\nseen in the past 30 years that have been running like this.</p>\n<h2 id=\"disk-failures-aren-t-that-common\">Disk failures aren't that common</h2>\n<p>Rhetoric about processes going down due to a hardware failure are\nprobably overblown. Hard drives don’t crash very often. They don’t\nreally wear out as quickly as they used to, and you can be running a\nsystem for years before anything even remotely concerning happens.</p>\n<h2 id=\"auto-deployment-is-better-than-manual\">Auto-deployment is better than manual</h2>\n<p>When you start to iterate a little bit quicker, you get bored of\nmanually building and copying and restarting the binary on the\nsystem. This is especially noticeable if you forget the steps later\non.</p>\n<!-- Implementing Auto-Deployment -->\n<p>If you’re a little bit more advanced you might have some special\nscripts or post-merge git hooks, so that when you push to your repo it\nwould apply to the same machine and you have some associated token on\nyour CI machine that is capable of uploading a binary and running a\ncommand like copy and restart (e.g. SSH key or API\nkey). Alternatively, you might implement a polling system on the\nactual production system which will check if any updates have occurred\nin get and if so pull down a new binary. This is how we were doing\nthings in e.g. 2013.</p>\n<h2 id=\"backups-become-worth-it\">Backups become worth it</h2>\n<p>Eventually, if you're lucky, your service starts to become slightly\nmore important; maybe it’s used in business and people actually are\nusing it and storing valuable things in the database. You start to\nthink that back-ups are a good idea and worth the investment.</p>\n<!-- Redundancy of DB -->\n<p>You probably also have a script to back up the database, or replicate\nit on a separate machine, for redundancy.</p>\n<h2 id=\"deployment-staging\">Deployment staging</h2>\n<p>Eventually, you might have a staged deployment strategy. So you might\nhave a developer testing machine, you might have a QA machine, a\nstaging machine, and finally a production machine. All of these are\nconfigured in pretty much the same way, but they are deployed at\ndifferent times and probably the system administrator is the only one\nwith access to deploy to production.</p>\n<!-- Continuum -->\n<p>It’s clear by this point that I’m describing a continuum from &quot;hobby\nproject&quot; to &quot;enterprise serious business synergy solutions&quot;.</p>\n<h2 id=\"packaging-with-docker-is-good\">Packaging with Docker is good</h2>\n<p>Docker effectively leads to collapsing all of your system dependencies\nfor your binary to run into one contained package. This is good,\nbecause dependency management is hell. It's also highly wasteful,\nbecause its level of granularity is very wide. But this is a trade-off\nwe accept for the benefits.</p>\n<h2 id=\"custodians-multiple-processes-are-useful\">Custodians/multiple processes are useful</h2>\n<p>Docker doesn’t have much to say about starting and restarting\nservices. I’ve explored using CoreOS with the hosting provider Digital\nOcean, and simply running a fresh virtual machine, with the given\nDocker image.</p>\n<p>However, you quickly run into the problem of starting up and tearing\ndown:</p>\n<ul>\n<li>When you start the service, you need certain liveness checks\nand health checks, so if the service fails to start then you should\nnot stop the existing service from running, for example. You should\nkeep the existing ones running.</li>\n<li>If the process fails at any time during running then you should also\nrestart the process. I thought about this point a lot, and came to the\nconclusion that it’s better to have your process be restarted than to\nassume that the reason it failed was so dangerous that the process\nshouldn’t start again. Probably it’s more likely that there is an\nexception or memory issue that happened in a pathological case which\nyou can investigate in your logging system. But it doesn’t mean that\nyour users should suffer by having downtime.</li>\n<li>The natural progression of this functionality is to support\ndifferent rollout strategies. Do you want to switch everything to the\nnew system in one go, do you want it to be deployed piece-by-piece?</li>\n</ul>\n<!-- Summary: You Realise Worth Of Ops Tools -->\n<p>It’s hard to fully appreciate the added value of ops systems like\nKubernetes, Istio/Linkerd, Argo CD, Prometheus, Terraform, etc. until\nyou decide to design a complete architecture yourself, from scratch,\nthe way you want it to work in the long term.</p>\n<h2 id=\"kubernetes-provides-exactly-that\">Kubernetes provides exactly that</h2>\n<p>What system happens to accept Docker images, provide custodianship,\nroll out strategies, and trivial redeploy? Kubernetes.</p>\n<p>It provides this classical monitoring and custodian responsibilities\nthat plenty of other systems have done in the past. However, unlike\nsimply running a process and testing if it’s fine and then turning off\nanother process, Kubernetes buys into Docker all the way.  Processes\nare isolated from each other, in both the network on the file\nsystem. Therefore, you can very reliably start and stop the services\non the same machine. Nothing about a process's machine state is\npersistent, therefore you are forced to design your programs in a way\nthat state is explicitly stored either ephemerally, or elsewhere.</p>\n<!-- Cloud Managed Databases Make This Practical -->\n<p>In the past it might be a little bit scarier to have your database\nrunning in such system, what if it automatically wipes out the\ndatabase process? With today’s cloud base deployments, it's more\ncommon to use a managed database such as that provided by Amazon,\nDigital Ocean, Google or Azure. The whole problem of updating and\nbacking up your database can pretty much be put to one\nside. Therefore, you are free to mess with the configuration or\ntopology of your cluster as much as you like without affecting your\ndatabase.</p>\n<h2 id=\"declarative-is-good-vendor-lock-in-is-bad\">Declarative is good, vendor lock-in is bad</h2>\n<p>A very appealing feature of a deployment system like Kubernetes is\nthat everything is automatic and declarative. You stick all of your\nconfiguration in simple YAML files (which is also a curse because YAML\nhas its own warts and it's not common to find formal schemas for it).\nThis is also known as &quot;infrastructure as code&quot;.</p>\n<p>Ideally, you should have as much as possible about your infrastructure\nin code checked in to a repo so that you can reproduce it and track\nit.</p>\n<p>There is also a much more straight-forward path to migrate from one\nservice provider to another service provider. Kubernetes is supported\non all the major service providers (Google, Amazon, Azure), therefore\nyou are less vulnerable to vendor lock-in. They also all provide\nmanaged databases that are standard (PostgreSQL, for example) with\ntheir normal wire protocols. If you were using the vendor-specific\nAPIs to achieve some of this, you'd be stuck on one vendor. I, for\nexample, am not sure whether to go with Amazon or Azure on a big\npersonal project right now. If I use Kubernetes, I am mitigating risk.</p>\n<p>With something like Terraform you can go one step further, in which\nyou write code that can create your cluster completely from\nscratch. This is also more vendor independent/mitigated.</p>\n<h2 id=\"more-advanced-rollout\">More advanced rollout</h2>\n<p>Your load balancer and your DNS can also be in code. Typically a load\nbalancer that does the job is nginx. However, for more advanced\ndeployments such as A/B or green/blue deployments, you may need\nsomething more advanced like Istio or Linkerd.</p>\n<p>Do I really want to deploy a new feature to all of my users? Maybe,\nthat might be easier. Do I want to deploy a different way of marketing\nmy product on the website to all users at once? If I do that, then I\ndon’t exactly know how effective it is. So, I could perhaps do a\ndeployment in which half of my users see one page and half of the\nusers see another page. These kinds of deployments are\nstraight-forwardly achieved with Istio/Linkerd-type service meshes,\nwithout having to change any code in your app.</p>\n<h2 id=\"relationship-between-code-and-deployed-state\">Relationship between code and deployed state</h2>\n<p>Let's think further than this.</p>\n<p>You've set up your cluster with your provider, or Terraform. You've\nset up your Kubernetes deployments and services. You've set up your CI\nto build your project, produce a Docker image, and upload the images\nto your registry. So far so good.</p>\n<p>Suddenly, you’re wondering, how do I actually deploy this? How do I\ncall Kubernetes, with the correct credentials, to apply this new\nDoctor image to the appropriate deployment?</p>\n<p>Actually, this is still an ongoing area of innovation. An obvious way\nto do it is: you put some details on your CI system that has access to\nrun kubectl, then set the image with the image name and that will try\nto do a deployment. Maybe the deployment fails, you can look at that\nresult in your CI dashboard.</p>\n<p>However, the question comes up as what is currently actually deployed\non production? Do we really have infrastructure as code here?</p>\n<p>It’s not that I edited the file and that update suddenly got\nreflected. There’s no file anywhere in Git that contains what the\ncurrent image is. Head scratcher.</p>\n<p>Ideally, you would have a repository somewhere which states exactly\nwhich image should be deployed right now. And if you change it in a\ncommit, and then later revert that commit, you should expect the\nproduction is also reverted to reflect the code, right?</p>\n<h2 id=\"argocd\">ArgoCD</h2>\n<p>One system which attempts to address this is ArgoCD. They implement\nwhat they call &quot;GitOps&quot;. All state of the system is reflected in a Git\nrepo somewhere. In Argo CD, after your GitHub/Gitlab/Jenkins/Travis CI\nsystem has pushed your Docker image to the Docker repository, it makes\na gRPC call to Argo, which becomes aware of the new image. As an\nadmin, you can now trivially look in the UI and click &quot;Refresh&quot; to\nredeploy the new version.</p>\n<h2 id=\"infra-as-code\">Infra-as-code</h2>\n<p>The common running theme in all of this is\ninfrastructure-as-code. It’s immutability. It’s declarative. It’s\nremoving the number of steps that the human has to do or care\nabout. It’s about being able to rewind. It’s about redundancy. And\nit’s about scaling easily.</p>\n<!-- Circling Back -->\n<p>When you really try to architect your own system, and your business\nwill lose money in the case of ops mistakes, then you start to think\nthat all of these advantages of infrastructure as code start looking\nreally attractive.</p>\n<p>But before you really sit down and think about this stuff, however, it\nis pretty hard to empathise or sympathise with the kind of concerns\nthat people using these systems have.</p>\n<!-- Downsides/Tax -->\n<p>There are some downsides to these tools, as with any:</p>\n<ul>\n<li>Docker is quite wasteful of time and space</li>\n<li>Kubernetes is undoubtedly complex, and leans heavily on YAML</li>\n<li><a href=\"https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/\">All abstractions are leaky</a>,\ntherefore tools like this all leak</li>\n</ul>\n<h2 id=\"where-the-dev-meets-the-ops\">Where the dev meets the ops</h2>\n<p>Now that I’ve started looking into these things and appreciating their\nuse, I interact a lot more with the ops side of our DevOps team at work,\nand I can also be way more helpful in assisting them with the\ninformation that they need, and also writing apps which anticipate the\nkind of deployment that is going to happen. The most difficult\nchallenge typically is metrics and logging, for run-of-the-mill apps,\nI’m not talking about high-performance apps.</p>\n<!-- An Exercise -->\n<p>One way way to bridge the gap between your ops team and dev team,\ntherefore, might be an exercise meeting in which you do have a dev\nperson literally sit down and design an app architecture and\ninfrastructure, from the ground up using the existing tools that we\nhave that they are aware of and then your ops team can point out the\nadvantages and disadvantages of their proposed solution. Certainly,\nI think I would have benefited from such a mentorship, even for an\nhour or two.</p>\n<!-- Head-In-The-Sand Also Works -->\n<p>It may be that your dev team and your ops team are completely separate\nand everybody’s happy. The devs write code, they push it, and then it\nmagically works in production and nobody has any issues. That’s\ncompletely fine. If anything it would show that you have a very good\nprocess. In fact, that’s pretty much how I’ve worked for the past\neight years at this company.</p>\n<p>However, you could derive some benefit if your teams are having\ndifficulty communicating.</p>\n<p>Finally, the tools in the ops world aren't perfect, and they're made\nby us devs. If you have a hunch that you can do better than these\ntools, you should learn more about them, and you might be right.</p>\n<h2 id=\"what-we-do\">What we do</h2>\n<p>FP Complete are using a great number of these tools, and we're writing\nour own, too. If you'd like to know more, email use at\n<a href=\"mailto:sales@fpcomplete.com\">sales@fpcomplete.com</a>.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/",
        "slug": "devops-for-developers",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "DevOps for (Skeptical) Developers",
        "description": null,
        "updated": null,
        "date": "2020-08-16",
        "year": 2020,
        "month": 8,
        "day": 16,
        "taxonomies": {
          "categories": [
            "functional programming",
            "devops"
          ]
        },
        "extra": {
          "author": "Chris Done",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/devops-for-developers/",
        "components": [
          "blog",
          "devops-for-developers"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "on-being-skeptical",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#on-being-skeptical",
            "title": "On being skeptical",
            "children": []
          },
          {
            "level": 2,
            "id": "the-humble-app",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#the-humble-app",
            "title": "The humble app",
            "children": []
          },
          {
            "level": 2,
            "id": "disk-failures-aren-t-that-common",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#disk-failures-aren-t-that-common",
            "title": "Disk failures aren't that common",
            "children": []
          },
          {
            "level": 2,
            "id": "auto-deployment-is-better-than-manual",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#auto-deployment-is-better-than-manual",
            "title": "Auto-deployment is better than manual",
            "children": []
          },
          {
            "level": 2,
            "id": "backups-become-worth-it",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#backups-become-worth-it",
            "title": "Backups become worth it",
            "children": []
          },
          {
            "level": 2,
            "id": "deployment-staging",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#deployment-staging",
            "title": "Deployment staging",
            "children": []
          },
          {
            "level": 2,
            "id": "packaging-with-docker-is-good",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#packaging-with-docker-is-good",
            "title": "Packaging with Docker is good",
            "children": []
          },
          {
            "level": 2,
            "id": "custodians-multiple-processes-are-useful",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#custodians-multiple-processes-are-useful",
            "title": "Custodians/multiple processes are useful",
            "children": []
          },
          {
            "level": 2,
            "id": "kubernetes-provides-exactly-that",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#kubernetes-provides-exactly-that",
            "title": "Kubernetes provides exactly that",
            "children": []
          },
          {
            "level": 2,
            "id": "declarative-is-good-vendor-lock-in-is-bad",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#declarative-is-good-vendor-lock-in-is-bad",
            "title": "Declarative is good, vendor lock-in is bad",
            "children": []
          },
          {
            "level": 2,
            "id": "more-advanced-rollout",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#more-advanced-rollout",
            "title": "More advanced rollout",
            "children": []
          },
          {
            "level": 2,
            "id": "relationship-between-code-and-deployed-state",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#relationship-between-code-and-deployed-state",
            "title": "Relationship between code and deployed state",
            "children": []
          },
          {
            "level": 2,
            "id": "argocd",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#argocd",
            "title": "ArgoCD",
            "children": []
          },
          {
            "level": 2,
            "id": "infra-as-code",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#infra-as-code",
            "title": "Infra-as-code",
            "children": []
          },
          {
            "level": 2,
            "id": "where-the-dev-meets-the-ops",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#where-the-dev-meets-the-ops",
            "title": "Where the dev meets the ops",
            "children": []
          },
          {
            "level": 2,
            "id": "what-we-do",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#what-we-do",
            "title": "What we do",
            "children": []
          }
        ],
        "word_count": 2614,
        "reading_time": 14,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/our-history-containerization.md",
        "content": "<p>FP Complete has been working with containerization (or OS-level virtualization) since before it was popularized by Docker.  What follows is a brief history of how and why we got started using containers, and how our use of containerization has evolved as new technology has emerged.</p>\n<h2 id=\"brief-history\">Brief history</h2>\n<p>Our first foray into containerization started at the beginning of the company, when we were building a web-based integrated development environment for Haskell.  We needed a secure and cost-effective way to be able to compile and run Haskell code on the server side.  While giving each active user their own virtual machine with dedicated CPU and memory would have satisfied the first requirement (security), it would have been far from cost effective.  GHC, the de-facto standard Haskell compiler, is notoriously resource hungry, so the VM would have to be quite large (it's not uncommon to need 4 GB or more of RAM to compile a fairly straightforward piece of software).  We needed a way to share CPU and memory resources between multiple users securely and be able to shift load around a cluster of virtual machines to keep usage balanced and avoid one heavy user from impacting the experience of others users on the same VM.  This sounds like a job for container orchestration!  Unfortunately, Docker didn't exist yet, let alone Kubernetes.  The state of the art for Linux containers at the time was LXC, which was mostly a collection of shell scripts that helped with using the Linux kernel features that underly all Linux container solutions, but at a much lower level than Docker.  On top of this we built everything we needed to distribute &quot;images&quot; of a base filesystem plus overlay for local changes, isolated container networks, and ability to shift load based on VM and container utilization -- that is, many of the things Docker and Kubernetes do now, but tailored specifically for our application's needs.</p>\n<p>When Docker came on the scene, we embraced it despite some early growing pains, since it was much easier to use and more general purpose than our &quot;bespoke&quot; system and we thought it likely that it would soon become a de-facto standard, which is exactly what happened.  For internal and customer solutions, Docker allowed us to create much more nimble and efficient deployment solutions that satisfied the requirement for <a href=\"https://www.fpcomplete.com/devops/immutable-infrastructure/\">immutable infrastructure</a>.  Prior to Docker, we achieved immutability by building VM images and spinning up virtual machines; a much slower and heavier process than building a Docker image and running it on an already-provisioned VM.  This also allowed us to run multiple applications isolated from one another on a single VM without worry of interference with each other.</p>\n<p>Finally Kubernetes arrived.  While it was not the first orchestration platform, it was the first that wholeheartedly standardized on Docker containers.  Once again we embraced it, despite some early growing pains, due to its ease of use, multi-cloud support, fast pace of improvement, and backing of a major company (Google).  We once again bet that Kubernetes would become the de-facto standard, which is again exactly what happened.  With Kubernetes, instead of having to think about which VM a container would run on, we can have a cluster of general-purpose nodes and let the orchestrator worry about what runs on which node.  This lets us squeeze yet more efficiency out of our resources.  Due to its ease of use and built-in support for common rollout strategies, we can give developers the ability to deploy their apps directly, and since it is so easy to tie into CI/CD pipelines we can drastically simplify automated deployment processes.</p>\n<p>Going forward, we continue to keep up with the latest developments in containerization and are constantly evaluating new and alternative technologies, to stay on the forefront of DevOps.</p>\n<h2 id=\"why-we-really-like-it\">Why we really like it</h2>\n<ul>\n<li>\n<p>Supports <a href=\"https://www.fpcomplete.com/devops/immutable-infrastructure/\">immutable infrastructure</a>.</p>\n</li>\n<li>\n<p>Fast build and deployment processes.</p>\n</li>\n<li>\n<p>Low overhead and efficient use of compute resources.</p>\n</li>\n<li>\n<p>Easy integration with CI/CD pipelines.</p>\n</li>\n<li>\n<p>Isolation of applications from others running on the same machine.</p>\n</li>\n<li>\n<p>Bundles dependencies with the application, so they can be tested together and there's no risk of deploying to an incorrect environment.</p>\n</li>\n<li>\n<p>Developers on various platforms can build and test the application in a consistent environment.</p>\n</li>\n</ul>\n<h2 id=\"limitations-of-the-technology\">Limitations of the technology</h2>\n<ul>\n<li>\n<p>Containers and container orchestration are most mature on Linux, although Docker and Kubernetes do now support running Windows containers on machines running Windows, and most modern server operating system have support for some kind of containerization (but not necessarily Docker or Kubernetes).</p>\n</li>\n<li>\n<p>Containers and container orchestration add additional layers of abstraction and complexity.  This can, at times, make diagnosing problems more difficult.</p>\n</li>\n<li>\n<p>Legacy applications can be tricky to containerize since they assume they are running on a persistent machine rather than an ephemeral one.  While this can be mitigated using persistent volumes, it makes the containerization strategy less straightforward.</p>\n</li>\n<li>\n<p>While properly configured containers are relatively secure, all containers running on a host share a single operating system kernel which means there is greater risk that a process can use a security vulnerability to &quot;break out&quot; of its container than when using VMs.</p>\n</li>\n</ul>\n<h2 id=\"resources\">Resources</h2>\n<p>From FP Complete:</p>\n<ul>\n<li><a href=\"https://www.fpcomplete.com/devops/containerization/\">Introduction to Containerization concepts</a></li>\n<li><a href=\"https://www.fpcomplete.com/devops/immutable-infrastructure/\">Introduction to Immutable Infrastructure concepts</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/deploying_haskell_apps_with_kubernetes/\">Webinar: Deploying Haskell apps with Kubernetes</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/2018/07/deploying-rust-with-docker-and-kubernetes/\">Blog post: Deploying rust with Docker and Kubernetes</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/2017/02/immutability-docker-haskells-st-type/\">Blog post: Immutability, Docker, and Haskell's ST type</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/2017/01/containerize-legacy-app/\">Blog post: Containerizing a legacy application: an overview</a></li>\n</ul>\n<p>From the web:</p>\n<ul>\n<li><a href=\"https://www.docker.com/resources/what-container\">What is a container?</a></li>\n<li><a href=\"https://www.docker.com/get-started\">Get started with Docker</a></li>\n<li><a href=\"https://kubernetes.io/docs/concepts/\">Kubernetes concepts</a></li>\n<li><a href=\"https://kubernetes.io/docs/setup/\">Getting started with Kubernetes</a></li>\n</ul>\n",
        "permalink": "https://www.fpcomplete.com/blog/our-history-containerization/",
        "slug": "our-history-containerization",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Our history with containerization",
        "description": "FP Complete has a long history of working with containers, beginning before Docker existed and staying ahead of advances in the technology.",
        "updated": null,
        "date": "2020-08-13",
        "year": 2020,
        "month": 8,
        "day": 13,
        "taxonomies": {
          "categories": [
            "devops"
          ],
          "tags": [
            "devops",
            "docker",
            "kubernetes"
          ]
        },
        "extra": {
          "author": "FP Complete Team",
          "blogimage": "/images/blog/cloud-computing.jpg"
        },
        "path": "blog/our-history-containerization/",
        "components": [
          "blog",
          "our-history-containerization"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "brief-history",
            "permalink": "https://www.fpcomplete.com/blog/our-history-containerization/#brief-history",
            "title": "Brief history",
            "children": []
          },
          {
            "level": 2,
            "id": "why-we-really-like-it",
            "permalink": "https://www.fpcomplete.com/blog/our-history-containerization/#why-we-really-like-it",
            "title": "Why we really like it",
            "children": []
          },
          {
            "level": 2,
            "id": "limitations-of-the-technology",
            "permalink": "https://www.fpcomplete.com/blog/our-history-containerization/#limitations-of-the-technology",
            "title": "Limitations of the technology",
            "children": []
          },
          {
            "level": 2,
            "id": "resources",
            "permalink": "https://www.fpcomplete.com/blog/our-history-containerization/#resources",
            "title": "Resources",
            "children": []
          }
        ],
        "word_count": 960,
        "reading_time": 5,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/cloud-deployment-models-advantages-and-disadvantages.md",
        "content": "<p>In this post we show a couple of options when it comes to a cloud\ndeployment model. Depending on the needs of your organization some\noptions may suit you better than others.</p>\n<h1 id=\"private-cloud\">Private Cloud</h1>\n<p>A private cloud is cloud infrastructure that only members of your organization\ncan utilize. It is typically owned and managed by the organization itself and\nis hosted on premises but it could also be managed by a third party in a secure\ndatacenter. This deployment model is best suited for organizations that deal\nwith sensitive data and/or are required to uphold certain security standards by\nvarious regulations.</p>\n<p>Advantages:</p>\n<ul>\n<li>Organization specific</li>\n<li>High degree of security and level of control</li>\n<li>Ability to choose your resources (ie. specialized hardware)</li>\n</ul>\n<p>Disadvantages:</p>\n<ul>\n<li>Lack of elasticity and capacity to scale (bursts)</li>\n<li>Higher cost</li>\n<li>Requires a significant amount of engineering effort</li>\n</ul>\n<h1 id=\"public-cloud\">Public Cloud</h1>\n<p>Public cloud refers to cloud infrastructure that is located and\naccessed over the public network. It provides a convenient way to\nburst and scale your project depending on the use and is typically\npay-per-use. Popular examples include <a href=\"https://aws.amazon.com\">Amazon AWS</a>,\n<a href=\"https://cloud.google.com/\">Google Cloud Platform</a> and <a href=\"https://azure.microsoft.com/\">Microsoft\nAzure</a>.</p>\n<p>Advantages:</p>\n<ul>\n<li>Scalability/Flexibility/Bursting</li>\n<li>Cost effective</li>\n<li>Ease of use</li>\n</ul>\n<p>Disadvantages:</p>\n<ul>\n<li>Shared resources</li>\n<li>Operated by third party</li>\n<li>Unreliability</li>\n<li>Less secure</li>\n</ul>\n<h1 id=\"hybrid-cloud\">Hybrid Cloud</h1>\n<p>This type of cloud infrastructure assumes that you are hosting your system both\non private and public cloud . One use case might be regulation requiring data\nto be stored in a locked down private data center but have the application\nprocessing parts available on the public cloud and talking to the private\ncomponents over a secure tunnel.</p>\n<p>Another example is hosting most of the system inside a private cloud and having\na clone of the system on the public cloud to allow for rapid scaling and\naccommodating bursts of new usage that would otherwise not be possible on the\nprivate cloud.</p>\n<p>Advantages:</p>\n<ul>\n<li>Cost effective</li>\n<li>Scalability/Flexibility</li>\n<li>Balance of convenience and security</li>\n</ul>\n<p>Disadvantages:</p>\n<ul>\n<li>Same disadvantages as the public cloud</li>\n</ul>\n<h1 id=\"multi-cloud\">Multi-Cloud</h1>\n<p>This option is a variant of the hybrid cloud but we refer to it when we mean\n&quot;using multiple public cloud providers&quot;. It is mostly used for mission critical\nsystems that want to minimize the amount of down time if a specific service on\na particular cloud goes down (e.g., the S3 outage of 2017 that took down a lot\nof web services with it). This option is arguably the most advanced option and\nsacrifices convenience for security and reliability. It requires significant\nexpertise and engineering effort to get right since most platforms vary widely\nbetween the type of resources and services that they provide in subtle ways.</p>\n<p>When chosing a cloud deployment model weigh the advantages and disadvantages of\neach option as it relates to your business objectives. </p>\n<p>If you liked this post you may also like: <a href=\"https://www.fpcomplete.com/blog/intro-to-devops-on-govcloud/\">Introduction to DevOps on AWS Gov Cloud</a></p>\n",
        "permalink": "https://www.fpcomplete.com/blog/cloud-deployment-models-advantages-and-disadvantages/",
        "slug": "cloud-deployment-models-advantages-and-disadvantages",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Cloud Deployment Models: Advantages and Disadvantages",
        "description": "Choosing the correct Cloud Deployment Model is crucial. Discover the advantages and disadvantages of each and how to choose the best one for your organization.",
        "updated": null,
        "date": "2020-08-07T13:41:00Z",
        "year": 2020,
        "month": 8,
        "day": 7,
        "taxonomies": {
          "categories": [
            "devops"
          ],
          "tags": [
            "devops"
          ]
        },
        "extra": {
          "author": "FP Complete Team",
          "blogimage": "/images/blog/deployment.jpg"
        },
        "path": "blog/cloud-deployment-models-advantages-and-disadvantages/",
        "components": [
          "blog",
          "cloud-deployment-models-advantages-and-disadvantages"
        ],
        "summary": null,
        "toc": [
          {
            "level": 1,
            "id": "private-cloud",
            "permalink": "https://www.fpcomplete.com/blog/cloud-deployment-models-advantages-and-disadvantages/#private-cloud",
            "title": "Private Cloud",
            "children": []
          },
          {
            "level": 1,
            "id": "public-cloud",
            "permalink": "https://www.fpcomplete.com/blog/cloud-deployment-models-advantages-and-disadvantages/#public-cloud",
            "title": "Public Cloud",
            "children": []
          },
          {
            "level": 1,
            "id": "hybrid-cloud",
            "permalink": "https://www.fpcomplete.com/blog/cloud-deployment-models-advantages-and-disadvantages/#hybrid-cloud",
            "title": "Hybrid Cloud",
            "children": []
          },
          {
            "level": 1,
            "id": "multi-cloud",
            "permalink": "https://www.fpcomplete.com/blog/cloud-deployment-models-advantages-and-disadvantages/#multi-cloud",
            "title": "Multi-Cloud",
            "children": []
          }
        ],
        "word_count": 486,
        "reading_time": 3,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/understanding-cloud-auth.md",
        "content": "<p>The topics of authentication and authorization usually appear simple but turn out to hide significant complexity. That's because, at its core, auth is all about answering two questions:</p>\n<ul>\n<li>Who are you</li>\n<li>What are you allowed to do</li>\n</ul>\n<p>However, the devil is in the details. Seasoned IT professionals, software developers, and even typical end users are fairly accustomed at this point to many of the most common requirements and pain points around auth.</p>\n<p>Cloud authentication and authorization is not drastically different from non-cloud systems, at least in principle. However, there are a few things about the cloud and its common use cases that introduce some curve balls:</p>\n<ul>\n<li>As with most auth systems, cloud providers each have their own idiosyncracies</li>\n<li>Cloud auth systems have almost always been designed from the outset to work API first, and interact with popular web technologies</li>\n<li>Security is usually taken very seriously in cloud, leading to workflows arguably more complex than other systems</li>\n<li>Cloud services themselves typically need some method to authenticate to the cloud, e.g. a virtual machine gaining access to private blob storage</li>\n<li>Many modern DevOps tools are commonly deployed to cloud systems, and introduce extra layers of complexity and indirection</li>\n</ul>\n<p>This blog post series is going to focus on the full picture of authentication and authorization, focusing on a cloud mindset. There is significant overlap with non-cloud systems in this, but we'll be covering those details as well to give a complete picture. Once we have those concepts and terms in place, we'll be ready to tackle the quirks of individual cloud providers and commonly used tooling.</p>\n<h2 id=\"goals-of-authentication\">Goals of authentication</h2>\n<p>We're going to define authentication as proving your identity to a service provider. A service provider can be anything from a cloud provider offering virtual machines, to your webmail system, to a bouncer at a bartender who has your name on a list. The identity is an equally flexible concept, and could be &quot;my email address&quot; or &quot;my user ID in a database&quot; or &quot;my full name.&quot;</p>\n<p>To help motivate the concepts we'll be introducing, let's understand what goals we're trying to achieve with typical authentication systems.</p>\n<ul>\n<li>Allow a user to prove who he/she is</li>\n<li>Minimize the number of passwords a user has to memorize</li>\n<li>Minimize the amount of work IT administrator have to do to create new user accounts, maintain them, and ultimately shut them down\n<ul>\n<li>That last point is especially important; no one wants the engineer who was just fired to still be able to authenticate to one of the systems</li>\n</ul>\n</li>\n<li>Provide security against common attack vectors, like compromised passwords or lost devices</li>\n<li>Provide a relatively easy-to-use method for user authentication</li>\n<li>Allow a computer program/application/service (lets call these all apps) to prove what it is</li>\n<li>Provide a simple way to allocate, securely transmit, and store credentials necessary for those proofs</li>\n<li>Ensure that credentials can be revoked when someone leaves a company or an app is no longer desired (or is compromised)</li>\n</ul>\n<h2 id=\"goals-of-authorization\">Goals of authorization</h2>\n<p>Once we know the identity of something or someone, the next question is: what are they allowed to do? That's where authorization comes into play. A good authorization provides these kinds of features:</p>\n<ul>\n<li>Fine grained control, when necessary, of who can do what</li>\n<li>Ability to grant common sets of permissions as a bundle, avoiding tedium and mistakes</li>\n<li>A centralized collection of authorization rules</li>\n<li>Ability to revoke a permission, and see that change propagated quickly to multiple systems</li>\n<li>Ability to delegate permissions from one identity to another\n<ul>\n<li>For example: if I'm allowed to read a file on some cloud storage server, it would be nice if I could let my mail client do that too, without the mail program pretending it's me</li>\n</ul>\n</li>\n<li>To avoid mistakes, it would be nice to assume a smaller set of permissions when performing some operations\n<ul>\n<li>For example: as a super user/global admin/root user, I'd like to be able to say &quot;I don't want to accidentally delete systems files right now&quot;</li>\n</ul>\n</li>\n</ul>\n<p>In simple systems, the two concepts of authentication and authorization is straightforward. For example, on a single-user computer system, my username would be my identity, I would authenticate using my password, and as that user I would be authorized to do anything on the computer system.</p>\n<p>However, most modern systems end up with many additional layers of complexity. Let's step through what some of these concepts are.</p>\n<h2 id=\"users-and-policies\">Users and policies</h2>\n<p>A basic concept of authentication would be a <em>user</em>. This typically would refer to a real human being accessing some service. Depending on the system, they may use identifiers like usernames or email addresses. User accounts are often times given to non-users, like automated processes or Continuous Integration (CI) jobs. However, most modern systems would recommend using a service account (discussed below) or similar instead.</p>\n<p>Sometimes, the user is the end of the story. When I log into my personal Gmail account, I'm allowed to read and write emails in that account. However, when dealing with multiuser shared systems, some form of permissions management comes along as well. Most cloud providers have a robust and sophisticated set of policies, where you can specify fine-grained individual permissions within a policy.</p>\n<p>As an example, with AWS, the S3 file storage service provides an array of individual actions from the obvious (read, write, and delete an object) to the more obscure (like setting retention policies on an object). You can also specify which files can be affected by these permissions, allowing a user to, for example, have read and write access in one directory, but read-only access in another.</p>\n<p>Managing all of these individual permissions each time for each user is tedious and error prone. It makes it difficult to understand what a user can actually do. Common practice is to create a few policies across your organization, and assign them appropriately to each user, trying to minimize the amount of permissions granted out.</p>\n<h2 id=\"groups\">Groups</h2>\n<p>Within the world of authorization, groups are a natural extensions of users and policies. Odds are you'll have multiple users and multiple policies. And odds are that you're likely to have groups of users who need to have similar sets of policy documents. You <em>could</em> create a large master policy that encompasses the smaller policies, but that could be difficult to maintain. You could also apply each individual policy document to each user, but that's difficult to keep track of.</p>\n<p>Instead, with groups, you can assign multiple policies to a group, and multiple groups to a user. If you have a billing team that needs access to the billing dashboard, plus the list of all users in the system, you may have a <code>BillingDashboard</code> policy as well as a <code>ListUsers</code> policy, and assign both policies to a <code>BillingTeam</code> group. You may then also assign the <code>ListUsers</code> policy to the <code>Operators</code> group.</p>\n<h2 id=\"roles\">Roles</h2>\n<p>There's a downside with this policies and groups setup described above. Even if I'm a superadmin on my cloud account, I may not want to have the responsibility of all those powers at all times. It's far too easy to accidentally destroy vital resources like a database server. Often, we would like to artificially limit our permissions while operating with a service.</p>\n<p>Roles allow us to do this. With roles, we create a named role for some set of operations, assign a set of policies to it, and provide some way for users to <em>assume</em> that role. When you assume that role, you can perform actions using that set of permissions, but audit trails will still be able to trace back to the original user who performed the actions.</p>\n<p>Arguably a cloud best practice is to grant users only enough permissions to assume various roles, and otherwise unable to perform any meaningful actions. This forces a higher level of stated intent when interacting with cloud APIs.</p>\n<h2 id=\"service-accounts\">Service accounts</h2>\n<p>Some cloud providers and tools support the concept of a service account. While users <em>can</em> be used for both real human beings and services, there is often a mismatch. For example, we typically want to enable multi-factor authentication on real user accounts, but alternative authentication schemes on services.</p>\n<p>One approach to this is service accounts. Service accounts vary among different providers, but typically allow defining some kind of service, receiving some secure token or password, and assigning either roles or policies to that service account.</p>\n<p>In some cases, such as Amazon's EC2, you can assign roles directly to cloud machines, allowing programs running on those machines to easily and securely assume those roles, without needing to store any kinds of tokens or secrets. This concept nicely ties in with roles for users, making role-based management of both users and services and emerging best practice in industry.</p>\n<h2 id=\"rbac-vs-acl\">RBAC vs ACL</h2>\n<p>The system described above is known as Role Based Access Control, or RBAC. Many people are likely familiar with the related concept known as Access Control Lists, or ACL. With ACLs, administrators typically have more work to do, specifically managing large numbers of resources and assigning users to each of those per-resource lists. Using groups or roles significantly simplifies the job of the operator, and reduces the likelihood of misapplied permissions.</p>\n<h2 id=\"single-sign-on\">Single sign-on</h2>\n<p>Most modern DevOps platforms have multiple systems, each requiring separate authentication. For example, in a modern Kubernetes-based deployment, you're likely to have:</p>\n<ul>\n<li>The underlying cloud vendor\n<ul>\n<li>Both command line and web based access</li>\n</ul>\n</li>\n<li>Kubernetes itself\n<ul>\n<li>Both command line access and the Kubernetes Dashboard</li>\n</ul>\n</li>\n<li>A monitoring dashboard</li>\n<li>A log aggregation system</li>\n<li>Other company-specific services</li>\n</ul>\n<p>That's in addition to maintaining a company's standard directory, such as Active Directory or G Suite. Maintaining this level of duplication among user accounts is time consuming, costly, and dangerous. Furthermore, while it's reasonable to securely lock down a single account via MFA and other mechanisms, expecting users to maintain such information for all of these systems securely is unreasonable. And some of these systems don't even provide such security mechanisms.</p>\n<p>Instead, single sign-on provides a standards-based, secure, and simple method for authenticating to these various systems. In some cases, user accounts still need to be created in each individual system. In those cases, automated user provisioning is ideal. We'll talk about some of that in later posts. In other cases, like AWS's identity provider mechanism, it's possible for temporary identifiers to be generated on-the-fly for each SSO-based login, with roles assigned.</p>\n<p>Deeper questions arise about where permissions management is handled. Should the central directory, like Active Directory, maintain permissions information for all systems? Should a single role in the directory represent permissions information in all of the associated systems? Should a separate set of role mappings be maintained for each service?</p>\n<p>Typically, organizations end up including some of each, depending on the functionality available in the underlying tooling, and organizational discretion on how much information to include in a directory.</p>\n<h2 id=\"going-deeper\">Going deeper</h2>\n<p>What we've covered here sets the stage for understanding many cloud-specific authentication and authorization schemes. Going forward, we're going to cover a look into common auth protocols, followed by a review of specific cloud providers and tools, specifically AWS, Azure, and Kubernetes.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/",
        "slug": "understanding-cloud-auth",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Understanding cloud auth",
        "description": "Authentication and authorization are a core component to any secure system. In this overview post, we will begin analyzing common patterns in cloud auth",
        "updated": null,
        "date": "2020-07-29",
        "year": 2020,
        "month": 7,
        "day": 29,
        "taxonomies": {
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "blogimage": "/images/blog/cloud-computing.jpg"
        },
        "path": "blog/understanding-cloud-auth/",
        "components": [
          "blog",
          "understanding-cloud-auth"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "goals-of-authentication",
            "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/#goals-of-authentication",
            "title": "Goals of authentication",
            "children": []
          },
          {
            "level": 2,
            "id": "goals-of-authorization",
            "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/#goals-of-authorization",
            "title": "Goals of authorization",
            "children": []
          },
          {
            "level": 2,
            "id": "users-and-policies",
            "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/#users-and-policies",
            "title": "Users and policies",
            "children": []
          },
          {
            "level": 2,
            "id": "groups",
            "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/#groups",
            "title": "Groups",
            "children": []
          },
          {
            "level": 2,
            "id": "roles",
            "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/#roles",
            "title": "Roles",
            "children": []
          },
          {
            "level": 2,
            "id": "service-accounts",
            "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/#service-accounts",
            "title": "Service accounts",
            "children": []
          },
          {
            "level": 2,
            "id": "rbac-vs-acl",
            "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/#rbac-vs-acl",
            "title": "RBAC vs ACL",
            "children": []
          },
          {
            "level": 2,
            "id": "single-sign-on",
            "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/#single-sign-on",
            "title": "Single sign-on",
            "children": []
          },
          {
            "level": 2,
            "id": "going-deeper",
            "permalink": "https://www.fpcomplete.com/blog/understanding-cloud-auth/#going-deeper",
            "title": "Going deeper",
            "children": []
          }
        ],
        "word_count": 1863,
        "reading_time": 10,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/understanding-devops-roles-and-responsibilities.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/understanding-devops-roles-and-responsibilities/",
        "slug": "understanding-devops-roles-and-responsibilities",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Understanding DevOps Roles and Responsibilities",
        "description": "Companies are implementing DevOps at an increasingly rapid rate. Discover the roles and responsibilities and how to implement DevOps into your latest project.",
        "updated": null,
        "date": "2020-07-24T13:12:00Z",
        "year": 2020,
        "month": 7,
        "day": 24,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "insights",
            "devops"
          ]
        },
        "extra": {
          "author": "FP Complete Team",
          "html": "hubspot-blogs/understanding-devops-roles-and-responsibilities.html",
          "blogimage": "/images/blog/exec-insight.jpg"
        },
        "path": "blog/understanding-devops-roles-and-responsibilities/",
        "components": [
          "blog",
          "understanding-devops-roles-and-responsibilities"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/preparing-for-cloud-computing-trends.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/preparing-for-cloud-computing-trends/",
        "slug": "preparing-for-cloud-computing-trends",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Preparing for Upcoming Cloud Computing Trends",
        "description": "Cloud Computing is growing at a rate 7 times faster than the rest of IT with no signs of slowing in the coming years. Discover all the trends businesses should be preparing for in order to succeed in 2020 and beyond. ",
        "updated": null,
        "date": "2020-07-24T11:05:00Z",
        "year": 2020,
        "month": 7,
        "day": 24,
        "taxonomies": {
          "categories": [
            "insights",
            "devops"
          ],
          "tags": [
            "devops"
          ]
        },
        "extra": {
          "author": "FP Complete Team",
          "html": "hubspot-blogs/preparing-for-cloud-computing-trends.html",
          "blogimage": "/images/blog/cloud-computing.jpg"
        },
        "path": "blog/preparing-for-cloud-computing-trends/",
        "components": [
          "blog",
          "preparing-for-cloud-computing-trends"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/cloud-preparation-checklist.md",
        "content": "<p>While moving to the cloud brings many benefits associated with it, we\nneed to also be aware of the pain points associated with such a move.\nThis post will discuss those pain points, provide ways to mitigate them, and\ngive you a checklist which can be used if you plan to migrate your\napplications to cloud. We will also discuss the advantages of\nmoving to the cloud.</p>\n<h2 id=\"common-pain-points\">Common pain points</h2>\n<p>One of the primary pain points in moving to the cloud is selecting the\nappropriate tools for a specific usecase. We have an abundance of tools\navailable, with many solving the same problem in different ways. To give\nyou a basic idea, this is the CNCF's (Cloud Native Computing\nFoundation) recommended path through the cloud native technologies:</p>\n<img src=\"/images/insights/cloud-prep-checklist/landscape.png\" alt=\"Cloud Native Landscape\" title=\"Cloud Native Landscape\" width=\"100%\">\n<p></p>\n<p>Picking the right tool is hard, and this is where having experience\nwith them comes in handy.</p>\n<p>Also, the existing knowledge of on-premises data centers may not be\ndirectly transferable when you plan to move to the cloud. An individual might\nhave to undergo a basic training to understand the terminology and the\nconcepts used by a particular cloud vendor. An on-premises system\nadministrator might be used to setting up firewalls via\n<a href=\"https://en.wikipedia.org/wiki/Iptables\">Iptables</a>, but he might also\nwant to consider using <a href=\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html\">Security\ngroups</a>\nif he plans to accomplish the same goals in the AWS ecosystem (for EC2 instances).</p>\n<p>Another point to consider while moving to the cloud is the ease with which you\ncan easily get locked in to a single vendor. You might start using\nAmazon's <a href=\"https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html\">Auto Scaling\nGroups</a>\nto automatically handle the load of your application. But when you plan\nto switch to another cloud vendor the migration might not be\nstraightforward. Switching between cloud services isn't easy, and if you want portability, you\nneed to make sure that your applications are built with a multi-cloud\nstrategy. This will allow you to easily switch between vendors if such a\nscenario arises. Taking advantage of containers and Kubernetes may give\nyou additional flexibility and ease portability between different cloud\nvendors.</p>\n<h2 id=\"advantages-of-moving\">Advantages of moving</h2>\n<p>Despite the pain points listed above, there are many advantages involved in\nmoving your applications to cloud. Note that even big media services\nprovider like\n<a href=\"https://netflixtechblog.com/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec\">Netflix</a>\nhas moved on to the cloud instead of building and managing their own\ndata center solution.</p>\n<h3 id=\"cost\">Cost</h3>\n<p>One of the primary advantages of leveraging the cloud is avoiding\nthe cost of building your\nown data center. Building a secure data center is not trivial. By\noffloading this activity to an external cloud provider, you can instead build your\napplications on top of the infrastructure provided by them. This not\nonly saves the initial capital expenditure but also saves headaches from\nreplacing hardware, such as replacing failing network switches. But note that\nswitching to the cloud will not magically save cost. Depending on your\napplication's architecture and workload, you have to be aware of the\nchoices you make and make sure that your choices are cost efficient.</p>\n<h3 id=\"uptime\">Uptime</h3>\n<p>Cloud vendors provide SLAs (Service Level Agreements) where they state\ninformation about uptime and the guarantees they make. This is a\nsnapshot from the Amazon Compute SLA:</p>\n<p><img src=\"/images/insights/cloud-prep-checklist/sla.png\" alt=\"SLA\" title=\"SLA\" /></p>\n<p>All major cloud providers have historically provided excellent uptime,\nespecially for applications that properly leverage availability zones.\nBut depending on a specific\nusecase/applications, you should define the acceptable uptime for your\napplication and make sure that your SLA matches with it. Also depending\non the requirements, you can architect your application such that it has\nmulti region deployments to provide a better uptime in case there is an\noutage in one region.</p>\n<h3 id=\"security-and-compliance\">Security and Compliance</h3>\n<p>Cloud deployments provide an extra benefit when working in regulated industries\nor with government projects. In many cases, cloud vendors provide regulation-compliant\nhardware.\nBy using cloud providers, we can take advantage of the various\ncompliance standards (eg: HIPAA, PCI etc) they meet.\nValidating an on-premises data center against such standards can be a time consuming,\nexpensive process. Relying on already validated hardware can be faster, cheaper, easier,\nand more reliable.</p>\n<p>Broadening the security topic, cloud vendors typically also provide\na wide range of additional security tools.</p>\n<p>Despite these boons,\nproper care must still be taken, and best practices must still be followed,\nto deploy an application securely.\nAlso, be aware that running on compliant hardware does not automatically\nensure compliance of the software. Code and infrastructure must still meet\nvarious standards.</p>\n<h3 id=\"ease-of-scaling\">Ease of scaling</h3>\n<p>With cloud providers, you can easily add and remove machines or add more\npower (RAM, CPU etc) to them. The ease with which you can horizontally and\nvertically scale your application without worrying about your\ninfrastructure is powerful, and can revolutionize how your approach\nhardware allocation. As your applications load increases,\nyou can easily scale up in a few minutes.</p>\n<p>One of the perhaps surprising benefits of this is that you don't need to\npreemptively scale up your hardware. Many cloud deployments are able\nto reduce the total compute capacity available in a cluster, relying\non the speed of cloud providers to scale up in response to increases in demand.</p>\n<h3 id=\"focus-on-problem-solving\">Focus on problem solving</h3>\n<p>With no efforts in maintaining the on-premises data center, you can\ninstead put your effort in your application and the problem it solves.\nThis allows you to focus on your core business problems and your\ncustomers.</p>\n<p>While not technically important, the cloud providers have energy\nefficient data centers and run it on better efficiency. As a case study,\n<a href=\"https://cloud.google.com/blog/topics/google-cloud-next/our-heads-in-the-cloud-but-were-keeping-the-earth-in-mind\">Google even uses machine learning technology to make its data centers\nmore\nefficient</a>.\nHence, it might be environmentally a better decision to run your\napplications on cloud.</p>\n<h2 id=\"getting-ready-for-cloud\">Getting ready for Cloud</h2>\n<p>Once you are ready for migrating to the cloud, you can plan for the next\nsteps and initiate the process. We have the following general checklist\nwhich we usually take and tailor it based on our clients requirements:</p>\n<h3 id=\"checklist\">Checklist</h3>\n<ul>\n<li>Make a list of your applications and dependencies which need to be\nmigrated.</li>\n<li>Benchmark your applications to establish cloud performance\nKPIs (Key Performance Indicators).</li>\n<li>List out any required compliance requirements for your\napplication and plan for ensuring it.</li>\n<li>Onboard relevant team members to the cloud service's use management\nsystem, ideally integrating with existing user directories and\nleveraging features like single sign on and automated user provisioning.</li>\n<li>Establish access controls to your cloud service, relying on role based\nauthorization techniques.</li>\n<li>Evaluate your migration options. You might want to re-architect it\nto take advantage of cloud-native technologies. Or you might simply\ndecide to shift the existing application without any changes.</li>\n<li>Create your migration plan in a Runbook.</li>\n<li>Have a rollback plan in case migration fails.</li>\n<li>Test your migration and rollback plans in a separate environment.</li>\n<li>Communicate about the migration to internal stakeholders and customers.</li>\n<li>Execute your cloud migration.</li>\n<li>Prune your on-premises infrastructure.</li>\n<li>Optimize your cloud infrastructure for your workloads.</li>\n</ul>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>I hope we were able to present you the challenges involved in\nmigration to cloud and how to prepare for them. We have helped various\ncompanies in migration and other devops services. Free feel to <a href=\"https://www.fpcomplete.com/contact-us/\">reach out to\nus</a> regarding any questions on\ncloud migrations or any of the other services.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/",
        "slug": "cloud-preparation-checklist",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Cloud preparation checklist",
        "description": "Considering a move to the cloud? Read up on cloud advantages, common pain points, and our recommended step by step process",
        "updated": null,
        "date": "2020-07-22",
        "year": 2020,
        "month": 7,
        "day": 22,
        "taxonomies": {
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Sibi Prabakaran",
          "blogimage": "/images/blog/cloud-computing.jpg"
        },
        "path": "blog/cloud-preparation-checklist/",
        "components": [
          "blog",
          "cloud-preparation-checklist"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "common-pain-points",
            "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#common-pain-points",
            "title": "Common pain points",
            "children": []
          },
          {
            "level": 2,
            "id": "advantages-of-moving",
            "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#advantages-of-moving",
            "title": "Advantages of moving",
            "children": [
              {
                "level": 3,
                "id": "cost",
                "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#cost",
                "title": "Cost",
                "children": []
              },
              {
                "level": 3,
                "id": "uptime",
                "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#uptime",
                "title": "Uptime",
                "children": []
              },
              {
                "level": 3,
                "id": "security-and-compliance",
                "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#security-and-compliance",
                "title": "Security and Compliance",
                "children": []
              },
              {
                "level": 3,
                "id": "ease-of-scaling",
                "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#ease-of-scaling",
                "title": "Ease of scaling",
                "children": []
              },
              {
                "level": 3,
                "id": "focus-on-problem-solving",
                "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#focus-on-problem-solving",
                "title": "Focus on problem solving",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "getting-ready-for-cloud",
            "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#getting-ready-for-cloud",
            "title": "Getting ready for Cloud",
            "children": [
              {
                "level": 3,
                "id": "checklist",
                "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#checklist",
                "title": "Checklist",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/cloud-preparation-checklist/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 1276,
        "reading_time": 7,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/devops-security-and-privacy-strategies.md",
        "content": "<p>DevOps Security and Privacy—FP Complete’s\ncomprehensive, easy to understand guide designed\nto help you understand why they’re so critical to\nthe safety of your DevOps strategy.</p>\n<p>The following is a transcription of a live\nwebinar given by <a href=\"https://www.fpcomplete.com/\">FP Complete</a>\nFounder and Chairman Aaron Contorer, on\n<a href=\"https://www.youtube.com/user/FPComplete\">FP Complete's YouTube Channel</a>.</p>\n<h2 id=\"introducing-aaron\">Introducing Aaron</h2>\n<p>I’m the Founder and Chairman of <a href=\"https://www.fpcomplete.com/\">FP Complete</a>,\nwhere we help companies use state-of-the-art tools and\ntechniques to produce secure, lightning-fast,\nfeature-rich software, faster and more often.</p>\n<p>Before founding FP Complete, I was an\nexecutive at Microsoft, where I served as program\nmanager for distributed systems, and general\nmanager of Visual C++, the leading software\ndevelopment tool at that time. Also, I \narchitected MSN’s move to Internet-based server\nsoftware, served as the full-time technology\nadviser to Bill Gates, and I founded and ran the\ncompany’s Productivity Tools Team for complex\nsoftware engineering projects.</p>\n<p>Okay, so enough about me. Let’s begin this\ndiscussion by recognizing our industry’s\nunfortunate—but preventable—reality:</p>\n<h2 id=\"breaches-are-happening-far-too-often\">Breaches are happening far too often</h2>\n<p>We all know how bad the state of the\nworld is within security and privacy\nright now. Projects are getting very\ncomplicated. And I—just as a sample—want\nto point out that this is a very typical\nbreach. Monzo said that for six months\nunauthorized people had access to\npeople’s secret code numbers, their pin\nnumbers. I’m not singling them out at\nall, but rather saying… “This is very\ntypical.” They’re a bank, and they\ncompromised this type of data for months\nand months.</p>\n<p>How does it happen? It’s not only\nbecause of logging and monitoring not\nbeing in place, although that can be a\nbig factor. It’s because of complexity.\nHonestly, we’re all trying very hard to\ndo our jobs, but users keep asking and\nexecutives keep asking for new features.\nAnd that integration just creates point\nafter point where problems can happen,\nand things get overlooked.</p>\n<h2 id=\"opportunities-for-penetration-are-everywhere\">Opportunities for penetration are everywhere</h2>\n<p>I would argue that today’s\napplications are more about assembling\nbuilding blocks than they are about just\nwriting new code. But every time you\nincrease that complexity by adding more\nbuilding blocks, you increase the number\nof interface points between\ncomponents—the number of places where\nsomebody might have done something wrong.\nAnd so we’re really creating a system of\nentry points between component A and\ncomponent B. But entry points—that sounds\nlike something I would compromise if I\nwere a security violator, right?\nFurthermore, we’re manually configuring\nour systems. People aren’t using\ncontinuous deployment. And so there is\nsome wizard who’s supposed to go set up\nthe latest server or integrate it with\nthe database or integrate it with the web\nwith a firewall or whatever they’re\nsupposed to do. Every manual step creates\nfurther opportunities for penetration,\nfor defects, because people are\nimperfect. Even the best person in your\nteam doing a process a hundred times\nmight do it wrong, one or two times. An\nautomated scanner is going to find that\ntime, and it’s going to break into your\nsystem before you know it.</p>\n<h2 id=\"let-s-talk-devsecops\">Let’s talk DevSecOps</h2>\n<p>DevSecOps—DevOps with security stuck\nright in the middle. And I think that’s a\ngood way of looking at this problem. We\nwant to integrate all the different parts\nof our engineering into one pool of\nautomation, and include security and\nquality assurance as part of that\nautomated process. We talked earlier\nabout automated testing being part of our\nbuilds. But we want to go much farther\nthan that, as technical teams. We want to\nstart from the beginning of our projects,\ntalking about how secure they need to be.\nWhat are the risks that they’re supposed\nto defend against or or not create? We\nwant every member of the team to\nunderstand that system downtime—because\nsomebody broke in and trashed it, or even\nworse privacy violations which you can\nnever undo, because when people’s\npersonal information has been published,\nyou can’t unpublish it—we need to let our\nteam members know that these are\npriorities and put them on the to-do list\nfor the project. And we can’t call it\nsomething done if the security part isn’t\ndone. It’s not something we tack on at\nthe end. We don’t build in unsecured,\ncrazy, poorly architected apps, and then\nat the end, ask someone to build a brick\nwall around them. Because as soon as one\nlittle person gets through the brick\nwall, it’s open season. So, we want the\nengineers to know everything they do\nshould be checked for security. That’s a\nculture change to say that it’s\neveryone’s job.</p>\n<p>We need to integrate quality assurance\nwith security, which means somebody is\nchecking the software we wrote for\nweaknesses; somebody is trying to break\nin or, at least, trying to run tools that\nwill show us common ways to break in and\nweather their presence.</p>\n<p>And we need to inspect our cloud\nsystems that are running to make sure\nthat our deployment, and our system\noperations and administration, is as\nsecure as we meant it to be. Did somebody\nomit a step? We want to discover that\nright away and fix it. Or, ideally,\nautomate the way we set up all of our\nsystems using, for example, an\norchestration software package to\nautomatically configure our servers, so\nit isn’t the case that late in the day,\npeople are more likely to make a mistake.\nBecause, well written scripts do just as\ngood a job even when they’re tired.</p>\n<p>And we want to make sure that all of\nour systems are updated and patched and\nnot tell people that security is a waste\nof time and they should get back to work\non features.</p>\n<h2 id=\"process-tips\">Process tips</h2>\n<p>To do all this, we need to have a\nsimple design. And I would encourage\npeople to focus on the idea that\nsimplicity and modular design are great\nways to make a system easier to check for\nsecurity holes.</p>\n<p>We want to make sure that credentials\nthat are used in our modular\nsystems—where one piece of software is\nlogging into another service or another\npiece of software database—are kept in\nproperly secured credential storage. A\ncommon form of security violations is you\nlook at somebody’s source code and… Oh\nlook! There’s the password for the\ndatabase server right there …because the\napp had to connect to the server. That’s\ninappropriate design. There are special\ncredential storage services—your team\nshould use them.</p>\n<p>And we want to make sure that quality\ncontrol remains central to our culture,\nas developers of software, and that\nincludes DevOps, that includes system\nadministration. Too often, we have a good\npiece of software, and then it’s deployed\nincorrectly. And that’s where the problem\noccurs. So if you’re going to test\nwhether your code is written properly,\nmaybe also test whether the servers\nconfigured properly, from time to time.\nIt’s time well spent.</p>\n<h2 id=\"how-to-strengthen-your-security\">How to strengthen your security</h2>\n<p>So how can you move forward on\nsecurity? The good news is, while it may\nsound like a scary and intimidating area,\nthere are lots of practical steps you can\ntake right now, and you don’t even have\nto take them all at the same time, you\ncan take them incrementally. Here are\nsome great steps though that I highly\nrecommend.</p>\n<p>One is that—in your engineering team,\nand if you have multiple teams—in each\nengineering team somebody is explicitly\nthe security person. Somebody knows that\nit’s their job to keep an eye out for\nsecurity issues and prevention and that\nif there’s a problem they’re the person\nwho’s going to hear about it. They should\nhave the power to look into anything they\nneed to make sure there isn’t a security\nhole in the system.</p>\n<p>Use best practices from other\ncompanies. This is a great idea\nthroughout all of DevOps, including\nDevSecOps. You don’t have to reinvent\nanything. You can learn best practices\nand get a checklist together of what\nother companies have found helpful to\nlook for to find opportunities to secure\nyour system incrementally. We just piece\nby piece chip away at the risks that are\npresent in our systems. We don’t have to\nwait until some magic day when all of\nsecurity happens at once.</p>\n<p>Teach your people about security. A\nlot of security problems happen because\none person didn’t realize… Who didn’t\nknow that you’re supposed to not put\npasswords in the source code where\neveryone can see them? Well, one person\ntyped a password into the source code,\nbut now it’s there for everyone. So be\nsure that training and security, and how\nimportant it is, and how to do it is\navailable to everyone in your team. And\nmake sure that there’s a checklist. Who\ntook the security training? Who’s not\nbeen to security training yet?</p>\n<p>Scary but true fact: You should,\naccording to Price Waterhouse Coopers, if\nyou want to be a normal IT operation, be\nspending 11 to 15% of your IT budget on\nsecurity overall. That’s a significant\nnumber. And I think we can all agree that\nwith more internet work and more\nimporting of modules and stuff, we, if\nanything, could be worried that that\nnumber is going to go up. So automation\nthrough DevOps is really a way to keep a\nlid on that number. But I wouldn’t think\nof it as a way to make that number drive\ndown towards zero. Security is everyone’s\njob, and it’s going to remain that\nway.</p>\n<p>Beyond that, I’d say use it use the\nother techniques we talked about earlier\nin this presentation. You don’t have to\nbe the next Equifax, of having no\nmonitoring. You don’t have to allow silly\nmistakes by having no automation. And you\ndon’t have to create more security holes\nby reinventing your own tools and\nprocesses using components. Reuse is your\nfriend.</p>\n<h2 id=\"7-tech-ideas-you-can-start-now\">7 tech ideas you can start now</h2>\n<p>I won’t spend too long on this, but I\nwanted this for people who are more\nhands-on or the people who are\nsupervising hands-on engineers. These are\nsome practical steps that you can take to\nstart turning on pieces of security,\nright now. Every one of these—except\nperhaps service-oriented architecture—is\nsomething that literally you could task\nsomebody to do this week or next\nweek.</p>\n<p>These are straightforward tasks.</p>\n<ol>\n<li>Ensure all databases have firewalls on them. They’re a common data breach source!</li>\n<li>Use a password manager to generate secure passwords; enable two-factor authentication.</li>\n<li>Use roles and policies to assign specific permissions to users and services instead of running everything from root credentials or privileged users.</li>\n<li>Use bastion hosts or VPNs to limit access to internal machines.</li>\n<li>Use service-oriented architecture (SOA) to break off components that need high privilege.</li>\n<li>Include code analysis tools in the dev process and enforce fixes prior to deployment.</li>\n<li>Test your servers with automated scanners for break-in vulnerabilities.</li>\n</ol>\n<h2 id=\"fast-to-market-reliable-and-secure\">Fast to market, reliable, and secure</h2>\n<p>It’s a winning formula!</p>\n<p>So, in short, you have a choice to\nturn on DevOps to use a lot of technology\nthat’s been solved, a lot of best\npractices and engineering techniques that\nhave already been solved and tested at\nnumerous other companies—clients of ours,\nfamous internet companies, everyone. When\nI say “everyone”, the truth is the\nminority of companies are already using\nproper DevOps. But enough companies that\nyou don’t have to be the first, you don’t\nhave to be the Pioneer. DevOps is a\nwinning formula that will get you to\nmarket faster, and more reliable, and\nwith better security. Or you could be the\nnext Equifax and the next Capital One,\nwhich is the default situation.</p>\n<h2 id=\"need-help-with-devops-security-and-privacy\">Need help with DevOps Security and Privacy?</h2>\n<p>FP Complete offers corporations its\nDevOps Success Program which offers\nadvanced Privacy and Security software\nengineering mentoring among many other\nmoving parts in the DevOps world.</p>\n<p>For more information, please <a href=\"https://www.fpcomplete.com/contact-us/\">contact us</a> or see our <a href=\"https://www.fpcomplete.com/devops/\">DevOps homepage</a>.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/",
        "slug": "devops-security-and-privacy-strategies",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "DevOps Security and Privacy Strategies",
        "description": "DevOps Security and Privacy—FP Complete’s comprehensive, easy to understand guide designed to help you understand why they’re so critical to the safety of your DevOps strategy. The following is a transcription of a live webinar given by FP Complete Founder and Chairman Aaron Contorer, on FP Complete’s YouTube Channel. I’m the Founder and Chairman of FP Complete, where we […]",
        "updated": null,
        "date": "2020-05-29",
        "year": 2020,
        "month": 5,
        "day": 29,
        "taxonomies": {
          "categories": [
            "devops"
          ],
          "tags": [
            "devops",
            "insights"
          ]
        },
        "extra": {
          "author": "Aaron Contorer",
          "blogimage": "/images/blog/network-security.jpg"
        },
        "path": "blog/devops-security-and-privacy-strategies/",
        "components": [
          "blog",
          "devops-security-and-privacy-strategies"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "introducing-aaron",
            "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/#introducing-aaron",
            "title": "Introducing Aaron",
            "children": []
          },
          {
            "level": 2,
            "id": "breaches-are-happening-far-too-often",
            "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/#breaches-are-happening-far-too-often",
            "title": "Breaches are happening far too often",
            "children": []
          },
          {
            "level": 2,
            "id": "opportunities-for-penetration-are-everywhere",
            "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/#opportunities-for-penetration-are-everywhere",
            "title": "Opportunities for penetration are everywhere",
            "children": []
          },
          {
            "level": 2,
            "id": "let-s-talk-devsecops",
            "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/#let-s-talk-devsecops",
            "title": "Let’s talk DevSecOps",
            "children": []
          },
          {
            "level": 2,
            "id": "process-tips",
            "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/#process-tips",
            "title": "Process tips",
            "children": []
          },
          {
            "level": 2,
            "id": "how-to-strengthen-your-security",
            "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/#how-to-strengthen-your-security",
            "title": "How to strengthen your security",
            "children": []
          },
          {
            "level": 2,
            "id": "7-tech-ideas-you-can-start-now",
            "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/#7-tech-ideas-you-can-start-now",
            "title": "7 tech ideas you can start now",
            "children": []
          },
          {
            "level": 2,
            "id": "fast-to-market-reliable-and-secure",
            "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/#fast-to-market-reliable-and-secure",
            "title": "Fast to market, reliable, and secure",
            "children": []
          },
          {
            "level": 2,
            "id": "need-help-with-devops-security-and-privacy",
            "permalink": "https://www.fpcomplete.com/blog/devops-security-and-privacy-strategies/#need-help-with-devops-security-and-privacy",
            "title": "Need help with DevOps Security and Privacy?",
            "children": []
          }
        ],
        "word_count": 2012,
        "reading_time": 11,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/rapid-devops-success.md",
        "content": "<p>Continuous integration and deployment, monitoring and logging, and security and privacy—FP Complete’s comprehensive, easy to understand guide designed to help you learn why those three DevOps strategies collectively create an environment where high-quality software can be developed quicker and more efficiently than ever before.</p>\n<p>Aaron Contorer, founder and chairman of FP Complete, presented the following webinar. Read below for a transcript of the video.</p>\n<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/5U11unR_py0\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n<h2 id=\"introducing-aaron\">Introducing Aaron</h2>\n<p>I’m the Founder and Chairman of <a href=\"https://www.fpcomplete.com/\">FP Complete</a>,\nwhere we help companies use\nstate-of-the-art tools and techniques to\nproduce secure, lightning-fast,\nfeature-rich software, faster and more\noften.</p>\n<p>Before founding FP Complete, I was an\nexecutive at Microsoft, where I served as\nprogram manager for distributed systems,\nand general manager of Visual C++, the\nleading software development tool at that\ntime. Also, I architected MSN’s move\nto Internet-based server software, served\nas the full-time technology adviser to\nBill Gates, and I founded and ran the\ncompany’s Productivity Tools Team for\ncomplex software engineering\nprojects.</p>\n<p>Okay, so enough about me. Let’s begin\nthis presentation by stating the\nobvious:</p>\n<h2 id=\"software-development-is-complicated\">Software development is complicated</h2>\n<p>As information technology and software\npeople, it’s easy to recognize how things\nare changing at an astonishing speed. To\nkeep pace, we need tools and processes\nthat allow us to rapidly deploy better\ncode more frequently with fewer errors.\nIs that a high bar to reach? Yes, of\ncourse, it is. But it absolutely must be\nmet—that is <em>if</em> you\nwant your company to survive.</p>\n<h2 id=\"inefficiencies-are-everywhere\">Inefficiencies are everywhere</h2>\n<p>In most companies, I would argue that\nthe information technology team and the\nsoftware engineering team are not totally\ntrusted by the rest of the company.</p>\n<p>Of course, I don’t mean they’re not\ntrusted as in they’re not good, smart\npeople. What I mean is that they don’t\nmeet their deadlines, leading to sprints\nbecoming longer than initially expected,\nultimately causing everyone to feel\nrushed and end results lacking in\nquality.</p>\n<h2 id=\"it-has-lost-management-s-trust\">IT has lost management’s trust</h2>\n<p>When management begins to not trust\nengineering and IT, a bad dynamic\ndevelops. No longer does the team get to\nfocus on building great things for their\nend-users. Instead, they’re forced to\nfocus on solving their struggles and\ndealing with interpersonal friction.</p>\n<p>Believe it or not, the problems we’re\nhaving aren’t people-problems. It’s not\nthat they lack good intentions or\nbrainpower.</p>\n<p>Instead, the problem is this:</p>\n<h2 id=\"modern-software-ancient-tech\">Modern software, ancient tech</h2>\n<p><strong>Modern software development can’t be performed using ancient technologies applied within simplistic workflows.</strong></p>\n<p>I often like to say…</p>\n<p><em>“The best craftsperson with a\nhandsaw cannot do woodworking as\nefficiently as a robotic cutting\ntool.”</em></p>\n<p>When we automate our work, it becomes\nfaster and easier to replicate. We don’t\nbuild in lots of mistakes. As a result,\nwe get to move on with our lives instead\nof going back and reworking things over\nand over again.</p>\n<p>When we automate with good tools and\nbetter processes programmed in, and we\nrepeat this same process every time,\neveryone can trust that our work will be\nperformed with quality, and our systems\nwill be more safe and secure.</p>\n<p>Sounds ideal, doesn’t it? Of course,\nit does.</p>\n<p>But how do you do it? How do you\nevolve from the environment you’re\noperating in today to the utopia DevOps\nstrategies will allow you to live and\nwork within well into the future?</p>\n<p>Learn more about how <a href=\"https://www.fpcomplete.com/devops/\">FP Complete does DevOps</a>.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/rapid-devops-success/",
        "slug": "rapid-devops-success",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Webinar Review: Learn Rapid DevOps Success",
        "description": "Continuous integration and deployment, monitoring and logging, and security and privacy—FP Complete’s comprehensive, easy to understand guide designed to help you learn why...",
        "updated": null,
        "date": "2020-05-29",
        "year": 2020,
        "month": 5,
        "day": 29,
        "taxonomies": {
          "categories": [
            "devops"
          ],
          "tags": [
            "devops",
            "insights"
          ]
        },
        "extra": {
          "author": "Aaron Contorer",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/rapid-devops-success/",
        "components": [
          "blog",
          "rapid-devops-success"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "introducing-aaron",
            "permalink": "https://www.fpcomplete.com/blog/rapid-devops-success/#introducing-aaron",
            "title": "Introducing Aaron",
            "children": []
          },
          {
            "level": 2,
            "id": "software-development-is-complicated",
            "permalink": "https://www.fpcomplete.com/blog/rapid-devops-success/#software-development-is-complicated",
            "title": "Software development is complicated",
            "children": []
          },
          {
            "level": 2,
            "id": "inefficiencies-are-everywhere",
            "permalink": "https://www.fpcomplete.com/blog/rapid-devops-success/#inefficiencies-are-everywhere",
            "title": "Inefficiencies are everywhere",
            "children": []
          },
          {
            "level": 2,
            "id": "it-has-lost-management-s-trust",
            "permalink": "https://www.fpcomplete.com/blog/rapid-devops-success/#it-has-lost-management-s-trust",
            "title": "IT has lost management’s trust",
            "children": []
          },
          {
            "level": 2,
            "id": "modern-software-ancient-tech",
            "permalink": "https://www.fpcomplete.com/blog/rapid-devops-success/#modern-software-ancient-tech",
            "title": "Modern software, ancient tech",
            "children": []
          }
        ],
        "word_count": 592,
        "reading_time": 3,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/rust-devops.md",
        "content": "<p>On February 2, 2020, one of FP Complete's Lead Software Engineers—Mike McGirr—presented a webinar on using Rust for creating DevOps tooling.</p>\n<h2 id=\"webinar-outline\">Webinar Outline</h2>\n<p>FP Complete is hosting a functional programming\nwebinar on, “Learn Rapid Rust with DevOps Success\nStrategies.” A beginner’s guide including sample Rust\ndemonstration on writing your DevOps tools with Rust\nover Haskell. An introduction to Rust, with basic DevOps\nuse cases, and the library ecosystem, airing on\nFebruary 5th, 2020.</p>\n<p>The webinar will be hosted by Mike McGirr, a DevOps\nSoftware Engineer at FP Complete which will provide an\nabundance of Rust information with respect to\nfunctional programming and DevOps, featuring (safety,\nspeed and accuracy) that make it unique and contributes\nto its popularity, and its possible preference as a\nlanguage of choice for operating systems over Haskell,\nweb browsers and device drivers among others. The\nwebinar offers an interesting opportunity to learn and\nuse Rust in developing real world projects aside from\nHaskell or other functional programming languages\navailable today.</p>\n<h2 id=\"topics-covered\">Topics covered</h2>\n<p>During the webinar we will cover the following\ntopics:</p>\n<ul>\n<li>A quick intro and background into the Rust programming language</li>\n<li>Some scenarios and reasons why you would want to use Rust for writing your DevOps tooling (and some reasons why you wouldn’t)</li>\n<li>A small example of using the existing AWS libraries to create   a basic DevOps tool</li>\n<li>How to Integrate FP into your Organization</li>\n</ul>\n<p>Mike Mcgirr, a Lead Software Engineer at FP\nComplete,will help us understand reasoning that\nsupports using Rust over other functional programming\nlanguages offered in the market today.</p>\n<h2 id=\"more-about-your-host\">More about your host</h2>\n<p>The webinar will be hosted by Mike McGirr, a veteran\nDevOps Software Engineer at FP Complete. With years of\nexperience in DevOps software development, Mike will\nwalk us through a first in a series of Rust webinars\ndiscussing why we would, and how we could utilize Rust\nas a functional programming language to build DevOps\nover other functional programming languages available\nin the market today. Mike will also share with us a\nsmall example script written in Rust showing how Rust\nmay be used.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/rust-devops/",
        "slug": "rust-devops",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Rust with DevOps Success Strategies",
        "description": "Wednesday Feb 5th, 2020, at 10:00 AM PST. Webinar Outline: FP Complete is hosting a functional programming webinar on, “Learn Rapid Rust with DevOps Success Strategies.” A beginner’s guide including sample Rust demonstration on writing your DevOps tools with Rust over Hasell. An introduction to Rust, with basic DevOps use cases, and the library ecosystem, […]",
        "updated": null,
        "date": "2020-02-05",
        "year": 2020,
        "month": 2,
        "day": 5,
        "taxonomies": {
          "categories": [
            "functional programming",
            "devops"
          ],
          "tags": [
            "devops",
            "rust",
            "insights"
          ]
        },
        "extra": {
          "author": "Mike McGirr",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/rust-devops/",
        "components": [
          "blog",
          "rust-devops"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "webinar-outline",
            "permalink": "https://www.fpcomplete.com/blog/rust-devops/#webinar-outline",
            "title": "Webinar Outline",
            "children": []
          },
          {
            "level": 2,
            "id": "topics-covered",
            "permalink": "https://www.fpcomplete.com/blog/rust-devops/#topics-covered",
            "title": "Topics covered",
            "children": []
          },
          {
            "level": 2,
            "id": "more-about-your-host",
            "permalink": "https://www.fpcomplete.com/blog/rust-devops/#more-about-your-host",
            "title": "More about your host",
            "children": []
          }
        ],
        "word_count": 351,
        "reading_time": 2,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/what_is_govcloud.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2019/05/what_is_govcloud/",
        "slug": "what-is-govcloud",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "What is GovCloud?",
        "description": "Devops, FedRAMP Compliance, and Making your Migration to GovCloud Successful - What is GovCloud?",
        "updated": null,
        "date": "2019-05-28T17:54:00Z",
        "year": 2019,
        "month": 5,
        "day": 28,
        "taxonomies": {
          "tags": [
            "devops",
            "aws",
            "govcloud"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "J Boyer",
          "html": "hubspot-blogs/what_is_govcloud.html",
          "blogimage": "/images/blog/cloud-computing.jpg"
        },
        "path": "blog/2019/05/what_is_govcloud/",
        "components": [
          "blog",
          "2019",
          "05",
          "what_is_govcloud"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/deploying_haskell_apps_with_kubernetes.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/deploying_haskell_apps_with_kubernetes/",
        "slug": "deploying-haskell-apps-with-kubernetes",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Deploying Haskell Apps with Kubernetes",
        "description": "This webinar describes how to Deploy Haskell applications using Kubernetes. Topics to be discussed include creation of a Kube cluster using Terraform and Kops, describe pods, deployments, services, load balancers, etc., deployment of a built image using kubectl and deploy, and more.",
        "updated": null,
        "date": "2018-09-11T16:24:00Z",
        "year": 2018,
        "month": 9,
        "day": 11,
        "taxonomies": {
          "tags": [
            "haskell",
            "devops"
          ],
          "categories": [
            "functional programming",
            "devops"
          ]
        },
        "extra": {
          "author": "Robert Bobbett",
          "html": "hubspot-blogs/deploying_haskell_apps_with_kubernetes.html",
          "blogimage": "/images/blog/kubernetes.jpg"
        },
        "path": "blog/deploying_haskell_apps_with_kubernetes/",
        "components": [
          "blog",
          "deploying_haskell_apps_with_kubernetes"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/devsecops.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/devsecops/",
        "slug": "devsecops",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "DevSecOps - Putting the Sec in DevOps",
        "description": "With today's tremendous security pressures, DevOps teams are moving to continuous development and integration, but continuous security is harder to integrate. To better understand how to secure your DevOps and protect your network read on.",
        "updated": null,
        "date": "2018-07-18T13:11:00Z",
        "year": 2018,
        "month": 7,
        "day": 18,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Robert Bobbett",
          "html": "hubspot-blogs/devsecops.html",
          "blogimage": "/images/blog/network-security.jpg"
        },
        "path": "blog/devsecops/",
        "components": [
          "blog",
          "devsecops"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/deploying-rust-with-docker-and-kubernetes.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2018/07/deploying-rust-with-docker-and-kubernetes/",
        "slug": "deploying-rust-with-docker-and-kubernetes",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Deploying Rust with Docker and Kubernetes",
        "description": "Using a tiny Rust app to demonstrate deploying Rust with Docker and Kubernetes.",
        "updated": null,
        "date": "2018-07-17T14:36:00Z",
        "year": 2018,
        "month": 7,
        "day": 17,
        "taxonomies": {
          "tags": [
            "rust",
            "devops",
            "kubernetes"
          ],
          "categories": [
            "functional programming",
            "devops"
          ]
        },
        "extra": {
          "author": "Chris Allen",
          "html": "hubspot-blogs/deploying-rust-with-docker-and-kubernetes.html",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/2018/07/deploying-rust-with-docker-and-kubernetes/",
        "components": [
          "blog",
          "2018",
          "07",
          "deploying-rust-with-docker-and-kubernetes"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/devops-to-prepare-for-a-blockchain-world.md",
        "content": "<h2 id=\"introduction\">Introduction</h2>\n<p>As the world adopts blockchain technologies, your IT infrastructure — and its\npredictability — become critical. Many companies lack the levels of automation\nand control needed to survive in this high-opportunity, high-threat environment.</p>\n<p>Are your software, cloud, and server systems automated and robust enough? Do you\nhave enough quality control for both your development and your online operations?\nOr will you join the list of companies bruised by huge data breaches and loss o\nf control over their own computer systems? If you are involved in blockchain, or\nany industry for that matter, these are the questions you need to ask yourself.</p>\n<p>Blockchain will require you to put more information online than ever before,\ncreating huge exposures for organizations that do not have a handle on their\nsecurity. Modern DevOps technologies, including many open-source systems, offer\npowerful solutions that can improve your systems to a level suitable for use with\nblockchain.</p>\n<h2 id=\"are-companies-really-ready-for-blockchain-technology\">Are companies REALLY ready for Blockchain technology?</h2>\n<p>The answer to it is most of the companies are NOT and those who are need to audit\nor reevaluate whether they are. The reason is BlockChain puts data to public making\nit prone to outside attacks if systems are not hardenend and updated on timely\nmanner.</p>\n<p>Big companies such as Equifax had millions of records stolen, Heartland credit\nprocessing was hacked and eventually had to pay 110 million and Airbus A400M due \nto wrong installation of manual software patch resulted in death of everyone on\non the plain. These are few of many such big companies that was hacked due to poorly\nimplemented IT technology.</p>\n<p>Once hailed as unhackable, blockchains are now getting hacked. According to a MIT\ntechnology review, hackers have stolen nearly $2 billion worth of cryptocurrency\nsince the beginning of 2017.</p>\n<h2 id=\"big-question-why-companies-are-getting-hacked\">Big Question: Why Companies are getting hacked ?</h2>\n<p>Blockchain itself isn't always the problem. Sometimes the blockchain is  secure \nbut the IT infrastructure is not capable to supporting it. There are cases where \nopen firewalls, unencrypted data, poor testing and manual errors were reasons \nbehind the hacking.</p>\n<p>So, the question to ask is: Is the majority of your IT infrastructure secure \nand reliable enough to support Blockchain Technology ?</p>\n<h2 id=\"what-is-an-it-factory\">What is an IT Factory ?</h2>\n<p>IT factory as per <a href=\"https://www.fpcomplete.com/our-team/\">Aaron Contorer</a>, founder \nand Chariman of FP Complete is divided into 3 parts</p>\n<ol>\n<li>Development</li>\n<li>Deployment</li>\n<li>System Operations</li>\n</ol>\n<p>If IT factory is implemented properly at each stage it could result in a new and\nbetter IT services leading to a more reliable, scalable and secure environment.</p>\n<p>Deployment is a bridge that allows software running on a developer laptop all the\nway to a scalable system and running Ops for monitoring. With DevOps practice,\nwe can ensure all the three stages of IT factory implemented.</p>\n<p>But, the key to build a working IT factory is Automation that ensure each step\nin the deployment process is reliable. With microservices architecture ,building\nand testing a reliable containerized based system is much easier now compared to\nthe earlier days.</p>\n<p>The only way to ensure a reliable, reproducible system is if companies start\nautomating each step of their software life cycle journey. Companies that are ensuring\ngood DevOps practices have a robust IT infrastructure compared to those that are\nNOT.</p>\n<h2 id=\"devops-for-blockchain\">DevOps for Blockchain</h2>\n<p>DevOps tools helps BlockChain better as it can ensure all code is tracked, tested,\ndeployed automatically, audited and Quality Assurance tested along each stage of\nthe delivery pipeline.</p>\n<p>The other benefits of having DevOps methods implemented in BlockChain is that it \nreduces the overall operational cost to companies, speeds up the overall pace of \nsoftware development and release cycle, improves the software quality and increases\nthe productivity.</p>\n<p>The following DevOps methods, if implemented in Blockchain, can be very helpful</p>\n<p><strong>1. Engineer for Safety</strong></p>\n<ul>\n<li>With proper version control tool like GITHUB , source code can be viewed,\ntracked with proper history of all changes to the base</li>\n<li>Development tools used by developers should be of the same version, should be\ntracked and should be  uniform across the project</li>\n<li>Continuous Integration (CI) pipeline must be implemented at the development\nstage to ensure nothing breaks on each commit. There are tools such as Jenkins,\nBamboo, Code Pipeline and many more that can help in setting up a proper CI .</li>\n<li>Each commit should be properly tested using test case management system with\nproper unit test cases for each commit</li>\n<li>Each Project should also have an Issue tracking system like JIRA, GITLAB etc\nto ensure all requests are properly tracked and closed.</li>\n</ul>\n<p><strong>2. Deploy for Safety</strong></p>\n<ul>\n<li>Continuous Deployment via DevOps tools to ensure code is automatically deployed\nto each environment</li>\n<li>Each environment (Development, Testing, DR, Production) should be a replica\nof each other</li>\n<li>Allow automation to setup all relevant infrastructure related to allow successful\ndeployment of code</li>\n<li>Setup infrastructure as code (IAC) to provision infrastructure that helps in\nreducing manual errors</li>\n<li>Sanity of each deployment by running test cases to ensure each component is\nfunctioning as expected</li>\n<li>Running Security testing after each Deployment on each environment</li>\n<li>Ensure system can be  RollBack/Rollforward without any manual intervention like\nCanary/Blue-Green Deployment</li>\n<li>Use container based deployments that provide more reliability for deployments</li>\n</ul>\n<p><strong>3. Operate for Safety</strong></p>\n<ul>\n<li>Set up Continuous Automated Monitoring and Logging</li>\n<li>Set up Anomaly detection and alerting mechanism</li>\n<li>Set up Automated Response and Recovery for any failures</li>\n<li>Ensure a Highly Available and scalable system for reliability</li>\n<li>Ensure data is encrypted for all outbound and inbound communication</li>\n<li>Ensure separation of admin powers, database powers, deployment powers , user \naccess etc. The more the powers are separated the lesser the risk</li>\n</ul>\n<p><strong>4. Separate for Safety</strong></p>\n<ul>\n<li>Separate each system internally from each other by using multiple small networks.\nFor Eg: database/backend on private subnets while UI on public subnets</li>\n<li>Set Internal and MutFirewalls ensure the database systems are protected with no access</li>\n<li>Separate Responsibility and credentials for reduce risk of exposure</li>\n</ul>\n<p><strong>5. Human systems</strong></p>\n<p>Despite keeping hardware and software checks, most the breaking of blockchain\nsystems today has happened because of &quot;People&quot; or &quot;Human Errors&quot;.</p>\n<p>Most people try hacks/workaround to get stuff working on production with no knowledge\non the impacts it could do on the system. Sometimes these stuff are not documented\nmaking it hard for the other person to fix it. Sometimes asking others to login\nto unauthorized systems by sharing credentials over calls paves a path for unsecure\nsystems</p>\n<p>To ensure companies must,</p>\n<ul>\n<li>Train people to STOP doing manual efforts to fix a broken system.</li>\n<li>Train people  NOT to do &quot;Social Engineering&quot; like asking colleagues \nto login to systems on their behalf, sharing passwords etc.</li>\n</ul>\n<p><strong>6. Quality Assurance</strong></p>\n<ul>\n<li>Need to review the Architectural as well as best practices are ensured in the\nproduct life cycle</li>\n<li>Need to ensure the code deploy pipeline has scope for penetration Testing</li>\n<li>Need to ensure there is weekly/monthly auditing of metrics, logs , systems to\ncheck for threats to the systems</li>\n<li>Each component and patch on system should be tested and approved by QA before\nrolling out to Production</li>\n<li>Companies could also hire third parties to audit their system on their behalf</li>\n</ul>\n<h2 id=\"how-to-get-there\">How to get there ?</h2>\n<p>The good news is &quot;IT IS POSSIBLE&quot;. There is no need for giant or all-in-one solutions.</p>\n<p>Companies that are starting fresh need  to start at the early phase of development\nto building a reliable system by focussing on above 6 points mentioned above. They\nneed to start thinking on all areas in the &quot;Plan and Design&quot; phase itself.</p>\n<p>For companies who are already on production or nearing production does not need\nto have to start fresh . They can start making incremental progress but it needs\nto start TODAY.</p>\n<p>Automation is the only SCIENCE in IT that can reduce errors and help towards building \na more and more reliable system. It will in the future save money and resources that \ncan be redirected to focus on other areas.</p>\n<p>To conclude, <a href=\"https://www.fpcomplete.com\">FP Complete</a> has been a leading consultant \non providing DevOps services. We excel at what we do and if you are looking to implement \nDevOps in your BlockChain. Please feel free to reach out to us for free consultations.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/devops-to-prepare-for-a-blockchain-world/",
        "slug": "devops-to-prepare-for-a-blockchain-world",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "DevOps to Prepare for a Blockchain World",
        "description": "This webinar describes how Devops can be used to prepare any company that is interested in adopting blockchain technology. Many companies lack the level of automation and control needed to survive in this high-opportunity, high-threat environment but DevOps technologies can offer powerful solutions.",
        "updated": null,
        "date": "2018-06-07T08:03:00Z",
        "year": 2018,
        "month": 6,
        "day": 7,
        "taxonomies": {
          "tags": [
            "devops",
            "blockchain"
          ],
          "categories": [
            "functional programming",
            "devops"
          ]
        },
        "extra": {
          "author": "FP Complete Team",
          "blogimage": "/images/blog/distributed-ledge.jpg"
        },
        "path": "blog/devops-to-prepare-for-a-blockchain-world/",
        "components": [
          "blog",
          "devops-to-prepare-for-a-blockchain-world"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "introduction",
            "permalink": "https://www.fpcomplete.com/blog/devops-to-prepare-for-a-blockchain-world/#introduction",
            "title": "Introduction",
            "children": []
          },
          {
            "level": 2,
            "id": "are-companies-really-ready-for-blockchain-technology",
            "permalink": "https://www.fpcomplete.com/blog/devops-to-prepare-for-a-blockchain-world/#are-companies-really-ready-for-blockchain-technology",
            "title": "Are companies REALLY ready for Blockchain technology?",
            "children": []
          },
          {
            "level": 2,
            "id": "big-question-why-companies-are-getting-hacked",
            "permalink": "https://www.fpcomplete.com/blog/devops-to-prepare-for-a-blockchain-world/#big-question-why-companies-are-getting-hacked",
            "title": "Big Question: Why Companies are getting hacked ?",
            "children": []
          },
          {
            "level": 2,
            "id": "what-is-an-it-factory",
            "permalink": "https://www.fpcomplete.com/blog/devops-to-prepare-for-a-blockchain-world/#what-is-an-it-factory",
            "title": "What is an IT Factory ?",
            "children": []
          },
          {
            "level": 2,
            "id": "devops-for-blockchain",
            "permalink": "https://www.fpcomplete.com/blog/devops-to-prepare-for-a-blockchain-world/#devops-for-blockchain",
            "title": "DevOps for Blockchain",
            "children": []
          },
          {
            "level": 2,
            "id": "how-to-get-there",
            "permalink": "https://www.fpcomplete.com/blog/devops-to-prepare-for-a-blockchain-world/#how-to-get-there",
            "title": "How to get there ?",
            "children": []
          }
        ],
        "word_count": 1354,
        "reading_time": 7,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/controlling-access-to-nomad-clusters.md",
        "content": "<p>In this blog post, we will learn how to control access to nomad.</p>\n<h2 id=\"introduction\">Introduction</h2>\n<p>Nomad is an application scheduler, that helps you schedule application-processes\nefficiently, across multiple servers, and keep your infrastructure costs low.\nNomad is capable of scheduling containers, virtual machines, as well as isolated\nforked processes.</p>\n<p>There are other schedulers available, such as Kubernetes, Mesos or Docker Swarm,\nbut each has different mechanisms for securing access. By following this post,\nyou will understand the main components in securing your Nomad cluster, but the\noverall idea is valid across any of the other schedulers available.</p>\n<p>One of Nomad's selling points, and why you could consider it over tools like\nKubernetes, is that you can schedule not only containers, but also QEMU\nimages, LXC, isolated <code>fork/exec</code> processes, and even Java applications in a\nchroot(!). All you need is a driver implemented for Nomad. On the other hand,\nits community is smaller than Kubernetes, so the tradeoffs have to be measured\non a project-by-project basis.</p>\n<p>We will start by deploying a test cluster and configuring access control lists\n(ACLs).</p>\n<h2 id=\"overview\">Overview</h2>\n<ul>\n<li>Nomad uses tokens to authenticate client requests.</li>\n<li>Each token is associated with policies.</li>\n<li>Policies are a collection of rules to allow or deny operations on resources.</li>\n</ul>\n<p>In this tutorial, we will:</p>\n<ol>\n<li>Setup our environment to run nomad inside a Vagrant virtual machine for running experiments</li>\n<li>We generate a root/admin token (usually known as the &quot;management&quot; token) and activate ACLs</li>\n<li>Using the management token, we add a new &quot;non-admin&quot; policy and create a token associated with this new policy</li>\n<li>Use the &quot;non-admin&quot; token to demonstrate access control.</li>\n</ol>\n<h2 id=\"setup-the-environment\">Setup the environment</h2>\n<p>Pre-requisites:</p>\n<ul>\n<li>POSIX shell, such as GNU Bash</li>\n<li>Vagrant &gt; <code>2.0.1</code></li>\n<li>Nomad demo <a href=\"https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/Vagrantfile\"><code>Vagrantfile</code></a></li>\n</ul>\n<p>We will run everything from within a virtual machine with all the necessary\nconfiguration and applications. Execute the following commands on your shell:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>$ cd $(mktemp --directory)\n$ curl -LO https://raw.githubusercontent.com/hashicorp/nomad/master/demo/vagrant/Vagrantfile\n$ vagrant up\n    ...\n    lines and lines of Vagrant output\n    this might take a while\n    ...\n$ vagrant ssh\n    ...\n    Message of the day greeting from VM\n    Anything after this point is being executed inside the virtual machine\n    ...\nvagrant@nomad:~$ nomad version\nNomad vX.X.X\nvagrant@nomad:~$ uname -n\nnomad\n</code></pre>\n<p>Depending on your system and the version of <code>Vagrantfile</code> used, the prompt may\nbe different.</p>\n<h2 id=\"setup-nomad\">Setup Nomad</h2>\n<p>We configure nomad to execute both as server and client for convenience, as\nopposed to a production environment where the server is remote and client is\nlocal to each machine or node. Create a <code>nomad-agent.conf</code> with the following\ncontents:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>bind_addr = &quot;0.0.0.0&quot;\ndata_dir = &quot;/var/lib/nomad&quot;\nregion = &quot;global&quot;\nacl {\n  enabled = true\n}\nserver {\n  enabled              = true\n  bootstrap_expect     = 1\n  authoritative_region = &quot;global&quot;\n}\nclient {\n  enabled = true\n}\n</code></pre>\n<p>Then, execute:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>vagrant@nomad:~$ sudo nomad agent -config=nomad-agent.conf # sudo is needed to run as a client\n</code></pre>\n<p>You should see output indicating that Nomad is running.</p>\n<blockquote>\n<p>Clients need root access to be able to execute processes, while servers only\ncommunicate to synchronize state.</p>\n</blockquote>\n<h2 id=\"acl-bootstrap\">ACL Bootstrap</h2>\n<p>On another terminal, after running <code>vagrant ssh</code> from our temporary working\ndirectory, run the following command:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>vagrant@nomad:~$ nomad acl bootstrap\n\nAccessor ID  = 2f34299b-0403-074d-83e2-60511341a54c\nSecret ID    = 9fff6a06-b991-22db-7fed-55f17918e846\nName         = Bootstrap Token\nType         = management\nGlobal       = true\nPolicies     = n/a\nCreate Time  = 2018-02-14 19:09:23.424119008 +0000 UTC\nCreate Index = 13\nModify Index = 13\n</code></pre>\n<p>This <code>Secret ID</code> is our <code>management</code> (admin) token. This token is valid globally\nand all operations are permitted. No policies are necessary while authenticating\nwith the management token, and so, none are configured by default.</p>\n<p>It is important to copy the <code>Accessor ID</code> and <code>Secret ID</code> to some file, for\nsafekeeping, as we will need these values later. For a production environment,\nit is safest to store these in a separate vault permanently.</p>\n<p>Once ACLs are on, all operations are denied <em>unless</em> a valid token is provided\nwith each request, and the operation we want is allowed by a policy associated\nwith the provided token.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>vagrant@nomad:~$ nomad node-status\nError querying node status: Unexpected response code: 403 (Permission denied)\n\nvagrant@nomad:~$ export NOMAD_TOKEN='9fff6a06-b991-22db-7fed-55f17918e846' # Secret ID, above\nvagrant@nomad:~$ nomad node-status\n\nID        DC   Name   Class   Drain  Status\n1f638a17  dc1  nomad  &lt;none&gt;  false  ready\n</code></pre><h2 id=\"designing-policies\">Designing policies</h2>\n<p>Policies are a collection of (ideally, non-overlapping) roles, that provide\naccess to different operations. The table below shows typical users of a Nomad\ncluster.</p>\n<table><thead><tr><th>Role</th><th>Namespace</th><th>Agent</th><th>Node</th><th>Remarks</th></tr></thead><tbody>\n<tr><td>Anonymous</td><td><code>deny</code></td><td><code>deny</code></td><td><code>deny</code></td><td>Unnecessary, as token-less requests are denied all operations.</td></tr>\n<tr><td>Developer</td><td><code>write</code></td><td><code>deny</code></td><td><code>read</code></td><td>Developers are permitted to debug their applications, but not to perform cluster management</td></tr>\n<tr><td>Logger</td><td><code>list-jobs</code>, <code>read-logs</code></td><td><code>deny</code></td><td><code>read</code></td><td>Automated log aggregators or analyzers that need read access to logs</td></tr>\n<tr><td>Job requester</td><td><code>submit-job</code></td><td><code>deny</code></td><td><code>deny</code></td><td>CI systems create new jobs, but don't interact with running jobs.</td></tr>\n<tr><td>Infrastructure</td><td><code>read</code></td><td><code>write</code></td><td><code>write</code></td><td>DevOps teams perform cluster management but seldom need to interact with running jobs.</td></tr>\n</tbody></table>\n<blockquote>\n<p>For namespace access, <code>read</code> is equivalent to\n<code>[read-job, list-jobs]</code>. <code>write</code> is equivalent to\n<code>[list-jobs, read-job, submit-job, read-logs, read-fs, dispatch-job]</code>.</p>\n</blockquote>\n<blockquote>\n<p>In the event that operators do need to have access to namespaces, one can\nalways create a token that has <em>both</em> Developer and Infrastructure policies\nattached. This is equivalent to having a <code>management</code> token.</p>\n</blockquote>\n<p>We have left out multi-region and multi-namespace setups here. We have assumed\neverything to be running under the <code>default</code> namespace. It should be noted that\non production deployments, with much larger needs, the policies could be\ndesigned per-namespace, and tracked between regions.</p>\n<h2 id=\"policy-specification\">Policy specification</h2>\n<p>Policies are expressed by a combination of rules Note that the <code>deny</code> rule will\npreside over any conflicting capability.</p>\n<p>Nomad accepts a JSON payload with the name and description of a policy, along\nwith a <em>quoted</em> JSON or HCL document with rules, like the following.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>{\n  &quot;Description&quot;: &quot;Agent and node management&quot;,\n  &quot;Name&quot;: &quot;infrastructure&quot;,\n  &quot;Rules&quot;: &quot;{\\&quot;agent\\&quot;:{\\&quot;policy\\&quot;:\\&quot;write\\&quot;},\\&quot;node\\&quot;:{\\&quot;policy\\&quot;:\\&quot;write\\&quot;}}&quot;\n}\n</code></pre>\n<p>This policy matches what we have in the table above.\nCreate an <code>infrastructure.json</code> with the content above for use in the next step.</p>\n<blockquote>\n<p>TIP:</p>\n<p>To avoid error-prone quoting, one could write the policies in YAML:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>Name: infrastructure\nDescription: Agent and node management\nRules:\n  agent:\n    policy: write\n  node:\n    policy: write\n</code></pre>\n<p>And then, convert them to JSON with the necessary quoting, by:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>$ yaml2json &lt; infrastructure.yaml | jq '.Rules = (.Rules | @text)' &gt; infrastructure.json\n</code></pre></blockquote>\n<h2 id=\"adding-a-policy\">Adding a policy</h2>\n<p>To add the policy, simply make an HTTP POST request to the server. The\n<code>NOMAD_TOKEN</code> below is the &quot;management&quot; token that we first created.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>vagrant@nomad:~$ curl \\\n    --request POST \\\n    --data @infrastructure.json \\\n    --header &quot;X-Nomad-Token: ${NOMAD_TOKEN}&quot; \\\n    https://127.0.0.1:4646/v1/acl/policy/infrastructure\n\nvagrant@nomad:~$ nomad acl policy list\nName            Description\ninfrastructure  Agent and node management\n\nvagrant@nomad:~$ nomad acl policy info infrastructure\nName        = infrastructure\nDescription = Agent and node management\nRules       = {&quot;agent&quot;:{&quot;policy&quot;:&quot;write&quot;},&quot;node&quot;:{&quot;policy&quot;:&quot;write&quot;}}\nCreateIndex = 425\nModifyIndex = 425\n</code></pre><h2 id=\"creating-a-token-for-a-policy\">Creating a token for a policy</h2>\n<p>We now create a token for the <code>infrastructure</code> policy, and attempt a few operations\nwith it:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code>vagrant@nomad:~$ nomad acl token create \\\n    -name='devops-team' \\\n    -type='client' \\\n    -global='true' \\\n    -policy='infrastructure'\n\nAccessor ID  = 927ea7a4-e689-037f-be89-54a2cdbd338c\nSecret ID    = 26832c8d-9315-c1ef-aabf-2058c8632da8\nName         = devops-team\nType         = client\nGlobal       = true\nPolicies     = [infrastructure]\nCreate Time  = 2018-02-15 19:53:59.97900843 +0000 UTC\nCreate Index = 432\nModify Index = 432\n\nvagrant@nomad:~$ export NOMAD_TOKEN='26832c8d-9315-c1ef-aabf-2058c8632da8' # change the token to the new one with the &quot;infrastructure&quot; policy attached\nvagrant@nomad:~$ nomad status\nError querying jobs: Unexpected response code: 403 (Permission denied)\n\nvagrant@nomad:~$ nomad node-status\nID        DC   Name   Class   Drain  Status\n1f638a17  dc1  nomad  &lt;none&gt;  false  ready\n</code></pre>\n<p>As you can see, anyone with the <code>devops-team</code> token will be allowed to\nrun operations on nodes, but not on jobs -- i.e. on namespace resources.</p>\n<h2 id=\"where-to-go-next\">Where to go next</h2>\n<p>The example above demonstrates adding one of the policies from our list at the\nbeginning. Adding the rest of them and trying different commands could be a\ngood exercise.</p>\n<p>As a reference, the FP Complete team maintains a\n<a href=\"https://github.com/fpco/nomad-acl-policies\">repository</a> with\npolicies ready for use.</p>\n<h4 id=\"related-articles\">Related articles</h4>\n<ul>\n<li><a href=\"https://www.fpcomplete.com/blog/2016/11/devops-best-practices-immutability/\">DevOps best practices: immutability</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/\">How to implement containers to streamline your DevOps workflow</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/2016/05/stack-security-gnupg-keys/\">Stack security: GnuPG keys</a></li>\n</ul>\n",
        "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/",
        "slug": "controlling-access-to-nomad-clusters",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Controlling access to Nomad clusters",
        "description": "Learn how to control access to your Nomad clusters on a per-role basis. This will get you the benefits of application schedulers such as Nomad and Kubernetes with all the security guarantees your services need but without the complex and lengthy setup that some other popular tools demand.",
        "updated": null,
        "date": "2018-05-17T13:21:00Z",
        "year": 2018,
        "month": 5,
        "day": 17,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "FP Complete Team",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/2018/05/controlling-access-to-nomad-clusters/",
        "components": [
          "blog",
          "2018",
          "05",
          "controlling-access-to-nomad-clusters"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "introduction",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#introduction",
            "title": "Introduction",
            "children": []
          },
          {
            "level": 2,
            "id": "overview",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#overview",
            "title": "Overview",
            "children": []
          },
          {
            "level": 2,
            "id": "setup-the-environment",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#setup-the-environment",
            "title": "Setup the environment",
            "children": []
          },
          {
            "level": 2,
            "id": "setup-nomad",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#setup-nomad",
            "title": "Setup Nomad",
            "children": []
          },
          {
            "level": 2,
            "id": "acl-bootstrap",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#acl-bootstrap",
            "title": "ACL Bootstrap",
            "children": []
          },
          {
            "level": 2,
            "id": "designing-policies",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#designing-policies",
            "title": "Designing policies",
            "children": []
          },
          {
            "level": 2,
            "id": "policy-specification",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#policy-specification",
            "title": "Policy specification",
            "children": []
          },
          {
            "level": 2,
            "id": "adding-a-policy",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#adding-a-policy",
            "title": "Adding a policy",
            "children": []
          },
          {
            "level": 2,
            "id": "creating-a-token-for-a-policy",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#creating-a-token-for-a-policy",
            "title": "Creating a token for a policy",
            "children": []
          },
          {
            "level": 2,
            "id": "where-to-go-next",
            "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#where-to-go-next",
            "title": "Where to go next",
            "children": [
              {
                "level": 4,
                "id": "related-articles",
                "permalink": "https://www.fpcomplete.com/blog/2018/05/controlling-access-to-nomad-clusters/#related-articles",
                "title": "Related articles",
                "children": []
              }
            ]
          }
        ],
        "word_count": 1393,
        "reading_time": 7,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/continuous-integration-delivery-best-practices.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/continuous-integration-delivery-best-practices/",
        "slug": "continuous-integration-delivery-best-practices",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Best practices when implementing continuous integration and delivery",
        "description": "Although, there are countless reasons to ditch the old ways of development and adopt DevOps practices, the change from one to the another can be an intimidating task. Use these best practices to ensure your company succeeds during these transitions. ",
        "updated": null,
        "date": "2018-04-11T12:49:00Z",
        "year": 2018,
        "month": 4,
        "day": 11,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "devops",
            "kub360"
          ]
        },
        "extra": {
          "author": "Deni Bertovic",
          "html": "hubspot-blogs/continuous-integration-delivery-best-practices.html",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/continuous-integration-delivery-best-practices/",
        "components": [
          "blog",
          "continuous-integration-delivery-best-practices"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/fintech-best-practices-devops-priorities-for-financial-technology-applications.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/fintech-best-practices-devops-priorities-for-financial-technology-applications/",
        "slug": "fintech-best-practices-devops-priorities-for-financial-technology-applications",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "FinTech best practices: DevOps Priorities for Financial Technology Applications",
        "description": "Modern software development is complicated, but developing software for the FinTech industry adds a whole new dimension of complexity. Adopting modern DevOps principals will ensure your software adheres to FinTech best practices. This blog explains how you can get started and be successful.",
        "updated": null,
        "date": "2018-04-05T12:21:00Z",
        "year": 2018,
        "month": 4,
        "day": 5,
        "taxonomies": {
          "tags": [
            "devops",
            "fintech"
          ],
          "categories": [
            "devops",
            "kube360"
          ]
        },
        "extra": {
          "author": "Aaron Contorer",
          "html": "hubspot-blogs/fintech-best-practices-devops-priorities-for-financial-technology-applications.html",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/fintech-best-practices-devops-priorities-for-financial-technology-applications/",
        "components": [
          "blog",
          "fintech-best-practices-devops-priorities-for-financial-technology-applications"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/recover-your-elasticsearch.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2018/04/recover-your-elasticsearch/",
        "slug": "recover-your-elasticsearch",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Recover your Elasticsearch",
        "description": "When using Elasticsearch you may run into cluster problems that could lose data because of a corrupt index. All is not lost because there are ways to recover your Elasticsearch. Find out how to bring the cluster to a healthy state with minimal or no data loss in such situation. ",
        "updated": null,
        "date": "2018-04-03T13:42:00Z",
        "year": 2018,
        "month": 4,
        "day": 3,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Alexey Kuleshevich",
          "html": "hubspot-blogs/recover-your-elasticsearch.html",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/2018/04/recover-your-elasticsearch/",
        "components": [
          "blog",
          "2018",
          "04",
          "recover-your-elasticsearch"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/without-performance-tests-we-will-have-a-bad-time-forever.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/without-performance-tests-we-will-have-a-bad-time-forever/",
        "slug": "without-performance-tests-we-will-have-a-bad-time-forever",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Without performance tests, we will have a bad time, forever",
        "description": "When writing Haskell software code, you cannot assume performance is optimized. You must utilize automated testing and eliminate human inspection.  Performance regression is not an option, or you will have a bad day.",
        "updated": null,
        "date": "2018-03-15T11:36:00Z",
        "year": 2018,
        "month": 3,
        "day": 15,
        "taxonomies": {
          "categories": [
            "devops"
          ],
          "tags": [
            "haskell"
          ]
        },
        "extra": {
          "author": "Niklas Hambüchen",
          "html": "hubspot-blogs/without-performance-tests-we-will-have-a-bad-time-forever.html",
          "blogimage": "/images/blog/qa.jpg"
        },
        "path": "blog/without-performance-tests-we-will-have-a-bad-time-forever/",
        "components": [
          "blog",
          "without-performance-tests-we-will-have-a-bad-time-forever"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/how-to-implement-containers-to-streamline-your-devops-workflow.md",
        "content": "<h1 id=\"what-are-docker-containers\">What are Docker Containers?</h1>\n<p>Docker containers are a form of &quot;lightweight&quot; virtualization They allow a\nprocess or process group to run in an environment with its own file system,\nsomewhat like <code>chroot</code> jails , and also with its own process table, users and\ngroups and, optionally, virtual network and resource limits. For most purposes,\nthe processes in a container think they have an entire OS to themselves and do\nnot have access to anything outside the container (unless explicitly granted).\nThis lets you precisely control the environment in which your processes run,\nallows multiple processes on the same (virtual) machine that have completely\ndifferent (even conflicting) requirements, and significantly increases isolation\nand container security.</p>\n<p>In addition to containers, Docker makes it easy to build and distribute images\nthat wrap up an application with its complete runtime environment.</p>\n<p>For more information, see \n<a href=\"https://www.cio.com/article/2924995/software/what-are-containers-and-why-do-you-need-them.html\">What are containers and why do you need them?</a> \nand \n<a href=\"https://containerjournal.com/2017/01/11/containers-devops-anyway/\">What Do Containers Have to Do with DevOps, Anyway?</a>.</p>\n<h1 id=\"containers-vs-virtual-machines-vms\">Containers vs Virtual Machines (VMs)</h1>\n<p>The difference between the &quot;lightweight&quot; virtualization of containers and\n&quot;heavyweight&quot; virtualization of VMs boils down to that, for the former, the\nvirtualization happens at the kernel level while for the latter it happens at\nthe hypervisor level. In other words, all the containers on a machine share the\nsame kernel, and code in the kernel isolates the containers from each other\nwhereas each VM acts like separate hardware and has its own kernel.</p>\n<img alt=\"Docker Carrying Haskell.jpg\" sizes=\"(max-width: 320px) 100vw, 320px\" src=\"/images/hubspot/4536576cadee37e3ea1e0a35a83a97a55015af6773242ecda5f919a7f1628cc5.jpeg\" srcset=\"/images/hubspot/04a7b5b957c890331f8535859d7c8528eadf4d83c82ae65e86ea28fea6f82898.jpeg 160w, /images/hubspot/4536576cadee37e3ea1e0a35a83a97a55015af6773242ecda5f919a7f1628cc5.jpeg 320w, /images/hubspot/1652b04e09bee96b23e47adb5830543a1feac5a48d5488b22602cec12a1b131d.jpeg 480w, /images/hubspot/4a5e5498d817ee00db5fdc27b5827a41a41d07253d95f73e093809cd27d6ea45.jpeg 640w, /images/hubspot/d77567fd61f4146be574d81e707b90ca7f80f3005770d6ef527ff656eb9b913d.jpeg 800w, /images/hubspot/f9971781a2d67ed9b0b30a5798652fdc1975985603d9fde0b60bf89de73faa7a.jpeg 960w\" style=\"width: 320px; margin: 0px 0px 10px 10px; letter-spacing: -0.08px; float: right;\" width=\"320\">\n<p>Containers are much less resource intensive than VMs because they do not need\nto be allocated exclusive memory and file system space or have the overhead of\nrunning an entire operating system. This makes it possible to run many more\ncontainers on a machine than you would VMs. Containers start nearly as fast as\nregular processes (you don't have to wait for the OS to boot), and parts of the\nhost's file system can be easily &quot;mounted&quot; into the container's file system\nwithout any additional overhead of network file system protocols.</p>\n<p>On the other hand, isolation is less guaranteed. If not careful, you can\noversubscribe a machine by running containers that need more resources than the\nmachine has available (this can be mitigated by setting appropriate resource\nlimits on containers). While containers security is an improvement over normal\nprocesses, the shared kernel means the attack surface is greater and there is\nmore risk of leakage between containers than there is between VMs.</p>\n<p>For more information, see <a href=\"https://blog.netapp.com/blogs/containers-vs-vms/\">Docker containers vs. virtual machines: What's the\ndifference?</a> and <a href=\"https://www.fpcomplete.com/blog/2016/11/devops-best-practices-immutability/\">DevOps Best\nPractices: Immutability</a></p>\n<h1 id=\"how-docker-containers-enhance-continuous-delivery-pipelines\">How Docker Containers Enhance Continuous Delivery Pipelines</h1>\n<p>There are, broadly, two areas where containers fit into your devops\nworkflow: for builds, and for deployment. They are often used together,\nbut do not have to be.</p>\n<h3 id=\"builds\">Builds</h3>\n<ul>\n<li>\n<p><strong>Synchronizing build environments:</strong> It can be difficult to keep\nbuild environments synchronized between developers and CI/CD\nservers, which can lead to unexpected build failures or changes in\nbehaviour . Docker images let you specify <em>exactly</em> the build tools,\nlibraries, and other dependencies (including their versions)\nrequired without needing to install them on individual machines, and\ndistribute those images easily. This way you can be sure that\neveryone is using exactly the same build environment.</p>\n</li>\n<li>\n<p><strong>Managing changes to build environments:</strong> Managing changes to\nbuild environments can also be difficult, since you need to roll\nthose out to all developers and build servers at the right time.\nThis can be especially tricky when there are multiple branches of\ndevelopment some of which may need older or newer environments than\neach other. With Docker, you can specify a particular version of the\nbuild image along with the source code, which means a particular\nrevision of the source code will always build in the right\nenvironment.</p>\n</li>\n<li>\n<p><strong>Isolating build environments:</strong> One CI/CD server may have to build\nmultiple projects, which may have conflicting requirements for build\ntools, libraries, and other dependencies. By running each build in\nits own ephemeral container created from potentially different\nDocker images, you can be certain that these builds environments\nwill not interfere with each other.</p>\n</li>\n</ul>\n<h3 id=\"deployment\">Deployment</h3>\n<ul>\n<li>\n<p><strong>Runtime environment bundled with application :</strong> The CD system\nbuilds a complete Docker image which bundles the application's\nenvironment with the application itself and then deploys the whole\nimage as one &quot;atomic&quot; step. There is no chance for configuration\nmanagement scripts to fail at deployment time, and no risk of the\nsystem configuration to be out of sync.</p>\n</li>\n<li>\n<p><strong>Preventing malicious changes:</strong> Container security is improved by\nusing immutable SHA digests to identify Docker images, which means\nthere is no way for a malicious actor to inject malware into your\napplication or its environment.</p>\n</li>\n<li>\n<p><strong>Easily roll back to a previous version:</strong> All it takes to roll\nback is to deploy a previous version of the Docker image. There is\nno worrying about system configuration changes needing to be\nmanually rolled back.</p>\n</li>\n<li>\n<p><strong>Zero downtime rollouts:</strong> In conjunction with container\norchestration tools like Kubernetes, it is easily to roll out new\nimage versions with zero downtime.</p>\n</li>\n<li>\n<p><strong>High availability and horizontal scaling:</strong> Container\norchestration tools like Kubernetes make it easy to distribute the\nsame image to containers on multiple servers, and add/remove\nreplicas at will or automatically.</p>\n</li>\n<li>\n<p><strong>Sharing a server between multiple applications:</strong> Multiple\napplications, or multiple versions of the same application (e.g. a\ndev and qa deployment), can run on the same server even if they have\nconflicting dependencies, since their runtime environments are\ncompletely separate.</p>\n</li>\n<li>\n<p><strong>Isolating applications:</strong> When multiple applications are deployed\nto a server in containers, they are isolated from one another.\nContainer security means each has its own file system, processes,\nand users there is less risk that they interfere with each other\nintentionally. When data <em>does</em> need to be shared between\napplications, parts of the host file system can be mounted into\nmultiple containers, but this is something you have full control\nover.</p>\n</li>\n</ul>\n<p>For more information, see:</p>\n<ul>\n<li><a href=\"https://www.fpcomplete.com/blog/2017/03/continuous-integration/\">Continuous Integration: An Overview</a></li>\n<li><a href=\"https://docs.microsoft.com/en-us/dotnet/standard/containerized-lifecycle-architecture/docker-application-lifecycle/containers-foundation-for-devops-collaboration\">Containers as the foundation for DevOps collaboration</a></li>\n<li><a href=\"https://www.sumologic.com/blog/devops/how-containerization-enables-devops/\">Docker and DevOps -- Enabling DevOps Teams Through Containerization</a>.</li>\n</ul>\n<h1 id=\"implementing-containers-into-your-devops-workflow\">Implementing Containers into Your DevOps Workflow</h1>\n<p>Containers can be integrated into your DevOps toolchain incrementally.\nOften it makes sense to start with the build environment, and then move\non to the deployment environment. This is a very broad overview of the\nsteps for a simple approach, without delving into the technical details\nvery much or covering all the possible variations.</p>\n<h3 id=\"requirements\">Requirements</h3>\n<ul>\n<li>Docker Engine installed on build servers and/or application servers</li>\n<li>Access to a Docker Registry. This is where Docker images are stored\nand pulled. There are numerous services that provide registries, and\nit's also easy to run your own.</li>\n</ul>\n<h3 id=\"containerizing-the-build-environment\">Containerizing the build environment</h3>\n<p>Many CI/CD systems now include built-in Docker support or easily enable\nit through plugins, but   <code>docker</code>   is a command-line application which\ncan be called from any build script even if your CI/CD system does not\nhave explicit support.</p>\n<ol>\n<li>\n<p>Determine your build environment requirements and write\na <code>Dockerfile</code> based on an existing Docker image, which is the\nspecification used to build an image for build containers. If you\nalready use a configuration management tool, you can use it within\nthe Dockerfile. Always specify precise versions of base images and\ninstalled packages so that image builds are consistent and upgrades\nare deliberate.</p>\n</li>\n<li>\n<p>Build the image using   <code>docker build</code>   and push it to the Docker\nregistry using   <code>docker push</code> .</p>\n</li>\n<li>\n<p>Create a <code>Dockerfile</code> for the application that is based on the build\nimage (specify the exact version of the base build image). This file\nbuilds the application, adds any required runtime dependencies that\naren't in the build image, and tests the application. A multi-stage\n <code>Dockerfile</code>  can be used if you don't want the application\ndeployment image to include all the build dependencies.</p>\n</li>\n<li>\n<p>Modify CI build scripts to build the application image and push it\nto the Docker registry. The image should be tagged with the build\nnumber, and possibly additional information such as the name of the\nbranch.</p>\n</li>\n<li>\n<p>If you are not yet ready to deploy with Docker, you can extract the\nbuild artifacts from the resulting Docker image.</p>\n</li>\n</ol>\n<p>It is best to <em>also</em> integrate building the build image itself into your\ndevops automation tools.</p>\n<h3 id=\"containerizing-deployment\">Containerizing deployment</h3>\n<p>This can be easier if your CD tool has support for Docker, but that is\nby no means necessary. We also recommend deploying to a container\norchestration system such as Kubernetes in most cases.</p>\n<p>Half the work has already been done, since the build process creates and\npushes an image containing the application and its environment.</p>\n<ul>\n<li>\n<p>If using Docker directly, now it's a matter of updating deployment\nscripts to use   <code>docker run</code>   on the application server with the\nimage and tag that was pushed in the previous section (after\nstopping any existing container). Ideally your application accepts\nits configuration via environment variables, in which case you use\nthe   <code>-e</code>   argument to specify those values depending on which\nstage is being deployed. If a configuration file are used, write it\nto the host file system and then use the   <code>-v</code>   argument to mount\nit to the correct path in the container.</p>\n</li>\n<li>\n<p>If using a container orchestration system such as Kubernetes, you\nwill typically have the deployment script connect to the\norchestration API endpoint to trigger an image update (e.g. using  \n<code>kubectl set image</code> , a Helm chart, or better yet, a\n<code>kustomization</code>.).</p>\n</li>\n</ul>\n<p>Once deployed, tools such as Prometheus are well suited to docker\ncontainer monitoring and alerting, but this can be plugged into existing\nmonitoring systems as well.</p>\n<p>FP Complete has implemented this kind of DevOps workflow, and\nsignificantly more complex ones, for many clients and would love to\ncount you among them! See our <a href=\"https://www.fpcomplete.com/devops/\">Devops Services</a> page.</p>\n<p>For more information, see <a href=\"https://techbeacon.com/how-secure-container-lifecycle\">How to secure the container\nlifecycle</a> and <a href=\"https://www.fpcomplete.com/blog/2017/01/containerize-legacy-app/\">Containerizing\na legacy application: an\noverview</a>.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/",
        "slug": "how-to-implement-containers-to-streamline-your-devops-workflow",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "How to Implement Containers to Streamline Your DevOps Workflow",
        "description": "Many technology companies have been rapidly implementing Docker Containers to enhance their continuous delivery pipeline. However, implementing containers into your DevOps workflow can be difficult. Learn how to execute this process efficiently and securely here. ",
        "updated": null,
        "date": "2018-01-31T08:00:00Z",
        "year": 2018,
        "month": 1,
        "day": 31,
        "taxonomies": {
          "tags": [
            "devops",
            "docker"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Emanuel Borsboom",
          "blogimage": "/images/blog/container.jpg"
        },
        "path": "blog/how-to-implement-containers-to-streamline-your-devops-workflow/",
        "components": [
          "blog",
          "how-to-implement-containers-to-streamline-your-devops-workflow"
        ],
        "summary": null,
        "toc": [
          {
            "level": 1,
            "id": "what-are-docker-containers",
            "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/#what-are-docker-containers",
            "title": "What are Docker Containers?",
            "children": []
          },
          {
            "level": 1,
            "id": "containers-vs-virtual-machines-vms",
            "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/#containers-vs-virtual-machines-vms",
            "title": "Containers vs Virtual Machines (VMs)",
            "children": []
          },
          {
            "level": 1,
            "id": "how-docker-containers-enhance-continuous-delivery-pipelines",
            "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/#how-docker-containers-enhance-continuous-delivery-pipelines",
            "title": "How Docker Containers Enhance Continuous Delivery Pipelines",
            "children": [
              {
                "level": 3,
                "id": "builds",
                "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/#builds",
                "title": "Builds",
                "children": []
              },
              {
                "level": 3,
                "id": "deployment",
                "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/#deployment",
                "title": "Deployment",
                "children": []
              }
            ]
          },
          {
            "level": 1,
            "id": "implementing-containers-into-your-devops-workflow",
            "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/#implementing-containers-into-your-devops-workflow",
            "title": "Implementing Containers into Your DevOps Workflow",
            "children": [
              {
                "level": 3,
                "id": "requirements",
                "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/#requirements",
                "title": "Requirements",
                "children": []
              },
              {
                "level": 3,
                "id": "containerizing-the-build-environment",
                "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/#containerizing-the-build-environment",
                "title": "Containerizing the build environment",
                "children": []
              },
              {
                "level": 3,
                "id": "containerizing-deployment",
                "permalink": "https://www.fpcomplete.com/blog/how-to-implement-containers-to-streamline-your-devops-workflow/#containerizing-deployment",
                "title": "Containerizing deployment",
                "children": []
              }
            ]
          }
        ],
        "word_count": 1752,
        "reading_time": 9,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/signs-your-business-needs-a-devops-consultant.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/signs-your-business-needs-a-devops-consultant/",
        "slug": "signs-your-business-needs-a-devops-consultant",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Signs Your Business Needs a DevOps Consultant",
        "description": "Today’s business challenges cause issues with traditional deployment models. Find out why a DevOps consultant may be right for you. ",
        "updated": null,
        "date": "2018-01-18T15:06:00Z",
        "year": 2018,
        "month": 1,
        "day": 18,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "insights",
            "devops"
          ]
        },
        "extra": {
          "author": "Aaron Contorer",
          "html": "hubspot-blogs/signs-your-business-needs-a-devops-consultant.html",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/signs-your-business-needs-a-devops-consultant/",
        "components": [
          "blog",
          "signs-your-business-needs-a-devops-consultant"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/devops-value-how-to-measure-the-success-of-devops.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/devops-value-how-to-measure-the-success-of-devops/",
        "slug": "devops-value-how-to-measure-the-success-of-devops",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "DevOps Value: How to Measure the Success of DevOps",
        "description": "Faster time to market and lower failure rate are the beginning of the many benefits DevOps offers companies. Discover the measurable metrics and KPIs, as well as the true business value DevOps offers.",
        "updated": null,
        "date": "2018-01-04T13:51:00Z",
        "year": 2018,
        "month": 1,
        "day": 4,
        "taxonomies": {
          "categories": [
            "devops",
            "insights"
          ],
          "tags": [
            "devops"
          ]
        },
        "extra": {
          "author": "Robert Bobbett",
          "html": "hubspot-blogs/devops-value-how-to-measure-the-success-of-devops.html",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/devops-value-how-to-measure-the-success-of-devops/",
        "components": [
          "blog",
          "devops-value-how-to-measure-the-success-of-devops"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/nat-gateways-in-amazon-govcloud.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/nat-gateways-in-amazon-govcloud/",
        "slug": "nat-gateways-in-amazon-govcloud",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "NAT Gateways in Amazon GovCloud",
        "description": "Since AWS GovCloud has no managed NAT gateways this task is left for you to set up. This post is the third in a series to explain how you can make it work.",
        "updated": null,
        "date": "2017-11-30T14:25:00Z",
        "year": 2017,
        "month": 11,
        "day": 30,
        "taxonomies": {
          "categories": [
            "devops",
            "kube360"
          ],
          "tags": [
            "devops",
            "aws",
            "govcloud"
          ]
        },
        "extra": {
          "author": "Yghor Kerscher",
          "html": "hubspot-blogs/nat-gateways-in-amazon-govcloud.html",
          "blogimage": "/images/blog/cloud-computing.jpg"
        },
        "path": "blog/nat-gateways-in-amazon-govcloud/",
        "components": [
          "blog",
          "nat-gateways-in-amazon-govcloud"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/my-devops-journey-and-how-i-became-a-recovering-it-operations-manager.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/my-devops-journey-and-how-i-became-a-recovering-it-operations-manager/",
        "slug": "my-devops-journey-and-how-i-became-a-recovering-it-operations-manager",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "My DevOps Journey and How I Became a Recovering IT Operations Manager",
        "description": "Learn how containerization and automated deployments laid the groundwork for what would become know as DevOps for a Fortune 500 IT company.",
        "updated": null,
        "date": "2017-11-15T13:30:00Z",
        "year": 2017,
        "month": 11,
        "day": 15,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "devops",
            "insights"
          ]
        },
        "extra": {
          "author": "Steve Bogdan",
          "html": "hubspot-blogs/my-devops-journey-and-how-i-became-a-recovering-it-operations-manager.html",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/my-devops-journey-and-how-i-became-a-recovering-it-operations-manager/",
        "components": [
          "blog",
          "my-devops-journey-and-how-i-became-a-recovering-it-operations-manager"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/amazon-govcloud-has-no-route53-how-to-solve-this.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/amazon-govcloud-has-no-route53-how-to-solve-this/",
        "slug": "amazon-govcloud-has-no-route53-how-to-solve-this",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Amazon GovCloud has no Route53! How to solve this?",
        "description": "Since Route53 is not yet available on Amazon GovCloud you need to find a different way to create custom DNS records for your services. We tell you how. ",
        "updated": null,
        "date": "2017-11-08T14:12:00Z",
        "year": 2017,
        "month": 11,
        "day": 8,
        "taxonomies": {
          "tags": [
            "devops",
            "aws",
            "govcloud"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Yghor Kerscher",
          "html": "hubspot-blogs/amazon-govcloud-has-no-route53-how-to-solve-this.html",
          "blogimage": "/images/blog/cloud-computing.jpg"
        },
        "path": "blog/amazon-govcloud-has-no-route53-how-to-solve-this/",
        "components": [
          "blog",
          "amazon-govcloud-has-no-route53-how-to-solve-this"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/intro-to-devops-on-govcloud.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/intro-to-devops-on-govcloud/",
        "slug": "intro-to-devops-on-govcloud",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Intro to Devops on GovCloud",
        "description": "If you have strict compliance criteria that require you to use AWS GovCloud, there are some obstacles you will encounter that we will help you address.",
        "updated": null,
        "date": "2017-10-26T11:02:00Z",
        "year": 2017,
        "month": 10,
        "day": 26,
        "taxonomies": {
          "categories": [
            "devops"
          ],
          "tags": [
            "devops",
            "govcloud"
          ]
        },
        "extra": {
          "author": "J Boyer",
          "html": "hubspot-blogs/intro-to-devops-on-govcloud.html",
          "blogimage": "/images/blog/cloud-computing.jpg"
        },
        "path": "blog/intro-to-devops-on-govcloud/",
        "components": [
          "blog",
          "intro-to-devops-on-govcloud"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/credstash.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2017/08/credstash/",
        "slug": "credstash",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Manage Secrets on AWS with credstash and terraform",
        "description": "Managing secrets is hard. Moving them around securely is even harder. Learn how to get secrets to the Cloud with terraform and credstash.",
        "updated": null,
        "date": "2017-08-28T15:00:00Z",
        "year": 2017,
        "month": 8,
        "day": 28,
        "taxonomies": {
          "tags": [
            "devops",
            "aws"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Alexey Kuleshevich",
          "html": "hubspot-blogs/credstash.html",
          "blogimage": "/images/blog/aws.jpg"
        },
        "path": "blog/2017/08/credstash/",
        "components": [
          "blog",
          "2017",
          "08",
          "credstash"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/functional-programming-and-modern-devops.md",
        "content": "<p>In this presentation, Aaron Contorer presents on how modern tools can\nbe used to reach the Engineering sweet spot.</p>\n<iframe width=\"100%\" height=\"315\"\nsrc=\"https://www.youtube.com/embed/ybSBCVhVWs8\" frameborder=\"0\"\nallow=\"accelerometer; autoplay; encrypted-media; gyroscope;\npicture-in-picture\" allowfullscreen></iframe>\n<br>\n<br>\n<h2 id=\"do-you-know-fp-complete\">Do you know FP Complete</h2>\n<p>At FP Complete, we do so many things to help companies it’s hard to\nencapsulate our impact in a few words. They say a picture is worth a\nthousand words, so a video has to be worth 10,000 words (at\nleast). Therefore, to tell all we can in as little time as possible,\ncheck out our explainer video. It’s only 108 seconds to get the full\nstory of FP Complete.</p>\n<iframe allowfullscreen=\n            \"allowfullscreen\" height=\"315\" src=\n            \"https://www.youtube.com/embed/JCcuSn_lFKs\"\n            target=\"_blank\" width=\n            \"100%\"></iframe>\n<br>\n<br>\n<p>Reach us to on <a href=\"mailto:sales@fpcomplete.com\">sales@fpcomplete.com</a> if you have suggestions or if\nyou would like to learn more about FP Complete and the services we\noffer.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/functional-programming-and-modern-devops/",
        "slug": "functional-programming-and-modern-devops",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Functional Programming and Modern DevOps",
        "description": "In this presentation, Aaron Contorer presents on how modern tools can be used to reach the Engineering sweet spot.",
        "updated": null,
        "date": "2017-08-11",
        "year": 2017,
        "month": 8,
        "day": 11,
        "taxonomies": {
          "tags": [
            "devops",
            "haskell",
            "insights"
          ],
          "categories": [
            "functional programming",
            "devops"
          ]
        },
        "extra": {
          "author": "Aaron Contorer",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/functional-programming-and-modern-devops/",
        "components": [
          "blog",
          "functional-programming-and-modern-devops"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "do-you-know-fp-complete",
            "permalink": "https://www.fpcomplete.com/blog/functional-programming-and-modern-devops/#do-you-know-fp-complete",
            "title": "Do you know FP Complete",
            "children": []
          }
        ],
        "word_count": 162,
        "reading_time": 1,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/continuous-integration.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2017/03/continuous-integration/",
        "slug": "continuous-integration",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Continuous Integration: an overview",
        "description": "Continuous integration makes development teams more productive and releases less stressful. Catch regressions quickly and deploy applications automatically.",
        "updated": null,
        "date": "2017-03-03T17:11:00Z",
        "year": 2017,
        "month": 3,
        "day": 3,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Emanuel Borsboom",
          "html": "hubspot-blogs/continuous-integration.html",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/2017/03/continuous-integration/",
        "components": [
          "blog",
          "2017",
          "03",
          "continuous-integration"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/immutability-docker-haskells-st-type.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2017/02/immutability-docker-haskells-st-type/",
        "slug": "immutability-docker-haskells-st-type",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Immutability, Docker, and Haskell's ST type",
        "description": "Immutability in software development is a well known constant in functional programming but is relatively new in modern devops and the parallels are worth examining.",
        "updated": null,
        "date": "2017-02-13T15:24:00Z",
        "year": 2017,
        "month": 2,
        "day": 13,
        "taxonomies": {
          "tags": [
            "haskell",
            "docker",
            "devops"
          ],
          "categories": [
            "functional programming",
            "devops"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "html": "hubspot-blogs/immutability-docker-haskells-st-type.html",
          "blogimage": "/images/blog/docker.jpg"
        },
        "path": "blog/2017/02/immutability-docker-haskells-st-type/",
        "components": [
          "blog",
          "2017",
          "02",
          "immutability-docker-haskells-st-type"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/quickcheck.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2017/01/quickcheck/",
        "slug": "quickcheck",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "QuickCheck and Magic of Testing",
        "description": "Discover the power of random testing in Haskell with QuickCheck. Learn how to use function properties and software specification to write bug-free software.",
        "updated": null,
        "date": "2017-01-24T14:24:00Z",
        "year": 2017,
        "month": 1,
        "day": 24,
        "taxonomies": {
          "categories": [
            "devops"
          ],
          "tags": [
            "haskell"
          ]
        },
        "extra": {
          "author": "Alexey Kuleshevich",
          "html": "hubspot-blogs/quickcheck.html",
          "blogimage": "/images/blog/qa.jpg"
        },
        "path": "blog/2017/01/quickcheck/",
        "components": [
          "blog",
          "2017",
          "01",
          "quickcheck"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/containerize-legacy-app.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2017/01/containerize-legacy-app/",
        "slug": "containerize-legacy-app",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Containerizing a legacy application: an overview",
        "description": "Running your legacy apps in Docker containers takes the pain out of deployment and puts you on a path to modern practices.  Learn what is involved in containerizing your app.",
        "updated": null,
        "date": "2017-01-12T15:45:00Z",
        "year": 2017,
        "month": 1,
        "day": 12,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Emanuel Borsboom",
          "html": "hubspot-blogs/containerize-legacy-app.html",
          "blogimage": "/images/blog/container.jpg"
        },
        "path": "blog/2017/01/containerize-legacy-app/",
        "components": [
          "blog",
          "2017",
          "01",
          "containerize-legacy-app"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/devops-best-practices-multifaceted-testing.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2016/11/devops-best-practices-multifaceted-testing/",
        "slug": "devops-best-practices-multifaceted-testing",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Devops best practices: Multifaceted Testing",
        "description": ".",
        "updated": null,
        "date": "2016-11-28T18:00:00Z",
        "year": 2016,
        "month": 11,
        "day": 28,
        "taxonomies": {
          "categories": [
            "devops"
          ],
          "tags": [
            "devops"
          ]
        },
        "extra": {
          "author": "Aaron Contorer",
          "html": "hubspot-blogs/devops-best-practices-multifaceted-testing.html",
          "blogimage": "/images/blog/qa.jpg"
        },
        "path": "blog/2016/11/devops-best-practices-multifaceted-testing/",
        "components": [
          "blog",
          "2016",
          "11",
          "devops-best-practices-multifaceted-testing"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/devops-best-practices-immutability.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2016/11/devops-best-practices-immutability/",
        "slug": "devops-best-practices-immutability",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Devops best practices: Immutability",
        "description": ".",
        "updated": null,
        "date": "2016-11-13T18:00:00Z",
        "year": 2016,
        "month": 11,
        "day": 13,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Aaron Contorer",
          "html": "hubspot-blogs/devops-best-practices-immutability.html",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/2016/11/devops-best-practices-immutability/",
        "components": [
          "blog",
          "2016",
          "11",
          "devops-best-practices-immutability"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/docker-demons-pid1-orphans-zombies-signals.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2016/10/docker-demons-pid1-orphans-zombies-signals/",
        "slug": "docker-demons-pid1-orphans-zombies-signals",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Docker demons: PID-1, orphans, zombies, and signals",
        "description": ".",
        "updated": null,
        "date": "2016-10-05T02:00:00Z",
        "year": 2016,
        "month": 10,
        "day": 5,
        "taxonomies": {
          "tags": [
            "devops",
            "docker"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "html": "hubspot-blogs/docker-demons-pid1-orphans-zombies-signals.html",
          "blogimage": "/images/blog/docker.jpg"
        },
        "path": "blog/2016/10/docker-demons-pid1-orphans-zombies-signals/",
        "components": [
          "blog",
          "2016",
          "10",
          "docker-demons-pid1-orphans-zombies-signals"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/docker-split-images.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2015/12/docker-split-images/",
        "slug": "docker-split-images",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "The split-image approach to building minimal runtime Docker images",
        "description": ".",
        "updated": null,
        "date": "2015-12-15T00:00:00Z",
        "year": 2015,
        "month": 12,
        "day": 15,
        "taxonomies": {
          "tags": [
            "devops",
            "docker"
          ],
          "categories": [
            "devops"
          ]
        },
        "extra": {
          "author": "Emanuel Borsboom",
          "html": "hubspot-blogs/docker-split-images.html",
          "blogimage": "/images/blog/docker.jpg"
        },
        "path": "blog/2015/12/docker-split-images/",
        "components": [
          "blog",
          "2015",
          "12",
          "docker-split-images"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/kubernetes.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2015/11/kubernetes/",
        "slug": "kubernetes",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Kubernetes for Haskell Services",
        "description": ".",
        "updated": null,
        "date": "2015-11-19T19:00:00Z",
        "year": 2015,
        "month": 11,
        "day": 19,
        "taxonomies": {
          "categories": [
            "devops"
          ],
          "tags": [
            "haskell",
            "kubernetes"
          ]
        },
        "extra": {
          "author": "Tim Dysinger",
          "html": "hubspot-blogs/kubernetes.html",
          "blogimage": "/images/blog/kubernetes.jpg"
        },
        "path": "blog/2015/11/kubernetes/",
        "components": [
          "blog",
          "2015",
          "11",
          "kubernetes"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/distributing-packages-without-sysadmin.md",
        "content": "",
        "permalink": "https://www.fpcomplete.com/blog/2015/05/distributing-packages-without-sysadmin/",
        "slug": "distributing-packages-without-sysadmin",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Distributing our packages without a sysadmin",
        "description": ".",
        "updated": null,
        "date": "2015-05-13T00:00:00Z",
        "year": 2015,
        "month": 5,
        "day": 13,
        "taxonomies": {
          "tags": [
            "devops"
          ],
          "categories": [
            "insights",
            "devops"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "html": "hubspot-blogs/distributing-packages-without-sysadmin.html",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/2015/05/distributing-packages-without-sysadmin/",
        "components": [
          "blog",
          "2015",
          "05",
          "distributing-packages-without-sysadmin"
        ],
        "summary": null,
        "toc": [],
        "word_count": 0,
        "reading_time": 0,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      }
    ]
  },
  {
    "name": "functional programming",
    "slug": "functional-programming",
    "permalink": "https://www.fpcomplete.com/categories/functional-programming/",
    "pages": [
      {
        "relative_path": "blog/collect-rust-traverse-haskell-scala.md",
        "content": "<p>There's a running joke in the functional programming community. Any Scala program can be written by combining the <code>traverse</code> function the correct number of times. This blog post is dedicated to that joke.</p>\n<p>In Rust, the <code>Iterator</code> trait defines a stream of values of a specific type. Many common types provide an <code>Iterator</code> interface. And the built in <code>for</code> loop construct works directly with the <code>Iterator</code> trait. Using that, we can easily do something like &quot;print all the numbers in a <code>Vec</code>&quot;:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec: </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#268bd2;\">i32</span><span style=\"color:#657b83;\">&gt; = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> num </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> myvec {\n        </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, num);\n    }\n}\n</span></code></pre>\n<p>Let's say we want to do something a bit different: double every value in the <code>Vec</code>. The most idiomatic and performant way to do that in Rust is with mutable references, e.g.:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> myvec: </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#268bd2;\">i32</span><span style=\"color:#657b83;\">&gt; = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> num </span><span style=\"color:#859900;\">in &amp;</span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> myvec {\n        *num *= </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">;\n    }\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, myvec);\n}\n</span></code></pre>\n<p>Since we're dedicating this post to functional programmers, it's worth nothing: this looks decidedly not-functional. &quot;Take a collection and apply a function over each value&quot; is well understood in FP circles—and increasingly in non-FP circles—as a <code>map</code>. Or using more category-theoretic nomenclature, it's a <code>Functor</code>. Fortunately, Rust provides a <code>map</code> method for <code>Iterator</code>s. Unfortunately, unlike Scala or Haskell, <code>map</code> doesn't work on data types like <code>Vec</code>. Let's compare, using Haskell:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">list </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#268bd2;\">Int</span><span style=\"color:#657b83;\">]\nlist </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">]\n\n</span><span style=\"color:#b58900;\">main </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">IO </span><span style=\"color:#859900;\">()\n</span><span style=\"color:#657b83;\">main </span><span style=\"color:#859900;\">= do\n  let</span><span style=\"color:#657b83;\"> newList </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#cb4b16;\">Int</span><span style=\"color:#657b83;\">]\n      newList </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> map (* </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">) list\n  print newList\n</span></code></pre>\n<p>The <code>map</code> function from the <code>Functor</code> typeclass works directly on a list. It produces a new list with the function applied to each value. Let's try to do the most equivalent thing in Rust:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec: </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#268bd2;\">i32</span><span style=\"color:#657b83;\">&gt; = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> new_vec: </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#268bd2;\">i32</span><span style=\"color:#657b83;\">&gt; = myvec.</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">x</span><span style=\"color:#657b83;\">| x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, new_vec);\n}\n</span></code></pre>\n<p>This fails with the error message:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">no method named `map` found for struct `std::vec::Vec&lt;i32&gt;` in the current scope\n</span></code></pre>\n<p>That's because, in Rust, <code>map</code> applies to the <code>Iterator</code> itself, not the underlying data structures. In order to use <code>map</code> on a <code>Vec</code>, we have to:</p>\n<ol>\n<li>Convert the <code>Vec</code> into an <code>Iterator</code></li>\n<li>Perform the <code>map</code> on the <code>Iterator</code></li>\n<li>Convert the <code>Iterator</code> back into a <code>Vec</code></li>\n</ol>\n<p>(1) can be performed using the <code>IntoIterator</code> trait, which provides a method named <code>into_iter</code>. And for (3), we could write our own <code>for</code> loop that fills up a <code>Vec</code>. But the right way is to use the <code>FromIterator</code> trait. And the easiest way to do that is with the <code>collect</code> method on <code>Iterator</code>.</p>\n<h2 id=\"using-fromiterator-and-collect\">Using FromIterator and collect</h2>\n<p>Let's write a program that properly uses <code>map</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec: </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#268bd2;\">i32</span><span style=\"color:#657b83;\">&gt; = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> new_vec: </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#268bd2;\">i32</span><span style=\"color:#657b83;\">&gt; = myvec\n        .</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">()\n        .</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">x</span><span style=\"color:#657b83;\">| x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">)\n        .</span><span style=\"color:#859900;\">collect</span><span style=\"color:#657b83;\">();\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, new_vec);\n}\n</span></code></pre>\n<p>Fairly straightforward, and our 3 steps turn into three chained method calls. Unfortunately, in practice, using <code>collect</code> is often not quite as straightforward as this. That's due to type inference. To see what I mean, let's take all of the type annotations out of the program above:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> new_vec = myvec\n        .</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">()\n        .</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">x</span><span style=\"color:#657b83;\">| x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">)\n        .</span><span style=\"color:#859900;\">collect</span><span style=\"color:#657b83;\">();\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, new_vec);\n}\n</span></code></pre>\n<p>This gives us the very friendly message:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">error[E0282]: type annotations needed\n --&gt; src\\main.rs:4:9\n  |\n4 |     let new_vec = myvec\n  |         ^^^^^^^ consider giving `new_vec` a type\n</span></code></pre>\n<p>The issue here is that we don't know which implementation of <code>FromIterator</code> we should be using. This is a problem that didn't exist in the pure FP world with <code>map</code> and <code>Functor</code>. In that world, <code>Functor</code>'s <code>map</code> is always &quot;shape preserving.&quot; When you <code>map</code> over a list in Haskell, the result will always be a list.</p>\n<p>That's not the case with the <code>IntoIterator</code>/<code>FromIterator</code> combination. <code>IntoIterator</code> destroys the original data structure, fully consuming it and producing an <code>Iterator</code>. Similarly, <code>FromIterator</code> produces a brand new data structure out of thin air, without any reference to the original data structure. Therefore, an explicit type annotation saying what the output type should be is necessary. In our program above, we did this by annotating <code>new_vec</code>. Another way is to use &quot;turbofish&quot; to annotate which <code>collect</code> to use:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> new_vec = myvec\n        .</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">()\n        .</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">x</span><span style=\"color:#657b83;\">| x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">)\n        .collect::&lt;</span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">&gt;&gt;();\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, new_vec);\n}\n</span></code></pre>\n<p>Note that we only needed indicate that we were collecting into a <code>Vec</code>. Rust's normal type inference was able to figure out:</p>\n<ul>\n<li>Which numeric type to use for the values</li>\n<li>That the original <code>myvec</code> should be a <code>Vec</code>, since it was produced by the <code>vec!</code> macro</li>\n</ul>\n<h2 id=\"side-effects-and-traverse\">Side effects and traverse</h2>\n<p>Alright, I want to announce to the world that I'll be doubling these values. It's easy to modify our <code>map</code>-using code in Rust to do this:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> new_vec = myvec\n        .</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">()\n        .</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">x</span><span style=\"color:#657b83;\">| {\n            </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">About to double </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, x);\n            x * </span><span style=\"color:#6c71c4;\">2\n        </span><span style=\"color:#657b83;\">})\n        .collect::&lt;</span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">&gt;&gt;();\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, new_vec);\n}\n</span></code></pre>\n<p>But Haskellers will warn you that this isn't quite so simple. <code>map</code> in Haskell is a pure function, meaning it doesn't allow for any side-effects (like printing to the screen). You can see this in action fairly easily:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">list </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#268bd2;\">Int</span><span style=\"color:#657b83;\">]\nlist </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">]\n\n</span><span style=\"color:#b58900;\">main </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">IO </span><span style=\"color:#859900;\">()\n</span><span style=\"color:#657b83;\">main </span><span style=\"color:#859900;\">= do\n  let</span><span style=\"color:#657b83;\"> newList </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#cb4b16;\">Int</span><span style=\"color:#657b83;\">]\n      newList </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\">\n        map\n          (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x </span><span style=\"color:#859900;\">-&gt; do</span><span style=\"color:#657b83;\">\n            putStrLn (</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">About to double </span><span style=\"color:#839496;\">&quot; </span><span style=\"color:#859900;\">++</span><span style=\"color:#657b83;\"> show x)\n            pure (x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">))\n          list\n  print newList\n</span></code></pre>\n<p>This code won't compile, due to the mismatch between an <code>Int</code> (a pure number) and an <code>IO Int</code> (an action with side effects which produces an <code>Int</code>):</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">Couldn&#39;t match type &#39;IO Int&#39; with &#39;Int&#39;\nExpected type: [Int]\nActual type: [IO Int]\n</span></code></pre>\n<p>Instead, we need to use <code>map</code>'s more powerful cousin, <code>traverse</code> (a.k.a. <code>mapM</code>, or &quot;monadic map&quot;). <code>traverse</code> allows us to perform a series of actions, and produce a new list with all of their results. This looks like:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">list </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#268bd2;\">Int</span><span style=\"color:#657b83;\">]\nlist </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">]\n\n</span><span style=\"color:#b58900;\">main </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">IO </span><span style=\"color:#859900;\">()\n</span><span style=\"color:#657b83;\">main </span><span style=\"color:#859900;\">= do</span><span style=\"color:#657b83;\">\n  newList </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\">\n    traverse\n      (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x </span><span style=\"color:#859900;\">-&gt; do</span><span style=\"color:#657b83;\">\n        putStrLn (</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">About to double </span><span style=\"color:#839496;\">&quot; </span><span style=\"color:#859900;\">++</span><span style=\"color:#657b83;\"> show x)\n        pure (x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">))\n      list\n  print newList\n</span></code></pre>\n<p>So why the difference between Haskell and Rust here? That's because Rust is not a pure language. Any function can perform side effects, like printing to the screen. Haskell, on the other hand, doesn't allow this, and therefore we need special helper functions like <code>traverse</code> to account for the potential side effects.</p>\n<p>I won't get into the philosophical differences between the two languages. Suffice it to say that both approaches have merit, and both have advantages and disadvantages. Let's see where the Rust approach &quot;breaks down&quot;, and how <code>FromIterator</code> steps up to the plate.</p>\n<h2 id=\"handling-failure\">Handling failure</h2>\n<p>In the example above with Haskell, we used side effects via the <code>IO</code> type. However, <code>traverse</code> isn't limited to working with <code>IO</code>. It can work with <em>many</em> different types, anything which is considered <code>Applicative</code>. And this covers many different common needs, including error handling. For example, we can change our program to not allow doubling &quot;big&quot; numbers greater than 5:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">list </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#268bd2;\">Int</span><span style=\"color:#657b83;\">]\nlist </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\">]\n\n</span><span style=\"color:#b58900;\">main </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">IO </span><span style=\"color:#859900;\">()\n</span><span style=\"color:#657b83;\">main </span><span style=\"color:#859900;\">= do\n  let</span><span style=\"color:#657b83;\"> newList </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\">\n        traverse\n          (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x </span><span style=\"color:#859900;\">-&gt;\n            if</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">&gt; </span><span style=\"color:#6c71c4;\">5\n              </span><span style=\"color:#859900;\">then </span><span style=\"color:#cb4b16;\">Left </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Not allowed to double big numbers</span><span style=\"color:#839496;\">&quot;\n              </span><span style=\"color:#859900;\">else </span><span style=\"color:#cb4b16;\">Right</span><span style=\"color:#657b83;\"> (x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">))\n          list\n  </span><span style=\"color:#859900;\">case</span><span style=\"color:#657b83;\"> newList </span><span style=\"color:#859900;\">of\n    </span><span style=\"color:#cb4b16;\">Left</span><span style=\"color:#657b83;\"> err </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> putStrLn err\n    </span><span style=\"color:#cb4b16;\">Right</span><span style=\"color:#657b83;\"> newList&#39; </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> print newList\n</span></code></pre>\n<p><code>Either</code> is a sum type, like an <code>enum</code> in Rust. It's equivalent to <code>Result</code> in Rust, but with different names. Instead of <code>Ok</code> and <code>Err</code>, we have <code>Right</code> (used by convention for success) and <code>Left</code> (used by convention for failure). The <code>Applicative</code> instance for it will stop processing when it encounters the first <code>Left</code>. So our program above will ultimately produce the output <code>Not allowed to double big numbers</code>. You can put as many values after the <code>6</code> in <code>list</code> as you want, and it will produce the same output. In fact, it will never even inspect those numbers.</p>\n<p>Coming back to Rust, let's first simply collect all of our <code>Result</code>s together into a single <code>Vec</code> to make sure the basics work:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> new_vec: </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#268bd2;\">i32</span><span style=\"color:#657b83;\">, </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">str</span><span style=\"color:#657b83;\">&gt;&gt; = myvec\n        .</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">()\n        .</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">x</span><span style=\"color:#657b83;\">| {\n            </span><span style=\"color:#859900;\">if</span><span style=\"color:#657b83;\"> x &gt; </span><span style=\"color:#6c71c4;\">5 </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Not allowed to double big numbers</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)\n            } </span><span style=\"color:#859900;\">else </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">)\n            }\n        })\n        .</span><span style=\"color:#859900;\">collect</span><span style=\"color:#657b83;\">();\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, new_vec);\n}\n</span></code></pre>\n<p>That makes sense. We've already seen that <code>.collect()</code> can take all the values in an <code>Iterator</code>'s stream and stick them into a <code>Vec</code>. And the <code>map</code> method is now generating <code>Result&lt;i32, &amp;str&gt;</code> values, so everything lines up.</p>\n<p>But this isn't the behavior we want. We want two changes:</p>\n<ul>\n<li><code>new_vec</code> should result in a <code>Result&lt;Vec&lt;i32&gt;, &amp;str&gt;</code>. In other words, it should result in either a single <code>Err</code> value, or a vector of successful results. Right now, it has a vector of success-or-failure values.</li>\n<li>We should immediately stop processing the original <code>Vec</code> once we see a value that's too big.</li>\n</ul>\n<p>To make it a bit more clear, it's easy enough to implement this with a <code>for</code> loop:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> new_vec = </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">::new();\n\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> myvec {\n        </span><span style=\"color:#859900;\">if</span><span style=\"color:#657b83;\"> x &gt; </span><span style=\"color:#6c71c4;\">5 </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Not allowed to double big numbers</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">);\n            </span><span style=\"color:#859900;\">return</span><span style=\"color:#657b83;\">;\n        } </span><span style=\"color:#859900;\">else </span><span style=\"color:#657b83;\">{\n            new_vec.</span><span style=\"color:#859900;\">push</span><span style=\"color:#657b83;\">(x);\n        }\n    }\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, new_vec);\n}\n</span></code></pre>\n<p>But now we've lost out on our <code>map</code> entirely, and we're dropping down to using explicit loops, mutation, and short-circuiting (via <code>return</code>). In other words, this code doesn't feel nearly as elegant to me.</p>\n<p>It turns out that our original code was almost perfect. Let's see a bit of magic, and then explain how it happend. Our previous version of the code used <code>map</code> and resulted in a <code>Vec&lt;Result&lt;i32, &amp;str&gt;&gt;</code>. And we wanted <code>Result&lt;Vec&lt;i32&gt;, &amp;str&gt;</code>. What happens if we simply change the type to what we want?</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> new_vec: </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#268bd2;\">i32</span><span style=\"color:#657b83;\">&gt;, </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">str</span><span style=\"color:#657b83;\">&gt; = myvec\n        .</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">()\n        .</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">x</span><span style=\"color:#657b83;\">| {\n            </span><span style=\"color:#859900;\">if</span><span style=\"color:#657b83;\"> x &gt; </span><span style=\"color:#6c71c4;\">5 </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Not allowed to double big numbers</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)\n            } </span><span style=\"color:#859900;\">else </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">)\n            }\n        })\n        .</span><span style=\"color:#859900;\">collect</span><span style=\"color:#657b83;\">();\n\n    </span><span style=\"color:#859900;\">match</span><span style=\"color:#657b83;\"> new_vec {\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(new_vec) </span><span style=\"color:#859900;\">=&gt; println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, new_vec),\n        </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(e) </span><span style=\"color:#859900;\">=&gt; println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, e),\n    }\n}\n</span></code></pre>\n<p>Thanks to the power of <code>FromIterator</code>, this simply works! To understand why, let's see some <a href=\"https://doc.rust-lang.org/stable/std/iter/trait.FromIterator.html#method.from_iter-15\">documentation on <code>FromIterator</code></a>:</p>\n<blockquote>\n<p>Takes each element in the <code>Iterator</code>: if it is an <code>Err</code>, no further elements are taken, and the <code>Err</code> is returned. Should no <code>Err</code> occur, a container with the values of each <code>Result</code> is returned.</p>\n</blockquote>\n<p>And suddenly it seems that Rust has implemented <code>traverse</code> all along! This extra flexibility in the <code>FromIterator</code> setup allows us to regain the short-circuiting error-handling behavior that FP people are familiar with in <code>traverse</code>.</p>\n<p>In contrast to <code>traverse</code>, we're still dealing with two different traits (<code>IntoIterator</code> and <code>FromIterator</code>), and there's nothing preventing these from being different types. Therefore, some kind of type annotation is still necessary. On the one hand, that could be seen as a downside of Rust's approach. On the other hand, it allows us to be more flexible in what types we generate, which we'll look at in the next section.</p>\n<p>And finally, it turns out we can use turbofish to rescue us yet again. For example:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> myvec = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\">];\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> new_vec = myvec\n        .</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">()\n        .</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">x</span><span style=\"color:#657b83;\">| {\n            </span><span style=\"color:#859900;\">if</span><span style=\"color:#657b83;\"> x &gt; </span><span style=\"color:#6c71c4;\">5 </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Not allowed to double big numbers</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)\n            } </span><span style=\"color:#859900;\">else </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(x * </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">)\n            }\n        })\n        .collect::&lt;</span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">&gt;, </span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">&gt;&gt;();\n\n    </span><span style=\"color:#859900;\">match</span><span style=\"color:#657b83;\"> new_vec {\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(new_vec) </span><span style=\"color:#859900;\">=&gt; println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, new_vec),\n        </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(e) </span><span style=\"color:#859900;\">=&gt; println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, e),\n    }\n}\n</span></code></pre><h2 id=\"different-fromiterator-impls\">Different FromIterator impls</h2>\n<p>So far, we've only seen two implementations of <code>FromIterator</code>: <code>Vec</code> and <code>Result</code>. There are many more available. One of my favorite is <code>HashMap</code>, which lets you collect a sequence of key/value pairs into a mapping.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::collections::HashMap;\n\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> people = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[\n        (</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Alice</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">),\n        (</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Bob</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">35</span><span style=\"color:#657b83;\">),\n        (</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Charlies</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">25</span><span style=\"color:#657b83;\">),\n    ].</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">().collect::&lt;HashMap&lt;</span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">, </span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">&gt;&gt;();\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Alice is: </span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, people.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Alice</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">));\n}\n</span></code></pre>\n<p>And due to how the <code>FromIterator</code> impl for <code>Result</code> works, you can layer these two together to collect a stream of <code>Result</code>s of pairs into a <code>Result&lt;HashMap&lt;_, _&gt;, _&gt;</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::collections::HashMap;\n\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> people = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">((</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Alice</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">)),\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">((</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Bob</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">35</span><span style=\"color:#657b83;\">)),\n        </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Uh-oh, this didn&#39;t work!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">),\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">((</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Charlies</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">25</span><span style=\"color:#657b83;\">)),\n    ].</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">().collect::&lt;</span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;HashMap&lt;</span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">, </span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">&gt;, </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">str</span><span style=\"color:#657b83;\">&gt;&gt;();\n\n    </span><span style=\"color:#859900;\">match</span><span style=\"color:#657b83;\"> people {\n        </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(e) </span><span style=\"color:#859900;\">=&gt; println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Error occurred: </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, e),\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(people) </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Alice is: </span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, people.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Alice</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">));\n        }\n    }\n}\n</span></code></pre><h2 id=\"validation\">Validation</h2>\n<p>In the Haskell world, we have two different concepts of error collection:</p>\n<ul>\n<li><code>Either</code>, which says &quot;stop on the first error&quot;</li>\n<li><code>Validation</code>, which says &quot;collect all of the errors together&quot;</li>\n</ul>\n<p><code>Validation</code> can be very useful for things like parsing web forms. You don't want to generate just the first failure, but collect all of the failures together for producing a more user-friendly experience. For fun, I decided to implement this in Rust as well:</p>\n<blockquote class=\"twitter-tweet\"><p lang=\"en\" dir=\"ltr\">I&#39;m tempted to write a Validation &quot;Applicative&quot; in Rust with a FromIterator impl to collect multiple Err values. I have no real need for this, but it still seems fun.</p>&mdash; Michael Snoyman (@snoyberg) <a href=\"https://twitter.com/snoyberg/status/1311639149171159041?ref_src=twsrc%5Etfw\">October 1, 2020</a></blockquote> <script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>\n<p>And as you'll see from the rest of that thread, a lot of the motivation for this blog post came from the Twitter replies.</p>\n<p>The implementation in Rust is fairly straightforward, and pretty easy to understand. I've <a href=\"https://github.com/snoyberg/validation-rs\">made it available on Github</a>. If there's any interest in seeing this as a crate, let me know in the issue tracker.</p>\n<p>To see this in action, let's modify our program above. First, I'll add the dependency to my <code>Cargo.toml</code> file:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">[</span><span style=\"color:#b58900;\">dependencies</span><span style=\"color:#657b83;\">.</span><span style=\"color:#b58900;\">validation</span><span style=\"color:#657b83;\">]\n</span><span style=\"color:#268bd2;\">git </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">https://github.com/snoyberg/validation-rs</span><span style=\"color:#839496;\">&quot;\n</span><span style=\"color:#268bd2;\">rev </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">0a7521f7022262bb00aea61761f76c3dd5ccefb5</span><span style=\"color:#839496;\">&quot;\n</span></code></pre>\n<p>And then modify the code to use the <code>Validation</code> enum instead of <code>Result</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::collections::HashMap;\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">validation::Validation;\n\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> people = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">((</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Alice</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">)),\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">((</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Bob</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">35</span><span style=\"color:#657b83;\">)),\n        </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Uh-oh, this didn&#39;t work!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">),\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">((</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Charlies</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#6c71c4;\">25</span><span style=\"color:#657b83;\">)),\n        </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">And neither did this!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">),\n    ].</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">().collect::&lt;Validation&lt;HashMap&lt;</span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">, </span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">&gt;, </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">str</span><span style=\"color:#657b83;\">&gt;&gt;&gt;();\n\n    </span><span style=\"color:#859900;\">match</span><span style=\"color:#657b83;\"> people.</span><span style=\"color:#859900;\">into_result</span><span style=\"color:#657b83;\">() {\n        </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(errs) </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Errors:</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">);\n            errs.</span><span style=\"color:#859900;\">into_iter</span><span style=\"color:#657b83;\">().</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">x</span><span style=\"color:#657b83;\">| </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, x)).</span><span style=\"color:#859900;\">collect</span><span style=\"color:#657b83;\">()\n        }\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(people) </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Alice is: </span><span style=\"color:#cb4b16;\">{:?}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, people.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Alice</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">));\n        }\n    }\n}\n</span></code></pre>\n<p><strong>Bonus</strong> Note the somewhat cheeky usage of <code>map</code> and <code>collect</code> to print out the errors. This is leveraging the <code>()</code> impl of <code>FromIterator</code>, which collects together a stream of <code>()</code> values into a single one.</p>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>I realize that was a bit of a rambling journey, but hopefully a fun one for Rustaceans, Haskellers, and Scala folk. Here are some of my takeaways:</p>\n<ul>\n<li>The <code>collect</code> method is very flexible</li>\n<li>There's no magic involved in <code>collect</code>, just the <code>FromIterator</code> trait and the behavior of the types that implement it\n<ul>\n<li>This was actually a big takeaway for me. I had somehow forgotten about <code>FromIterator</code> a few months back, and was nervous about what &quot;secret&quot; behavior <code>collect</code> was doing.</li>\n</ul>\n</li>\n<li>The downside of <code>collect</code> is that, since it's not structure preserving like <code>map</code> or <code>traverse</code>, you'll sometimes need type annotations\n<ul>\n<li>Get used to turbofish!</li>\n</ul>\n</li>\n<li>There are lots of useful impls of <code>FromIterator</code> available</li>\n</ul>\n<p>If you enjoyed this post, you may also want to check out these related topics:</p>\n<ul>\n<li><a href=\"https://www.fpcomplete.com/blog/2017/07/iterators-streams-rust-haskell/\">Iterators and Streams in Rust and Haskell</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/2018/07/streaming-utf8-haskell-rust/\">Streaming UTF-8 in Haskell and Rust</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/rust-devops/\">Rust with DevOps Success Strategies</a></li>\n<li><a href=\"https://www.fpcomplete.com/rust/crash-course/\">Rust Crash Course eBook</a></li>\n</ul>\n<p>If you're interested in learning more about FP Complete's <a href=\"services/#managedit\">consulting and training services</a>, please <a href=\"https://www.fpcomplete.com/contact-us/\">contact us</a> to talk with our team about how we can help you succeed with Rust, DevOps, and functional programming.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/collect-rust-traverse-haskell-scala/",
        "slug": "collect-rust-traverse-haskell-scala",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Collect in Rust, traverse in Haskell and Scala",
        "description": "In this post, we'll analyze the collect method in Rust, powered by the Iterator and FromIterator traits, together with a comparison against the traverse function from Haskell and Scala",
        "updated": null,
        "date": "2020-10-06",
        "year": 2020,
        "month": 10,
        "day": 6,
        "taxonomies": {
          "tags": [
            "rust",
            "haskell",
            "scala",
            "functional programming"
          ],
          "categories": [
            "functional programming"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "image": "images/blog/traverse-turtles.png",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/collect-rust-traverse-haskell-scala/",
        "components": [
          "blog",
          "collect-rust-traverse-haskell-scala"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "using-fromiterator-and-collect",
            "permalink": "https://www.fpcomplete.com/blog/collect-rust-traverse-haskell-scala/#using-fromiterator-and-collect",
            "title": "Using FromIterator and collect",
            "children": []
          },
          {
            "level": 2,
            "id": "side-effects-and-traverse",
            "permalink": "https://www.fpcomplete.com/blog/collect-rust-traverse-haskell-scala/#side-effects-and-traverse",
            "title": "Side effects and traverse",
            "children": []
          },
          {
            "level": 2,
            "id": "handling-failure",
            "permalink": "https://www.fpcomplete.com/blog/collect-rust-traverse-haskell-scala/#handling-failure",
            "title": "Handling failure",
            "children": []
          },
          {
            "level": 2,
            "id": "different-fromiterator-impls",
            "permalink": "https://www.fpcomplete.com/blog/collect-rust-traverse-haskell-scala/#different-fromiterator-impls",
            "title": "Different FromIterator impls",
            "children": []
          },
          {
            "level": 2,
            "id": "validation",
            "permalink": "https://www.fpcomplete.com/blog/collect-rust-traverse-haskell-scala/#validation",
            "title": "Validation",
            "children": []
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/collect-rust-traverse-haskell-scala/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 2585,
        "reading_time": 13,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/of-course-it-compiles-right.md",
        "content": "<p>I recently joined <a href=\"https://lambda.show/episodes/michael-snoyman-from-haskell-to-rust\">Matt Moore on LambdaShow</a>. We spent some time discussing Rust, and one point I made was that, in my experience with Rust, ergonomics go something like this:</p>\n<ul>\n<li>Beginner: oh cool, that worked, no problem</li>\n<li>Advanced beginner: wait... why exactly did that work 99 other times? Why is it failing this time? I'm so confused!</li>\n<li>Intermediate/advanced: oh, now I understand things really well, that's convenient</li>\n</ul>\n<p>That may seem a bit abstract. Fortunately for me, an example of that popped up almost immediately after the post went live. This is my sheepish blog post explaining how I fairly solidly misunderstood something about the borrow checker. Hopefully it will help others.</p>\n<p>Two weeks back, I wrote an offhand tweet with a bit of a code puzzle:</p>\n<blockquote class=\"twitter-tweet\"><p lang=\"en\" dir=\"ltr\">This program _looks_ like it could segfault by using a pointer to a dropped String. Who wants to guess what it actually does? <a href=\"https://t.co/gurHjdh2A7\">pic.twitter.com/gurHjdh2A7</a></p>&mdash; Michael Snoyman (@snoyberg) <a href=\"https://twitter.com/snoyberg/status/1301579676423462914?ref_src=twsrc%5Etfw\">September 3, 2020</a></blockquote>\n<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>\n<p>I thought this was a slightly tricky case of ownership, and hoped it would help push people to a more solid understanding of the topic. Soon after, I got a reply that gave the solution I had expected:</p>\n<blockquote class=\"twitter-tweet\" data-conversation=\"none\"><p lang=\"en\" dir=\"ltr\">Without the RefCell it would fail to compile, because you have a live &amp; with hello, then you try to borrow all_tags as &amp;mut, despite an immutable reference. Since we use RefCell, this becomes a runtime panic instead.</p>&mdash; Lúcás Meier (@cronokirby) <a href=\"https://twitter.com/cronokirby/status/1301582898701758465?ref_src=twsrc%5Etfw\">September 3, 2020</a></blockquote>\n<p>But then the twist: a question that made me doubt my own sanity.</p>\n<blockquote class=\"twitter-tweet\" data-conversation=\"none\"><p lang=\"en\" dir=\"ltr\">But I don&#39;t quite understand how it does that at runtime. All we keep is a plain reference, not a `Ref` (which keeps count). The `Ref` should actually get destroyed, but somehow it won&#39;t because of the &amp;str we keep. I studied the code quite a bit, but it still looks like magic.</p>&mdash; Eskimo Coder (@tuxkimo) <a href=\"https://twitter.com/tuxkimo/status/1304363365314236416?ref_src=twsrc%5Etfw\">September 11, 2020</a></blockquote>\n<p>This led me to filing <a href=\"https://github.com/rust-lang/rust/issues/76601\">a bogus bug report with the Rust team</a>. Fortunately for me, Jonas Schievink had mercy and quickly pointed me to the <a href=\"https://doc.rust-lang.org/reference/destructors.html?highlight=temporary,life#temporary-lifetime-extension\">documentation on temporary lifetime extension</a>, which explains the whole situation.</p>\n<p>If you've read this much, and everything made perfect sense, congratulations! You probably don't need to bother reading the rest of the post. But if anything is unclear, keep reading. I'll try to make this as clear as possible.</p>\n<p>And if the explanation below still doesn't make sense, may I recommend FP Complete's <a href=\"https://www.fpcomplete.com/rust/crash-course/\">Rust Crash Course eBook</a> to brush up on ownership?</p>\n<h2 id=\"borrow-rules\">Borrow rules</h2>\n<p>Arguably the key feature of Rust is its borrow checker. One of the core rules of the borrow checker is that you cannot access data that is mutably referenced elsewhere. Or said more directly: you can either immutably borrow data multiple times, or mutably borrow it once, but not both at the same time. Usually, we let the borrow checker enforce this rule. And it enforces that rule at compile time.</p>\n<p>However, there are some situations where a statically checked rule like that is too restrictive. In such cases, the Rust standard library provides <em>cells</em>, which let you move this borrow checking from compile time (via static analysis) to runtime (via dynamic counters). This is known as <em>interior mutability</em>. And a common type for this is a <a href=\"https://doc.rust-lang.org/stable/std/cell/struct.RefCell.html\"><code>RefCell</code></a>.</p>\n<p>With a <code>RefCell</code>, the checking occurs at runtime. Let's demonstrate how that works. First, consider this program that fails to compile:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age: </span><span style=\"color:#268bd2;\">u32 </span><span style=\"color:#657b83;\">= </span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">;\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_ref: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">u32 </span><span style=\"color:#657b83;\">= </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">age;\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_mut_ref: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#586e75;\">mut </span><span style=\"color:#268bd2;\">u32 </span><span style=\"color:#657b83;\">= </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age;\n    *age_mut_ref += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Happy birthday, you&#39;re </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\"> years old!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, age_ref);\n}\n</span></code></pre>\n<p>We try to take both an immutable reference <em>and</em> a mutable reference to the value <code>age</code> simultaneously. This doesn't work out too well:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">error[E0502]: cannot borrow `age` as mutable because it is also borrowed as immutable\n --&gt; src\\main.rs:6:33\n  |\n4 |     let age_ref: &amp;u32 = &amp;age;\n  |                         ---- immutable borrow occurs here\n5 |\n6 |     let age_mut_ref: &amp;mut u32 = &amp;mut age;\n  |                                 ^^^^^^^^ mutable borrow occurs here\n...\n9 |     println!(&quot;Happy birthday, you&#39;re {} years old!&quot;, age_ref);\n  |                                                      ------- immutable borrow later used here\n</span></code></pre>\n<p>The right thing to do is to fix this code. But let's do the wrong thing! Instead of trying to fix it correctly, we're going to use <code>RefCell</code> to replace our compile time checks (which prevent the code from building) with runtime checks (which allow the code to build, and then fail at runtime). Let's check that out:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::cell::{Ref, RefMut, RefCell};\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age: RefCell&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = RefCell::new(</span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_ref: Ref&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">();\n\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age_mut_ref: RefMut&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow_mut</span><span style=\"color:#657b83;\">();\n    *age_mut_ref += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Happy birthday, you&#39;re </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\"> years old!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, age_ref);\n}\n</span></code></pre>\n<p>It's instructive to compare this code with the previous code. It looks remarkably similar! We're replaced <code>&amp;u32</code> with <code>Ref&lt;u32&gt;</code>, <code>&amp;mut u32</code> with <code>RefMut&lt;u32&gt;</code>, and <code>&amp;age</code> and <code>&amp;mut age</code> with <code>age.borrow()</code> and <code>age.borrow_mut()</code>, respectively. You may be wondering: what are those <code>Ref</code> and <code>RefMut</code> things? Hold that thought.</p>\n<p>This code surprisingly compiles. And here's the runtime output (using Rust Nightly, which gives a slightly nicer error message):</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">thread &#39;main&#39; panicked at &#39;already borrowed: BorrowMutError&#39;, src\\main.rs:7:44\n</span></code></pre>\n<p>That looks a lot like the error message we saw above from the compiler. That's no accident: these are the same error showing up in two different ways.</p>\n<h2 id=\"ref-and-refmut\">Ref and RefMut</h2>\n<p>Our code panics when it calls <code>age.borrow_mut()</code>. Something seems to <em>know</em> that the <code>age_ref</code> variable exists. And in fact, that's basically true. When we called <code>age.borrow()</code>, a counter on the <code>RefCell</code> was incremented. As long as <code>age_ref</code> stays alive, that counter will remain active. When <code>age_ref</code> goes out of scope, the <code>Ref&lt;u32&gt;</code> will be dropped, and the drop will cause the counter to be decremented. The same logic applies to the <code>age_mut_ref</code>. Let's make two modifications to our code. First, there's no need to call <code>age.borrow()</code> before <code>age.borrow_mut()</code>. Let's slightly rearrange the code:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::cell::{Ref, RefMut, RefCell};\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age: RefCell&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = RefCell::new(</span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age_mut_ref: RefMut&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow_mut</span><span style=\"color:#657b83;\">();\n    *age_mut_ref += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_ref: Ref&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">();\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Happy birthday, you&#39;re </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\"> years old!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, age_ref);\n}\n</span></code></pre>\n<p>This compiles, but still gives a runtime error. However, it's a slightly different one:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">thread &#39;main&#39; panicked at &#39;already mutably borrowed: BorrowError&#39;, src\\main.rs:8:33\n</span></code></pre>\n<p>Now the problem is that, when we try to call <code>age.borrow()</code>, the <code>age_mut_ref</code> is still active. Fortunately, we can fix that by manually dropping it before the <code>age.borrow()</code> call:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::cell::{Ref, RefMut, RefCell};\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age: RefCell&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = RefCell::new(</span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age_mut_ref: RefMut&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow_mut</span><span style=\"color:#657b83;\">();\n    *age_mut_ref += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n    std::mem::drop(age_mut_ref);\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_ref: Ref&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">();\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Happy birthday, you&#39;re </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\"> years old!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, age_ref);\n}\n</span></code></pre>\n<p>And finally, our program not only compiles, but runs successfully! Now I know that I'm 31 years old! (Or at least I wish I still was.)</p>\n<p>We have another mechanism for forcing the value to drop: an inner block. If we create a block within the <code>main</code> function, it will have its own scope, and the <code>age_mut_ref</code> will automatically be dropped, no need for <code>std::mem::drop</code>. That looks like this:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::cell::{Ref, RefMut, RefCell};\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age: RefCell&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = RefCell::new(</span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">);\n\n    {\n        </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age_mut_ref: RefMut&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow_mut</span><span style=\"color:#657b83;\">();\n        *age_mut_ref += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n    }\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_ref: Ref&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">();\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Happy birthday, you&#39;re </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\"> years old!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, age_ref);\n}\n</span></code></pre>\n<p>Once again, this compiles and runs. Looking back, we can hopefully now understand why <code>Ref</code> and <code>RefMut</code> are necessary. If <code>.borrow()</code> and <code>.borrow_mut()</code> simply returned actual references (immutable or mutable), there would be no <code>struct</code> with a <code>Drop</code> impl to ensure that the internal counters in <code>RefCell</code> were decremented when they go out of scope. So the world now makes sense.</p>\n<h2 id=\"no-reference-without-a-ref\">No reference without a Ref</h2>\n<p>Here's something cool: you can borrow a normal reference (e.g. <code>&amp;u32</code>) from a <code>Ref</code> (e.g. <code>Ref&lt;u32&gt;</code>). Check this out:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::cell::{Ref, RefMut, RefCell};\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age: RefCell&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = RefCell::new(</span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">);\n\n    {\n        </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age_mut_ref: RefMut&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow_mut</span><span style=\"color:#657b83;\">();\n        *age_mut_ref += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n    }\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_ref: Ref&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">();\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_reference: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">u32 </span><span style=\"color:#657b83;\">= </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">age_ref;\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Happy birthday, you&#39;re </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\"> years old!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, age_reference);\n}\n</span></code></pre>\n<p><code>age_ref</code> is a <code>Ref&lt;u32&gt;</code>, but <code>age_reference</code> is a <code>&amp;u32</code>. This is a compile-time-checked reference. We're now saying that the lifetime of <code>age_reference</code> cannot outlive the lifetime of <code>age_ref</code>. As it stands, that's true, and everything compiles and runs correctly. But we can break that really easily using either <code>std::mem::drop</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::cell::{Ref, RefMut, RefCell};\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age: RefCell&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = RefCell::new(</span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">);\n\n    {\n        </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age_mut_ref: RefMut&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow_mut</span><span style=\"color:#657b83;\">();\n        *age_mut_ref += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n    }\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_ref: Ref&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">();\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_reference: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">u32 </span><span style=\"color:#657b83;\">= </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">age_ref;\n    std::mem::drop(age_ref);\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Happy birthday, you&#39;re </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\"> years old!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, age_reference);\n}\n</span></code></pre>\n<p>Or by using inner blocks:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::cell::{Ref, RefMut, RefCell};\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age: RefCell&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = RefCell::new(</span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">);\n\n    {\n        </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age_mut_ref: RefMut&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow_mut</span><span style=\"color:#657b83;\">();\n        *age_mut_ref += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n    }\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_reference: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">u32 </span><span style=\"color:#657b83;\">= {\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_ref: Ref&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">();\n        </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">age_ref\n    };\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Happy birthday, you&#39;re </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\"> years old!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, age_reference);\n}\n</span></code></pre>\n<p>The latter results in the error message:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">error[E0597]: `age_ref` does not live long enough\n  --&gt; src\\main.rs:12:9\n   |\n10 |     let age_reference: &amp;u32 = {\n   |         ------------- borrow later stored here\n11 |         let age_ref: Ref&lt;u32&gt; = age.borrow();\n12 |         &amp;age_ref\n   |         ^^^^^^^^ borrowed value does not live long enough\n13 |     };\n   |     - `age_ref` dropped here while still borrowed\n</span></code></pre>\n<p>This makes sense hopefully: <code>age_reference</code> is borrowing from <code>age_ref</code>, and therefore cannot outlive it.</p>\n<h2 id=\"the-false-fail\">The false fail</h2>\n<p>Alright, our inner block currently looks like this, and refuses to compile:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_reference: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">u32 </span><span style=\"color:#657b83;\">= {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_ref: Ref&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">();\n    </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">age_ref\n};\n</span></code></pre>\n<p><code>age_ref</code> is really a useless temporary variable inside that block. I assign a value to it, and then immediately borrow from that variable and never use it again. It should have no impact on our program to combine that into a single line within a block, right? Wrong. Check out this program:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::cell::{RefMut, RefCell};\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age: RefCell&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = RefCell::new(</span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\">);\n\n    {\n        </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> age_mut_ref: RefMut&lt;</span><span style=\"color:#268bd2;\">u32</span><span style=\"color:#657b83;\">&gt; = age.</span><span style=\"color:#859900;\">borrow_mut</span><span style=\"color:#657b83;\">();\n        *age_mut_ref += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n    }\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_reference: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">u32 </span><span style=\"color:#657b83;\">= {\n        </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">()\n    };\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Happy birthday, you&#39;re </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\"> years old!</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, age_reference);\n}\n</span></code></pre>\n<p>This looks almost identical to the code above. But this code compiles and runs successfully. What gives?!? It turns out, creating our temporary variable wasn't quite as meaningless as we thought. That's thanks to something called <a href=\"https://doc.rust-lang.org/reference/destructors.html?highlight=temporary,life#temporary-lifetime-extension\"><em>temporary lifetime extension</em></a>. Let me start with a caveat from the docs themselves:</p>\n<blockquote>\n<p><strong>Note</strong>: The exact rules for temporary lifetime extension are subject to change. This is describing the current behavior only.</p>\n</blockquote>\n<p>With that out of the way, let's quote once more from the docs:</p>\n<blockquote>\n<p>The temporary scopes for expressions in <code>let</code> statements are sometimes <em>extended</em> to the scope of the block containing the <code>let</code> statement. This is done when the usual temporary scope would be too small, based on certain syntactic rules.</p>\n</blockquote>\n<p>OK, I'm all done quoting. The documentation there is pretty good at explaining things. For our case above, let's look at the code in question:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> age_reference: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">u32 </span><span style=\"color:#657b83;\">= {\n    </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">age.</span><span style=\"color:#859900;\">borrow</span><span style=\"color:#657b83;\">()\n};\n</span></code></pre>\n<p><code>age.borrow()</code> create a value of type <code>Ref&lt;u32&gt;</code>. What variable holds that value? Trick question: there isn't one. This value is <em>temporary</em>. We use temporary values in programming all the time. In <code>(1 + 2) + 5</code>, the expression <code>1 + 2</code> generates a temporary <code>3</code>, which is then added to <code>5</code> and thrown away. Normally these temporaries aren't terribly interesting.</p>\n<p>But in the context of lifetimes and borrow checkers, they are. Taken at the most literal, <code>{ &amp;age.borrow() }</code> should behave as follows:</p>\n<ul>\n<li>Create a new block</li>\n<li>Call <code>age.borrow()</code> to get a <code>Ref&lt;u32&gt;</code></li>\n<li>That <code>Ref&lt;u32&gt;</code> is owned by the block around this expression</li>\n<li>Borrow a reference to that <code>Ref&lt;u32&gt;</code></li>\n<li>Try to return that reference as the result of the block</li>\n<li>Realize that reference refers to a value that was dropped with the block, and therefore lifetime rules are violated</li>\n</ul>\n<p>But this kind of thing would pop up all the time! Consider the incredibly simple examples from the docs that I promised not to quote from anymore (borrowing code snippets is different, OK?):</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> x = </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#586e75;\">mut </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\">;\n</span><span style=\"color:#93a1a1;\">// Usually a temporary would be dropped by now, but the temporary for `0` lives\n// to the end of the block.\n</span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, x);\n</span></code></pre>\n<p>It turns out that strictly following lexical scoping rules for lifetimes wouldn't be ergonomic. So there's a special case to make it feel right.</p>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>Firstly, I hope this was a good example of my comment about ergonomics. I never would have thought about <code>let x = &amp;mut 0</code> as a beginner: yeah, sure, I can borrow a reference to a number. Cool. Then, with a bit more experience, it suddenly seems shocking: what's the lifetime of <code>0</code>? And finally, with just a bit more experience (and the kind help of Rust issue tracker maintainers), it makes sense again.</p>\n<p>Secondly, I hope this semi-deep dive into how <code>RefCell</code> moves borrow rule checking to runtime helps elucidate some things. In my opinion, this was one of the harder concepts to grok in my Rust learning journey.</p>\n<p>Thirdly, I hope seeing the temporary lifetime extension rules helps clarify why some things work that you thought wouldn't. I know I've been in the middle of writing something before, been surprised the borrow checker didn't punch me in the face, and then happily went on my way instead of questioning why everything went better than expected.</p>\n<p>The tweets I started this off with discuss a more advanced version than I covered in the rest of the post. I'd recommend going back to the top and making sure the code and explanations all make sense.</p>\n<p>Want to learn more about Rust? Check out FP Complete's <a href=\"https://www.fpcomplete.com/rust/crash-course/\">Rust Crash Course</a>, or read about our <a href=\"/services/#managedit\">training courses</a>. Also, you may be interested in these related posts:</p>\n<ul>\n<li><a href=\"https://www.fpcomplete.com/blog/different-levels-async-rust/\">Different levels of async in Rust</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/2018/10/is-rust-functional/\">Is Rust functional?</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/rust-for-devops-tooling/\">Using Rust for DevOps tooling</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/serverless-rust-wasm-cloudflare/\">Serverless Rust using WASM and CloudFlare</a></li>\n</ul>\n",
        "permalink": "https://www.fpcomplete.com/blog/of-course-it-compiles-right/",
        "slug": "of-course-it-compiles-right",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Rust: Of course it compiles, right?",
        "description": "An example of some code that looks like it should fail at runtime, then looks like it should fail at compile time, and why rustc lets it slide",
        "updated": null,
        "date": "2020-09-21",
        "year": 2020,
        "month": 9,
        "day": 21,
        "taxonomies": {
          "tags": [
            "rust"
          ],
          "categories": [
            "functional programming"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "image": "images/blog/of-course.png",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/of-course-it-compiles-right/",
        "components": [
          "blog",
          "of-course-it-compiles-right"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "borrow-rules",
            "permalink": "https://www.fpcomplete.com/blog/of-course-it-compiles-right/#borrow-rules",
            "title": "Borrow rules",
            "children": []
          },
          {
            "level": 2,
            "id": "ref-and-refmut",
            "permalink": "https://www.fpcomplete.com/blog/of-course-it-compiles-right/#ref-and-refmut",
            "title": "Ref and RefMut",
            "children": []
          },
          {
            "level": 2,
            "id": "no-reference-without-a-ref",
            "permalink": "https://www.fpcomplete.com/blog/of-course-it-compiles-right/#no-reference-without-a-ref",
            "title": "No reference without a Ref",
            "children": []
          },
          {
            "level": 2,
            "id": "the-false-fail",
            "permalink": "https://www.fpcomplete.com/blog/of-course-it-compiles-right/#the-false-fail",
            "title": "The false fail",
            "children": []
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/of-course-it-compiles-right/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 2436,
        "reading_time": 13,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/where-rust-fits-in-your-organization.md",
        "content": "<p>Rust is a relatively new and promising language that offers improvements in software\nin terms of safety and speed. We'll cover if adopting Rust into your organization\nmakes sense and where you would want to add it to an existing software stack.</p>\n<h2 id=\"advantages-of-rust\">Advantages of Rust</h2>\n<h3 id=\"background\">Background</h3>\n<p>Rust was originally created by Mozilla in order to replace C++ in the Firefox \nbrowser with a safer alternative.\nC++ is not a memory safe language, and for Mozilla memory safety issues were the \nmain culprit for numerous bugs and security vulnerabilities in the Firefox browser.</p>\n<p>To replace it Mozilla needed a language that would not require a runtime or a \ngarbage collector. No language existed at that time which reasonably met those\nrequirements, so instead Mozilla worked to implement their own language. \nOut of that endeavor sprung Rust.</p>\n<h3 id=\"adoption-and-use-beyond-mozilla\">Adoption and use beyond Mozilla</h3>\n<p>Since its creation the language has gained widespread adoption and use far\nbeyond Mozilla and the Firefox browser. \nThis is not surprising, as the language is generally considered to be superbly well\ndesigned, adopting many programming language advances that have been made in the\nlast 20 years. \nAdd to that it's incredibly fast - on the same level as idiomatic C and C++ code.</p>\n<h3 id=\"language-design\">Language Design</h3>\n<p>Another reason for its popularity and growing use is that Rust doesn't re-implement\nbug-causing language design choices.</p>\n<p>With Rust, errors induced by missing null checking and poor error handling, as \nwell as other classes of coding errors, are ruled out by the design of the language \nand the strong type checks by the Rust compiler.</p>\n<p>For example instead of allowing for things to be <code>null</code> or <code>nil</code>, Rust has enum \ntypes. Using these a Rust programmer can handle failure cases in \na reasonable and safe way with useful enum types like \n<a href=\"https://doc.rust-lang.org/std/option/enum.Option.html\"><code>Option</code></a> and\n<a href=\"https://doc.rust-lang.org/std/result/enum.Result.html\"><code>Result</code></a>.</p>\n<p>Compare this to a language like Go which doesn't provide this and instead implements\nthe <code>null</code> pointer. Doing so essentially creates a dangerous escape door out of \nthe type system that infects every type in the language. \nAs a result a Go programmer could easily forget to check for <code>null</code> and overlook \ncases where a <code>null</code> value could be returned.</p>\n<p>So if you have a Python 2 code base and you're trying to decide whether to \nre-implement it Go, use Rust instead!</p>\n<h2 id=\"rust-in-the-wild\">Rust in the wild</h2>\n<h3 id=\"rust-adoption-success-stories\">Rust Adoption Success Stories</h3>\n<p>In 2020 Rust was once again (for 5 years running!) the most loved programming \nlanguage according to the \n<a href=\"https://insights.stackoverflow.com/survey/2020#technology-most-loved-dreaded-and-wanted-languages-loved\">Stack Overflow developer survey</a>.</p>\n<p>Just because software developers love a language though, does not equate\nto success if you adopt the language into your organization.</p>\n<p>Some of the best success stories for companies that have adopted Rust come from \nthose that isolated some small but critical piece of their software and \nre-implemented it in Rust.</p>\n<p>In a large organization, Rust is extremely useful in a scenario like this where\na small but rate limiting piece of the software stack can be re-written in Rust.\nThis gives the organization the benefits of adopting Rust in terms of performant,\nfast software but without requiring them to adopt the language across the board.\nAnd because Rust doesn't bring its own competing runtime and garbage collector, \nit fits this role phenomenally well.</p>\n<h3 id=\"large-companies-that-count-themselves-as-rustaceans\">Large Companies that count themselves as Rustaceans</h3>\n<p>Large companies like Microsoft now expound on Rust being \n<a href=\"https://thenewstack.io/microsoft-rust-is-the-industrys-best-chance-at-safe-systems-programming/\">the future of safe software development</a>\nand have <a href=\"https://medium.com/the-innovation/how-microsoft-is-adopting-rust-e0f8816566ba\">adopted using it</a>.\nOther companies like Amazon have <a href=\"https://seanmonstar.com/post/617213413024759808/next-up-aws\">chosen Rust</a>\nmore and more for new critical pieces of cloud infrastructure software.</p>\n<p>Apple, Google, Facebook, Cloudflare, and Dropbox (to name a few) also all now count\nthemselves as Rust adopters.</p>\n<h2 id=\"cost-and-tradeoffs-of-rust\">Cost and Tradeoffs of Rust</h2>\n<h3 id=\"fighting-the-rust-compiler\">Fighting the Rust Compiler</h3>\n<p>One of the key reasons to use Rust is to limit (or completely eliminate) entire\nclasses of runtime bugs and errors. The drawback is that with Rust's strong type\nsystem and compile time checks, you will end up seeing a fair bit more compile time \nerrors with your code.  Some developers find this unnerving and become frustrated.\nThis is especially true if they're used to less safe languages (like Javascript or C++)\nthat ignore certain categories of programming mistakes at compile time and leave \nthem as surprises when the software is run.</p>\n<p>For some organizations, they're okay with this trade-off and the associated cost \nof discovering errors in production.\nIn these scenarios, it may be the case that the code being written is not \nincredibly critical and shipping buggy code to production is tolerable \n(to a certain degree). </p>\n<h3 id=\"development-time\">Development Time</h3>\n<p>Rust also brings with it a certain cost in terms of the time it takes to iterate\non and develop. This is something associated with all compiled languages and it's\nnot exclusive to Rust, but it's worth considering.\nRust might not be a good fit if your organization's projects are comprised of \nrelatively simple codebases where the added compile time is not worth it.</p>\n<h2 id=\"is-rust-right-for-your-organization\">Is Rust Right for Your Organization?</h2>\n<p>Rust is well suited to situations where having performant, resource efficient code\nmakes a huge difference for the larger overall product.\nIf your organization could benefit from isolating critical pieces of its software\nstack that meet this description, then you should consider adopting and using Rust.\nThe unique qualities of Rust mean that you don't need to adopt Rust across your \nentire organization to see a meaningful difference.</p>\n<p>In addition to that, Rust is seeing major adoption outside its original target \nuse case as a systems language. More and more it's being used for web servers,\nweb dev via Web Assembly, game development, and general purpose programming uses.\nRust has become a full stack language with huge range of supported use cases. </p>\n<p>If you'd like to know more about Rust and how adopting it could make a difference\nin your organization, then please reach out to FP Complete! \nIf you have a Rust project you want to get started, or if you would like \nRust training for your team, FP Complete can help.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/",
        "slug": "where-rust-fits-in-your-organization",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Where Rust fits in your organization",
        "description": "We'll cover if adopting Rust into your organization makes sense and where you would want to add it to an existing software stack.",
        "updated": null,
        "date": "2020-09-16",
        "year": 2020,
        "month": 9,
        "day": 16,
        "taxonomies": {
          "categories": [
            "functional programming"
          ],
          "tags": [
            "devops",
            "rust",
            "insights"
          ]
        },
        "extra": {
          "author": "Mike McGirr",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/where-rust-fits-in-your-organization/",
        "components": [
          "blog",
          "where-rust-fits-in-your-organization"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "advantages-of-rust",
            "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#advantages-of-rust",
            "title": "Advantages of Rust",
            "children": [
              {
                "level": 3,
                "id": "background",
                "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#background",
                "title": "Background",
                "children": []
              },
              {
                "level": 3,
                "id": "adoption-and-use-beyond-mozilla",
                "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#adoption-and-use-beyond-mozilla",
                "title": "Adoption and use beyond Mozilla",
                "children": []
              },
              {
                "level": 3,
                "id": "language-design",
                "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#language-design",
                "title": "Language Design",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "rust-in-the-wild",
            "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#rust-in-the-wild",
            "title": "Rust in the wild",
            "children": [
              {
                "level": 3,
                "id": "rust-adoption-success-stories",
                "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#rust-adoption-success-stories",
                "title": "Rust Adoption Success Stories",
                "children": []
              },
              {
                "level": 3,
                "id": "large-companies-that-count-themselves-as-rustaceans",
                "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#large-companies-that-count-themselves-as-rustaceans",
                "title": "Large Companies that count themselves as Rustaceans",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "cost-and-tradeoffs-of-rust",
            "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#cost-and-tradeoffs-of-rust",
            "title": "Cost and Tradeoffs of Rust",
            "children": [
              {
                "level": 3,
                "id": "fighting-the-rust-compiler",
                "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#fighting-the-rust-compiler",
                "title": "Fighting the Rust Compiler",
                "children": []
              },
              {
                "level": 3,
                "id": "development-time",
                "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#development-time",
                "title": "Development Time",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "is-rust-right-for-your-organization",
            "permalink": "https://www.fpcomplete.com/blog/where-rust-fits-in-your-organization/#is-rust-right-for-your-organization",
            "title": "Is Rust Right for Your Organization?",
            "children": []
          }
        ],
        "word_count": 1050,
        "reading_time": 6,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/avoiding-duplicating-strings-rust.md",
        "content": "<p><em>Based on actual events.</em></p>\n<p>Let's say you've got a blog. The blog has a bunch of posts. Each post has a title and a set of tags. The metadata for these posts is all contained in TOML files in a single directory. (If you use <a href=\"https://getzola.org/\">Zola</a>, you're pretty close to that.) And now you need to generate a CSV file showing a matrix of blog posts and their tags. Seems like a great job for Rust!</p>\n<p>In this post, we're going to:</p>\n<ul>\n<li>Explore how we'd solve this (fairly simple) problem</li>\n<li>Investigate how Rust's types tell us a lot about memory usage</li>\n<li>Play with some nice and not-so-nice ways to optimize our program</li>\n</ul>\n<p>Onwards!</p>\n<h2 id=\"program-behavior\">Program behavior</h2>\n<p>We've got a bunch of TOML files sitting in the <code>posts</code> directory. Here are some example files:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#93a1a1;\"># devops-for-developers.toml\n</span><span style=\"color:#268bd2;\">title </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">DevOps for (Skeptical) Developers</span><span style=\"color:#839496;\">&quot;\n</span><span style=\"color:#268bd2;\">tags </span><span style=\"color:#657b83;\">= [</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">dev</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">devops</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">]\n\n</span><span style=\"color:#93a1a1;\"># rust-devops.toml\n</span><span style=\"color:#268bd2;\">title </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Rust with DevOps</span><span style=\"color:#839496;\">&quot;\n</span><span style=\"color:#268bd2;\">tags </span><span style=\"color:#657b83;\">= [</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">devops</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">rust</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">]\n</span></code></pre>\n<p>We want to create a CSV file that looks like this:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">Title</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">dev</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">devops</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">rust</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">streaming\nDevOps for (Skeptical) Developers</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">true</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">true</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">false</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">false\nRust with DevOps</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">false</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">true</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">true</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">false\nServerless Rust using WASM and Cloudflare</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">false</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">true</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">true</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">false\nStreaming UTF-8 in Haskell and Rust</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">false</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">false</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">true</span><span style=\"color:#859900;\">,</span><span style=\"color:#268bd2;\">true\n</span></code></pre>\n<p>To make this happen, we need to:</p>\n<ul>\n<li>Iterate through the files in the <code>posts</code> directory</li>\n<li>Load and parse each TOML file</li>\n<li>Collect a set of all tags present in all posts</li>\n<li>Collect the parsed post information</li>\n<li>Create a CSV file from that information</li>\n</ul>\n<p>Not too bad, right?</p>\n<h2 id=\"setup\">Setup</h2>\n<p>You should make sure you've <a href=\"https://www.rust-lang.org/tools/install\">installed the Rust tools</a>. Then you can create a new empty project with <code>cargo new tagcsv</code>.</p>\n<p>Later on, we're going to play with some unstable language features, so let's opt into a nightly version of the compiler. To do this, create a <code>rust-toolchain</code> file containing:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">nightly-2020-08-29\n</span></code></pre>\n<p>Then add the following dependencies to your <code>Cargo.toml</code> file:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">[</span><span style=\"color:#b58900;\">dependencies</span><span style=\"color:#657b83;\">]\n</span><span style=\"color:#268bd2;\">csv </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">1.1.3</span><span style=\"color:#839496;\">&quot;\n</span><span style=\"color:#268bd2;\">serde </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">1.0.115</span><span style=\"color:#839496;\">&quot;\n</span><span style=\"color:#268bd2;\">serde_derive </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">1.0.115</span><span style=\"color:#839496;\">&quot;\n</span><span style=\"color:#268bd2;\">toml </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">0.5.6</span><span style=\"color:#839496;\">&quot;\n</span></code></pre>\n<p>OK, now we can finally work on some code!</p>\n<h2 id=\"first-version\">First version</h2>\n<p>We're going to use the <code>toml</code> crate to parse our metadata files. <code>toml</code> is built on top of <code>serde</code>, and we can conveniently use <code>serde_derive</code> to automatically derive a <code>Deserialize</code> implementation for a <code>struct</code> that represents that metadata. So we'll start off our program with:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">serde_derive::Deserialize;\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::collections::HashSet;\n\n#[</span><span style=\"color:#268bd2;\">derive</span><span style=\"color:#657b83;\">(Deserialize)]\n</span><span style=\"color:#268bd2;\">struct </span><span style=\"color:#b58900;\">Post </span><span style=\"color:#657b83;\">{\n    </span><span style=\"color:#268bd2;\">title</span><span style=\"color:#657b83;\">: String,\n    </span><span style=\"color:#268bd2;\">tags</span><span style=\"color:#657b83;\">: HashSet&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt;,\n}\n</span></code></pre>\n<p>Next, we'll define our <code>main</code> function to load the data:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), std::io::Error&gt; {\n    </span><span style=\"color:#93a1a1;\">// Collect all tags across all of the posts\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> all_tags: HashSet&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt; = HashSet::new();\n    </span><span style=\"color:#93a1a1;\">// And collect the individual posts\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> posts: </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">&lt;Post&gt; = </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">::new();\n\n    </span><span style=\"color:#93a1a1;\">// Read in the files in the posts directory\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> dir = std::fs::read_dir(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">posts</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> entry </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> dir {\n        </span><span style=\"color:#93a1a1;\">// Error handling\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> entry = entry</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n        </span><span style=\"color:#93a1a1;\">// Read the file contents as a String\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> contents = std::fs::read_to_string(entry.</span><span style=\"color:#859900;\">path</span><span style=\"color:#657b83;\">())</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n        </span><span style=\"color:#93a1a1;\">// Parse the contents with the toml crate\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> post: Post = toml::from_str(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">contents)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n        </span><span style=\"color:#93a1a1;\">// Add all of the tags to the all_tags set\n        </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> tag </span><span style=\"color:#859900;\">in &amp;</span><span style=\"color:#657b83;\">post.tags {\n            all_tags.</span><span style=\"color:#859900;\">insert</span><span style=\"color:#657b83;\">(tag.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">());\n        }\n        </span><span style=\"color:#93a1a1;\">// Update the Vec of posts\n</span><span style=\"color:#657b83;\">        posts.</span><span style=\"color:#859900;\">push</span><span style=\"color:#657b83;\">(post);\n    }\n    </span><span style=\"color:#93a1a1;\">// Generate the CSV output\n    </span><span style=\"color:#859900;\">gen_csv</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">all_tags, </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">posts)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())\n}\n</span></code></pre>\n<p>And finally, let's define our <code>gen_csv</code> function to take the set of tags and the <code>Vec</code> of posts and generate the output file:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">gen_csv</span><span style=\"color:#657b83;\">(</span><span style=\"color:#268bd2;\">all_tags</span><span style=\"color:#657b83;\">: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">HashSet&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt;, </span><span style=\"color:#268bd2;\">posts</span><span style=\"color:#657b83;\">: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">[Post]) -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), std::io::Error&gt; {\n    </span><span style=\"color:#93a1a1;\">// Open the file for output\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> writer = csv::Writer::from_path(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">tag-matrix.csv</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n\n    </span><span style=\"color:#93a1a1;\">// Generate the header, with the word &quot;Title&quot; and then all of the tags\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> header = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Title</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">];\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> tag </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> all_tags.</span><span style=\"color:#859900;\">iter</span><span style=\"color:#657b83;\">() {\n        header.</span><span style=\"color:#859900;\">push</span><span style=\"color:#657b83;\">(tag);\n    }\n    writer.</span><span style=\"color:#859900;\">write_record</span><span style=\"color:#657b83;\">(header)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n\n    </span><span style=\"color:#93a1a1;\">// Print out a separate row for each post\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> post </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> posts {\n        </span><span style=\"color:#93a1a1;\">// Create a record with the post title...\n        </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> record = </span><span style=\"color:#859900;\">vec!</span><span style=\"color:#657b83;\">[post.title.</span><span style=\"color:#859900;\">as_str</span><span style=\"color:#657b83;\">()];\n        </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> tag </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> all_tags {\n            </span><span style=\"color:#93a1a1;\">// and then a true or false for each tag name\n            </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> field = </span><span style=\"color:#859900;\">if</span><span style=\"color:#657b83;\"> post.tags.</span><span style=\"color:#859900;\">contains</span><span style=\"color:#657b83;\">(tag) {\n                </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">true</span><span style=\"color:#839496;\">&quot;\n            </span><span style=\"color:#657b83;\">} </span><span style=\"color:#859900;\">else </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">false</span><span style=\"color:#839496;\">&quot;\n            </span><span style=\"color:#657b83;\">};\n            record.</span><span style=\"color:#859900;\">push</span><span style=\"color:#657b83;\">(field);\n        }\n        writer.</span><span style=\"color:#859900;\">write_record</span><span style=\"color:#657b83;\">(record)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    }\n    writer.</span><span style=\"color:#859900;\">flush</span><span style=\"color:#657b83;\">()</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())\n}\n</span></code></pre>\n<p>Side note: it would be slightly nicer to alphabetize the set of tags, which you can do by collecting all of the tags into a <code>Vec</code> and then sorting it. I had that previously, but removed it in the code above to reduce incidental noise to the example. If you feel like having fun, try adding that back.</p>\n<p>Anyway, this program works exactly as we want, and produces a CSV file. Perfect, right?</p>\n<h2 id=\"let-the-types-guide-you\">Let the types guide you</h2>\n<p>I love type-driven programming. I love the idea that looking at the types tells you a lot about the behavior of your program. And in Rust, the types can often tell you about the <em>memory usage</em> of your program. I want to focus on two lines, and then prove a point with a third. Consider:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">tags: HashSet&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt;,\n</span></code></pre>\n<p>and</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> all_tags: HashSet&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt; = HashSet::new();\n</span></code></pre>\n<p>Firstly, I love the fact that the types tell us so much about expected behavior. The tags are a <em>set</em>: the order is unimportant, and there are no duplicates. That makes sense. We don't want to list &quot;devops&quot; twice in our set of all tags. And there's nothing inherently &quot;first&quot; or &quot;second&quot; about &quot;dev&quot; vs &quot;rust&quot;. And we know that tags are arbitrary pieces of textual data. Awesome.</p>\n<p>But what I <em>really</em> like here is that it tells us about memory usage. Each post has its own copy of each tag. So does the <code>all_tags</code> set. How do I know this? Easy: because that's exactly what <code>String</code> means. There's no possibility of data sharing, at all. If we have 200 posts tagged &quot;dev&quot;, we will have 201 copies of the string &quot;dev&quot; in memory (200 for the posts, once for the <code>all_tags</code>).</p>\n<p>And now that we've seen it in the types, we can see evidence of it in the implementation too:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">all_tags.</span><span style=\"color:#859900;\">insert</span><span style=\"color:#657b83;\">(tag.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">());\n</span></code></pre>\n<p>That <code>.clone()</code> bothered me when I first wrote it. And that's what got me to look at the types, which bothered me further.</p>\n<p>In reality, this is nothing to worry about. Even with 1,000 posts, averaging 5 tags, with each tag averaging 20 bytes, this will only take up an extra 100,000 bytes of memory. So optimizing this away is not a good use of our time. We're much better off doing something else.</p>\n<p>But I wanted to have fun. And if you're reading this post, I think you want to continue this journey too. Onwards!</p>\n<h2 id=\"rc\">Rc</h2>\n<p>This isn't the first solution I tried. But it's the first one that worked easily. So we'll start here.</p>\n<p>The first thing we have to change is our types. As long as we have <code>HashSet&lt;String&gt;</code>, we know for a fact that we'll have extra copies of the data. This seems like a nice use case for <code>Rc</code>. <code>Rc</code> uses reference counting to let multiple values share ownership of another value. Sounds like exactly what we want!</p>\n<p>My approach here is to use compiler-error-driven development, and I encourage you to play along with your own copy of the code. First, let's <code>use</code> <code>Rc</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::rc::Rc;\n</span></code></pre>\n<p>Next, let's change our definition of <code>Post</code> to use an <code>Rc&lt;String&gt;</code> instead of <code>String</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">#[</span><span style=\"color:#268bd2;\">derive</span><span style=\"color:#657b83;\">(Deserialize)]\n</span><span style=\"color:#268bd2;\">struct </span><span style=\"color:#b58900;\">Post </span><span style=\"color:#657b83;\">{\n    </span><span style=\"color:#268bd2;\">title</span><span style=\"color:#657b83;\">: String,\n    </span><span style=\"color:#268bd2;\">tags</span><span style=\"color:#657b83;\">: HashSet&lt;Rc&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt;&gt;,\n}\n</span></code></pre>\n<p>The compiler doesn't like this very much. We can't derive <code>Deserialize</code> for an <code>Rc&lt;String&gt;</code>. So instead, let's make a <code>RawPost</code> struct for the deserializing, and then dedicate <code>Post</code> for holding the data with <code>Rc&lt;String&gt;</code>. In other words:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">#[</span><span style=\"color:#268bd2;\">derive</span><span style=\"color:#657b83;\">(Deserialize)]\n</span><span style=\"color:#268bd2;\">struct </span><span style=\"color:#b58900;\">RawPost </span><span style=\"color:#657b83;\">{\n    </span><span style=\"color:#268bd2;\">title</span><span style=\"color:#657b83;\">: String,\n    </span><span style=\"color:#268bd2;\">tags</span><span style=\"color:#657b83;\">: HashSet&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt;,\n}\n\n</span><span style=\"color:#268bd2;\">struct </span><span style=\"color:#b58900;\">Post </span><span style=\"color:#657b83;\">{\n    </span><span style=\"color:#268bd2;\">title</span><span style=\"color:#657b83;\">: String,\n    </span><span style=\"color:#268bd2;\">tags</span><span style=\"color:#657b83;\">: HashSet&lt;Rc&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt;&gt;,\n}\n</span></code></pre>\n<p>And then, when parsing the <code>toml</code>, we'll parse into a <code>RawPost</code> type:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> post: RawPost = toml::from_str(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">contents)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n</span></code></pre>\n<p>If you're following along, you'll only have one error message at this point about <code>posts.push(post);</code> having a mismatch between <code>Post</code> and <code>RawPost</code>. But before we address that, let's make one more type change above. I want to make <code>all_tags</code> contain <code>Rc&lt;String&gt;</code>.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> all_tags: HashSet&lt;Rc&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt;&gt; = HashSet::new();\n</span></code></pre>\n<p>OK, now we've got some nice error messages about mismatches between <code>Rc&lt;String&gt;</code> and <code>String</code>. This is where we have to be careful. The easiest thing to do would be to simply wrap our <code>String</code>s in an <code>Rc</code> and end up with lots of copies of <code>String</code>. Let's implement the next bit incorrectly first to see what I'm talking about.</p>\n<p>At this point in our code rewrite, we've got a <code>RawPost</code>, and we need to:</p>\n<ul>\n<li>Add its tags to <code>all_tags</code></li>\n<li>Create a new <code>Post</code> value based on the <code>RawPost</code></li>\n<li>Add the <code>Post</code> to the <code>posts</code> <code>Vec</code></li>\n</ul>\n<p>Here's the simple and wasteful implementation:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> raw_post: RawPost = toml::from_str(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">contents)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n\n</span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> post_tags: HashSet&lt;Rc&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt;&gt; = HashSet::new();\n\n</span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> tag </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> raw_post.tags {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> tag = Rc::new(tag);\n    all_tags.</span><span style=\"color:#859900;\">insert</span><span style=\"color:#657b83;\">(tag.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">());\n    post_tags.</span><span style=\"color:#859900;\">insert</span><span style=\"color:#657b83;\">(tag);\n}\n\n</span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> post = Post {\n    title: raw_post.title,\n    tags: post_tags,\n};\nposts.</span><span style=\"color:#859900;\">push</span><span style=\"color:#657b83;\">(post);\n</span></code></pre>\n<p>The problem here is that we always keep the original <code>String</code> from the <code>RawPost</code>. If that tag is already present in the <code>all_tags</code> set, we don't end up using the same copy.</p>\n<p>There's an unstable method on <code>HashSet</code>s that helps us out here. <code>get_or_insert</code> will try to insert a value into a <code>HashSet</code>. If the value is already present, it will drop the new value and return a reference to the original value. If the value isn't present, the value is added to the <code>HashSet</code> and we get a reference back to it. Changing our code to use that is pretty easy:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> tag </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> raw_post.tags {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> tag = Rc::new(tag);\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> tag = all_tags.</span><span style=\"color:#859900;\">get_or_insert</span><span style=\"color:#657b83;\">(tag);\n    post_tags.</span><span style=\"color:#859900;\">insert</span><span style=\"color:#657b83;\">(tag.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">());\n}\n</span></code></pre>\n<p>We still end up with a <code>.clone()</code> call, but now it's a clone of an <code>Rc</code>, which is a cheap integer increment. No additional memory allocation required! Since this method is unstable, we also have to enable the feature by adding this at the top of your source file:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">#![</span><span style=\"color:#268bd2;\">feature</span><span style=\"color:#657b83;\">(hash_set_entry)]\n</span></code></pre>\n<p>And only one more change required. The signature for <code>gen_csv</code> is expecting a <code>&amp;HashSet&lt;String&gt;</code>. If you change that to <code>&amp;HashSet&lt;Rc&lt;String&gt;&gt;</code>, the code will compile and run correctly. Yay!</p>\n<p>In case you got lost with all of the edits above, here's the current version of <code>main.rs</code>:</p>\n<script src=\"https://gist.github.com/snoyberg/5d6e7a515f9065f80b74909920ed94db.js\"></script>\n<h2 id=\"quibbles\">Quibbles</h2>\n<p>I already told you that the original <code>HashSet&lt;String&gt;</code> version of the code is likely Good Enough™ for most cases. I'll tell you that, if you're really bothered by that overhead, the <code>HashSet&lt;Rc&lt;String&gt;&gt;</code> version if almost certainly the right call. So we should probably just stop here and end the blog post on a nice, safe note.</p>\n<p>But let's be bold and crazy. I don't actually like this version of the code that much, for two reasons:</p>\n<ol>\n<li>The <code>Rc</code> feels dirty here. <code>Rc</code> is great for weird lifetime situations with values. But in our case, we know that the <code>all_tags</code> set, which owns all of the tags, will always outlive the usage of the tags inside the <code>Post</code>s. So reference counting feels like an unnecessary overhead and obscuring the situation.</li>\n<li>As demonstrated before, it's all too easy to mess up with the <code>Rc&lt;String&gt;</code> version. You can accidentally bypass all of the memory saving benefits by using a new <code>String</code> instead of cloning a reference to an existing one.</li>\n</ol>\n<p>What I'd really like to do is to have <code>all_tags</code> be a <code>HashSet&lt;String&gt;</code> and own the tags themselves. And then, inside <code>Post</code>, I'd like to keep references to those tags. Unfortunately, this doesn't quite work. Can you foresee why? If not, don't worry, I didn't see it until the borrow checker told me how wrong I was a few times. Let's experience that joy together. And we'll do it with compiler-driven development again.</p>\n<p>The first thing I'm going to do is remove the <code>use std::rc::Rc;</code> statement. That leads to our first error: <code>Rc</code> isn't in scope for <code>Post</code>. We want to keep a <code>&amp;str</code> in this struct. But we have to be explicit about lifetimes when holding references in structs. So our code ends up as:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">struct </span><span style=\"color:#b58900;\">Post</span><span style=\"color:#657b83;\">&lt;</span><span style=\"color:#586e75;\">&#39;a</span><span style=\"color:#657b83;\">&gt; {\n    </span><span style=\"color:#268bd2;\">title</span><span style=\"color:#657b83;\">: String,\n    </span><span style=\"color:#268bd2;\">tags</span><span style=\"color:#657b83;\">: HashSet&lt;</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#586e75;\">&#39;a </span><span style=\"color:#268bd2;\">str</span><span style=\"color:#657b83;\">&gt;,\n}\n</span></code></pre>\n<p>The next error is about the definition of <code>all_tags</code> in <code>main</code>. That's easy enough: just take out the <code>Rc</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> all_tags: HashSet&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt; = HashSet::new();\n</span></code></pre>\n<p>This is easy! Similarly, <code>post_tags</code> is defined as a <code>HashSet&lt;Rc&lt;String&gt;&gt;</code>. In this case, we want to hold <code>&amp;str</code>s instead, so:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> post_tags: HashSet&lt;</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">str</span><span style=\"color:#657b83;\">&gt; = HashSet::new();\n</span></code></pre>\n<p>We no longer need to use <code>Rc::new</code> in the <code>for</code> loop, or clone the <code>Rc</code>. So our loop simplifies down to:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> tag </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> raw_post.tags {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> tag = all_tags.</span><span style=\"color:#859900;\">get_or_insert</span><span style=\"color:#657b83;\">(tag);\n    post_tags.</span><span style=\"color:#859900;\">insert</span><span style=\"color:#657b83;\">(tag);\n}\n</span></code></pre>\n<p>And (misleadingly), we just have one error message left: the signature for <code>gen_csv</code> still uses a <code>Rc</code>. We'll get rid of that with the new signature:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">gen_csv</span><span style=\"color:#657b83;\">(</span><span style=\"color:#268bd2;\">all_tags</span><span style=\"color:#657b83;\">: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">HashSet&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt;, </span><span style=\"color:#268bd2;\">posts</span><span style=\"color:#657b83;\">: </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">[Post]) -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), std::io::Error&gt; {\n</span></code></pre>\n<p>And we get an (IMO confusing) error message about <code>&amp;str</code> and <code>&amp;String</code> not quite lining up:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">error[E0277]: the trait bound `&amp;str: std::borrow::Borrow&lt;std::string::String&gt;` is not satisfied\n  --&gt; src\\main.rs:67:38\n   |\n67 |             let field = if post.tags.contains(tag) {\n   |                                      ^^^^^^^^ the trait `std::borrow::Borrow&lt;std::string::String&gt;` is not implemented for `&amp;str`\n</span></code></pre>\n<p>But this can be solved by explicitly asking for a <code>&amp;str</code> via the <code>as_str</code> method:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> field = </span><span style=\"color:#859900;\">if</span><span style=\"color:#657b83;\"> post.tags.</span><span style=\"color:#859900;\">contains</span><span style=\"color:#657b83;\">(tag.</span><span style=\"color:#859900;\">as_str</span><span style=\"color:#657b83;\">()) {\n</span></code></pre>\n<p>And you might think we're done. But this is where the &quot;misleading&quot; idea comes into play.</p>\n<h2 id=\"the-borrow-checker-wins\">The borrow checker wins</h2>\n<p>If you've been following along, you should now see an error message on your screen that looks something like:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">error[E0499]: cannot borrow `all_tags` as mutable more than once at a time\n  --&gt; src\\main.rs:35:23\n   |\n35 |             let tag = all_tags.get_or_insert(tag);\n   |                       ^^^^^^^^ mutable borrow starts here in previous iteration of loop\n\nerror[E0502]: cannot borrow `all_tags` as immutable because it is also borrowed as mutable\n  --&gt; src\\main.rs:46:13\n   |\n35 |             let tag = all_tags.get_or_insert(tag);\n   |                       -------- mutable borrow occurs here\n...\n46 |     gen_csv(&amp;all_tags, &amp;posts)?;\n   |             ^^^^^^^^^  ------ mutable borrow later used here\n   |             |\n   |             immutable borrow occurs here\n</span></code></pre>\n<p>I was <a href=\"https://twitter.com/snoyberg/status/1299253999800123392?s=20\">convinced</a> that the borrow checker was being overly cautious here. Why would a mutable borrow of <code>all_tags</code> to insert a tag into the set conflict with an immutable borrow of the tags inside the set? (If you already see my error, feel free to laugh at my naivete.) I could follow why I'd violated borrow check rules. Specifically: you can't have a mutable reference and any other reference live at the same time. But I didn't see how this was actually stopping my code from segfaulting.</p>\n<p>After a bit more thinking, it clicked. I realized that I had an invariant in my head which did not appear anywhere in my types. And therefore, the borrow checker was fully justified in saying my code was unsafe. What I realized is that I had implicitly been assuming that my mutations of the <code>all_tags</code> set would never delete any existing values in the set. I can look at my code and see that that's the case. However, the borrow checker doesn't play those kinds of games. It deals with types and facts. And in fact, my code was not provably correct.</p>\n<p>So now is really time to quit, and accept the <code>Rc</code>s, or even just the <code>String</code>s and wasted memory. We're all done. Please don't keep reading.</p>\n<h2 id=\"time-to-get-unsafe\">Time to get unsafe</h2>\n<p>OK, I lied. We're going to take one last step here. I'm not going to tell you this is a good idea. I'm not going to tell you this code is generally safe. I am going to tell you that it works in my testing, and that I refuse to commit it to the master branch of the project I'm working on.</p>\n<p>We've got two issues:</p>\n<ul>\n<li>We have an unstated invariant that we never delete tags from our <code>all_tags</code> <code>HashSet</code></li>\n<li>We need a mutable reference to the <code>HashSet</code> to insert, and that prevents taking immutable references for our tags</li>\n</ul>\n<p>Let's fix this. We're going to define a new <code>struct</code>, called an <code>AppendSet</code>, which only provides the ability to insert new tags, not delete old ones.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">struct </span><span style=\"color:#b58900;\">AppendSet</span><span style=\"color:#657b83;\">&lt;T&gt; {\n    </span><span style=\"color:#268bd2;\">inner</span><span style=\"color:#657b83;\">: HashSet&lt;T&gt;,\n}\n</span></code></pre>\n<p>We're going to provide three methods:</p>\n<ul>\n<li>A static method <code>new</code>, boring</li>\n<li>A <code>get_or_insert</code> that behaves just like <code>HashSet</code>'s, but only needs an immutable reference, not a mutable one</li>\n<li>An <code>inner</code> method that returns a reference to the internal <code>HashSet</code> so we can reuse its <code>Iterator</code> interface</li>\n</ul>\n<p>The first and last are really easy. <code>get_or_insert</code> is a bit more involved, let's just stub it out for now.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">impl</span><span style=\"color:#657b83;\">&lt;T&gt; </span><span style=\"color:#b58900;\">AppendSet</span><span style=\"color:#657b83;\">&lt;T&gt; {\n    </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">new</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#268bd2;\">Self </span><span style=\"color:#657b83;\">{\n        AppendSet {\n            inner: HashSet::new(),\n        }\n    }\n\n    </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">get_or_insert</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">self</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">t</span><span style=\"color:#657b83;\">: T) -&gt; </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">T\n    </span><span style=\"color:#859900;\">where</span><span style=\"color:#657b83;\">\n        T: Eq + std::hash::Hash,\n    {\n        </span><span style=\"color:#859900;\">unimplemented!</span><span style=\"color:#657b83;\">()\n    }\n\n    </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">inner</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">self</span><span style=\"color:#657b83;\">) -&gt; </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">HashSet&lt;T&gt; {\n        </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#d33682;\">self</span><span style=\"color:#657b83;\">.inner\n    }\n}\n</span></code></pre>\n<p>Next, we'll redefine <code>all_tags</code> as:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> all_tags: AppendSet&lt;</span><span style=\"color:#859900;\">String</span><span style=\"color:#657b83;\">&gt; = AppendSet::new();\n</span></code></pre>\n<p>Note that we no longer have the <code>mut</code> keyword here. We never need to mutate this thing... sort of. We'll interact with it via <code>get_or_insert</code>, which at least claims it doesn't mutate. The only other change we have to make is in the call to <code>gen_csv</code>, where we want to use the <code>inner()</code> method:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">gen_csv</span><span style=\"color:#657b83;\">(all_tags.</span><span style=\"color:#859900;\">inner</span><span style=\"color:#657b83;\">(), </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">posts)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n</span></code></pre>\n<p>And perhaps surprisingly, our code now compiles. There's only one thing left to do: implement that <code>get_or_insert</code> method. And this is where the dirtiness happens.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">get_or_insert</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#268bd2;\">self</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">t</span><span style=\"color:#657b83;\">: T) -&gt; </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">T\n</span><span style=\"color:#859900;\">where</span><span style=\"color:#657b83;\">\n    T: Eq + std::hash::Hash,\n{\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> const_ptr = </span><span style=\"color:#d33682;\">self </span><span style=\"color:#859900;\">as </span><span style=\"color:#268bd2;\">*const Self</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> mut_ptr = const_ptr </span><span style=\"color:#859900;\">as </span><span style=\"color:#586e75;\">*mut </span><span style=\"color:#268bd2;\">Self</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> this = </span><span style=\"color:#586e75;\">unsafe </span><span style=\"color:#657b83;\">{ </span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#586e75;\">mut </span><span style=\"color:#657b83;\">*mut_ptr };\n    this.inner.</span><span style=\"color:#859900;\">get_or_insert</span><span style=\"color:#657b83;\">(t)\n}\n</span></code></pre>\n<p>That's right, <code>unsafe</code> baby!</p>\n<img src=\"/images/blog/live-dangerously.jpg\" style=\"max-width:100%\">\n<p>This code absolutely works. I'm also fairly certain it won't <em>generally</em> work. We are very likely violating invariants of <code>HashSet</code>'s interface. As one simple example, we now have the ability to change the contents of a <code>HashSet</code> while there is an active iterate looping through it. I haven't investigated the internals of <code>HashSet</code>, but I wouldn't be surprised at all to find out this breaks some invariants.</p>\n<p><strong>NOTE</strong> To address one of these concerns: what if we modified the <code>inner</code> method on <code>AppendSet</code> to consume the <code>self</code> and return a <code>HashSet</code>? That would definitely help us avoid accidentally violating invariants. But it also won't compile. The <code>AppendSet</code> itself is immutably borrowed by the <code>Post</code> values, and therefore we cannot move it.</p>\n<p>So does this code work? It seems to. Will <code>AppendSet</code> generally work for similar problems? I have no idea. Will this code continue to work with future versions of the standard library with changes to <code>HashSet</code>'s implementation? I have no idea. In other words: <strong>don't use this code</strong>. But it sure was fun to write.</p>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>That was certainly a fun excursion. It's a bit disappointing to end up at the ideal solution requiring unsafe to work. But the <code>Rc</code> version is a really nice middle ground. And even the &quot;bad&quot; version isn't so bad.</p>\n<p>A theoretically better answer would be to use a data structure specifically designed for this use case. I didn't do any investigation to see if such things existed already. If you have any advice, please let me know!</p>\n<p>Check out <a href=\"https://www.fpcomplete.com/rust/\">other FP Complete Rust information</a>.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/",
        "slug": "avoiding-duplicating-strings-rust",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Avoiding duplicating strings in Rust",
        "description": "Let's go way over the top and try to avoid having duplicate strings in memory in Rust",
        "updated": null,
        "date": "2020-09-14",
        "year": 2020,
        "month": 9,
        "day": 14,
        "taxonomies": {
          "categories": [
            "functional programming"
          ],
          "tags": [
            "rust"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "image": "images/blog/live-dangerously.jpg",
          "blogimage": "/images/blog/functional.jpg"
        },
        "path": "blog/avoiding-duplicating-strings-rust/",
        "components": [
          "blog",
          "avoiding-duplicating-strings-rust"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "program-behavior",
            "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/#program-behavior",
            "title": "Program behavior",
            "children": []
          },
          {
            "level": 2,
            "id": "setup",
            "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/#setup",
            "title": "Setup",
            "children": []
          },
          {
            "level": 2,
            "id": "first-version",
            "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/#first-version",
            "title": "First version",
            "children": []
          },
          {
            "level": 2,
            "id": "let-the-types-guide-you",
            "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/#let-the-types-guide-you",
            "title": "Let the types guide you",
            "children": []
          },
          {
            "level": 2,
            "id": "rc",
            "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/#rc",
            "title": "Rc",
            "children": []
          },
          {
            "level": 2,
            "id": "quibbles",
            "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/#quibbles",
            "title": "Quibbles",
            "children": []
          },
          {
            "level": 2,
            "id": "the-borrow-checker-wins",
            "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/#the-borrow-checker-wins",
            "title": "The borrow checker wins",
            "children": []
          },
          {
            "level": 2,
            "id": "time-to-get-unsafe",
            "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/#time-to-get-unsafe",
            "title": "Time to get unsafe",
            "children": []
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/avoiding-duplicating-strings-rust/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 3191,
        "reading_time": 16,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/rust-for-devops-tooling.md",
        "content": "<p>A beginner's guide to writing your DevOps tools in Rust.</p>\n<h2 id=\"introduction\">Introduction</h2>\n<p>In this blog post we'll cover some basic DevOps use cases for Rust and why \nyou would want to use it.\nAs part of this, we'll also cover a few common libraries you will likely use\nin a Rust-based DevOps tool for AWS.</p>\n<p>If you're already familiar with writing DevOps tools in other languages,\nthis post will explain why you should try Rust.</p>\n<p>We'll cover why Rust is a particularly good choice of language to write your DevOps\ntooling and critical cloud infrastructure software in.\nAnd we'll also walk through a small demo DevOps tool written in Rust. \nThis project will be geared towards helping someone new to the language ecosystem \nget familiar with the Rust project structure.</p>\n<p>If you're brand new to Rust, and are interested in learning the language, you may want to start off with our <a href=\"https://www.fpcomplete.com/rust/crash-course/\">Rust Crash Course eBook</a>.</p>\n<h2 id=\"what-makes-the-rust-language-unique\">What Makes the Rust Language Unique</h2>\n<blockquote>\n<p>Rust is a systems programming language focused on three goals: safety, speed, \nand concurrency. It maintains these goals without having a garbage collector, \nmaking it a useful language for a number of use cases other languages aren’t \ngood at: embedding in other languages, programs with specific space and time \nrequirements, and writing low-level code, like device drivers and operating systems. </p>\n</blockquote>\n<p><em>The Rust Book (first edition)</em></p>\n<p>Rust was initially created by Mozilla and has since gained widespread adoption and\nsupport. As the quote from the Rust book alludes to, it was designed to fill the \nsame space that C++ or C would (in that it doesn’t have a garbage collector or a runtime).\nBut Rust also incorporates zero-cost abstractions and many concepts that you would\nexpect in a higher level language (like Go or Haskell).\nFor that, and many other reasons, Rust's uses have expanded well beyond that\noriginal space as low level safe systems language.</p>\n<p>Rust's ownership system is extremely useful in efforts to write correct and \nresource efficient code. Ownership is one of the killer features of the Rust \nlanguage and helps programmers catch classes of resource errors at compile time \nthat other languages miss or ignore.</p>\n<p>Rust is an extremely performant and efficient language, comparable to the speeds \nyou see with idiomatic everyday C or C++.\nAnd since there isn’t a garbage collector in Rust, it’s a lot easier to get \npredictable deterministic performance.</p>\n<h2 id=\"rust-and-devops\">Rust and DevOps</h2>\n<p>What makes Rust unique also makes it very useful for areas stemming from robots \nto rocketry, but are those qualities relevant for DevOps?\nDo we care if we have efficient executables or fine grained control over \nresources, or is Rust a bit overkill for what we typically need in DevOps?</p>\n<p><em>Yes and no</em></p>\n<p>Rust is clearly useful for situations where performance is crucial and actions \nneed to occur in a deterministic and consistent way. That obviously translates to \nlow-level places where previously C and C++ were the only game in town. \nIn those situations, before Rust, people simply had to accept the inherent risk and \nadditional development costs of working on a large code base in those languages.\nRust now allows us to operate in those areas but without the risk that C and C++\ncan add.</p>\n<p>But with DevOps and infrastructure programming we aren't constrained by those \nrequirements. For DevOps we've been able to choose from languages like Go, Python, \nor Haskell because we're not strictly limited by the use case to languages without \ngarbage collectors. Since we can reach for other languages you might argue \nthat using Rust is a bit overkill, but let's go over a few points to counter this.</p>\n<h3 id=\"why-you-would-want-to-write-your-devops-tools-in-rust\">Why you would want to write your DevOps tools in Rust</h3>\n<ul>\n<li>Small executables relative to other options like Go or Java</li>\n<li>Easy to port across different OS targets</li>\n<li>Efficient with resources (which helps cut down on your AWS bill) </li>\n<li>One of the fastest languages (even when compared to C)</li>\n<li>Zero cost abstractions - Rust is a low level performant language which also\ngives the us benefits of a high level language with its generics and abstractions.</li>\n</ul>\n<p>To elaborate on some of these points a bit further:</p>\n<h4 id=\"os-targets-and-cross-compiling-rust-for-different-architectures\">OS targets and Cross Compiling Rust for different architectures</h4>\n<p>For DevOps it's also worth mentioning the (relative) ease with which you can \nport your Rust code across different architectures and different OS's. </p>\n<p>Using the official Rust toolchain installer <code>rustup</code>, it's easy to get the \nstandard library for your target platform.\nRust <a href=\"https://doc.rust-lang.org/nightly/rustc/platform-support.html\">supports a great number of platforms</a>\nwith different tiers of support.\nThe docs for the <code>rustup</code> tool has <a href=\"https://rust-lang.github.io/rustup/cross-compilation.html\">a section</a>\ncovering how you can access pre-compiled artifacts for various architectures.\nTo install the target platform for an architecture (other than the host platform which is installed by default)\nyou simply need to run <code>rustup target add</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">$ rustup target add x86_64-pc-windows-msvc \ninfo: downloading component &#39;rust-std&#39; for &#39;x86_64-pc-windows-msvc&#39;\ninfo: installing component &#39;rust-std&#39; for &#39;x86_64-pc-windows-msvc&#39;\n</span></code></pre>\n<p>Cross compilation is already built into the Rust compiler by default. \nOnce the <code>x86_64-pc-windows-msvc</code> target is installed you can build for Windows \nwith the <code>cargo</code> build tool using the <code>--target</code> flag:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">cargo build --target=x86_64-pc-windows-msvc\n</span></code></pre>\n<p>(the default target is always the host architecture)</p>\n<p>If one of your dependencies links to a native (i.e. non-Rust) library, you will\nneed to make sure that those cross compile as well. Doing <code>rustup target add</code>\nonly installs the Rust standard library for that target. However for the other \ntools that are often needed when cross-compiling, there is the handy\n<a href=\"https://github.com/rust-embedded/cross\">github.com/rust-embedded/cross</a> tool.\nThis is essentially a wrapper around cargo which does all cross compilation in \ndocker images that have all the necessary bits (linkers) and pieces installed.</p>\n<h4 id=\"small-executables\">Small Executables</h4>\n<p>A key unique feature of Rust is that it doesn't need a runtime or a garbage collector.\nCompare this to languages like Python or Haskell: with Rust the lack of any runtime\ndependencies (Python), or system libraries (as with Haskell) is a huge advantage \nfor portability.</p>\n<p>For practical purposes, as far as DevOps is concerned, this portability means \nthat Rust executables are much easier to deploy than scripts.\nWith Rust, compared to Python or Bash, we don't need to set up the environment for \nour code ahead of time. This frees us up from having to worry if the runtime \ndependencies for the language are set up.</p>\n<p>In addition to that, with Rust you're able to produce 100% static executables for \nLinux using the MUSL libc (and by default Rust will statically link all Rust code). \nThis means that you can deploy your Rust DevOps tool's binaries across your Linux \nservers without having to worry if the correct <code>libc</code> or other libraries were \ninstalled beforehand.</p>\n<p>Creating static executables for Rust is simple. As we discussed before, when discussing\ndifferent OS targets, it's easy with Rust to switch the target you're building against.\nTo compile static executables for the Linux MUSL target all you need to do is add \nthe <code>musl</code> target with:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">$ rustup target add x86_64-unknown-linux-musl\n</span></code></pre>\n<p>Then you can using this new target to build your Rust project as a fully static \nexecutable with:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">$ cargo build --target x86_64-unknown-linux-musl\n</span></code></pre>\n<p>As a result of not having a runtime or a garbage collector, Rust executables \ncan be extremely small. For example, there is a common DevOps tool called \nCredStash that was originally written in Python but has since been \nported to Go (GCredStash) and now Rust (RuCredStash).</p>\n<p>Comparing the executable sizes of the Rust versus Go implementations of CredStash,\nthe Rust executable is nearly a quarter of the size of the Go variant. </p>\n<table><thead><tr><th>Implementation</th><th>Executable Size</th></tr></thead><tbody>\n<tr><td>Rust CredStash: (RuCredStash Linux amd64)</td><td>3.3 MB</td></tr>\n<tr><td>Go CredStash: (GCredStash Linux amd64 v0.3.5)</td><td>11.7 MB</td></tr>\n</tbody></table>\n<p>Project links:</p>\n<ul>\n<li><a href=\"https://github.com/psibi/rucredstash\">github.com/psibi/rucredstash</a></li>\n<li><a href=\"https://github.com/winebarrel/gcredstash\">github.com/winebarrel/gcredstash</a></li>\n</ul>\n<p>This is by no means a perfect comparison, and 8 MB may not seem like a lot, but\nconsider the advantage automatically of having executables that are a quarter of the \nsize you would typically expect. </p>\n<p>This cuts down on the size your Docker images, AWS AMI's, or Azure VM images need\nto be - and that helps speed up the time it takes to spin up new deployments.</p>\n<p>With a tool of this size, having an executable that is 75% smaller than it \nwould be otherwise is not immediately apparent. On this scale the difference, 8 MB,\nis still quite cheap.\nBut with larger tools (or collections of tools and Rust based software) the benefits\nadd up and the difference begins to be a practical and worthwhile consideration.</p>\n<p>The Rust implementation was also not strictly written with the resulting size of \nthe executable in mind. So if executable size was even more important of a \nfactor other changes could be made - but that's beyond the scope of this post.</p>\n<h4 id=\"rust-is-fast\">Rust is fast</h4>\n<p>Rust is very fast even for common idiomatic everyday Rust code. And not only that\nit's arguably easier to work with than with C and C++ and catch errors in your \ncode.</p>\n<p>For the Fortunes benchmark (which exercises the ORM, \ndatabase connectivity, dynamic-size collections, sorting, server-side templates, \nXSS countermeasures, and character encoding) Rust is second and third, only lagging \nbehind the first place C++ based framework by 4 percent. </p>\n<img src=\"/images/blog/techempower-benchmarks-round-19-fortunes.png\" style=\"max-width:95%\">\n<p>In the benchmark for database access for a single query Rust is first and second:</p>\n<img src=\"/images/blog/techempower-benchmarks-round-19-single-query.png\" style=\"max-width:95%\">\n<p>And in a composite of all the benchmarks Rust based frameworks are second and third place.</p>\n<img src=\"/images/blog/techempower-benchmarks-round-19-composite.png\" style=\"max-width:95%\">\n<p>Of course language and framework benchmarks are not real life, however this is \nstill a fair comparison of the languages as they relate to others (within the context \nand the focus of the benchmark).</p>\n<p>Source: <a href=\"https://www.techempower.com/benchmarks/\">https://www.techempower.com/benchmarks</a></p>\n<h3 id=\"why-would-you-not-want-to-write-your-devops-tools-in-rust\">Why would you not want to write your DevOps tools in Rust?</h3>\n<p>For medium to large projects, it’s important to have a type system and compile \ntime checks like those in Rust versus what you would find in something like Python\nor Bash.\nThe latter languages let you get away with things far more readily. This makes \ndevelopment much &quot;faster&quot; in one sense.</p>\n<p>Certain situations, especially those with involving small project codebases, would \nbenefit more from using an interpreted language. In these cases, being able to quickly \nchange pieces of the code without needing to re-compile and re-deploy the project\noutweighs the benefits (in terms of safety, execution speed, and portability)\nthat languages like Rust bring. </p>\n<p>Working with and iterating on a Rust codebase in those circumstances, with frequent\nbut small codebases changes, would be needlessly time-consuming\nIf you have a small codebase with few or no runtime dependencies, then it wouldn't\nbe worth it to use Rust.</p>\n<h2 id=\"demo-devops-project-for-aws\">Demo DevOps Project for AWS</h2>\n<p>We'll briefly cover some of the libraries typically used for an AWS focused \nDevOps tool in a walk-through of a small demo Rust project here. \nThis aims to provide a small example that uses some of the libraries you'll likely\nwant if you’re writing a CLI based DevOps tool in Rust. Specifically for this \nexample we'll show a tool that does some basic operations against AWS S3 \n(creating new buckets, adding files to buckets, listing the contents of buckets).</p>\n<h3 id=\"project-structure\">Project structure</h3>\n<p>For AWS integration we're going to utilize the <a href=\"https://www.rusoto.org/\">Rusoto</a> library.\nSpecifically for our modest demo Rust DevOps tools we're going to pull in the \n<a href=\"https://docs.rs/rusoto_core/0.45.0/rusoto_core/\">rusoto_core</a> and the \n<a href=\"https://docs.rs/rusoto_s3/0.45.0/rusoto_s3/\">rusoto_s3</a> crates (in Rust a <em>crate</em>\nis akin to a library or package).</p>\n<p>We're also going to use the <a href=\"https://docs.rs/structopt/0.3.16/structopt/\">structopt</a> crate\nfor our CLI options. This is a handy, batteries included CLI library that makes \nit easy to create a CLI interface around a Rust struct. </p>\n<p>The tool operates by matching the CLI option and arguments the user passes in \nwith a <a href=\"https://github.com/fpco/rust-aws-devops/blob/54d6cfa4bb7a9a15c2db52976f2b7057431e0c5e/src/main.rs#L211\"><code>match</code> expression</a>.</p>\n<p>We can then use this to match on that part of the CLI option struct we've defined \nand call the appropriate functions for that option.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">match</span><span style=\"color:#657b83;\"> opt {\n    Opt::Create { bucket: bucket_name } </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n        </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Attempting to create a bucket called: </span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, bucket_name);\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> demo = S3Demo::new(bucket_name);\n        </span><span style=\"color:#859900;\">create_demo_bucket</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">demo);\n    },\n</span></code></pre>\n<p>This matches on the <a href=\"https://github.com/fpco/rust-aws-devops/blob/54d6cfa4bb7a9a15c2db52976f2b7057431e0c5e/src/main.rs#L182\"><code>Create</code></a>\nvariant of the <code>Opt</code> enum. </p>\n<p>We then use <code>S3Demo::new(bucket_name)</code> to create a new <code>S3Client</code> which we can\nuse in the standalone <code>create_demo_bucket</code> function that we've defined \nwhich will create a new S3 bucket.</p>\n<p>The tool is fairly simple with most of the code located in \n<a href=\"https://github.com/fpco/rust-aws-devops/blob/54d6cfa4bb7a9a15c2db52976f2b7057431e0c5e/src/main.rs\">src/main.rs</a></p>\n<h3 id=\"building-the-rust-project\">Building the Rust project</h3>\n<p>Before you build the code in this project, you will need to install Rust. \nPlease follow <a href=\"https://www.rust-lang.org/tools/install\">the official install instructions here</a>.</p>\n<p>The default build tool for Rust is called Cargo. It's worth getting familiar \nwith <a href=\"https://doc.rust-lang.org/cargo/guide/\">the docs for Cargo</a>\nbut here's a quick overview for building the project.</p>\n<p>To build the project run the following from the root of the \n<a href=\"https://github.com/fpco/rust-aws-devops\">git repo</a>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">cargo build\n</span></code></pre>\n<p>You can then use <code>cargo run</code> to run the code or execute the code directly\nwith <code>./target/debug/rust-aws-devops</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">$ ./target/debug/rust-aws-devops \n\nRunning tool\nRustAWSDevops 0.1.0\nMike McGirr &lt;mike@fpcomplete.com&gt;\n\nUSAGE:\n    rust-aws-devops &lt;SUBCOMMAND&gt;\n\nFLAGS:\n    -h, --help       Prints help information\n    -V, --version    Prints version information\n\nSUBCOMMANDS:\n    add-object       Add the specified file to the bucket\n    create           Create a new bucket with the given name\n    delete           Try to delete the bucket with the given name\n    delete-object    Remove the specified object from the bucket\n    help             Prints this message or the help of the given subcommand(s)\n    list             Try to find the bucket with the given name and list its objects``\n</span></code></pre>\n<p>Which will output the nice CLI help output automatically created for us \nby <code>structopt</code>.</p>\n<p>If you're ready to build a release version (with optimizations turn on which \nwill make compilation take slightly longer) run the following:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">cargo build --release\n</span></code></pre><h2 id=\"conclusion\">Conclusion</h2>\n<p>As this small demo showed, it's not difficult to get started using Rust to write\nDevOps tools. And even then we didn't need to make a trade-off between ease of\ndevelopment and performant fast code. </p>\n<p>Hopefully the next time you're writing a new piece of DevOps software, \nanything from a simple CLI tool for a specific DevOps operation or you're writing \nthe next Kubernetes, you'll consider reaching for Rust.\nAnd if you have further questions about Rust, or need help implementing your Rust \nproject, please feel free to reach out to FP Complete for Rust engineering \nand training!</p>\n<p>Want to learn more Rust? Check out our <a href=\"https://www.fpcomplete.com/rust/crash-course/\">Rust Crash Course eBook</a>. And for more information, check out our <a href=\"https://www.fpcomplete.com/rust/\">Rust homepage</a>.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/",
        "slug": "rust-for-devops-tooling",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Using Rust for DevOps tooling",
        "description": "A beginner's guide to writing your DevOps tools in Rust.",
        "updated": null,
        "date": "2020-09-09",
        "year": 2020,
        "month": 9,
        "day": 9,
        "taxonomies": {
          "categories": [
            "functional programming",
            "devops"
          ],
          "tags": [
            "devops",
            "rust",
            "insights"
          ]
        },
        "extra": {
          "author": "Mike McGirr",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/rust-for-devops-tooling/",
        "components": [
          "blog",
          "rust-for-devops-tooling"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "introduction",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#introduction",
            "title": "Introduction",
            "children": []
          },
          {
            "level": 2,
            "id": "what-makes-the-rust-language-unique",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#what-makes-the-rust-language-unique",
            "title": "What Makes the Rust Language Unique",
            "children": []
          },
          {
            "level": 2,
            "id": "rust-and-devops",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#rust-and-devops",
            "title": "Rust and DevOps",
            "children": [
              {
                "level": 3,
                "id": "why-you-would-want-to-write-your-devops-tools-in-rust",
                "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#why-you-would-want-to-write-your-devops-tools-in-rust",
                "title": "Why you would want to write your DevOps tools in Rust",
                "children": [
                  {
                    "level": 4,
                    "id": "os-targets-and-cross-compiling-rust-for-different-architectures",
                    "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#os-targets-and-cross-compiling-rust-for-different-architectures",
                    "title": "OS targets and Cross Compiling Rust for different architectures",
                    "children": []
                  },
                  {
                    "level": 4,
                    "id": "small-executables",
                    "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#small-executables",
                    "title": "Small Executables",
                    "children": []
                  },
                  {
                    "level": 4,
                    "id": "rust-is-fast",
                    "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#rust-is-fast",
                    "title": "Rust is fast",
                    "children": []
                  }
                ]
              },
              {
                "level": 3,
                "id": "why-would-you-not-want-to-write-your-devops-tools-in-rust",
                "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#why-would-you-not-want-to-write-your-devops-tools-in-rust",
                "title": "Why would you not want to write your DevOps tools in Rust?",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "demo-devops-project-for-aws",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#demo-devops-project-for-aws",
            "title": "Demo DevOps Project for AWS",
            "children": [
              {
                "level": 3,
                "id": "project-structure",
                "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#project-structure",
                "title": "Project structure",
                "children": []
              },
              {
                "level": 3,
                "id": "building-the-rust-project",
                "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#building-the-rust-project",
                "title": "Building the Rust project",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/rust-for-devops-tooling/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 2540,
        "reading_time": 13,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/http-status-codes-async-rust.md",
        "content": "<p>This blog post is a direct follow up on my previous blog post on <a href=\"https://www.fpcomplete.com/blog/different-levels-async-rust/\">different levels of async in Rust</a>. You may want to check that one out before diving in here.</p>\n<p>Alright, so now we know that we can make our programs asynchronous by using non-blocking I/O calls. But last time we only saw examples that remained completely sequential, defeating the whole purpose of async. Let's change that with something more sophisticated.</p>\n<p>A few months ago I needed to ensure that all of the URLs for a domain name resolved to either a real web page (200 status code) or redirected to somewhere else with a real web page. To make that happen, I needed a program that would:</p>\n<ul>\n<li>Read all of the URLs in a text file, one URL per line</li>\n<li>Produce a CSV file containing the URL and its status code</li>\n</ul>\n<p>To make this simple, we're going to take a lot of shortcuts like:</p>\n<ul>\n<li>Hard-coding the input file path for the URLs</li>\n<li>Printing out the CSV output to standard output</li>\n<li>Using a simple <code>println!</code> for generating CSV output instead of using a library</li>\n<li>Allow any errors to crash the entire program\n<ul>\n<li>In fact, as you'll see later, we're really treating this as a requirement: if any HTTP requests have an error, the program <em>must</em> terminate with an error code so we know something went wrong</li>\n</ul>\n</li>\n</ul>\n<p>For the curious: the original version of this was a <a href=\"https://twitter.com/snoyberg/status/1265526242486468616\">really short Haskell program</a> that had these properties. For fun a few weeks back, I rewrote it in <a href=\"https://twitter.com/snoyberg/status/1296716412262780928?s=20\">two</a> <a href=\"https://twitter.com/snoyberg/status/1296718361766887424?s=20\">ways</a> in Rust, which ultimately led to this pair of blog posts.</p>\n<h2 id=\"fully-blocking\">Fully blocking</h2>\n<p>Like last time, I recommend following along with my code. I'll kick this off with <code>cargo new httpstatus</code>. And then to avoid further futzing with our <code>Cargo.toml</code>, let's add our dependencies preemptively:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">[</span><span style=\"color:#b58900;\">dependencies</span><span style=\"color:#657b83;\">]\n</span><span style=\"color:#268bd2;\">tokio </span><span style=\"color:#657b83;\">= { </span><span style=\"color:#268bd2;\">version </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">0.2.22</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">features </span><span style=\"color:#657b83;\">= [</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">full</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">] }\n</span><span style=\"color:#268bd2;\">reqwest </span><span style=\"color:#657b83;\">= { </span><span style=\"color:#268bd2;\">version </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">0.10.8</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">features </span><span style=\"color:#657b83;\">= [</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">blocking</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">] }\n</span><span style=\"color:#268bd2;\">async-channel </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">1.4.1</span><span style=\"color:#839496;\">&quot;\n</span><span style=\"color:#268bd2;\">is_type </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">0.2.1</span><span style=\"color:#839496;\">&quot;\n</span></code></pre>\n<p>That <code>features = [&quot;blocking&quot;]</code> should hopefully grab your attention. The <code>reqwest</code> library provides an optional, fully blocking API. That seems like a great place to get started. Here's a nice, simple program that does what we need:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#93a1a1;\">// To use .lines() before, just like last time\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::io::BufRead;\n\n</span><span style=\"color:#93a1a1;\">// We&#39;ll return _some_ kind of an error\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), </span><span style=\"color:#859900;\">Box</span><span style=\"color:#657b83;\">&lt;dyn std::error::Error&gt;&gt; {\n    </span><span style=\"color:#93a1a1;\">// Open the file for input\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = std::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">urls.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#93a1a1;\">// Make a buffered version so we can read lines\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> buffile = std::io::BufReader::new(file);\n\n    </span><span style=\"color:#93a1a1;\">// CSV header\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">URL,Status</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#93a1a1;\">// Create a client so we can make requests\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = reqwest::blocking::Client::new();\n\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> line </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> buffile.</span><span style=\"color:#859900;\">lines</span><span style=\"color:#657b83;\">() {\n        </span><span style=\"color:#93a1a1;\">// Error handling on reading the lines in the file\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> line = line</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n        </span><span style=\"color:#93a1a1;\">// Make a request and send it, getting a response\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> resp = client.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">line).</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">()</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n        </span><span style=\"color:#93a1a1;\">// Print the status code\n        </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\">,</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line, resp.</span><span style=\"color:#859900;\">status</span><span style=\"color:#657b83;\">().</span><span style=\"color:#859900;\">as_u16</span><span style=\"color:#657b83;\">());\n    }\n    </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())\n}\n</span></code></pre>\n<p>Thanks to Rust's <code>?</code> syntax, error handling is pretty easy here. In fact, there are basically no gotchas here. <code>reqwest</code> makes this code really easy to write!</p>\n<p>Once you put a <code>urls.txt</code> file together, such as the following:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">https://www.wikipedia.org\nhttps://www.wikipedia.org/path-the-does-not-exist\nhttp://wikipedia.org\n</span></code></pre>\n<p>You'll hopefully get output such as:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">URL,Status\nhttps://www.wikipedia.org,200\nhttps://www.wikipedia.org/path-the-does-not-exist,404\nhttp://wikipedia.org,200\n</span></code></pre>\n<p>The logic above is pretty easy to follow, and hopefully the inline comments explain anything confusing. With that idea in mind, let's up our game a bit.</p>\n<h2 id=\"ditching-the-blocking-api\">Ditching the blocking API</h2>\n<p>Let's first move away from the blocking API in <code>reqwest</code>, but still keep all of the sequential nature of the program. This involves four relatively minor changes to the code, all spelled out below:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::io::BufRead;\n\n</span><span style=\"color:#93a1a1;\">// First change: add the Tokio runtime\n</span><span style=\"color:#657b83;\">#[</span><span style=\"color:#268bd2;\">tokio</span><span style=\"color:#657b83;\">::</span><span style=\"color:#268bd2;\">main</span><span style=\"color:#657b83;\">]\n</span><span style=\"color:#93a1a1;\">// Second: turn this into an async function\n</span><span style=\"color:#657b83;\">async </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), </span><span style=\"color:#859900;\">Box</span><span style=\"color:#657b83;\">&lt;dyn std::error::Error&gt;&gt; {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = std::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">urls.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> buffile = std::io::BufReader::new(file);\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">URL,Status</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#93a1a1;\">// Third change: Now we make an async Client\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = reqwest::Client::new();\n\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> line </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> buffile.</span><span style=\"color:#859900;\">lines</span><span style=\"color:#657b83;\">() {\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> line = line</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n\n        </span><span style=\"color:#93a1a1;\">// Fourth change: We need to .await after send()\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> resp = client.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">line).</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">().await</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n\n        </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\">,</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line, resp.</span><span style=\"color:#859900;\">status</span><span style=\"color:#657b83;\">().</span><span style=\"color:#859900;\">as_u16</span><span style=\"color:#657b83;\">());\n    }\n    </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())\n}\n</span></code></pre>\n<p>The program is still fully sequential: we fully send a request, then get the response, before we move onto the next URL. But we're at least ready to start playing with different async approaches.</p>\n<h2 id=\"where-blocking-is-fine\">Where blocking is fine</h2>\n<p>IF you remember from last time, we had a bit of a philosophical discussion on the nature of blocking, and that ultimately some blocking is OK in a program. In order to both simplify what we do here, as well as provide some real-world recommendations, let's list all of the blocking I/O we're doing:</p>\n<ul>\n<li>Opening the file <code>urls.txt</code></li>\n<li>Reading lines from that file</li>\n<li>Outputting to <code>stdout</code> with <code>println!</code></li>\n<li>Implicitly closing the file descriptor</li>\n</ul>\n<p>Note that, even though we're sequentially running our HTTP requests right now, those are in fact using non-blocking I/O. Therefore, I haven't included anything related to HTTP in the list above. We'll start dealing with the sequential nature next.</p>\n<p>Returning to the four blocking I/O calls above, I'm going to make a bold statement: don't bother making them non-blocking. It's not actually terribly difficult to do the file I/O using <code>tokio</code> (we saw how last time). But we get virtually no benefit from doing so. The latency for local disk access, especially when we're talking a file as small as <code>urls.txt</code> is likely to be, and especially in contrast to a bunch of HTTP requests, is miniscule.</p>\n<p>Feel free to disagree with me, or to take on making those calls non-blocking as an exercise. But I'm going to focus instead on higher value targets.</p>\n<h2 id=\"concurrent-requests\">Concurrent requests</h2>\n<p>The real problem here is that we have sequential HTTP requests going on. Instead, we would much prefer to make our requests concurrently. If we assume there are 100 URLs, and each request takes 1 second (hopefully an overestimation), a sequential algorithm can at best finish in 100 seconds. However, a concurrent algorithm could in theory finish all 100 requests in just 1 second. In reality that's pretty unlikely to happen, but it is completely reasonable to expect a significant speedup factor, depending on network conditions, number of hosts you're connecting to, and other similar factors.</p>\n<p>So how exactly do we do concurrency with <code>tokio</code>? The most basic answer is the <code>tokio::spawn</code> function. This spawns a new <em>task</em> in the <code>tokio</code> runtime. This is similar in principle to spawning a new system thread. But instead, running and scheduling is managed by the runtime instead of the operating system. Let's take a first stab at spawning each HTTP request into its own task:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">tokio::spawn(async </span><span style=\"color:#586e75;\">move </span><span style=\"color:#657b83;\">{\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> resp = client.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">line).</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">().await</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\">,</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line, resp.</span><span style=\"color:#859900;\">status</span><span style=\"color:#657b83;\">().</span><span style=\"color:#859900;\">as_u16</span><span style=\"color:#657b83;\">());\n});\n</span></code></pre>\n<p>That looks nice, but we have a problem:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">error[E0277]: the `?` operator can only be used in an async block that returns `Result` or `Option` (or another type that implements `std::ops::Try`)\n  --&gt; src\\main.rs:16:24\n   |\n15 |           tokio::spawn(async move {\n   |  _________________________________-\n16 | |             let resp = client.get(&amp;line).send().await?;\n   | |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot use the `?` operator in an async block that returns `()`\n17 | |\n18 | |             println!(&quot;{},{}&quot;, line, resp.status().as_u16());\n19 | |         });\n   | |_________- this function should return `Result` or `Option` to accept `?`\n</span></code></pre>\n<p>Our task doesn't return a <code>Result</code>, and therefore has no way to complain about errors. This is actually indicating a far more serious issue, which we'll get to later. But for now, let's just pretend errors won't happen, and cheat a bit with <code>.unwrap()</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> resp = client.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">line).</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">().await.</span><span style=\"color:#859900;\">unwrap</span><span style=\"color:#657b83;\">();\n</span></code></pre>\n<p>This also fails, now with an ownership issue:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">error[E0382]: use of moved value: `client`\n  --&gt; src\\main.rs:15:33\n   |\n10 |       let client = reqwest::Client::new();\n   |           ------ move occurs because `client` has type `reqwest::async_impl::client::Client`, which does not implement the `Copy` trait\n</span></code></pre>\n<p>This one is easier to address. The <code>Client</code> is being shared by multiple tasks. But each task needs to make its own clone of the <code>Client</code>. If you read <a href=\"https://docs.rs/reqwest/0.10.8/reqwest/struct.Client.html\">the docs</a>, you'll see that this is recommended behavior:</p>\n<blockquote>\n<p>The <code>Client</code> holds a connection pool internally, so it is advised that you create one and <strong>reuse</strong> it.</p>\n<p>You do <strong>not</strong> have to wrap the <code>Client</code> it in an <code>Rc</code> or <code>Arc</code> to <strong>reuse</strong> it, because it already uses an <code>Arc</code> internally.</p>\n</blockquote>\n<p>Once we add this line before our <code>tokio::spawn</code>, our code will compile:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = client.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">();\n</span></code></pre>\n<p>Unfortunately, things fail pretty spectacularly at runtime:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">URL,Status\nthread &#39;thread &#39;tokio-runtime-workerthread &#39;tokio-runtime-worker&#39; panicked at &#39;&#39; panicked at &#39;tokio-runtime-workercalled `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: &quot;https://www.wikipedia.org/path-the-does-not-exist&quot;, source: hyper::Error(Connect, ConnectError(&quot;dns error&quot;, Custom { kind: Interrupted, error: JoinError::Cancelled })) }called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: &quot;https://www.wikipedia.org/&quot;, source: hyper::Error(Connect, ConnectError(&quot;dns error&quot;, Custom { kind: Interrupted, error: JoinError::Cancelled })) }&#39; panicked at &#39;&#39;, &#39;, called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: &quot;http://wikipedia.org/&quot;, source: hyper::Error(Connect, ConnectError(&quot;dns error&quot;, Custom { kind: Interrupted, error: JoinError::Cancelled })) }src\\main.rssrc\\main.rs&#39;, ::src\\main.rs1717:::241724\n</span></code></pre>\n<p>That's a big error message, but the important bit for us is a bunch of <code>JoinError::Cancelled</code> stuff all over the place.</p>\n<h2 id=\"wait-for-me\">Wait for me!</h2>\n<p>Let's talk through what's happening in our program:</p>\n<ol>\n<li>Initiate the Tokio runtime</li>\n<li>Create a <code>Client</code></li>\n<li>Open the file, start reading line by line</li>\n<li>For each line:\n<ul>\n<li>Spawn a new task</li>\n<li>That task starts making non-blocking I/O calls</li>\n<li>Those tasks go to sleep, to be rescheduled when data is ready</li>\n<li>When all is said and done, print out the CSV lines</li>\n</ul>\n</li>\n<li>Reach the end of the <code>main</code> function, which triggers the runtime to shut down</li>\n</ol>\n<p>The problem is that we reach (5) long before we finish (4). When this happens, all in-flight I/O will be cancelled, which leads to the error messages we saw above. Instead, we need to ensure we wait for each task to complete before we exit. The easiest way to do this is to call <code>.await</code> on the result of the <code>tokio::spawn</code> call. (Those results, by the way, are called <code>JoinHandle</code>s.) However, doing so immediately will completely defeat the purpose of our concurrent work, since we will once again be sequential!</p>\n<p>Instead, we want to spawn all of the tasks, and then wait for them all to complete. One easy way to achieve this is to put all of the <code>JoinHandle</code>s into a <code>Vec</code>. Let's look at the code. And since we've made a bunch of changes since our last complete code dump, I'll show you the full current status of our source file:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::io::BufRead;\n\n#[</span><span style=\"color:#268bd2;\">tokio</span><span style=\"color:#657b83;\">::</span><span style=\"color:#268bd2;\">main</span><span style=\"color:#657b83;\">]\nasync </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), </span><span style=\"color:#859900;\">Box</span><span style=\"color:#657b83;\">&lt;dyn std::error::Error&gt;&gt; {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = std::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">urls.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> buffile = std::io::BufReader::new(file);\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">URL,Status</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = reqwest::Client::new();\n\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> handles = </span><span style=\"color:#859900;\">Vec</span><span style=\"color:#657b83;\">::new();\n\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> line </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> buffile.</span><span style=\"color:#859900;\">lines</span><span style=\"color:#657b83;\">() {\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> line = line</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = client.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">();\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> handle = tokio::spawn(async </span><span style=\"color:#586e75;\">move </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> resp = client.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">line).</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">().await.</span><span style=\"color:#859900;\">unwrap</span><span style=\"color:#657b83;\">();\n\n            </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\">,</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line, resp.</span><span style=\"color:#859900;\">status</span><span style=\"color:#657b83;\">().</span><span style=\"color:#859900;\">as_u16</span><span style=\"color:#657b83;\">());\n        });\n        handles.</span><span style=\"color:#859900;\">push</span><span style=\"color:#657b83;\">(handle);\n    }\n\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> handle </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> handles {\n        handle.await</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    }\n    </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())\n}\n</span></code></pre>\n<p>And finally we have a concurrent program! This is actually pretty good, but it has two flaws we'd like to fix:</p>\n<ol>\n<li>It doesn't properly handle errors, instead just using <code>.unwrap()</code>. I mentioned this above, and said our usage of <code>.unwrap()</code> was indicating a &quot;far more serious issue.&quot; That issue was the fact that the result values from spawning subthreads are never noticed by the main thread, which is really the core issue causing the cancellation we discussed above. It's always nice when type-driven error messages indicate a runtime bug in our code!</li>\n<li>There's no limitation on the number of concurrent tasks we'll spawn. Ideally, we'd rather have a job queue approach, with a dedicated number of worker tasks. This will let our program behave better as we increase the number of URLs in our input file.</li>\n</ol>\n<p><strong>NOTE</strong> It would be possible in the program above to skip the <code>spawn</code>s and collect a <code>Vec</code> of <code>Future</code>s, then <code>await</code> on those. However, that would once again end up sequential in nature. Spawning allows all of those <code>Future</code>s to run concurrently, and be polled by the <code>tokio</code> runtime itself. It would also be possible to use <a href=\"https://docs.rs/futures/0.3.5/futures/future/fn.join_all.html\"><code>join_all</code></a> to poll all of the <code>Future</code>s, but it <a href=\"https://github.com/tokio-rs/tokio/issues/2401#issuecomment-612858572\">has some performance issues</a>. So best to stick with <code>tokio::spawn</code>.</p>\n<p>Let's address the simpler one first: proper error handling.</p>\n<h2 id=\"error-handling\">Error handling</h2>\n<p>The basic concept of error handling is that we want the errors from the spawned tasks to be detected in the main tasks, and then cause the application to exit. One way to handle that is to return the <code>Err</code> values from the spawned tasks directly, and then pick them up with the <code>JoinHandle</code> that <code>spawn</code> returns. This sounds nice, but naively implemented will result in checking the error responses one at a time. Instead, we'd rather fail early, by detecting that (for example) the 57th request failed and immediately terminating the application.</p>\n<p>You <em>could</em> do some kind of a &quot;tell me which is the first <code>JoinHandle</code> that's ready,&quot; but it's not the way I initially implemented it, and some quick Googling indicated <a href=\"https://github.com/tokio-rs/tokio/issues/2401\">you'd have to be careful about which library functions you use</a>. Instead, we'll try a different approach using an <code>mpsc</code> (multi-producer, single-consumer).</p>\n<p>Here's the basic idea. Let's pretend there are 100 URLs in the file. We'll spawn 100 tasks. Each of those tasks will write a single value onto the <code>mpsc</code> channel: a <code>Result&lt;(), Error&gt;</code>. Then, in the <code>main</code> task, we'll read 100 values off of the channel. If any of them are <code>Err</code>, we exit the program immediately. Otherwise, if we read off 100 <code>Ok</code> values, we exit successfully.</p>\n<p>Before we read the file, we don't know how many lines will be in it. So we're going to use an unbounded channel. This isn't generally recommended practice, but it ties in closely with my second complaint above: we're spawning a separate task for each line in the file instead of doing something more intelligent like a job queue. In other words, if we can safely spawn N tasks, we can safely have an unbounded channel of size N.</p>\n<p>Alright, let's see the code in question!</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::io::BufRead;\n\n#[</span><span style=\"color:#268bd2;\">tokio</span><span style=\"color:#657b83;\">::</span><span style=\"color:#268bd2;\">main</span><span style=\"color:#657b83;\">]\nasync </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), </span><span style=\"color:#859900;\">Box</span><span style=\"color:#657b83;\">&lt;dyn std::error::Error&gt;&gt; {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = std::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">urls.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> buffile = std::io::BufReader::new(file);\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">URL,Status</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = reqwest::Client::new();\n\n    </span><span style=\"color:#93a1a1;\">// Create the channel. tx will be the sending side (each spawned task),\n    // and rx will be the receiving side (the main task after spawning).\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#657b83;\">(tx, </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> rx) = tokio::sync::mpsc::unbounded_channel();\n\n    </span><span style=\"color:#93a1a1;\">// Keep track of how many lines are in the file, and therefore\n    // how many tasks we spawned\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> count = </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\">;\n\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> line </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> buffile.</span><span style=\"color:#859900;\">lines</span><span style=\"color:#657b83;\">() {\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> line = line</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = client.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">();\n        </span><span style=\"color:#93a1a1;\">// Each spawned task gets its own copy of tx\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> tx = tx.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">();\n        tokio::spawn(async </span><span style=\"color:#586e75;\">move </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#93a1a1;\">// Use a map to say: if the request went through\n            // successfully, then print it. Otherwise:\n            // keep the error\n            </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> msg = client.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">line).</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">().await.</span><span style=\"color:#859900;\">map</span><span style=\"color:#657b83;\">(|</span><span style=\"color:#268bd2;\">resp</span><span style=\"color:#657b83;\">| {\n                </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\">,</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line, resp.</span><span style=\"color:#859900;\">status</span><span style=\"color:#657b83;\">().</span><span style=\"color:#859900;\">as_u16</span><span style=\"color:#657b83;\">());\n            });\n            </span><span style=\"color:#93a1a1;\">// And send the message to the channel. We ignore errors here.\n            // An error during sending would mean that the receiving side\n            // is already closed, which would indicate either programmer\n            // error, or that our application is shutting down because\n            // another task generated an error.\n</span><span style=\"color:#657b83;\">            tx.</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">(msg).</span><span style=\"color:#859900;\">unwrap</span><span style=\"color:#657b83;\">();\n        });\n\n        </span><span style=\"color:#93a1a1;\">// Increase the count of spawned tasks\n</span><span style=\"color:#657b83;\">        count += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n    }\n\n    </span><span style=\"color:#93a1a1;\">// Drop the sending side, so that we get a None when\n    // calling rx.recv() one final time. This allows us to\n    // test some extra assertions below\n    </span><span style=\"color:#657b83;\">std::mem::drop(tx);\n\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> i = </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#859900;\">loop </span><span style=\"color:#657b83;\">{\n        </span><span style=\"color:#859900;\">match</span><span style=\"color:#657b83;\"> rx.</span><span style=\"color:#859900;\">recv</span><span style=\"color:#657b83;\">().await {\n            </span><span style=\"color:#93a1a1;\">// All senders are gone, which must mean that\n            // we&#39;re at the end of our loop\n            </span><span style=\"color:#859900;\">None =&gt; </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">assert_eq!</span><span style=\"color:#657b83;\">(i, count);\n                </span><span style=\"color:#859900;\">break Ok</span><span style=\"color:#657b83;\">(());\n            }\n            </span><span style=\"color:#93a1a1;\">// Something finished successfully, make sure\n            // that we haven&#39;t reached the final item yet\n            </span><span style=\"color:#859900;\">Some</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())) </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">assert!</span><span style=\"color:#657b83;\">(i &lt; count);\n            }\n            </span><span style=\"color:#93a1a1;\">// Oops, an error! Time to exit!\n            </span><span style=\"color:#859900;\">Some</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(e)) </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">assert!</span><span style=\"color:#657b83;\">(i &lt; count);\n                </span><span style=\"color:#859900;\">return Err</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">From</span><span style=\"color:#657b83;\">::from(e));\n            }\n        }\n        i += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n    }\n}\n</span></code></pre>\n<p>With this in place, we now have a proper concurrent program that does error handling correctly. Nifty! Before we hit the job queue, let's clean this up a bit.</p>\n<h2 id=\"workers\">Workers</h2>\n<p>The previous code works well. It allows us to spawn multiple worker tasks, and then wait for all of them to complete, handling errors when they occur. Let's generalize this! We're doing this now since it will make the final step in this blog post much easier.</p>\n<p>We'll put all of the code for this in a separate module of our project. The code will be mostly the same as what we had before, except we'll have a nice <code>struct</code> to hold onto our data, and we'll be more explicit about the error type. Put this code into <code>src/workers.rs</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">is_type::Is; </span><span style=\"color:#93a1a1;\">// fun trick, we&#39;ll look at it below\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::future::Future;\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">tokio::sync::mpsc;\n\n</span><span style=\"color:#93a1a1;\">/// Spawn and then run workers to completion, handling errors\n</span><span style=\"color:#586e75;\">pub </span><span style=\"color:#268bd2;\">struct </span><span style=\"color:#b58900;\">Workers</span><span style=\"color:#657b83;\">&lt;E&gt; {\n    </span><span style=\"color:#268bd2;\">count</span><span style=\"color:#657b83;\">: </span><span style=\"color:#268bd2;\">usize</span><span style=\"color:#657b83;\">,\n    </span><span style=\"color:#268bd2;\">tx</span><span style=\"color:#657b83;\">: mpsc::UnboundedSender&lt;</span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), E&gt;&gt;,\n    </span><span style=\"color:#268bd2;\">rx</span><span style=\"color:#657b83;\">: mpsc::UnboundedReceiver&lt;</span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), E&gt;&gt;,\n}\n\n</span><span style=\"color:#268bd2;\">impl</span><span style=\"color:#657b83;\">&lt;E: </span><span style=\"color:#859900;\">Send + </span><span style=\"color:#586e75;\">&#39;static</span><span style=\"color:#657b83;\">&gt; </span><span style=\"color:#b58900;\">Workers</span><span style=\"color:#657b83;\">&lt;E&gt; {\n    </span><span style=\"color:#93a1a1;\">/// Create a new Workers value\n    </span><span style=\"color:#586e75;\">pub </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">new</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#268bd2;\">Self </span><span style=\"color:#657b83;\">{\n        </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#657b83;\">(tx, rx) = mpsc::unbounded_channel();\n        Workers { count: </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\">, tx, rx }\n    }\n\n    </span><span style=\"color:#93a1a1;\">/// Spawn a new task to run inside this Workers\n    </span><span style=\"color:#586e75;\">pub </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">spawn</span><span style=\"color:#657b83;\">&lt;T&gt;(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#586e75;\">mut </span><span style=\"color:#268bd2;\">self</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">task</span><span style=\"color:#657b83;\">: T)\n    </span><span style=\"color:#859900;\">where\n        </span><span style=\"color:#93a1a1;\">// Make sure we can run the task\n</span><span style=\"color:#657b83;\">        T: Future + Send + </span><span style=\"color:#586e75;\">&#39;static</span><span style=\"color:#657b83;\">,\n        </span><span style=\"color:#93a1a1;\">// And a weird trick: make sure that the output\n        // from the task is Result&lt;(), E&gt;\n        // Equality constraints would make this much nicer\n        // See: https://github.com/rust-lang/rust/issues/20041\n        </span><span style=\"color:#268bd2;\">T::</span><span style=\"color:#657b83;\">Output: Is&lt;Type = </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), E&gt;&gt;,\n    {\n        </span><span style=\"color:#93a1a1;\">// Get a new copy of the send side\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> tx = </span><span style=\"color:#d33682;\">self</span><span style=\"color:#657b83;\">.tx.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">();\n        </span><span style=\"color:#93a1a1;\">// Spawn a new task\n        </span><span style=\"color:#657b83;\">tokio::spawn(async </span><span style=\"color:#586e75;\">move </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#93a1a1;\">// Run the provided task and get its result\n            </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> res = task.await;\n            </span><span style=\"color:#93a1a1;\">// Send the task to the channel\n            // This should never fail, so we panic if something goes wrong\n            </span><span style=\"color:#859900;\">match</span><span style=\"color:#657b83;\"> tx.</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">(res.</span><span style=\"color:#859900;\">into_val</span><span style=\"color:#657b83;\">()) {\n                </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(()) </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">(),\n                </span><span style=\"color:#93a1a1;\">// could use .unwrap, but that would require Debug constraint\n                </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">=&gt; panic!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">Impossible happend! tx.send failed</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">),\n            }\n        });\n        </span><span style=\"color:#93a1a1;\">// One more worker to wait for\n        </span><span style=\"color:#d33682;\">self</span><span style=\"color:#657b83;\">.count += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n    }\n\n    </span><span style=\"color:#93a1a1;\">/// Finish running all of the workers, exiting when the first one errors or all of them complete\n    </span><span style=\"color:#586e75;\">pub</span><span style=\"color:#657b83;\"> async </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">run</span><span style=\"color:#657b83;\">(</span><span style=\"color:#586e75;\">mut </span><span style=\"color:#268bd2;\">self</span><span style=\"color:#657b83;\">) -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), E&gt; {\n        </span><span style=\"color:#93a1a1;\">// Make sure we don&#39;t wait for ourself here\n        </span><span style=\"color:#657b83;\">std::mem::drop(</span><span style=\"color:#d33682;\">self</span><span style=\"color:#657b83;\">.tx);\n        </span><span style=\"color:#93a1a1;\">// How many workers have completed?\n        </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> i = </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\">;\n\n        </span><span style=\"color:#859900;\">loop </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#859900;\">match </span><span style=\"color:#d33682;\">self</span><span style=\"color:#657b83;\">.rx.</span><span style=\"color:#859900;\">recv</span><span style=\"color:#657b83;\">().await {\n                </span><span style=\"color:#859900;\">None =&gt; </span><span style=\"color:#657b83;\">{\n                    </span><span style=\"color:#859900;\">assert_eq!</span><span style=\"color:#657b83;\">(i, </span><span style=\"color:#d33682;\">self</span><span style=\"color:#657b83;\">.count);\n                    </span><span style=\"color:#859900;\">break Ok</span><span style=\"color:#657b83;\">(());\n                }\n                </span><span style=\"color:#859900;\">Some</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())) </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n                    </span><span style=\"color:#859900;\">assert!</span><span style=\"color:#657b83;\">(i &lt; </span><span style=\"color:#d33682;\">self</span><span style=\"color:#657b83;\">.count);\n                }\n                </span><span style=\"color:#859900;\">Some</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(e)) </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n                    </span><span style=\"color:#859900;\">assert!</span><span style=\"color:#657b83;\">(i &lt; </span><span style=\"color:#d33682;\">self</span><span style=\"color:#657b83;\">.count);\n                    </span><span style=\"color:#859900;\">return Err</span><span style=\"color:#657b83;\">(e);\n                }\n            }\n            i += </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">;\n        }\n    }\n}\n</span></code></pre>\n<p>Now in <code>src/main.rs</code>, we're going to get to focus on just our business logic... and error handling. Have a look at the new contents:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#93a1a1;\">// Indicate that we have another module\n</span><span style=\"color:#268bd2;\">mod </span><span style=\"color:#b58900;\">workers</span><span style=\"color:#657b83;\">;\n\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::io::BufRead;\n\n</span><span style=\"color:#93a1a1;\">/// Create a new error type to handle the two ways errors can happen.\n</span><span style=\"color:#657b83;\">#[</span><span style=\"color:#268bd2;\">derive</span><span style=\"color:#657b83;\">(Debug)]\n</span><span style=\"color:#268bd2;\">enum </span><span style=\"color:#b58900;\">AppError </span><span style=\"color:#657b83;\">{\n    </span><span style=\"color:#cb4b16;\">IO</span><span style=\"color:#657b83;\">(std::io::Error),\n    Reqwest(reqwest::Error),\n}\n\n</span><span style=\"color:#93a1a1;\">// And now implement some boilerplate From impls to support ? syntax\n</span><span style=\"color:#268bd2;\">impl </span><span style=\"color:#859900;\">From</span><span style=\"color:#657b83;\">&lt;std::io::Error&gt; </span><span style=\"color:#859900;\">for </span><span style=\"color:#b58900;\">AppError </span><span style=\"color:#657b83;\">{\n    </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">from</span><span style=\"color:#657b83;\">(</span><span style=\"color:#268bd2;\">e</span><span style=\"color:#657b83;\">: std::io::Error) -&gt; </span><span style=\"color:#268bd2;\">Self </span><span style=\"color:#657b83;\">{\n        AppError::</span><span style=\"color:#cb4b16;\">IO</span><span style=\"color:#657b83;\">(e)\n    }\n}\n\n</span><span style=\"color:#268bd2;\">impl </span><span style=\"color:#859900;\">From</span><span style=\"color:#657b83;\">&lt;reqwest::Error&gt; </span><span style=\"color:#859900;\">for </span><span style=\"color:#b58900;\">AppError </span><span style=\"color:#657b83;\">{\n    </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">from</span><span style=\"color:#657b83;\">(</span><span style=\"color:#268bd2;\">e</span><span style=\"color:#657b83;\">: reqwest::Error) -&gt; </span><span style=\"color:#268bd2;\">Self </span><span style=\"color:#657b83;\">{\n        AppError::Reqwest(e)\n    }\n}\n\n#[</span><span style=\"color:#268bd2;\">tokio</span><span style=\"color:#657b83;\">::</span><span style=\"color:#268bd2;\">main</span><span style=\"color:#657b83;\">]\nasync </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), AppError&gt; {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = std::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">urls.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> buffile = std::io::BufReader::new(file);\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">URL,Status</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = reqwest::Client::new();\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> workers = workers::Workers::new();\n\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> line </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> buffile.</span><span style=\"color:#859900;\">lines</span><span style=\"color:#657b83;\">() {\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> line = line</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = client.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">();\n        </span><span style=\"color:#93a1a1;\">// Use workers.spawn, and no longer worry about results\n        // ? works just fine inside!\n</span><span style=\"color:#657b83;\">        workers.</span><span style=\"color:#859900;\">spawn</span><span style=\"color:#657b83;\">(async </span><span style=\"color:#586e75;\">move </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> resp = client.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">line).</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">().await</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n            </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\">,</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line, resp.</span><span style=\"color:#859900;\">status</span><span style=\"color:#657b83;\">().</span><span style=\"color:#859900;\">as_u16</span><span style=\"color:#657b83;\">());\n            </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())\n        })\n    }\n\n    </span><span style=\"color:#93a1a1;\">// Wait for the workers to complete\n</span><span style=\"color:#657b83;\">    workers.</span><span style=\"color:#859900;\">run</span><span style=\"color:#657b83;\">().await\n}\n</span></code></pre>\n<p>There's more noise around error handling, but overall the code is easier to understand. Now that we have that out of the way, we're finally ready to tackle the last piece of this...</p>\n<h2 id=\"job-queue\">Job queue</h2>\n<p>Let's review again at a high level how we do error handling with workers. We set up a channel to allow each worker task to send its results to a single receiver, the main task. We used <code>mpsc</code>, or &quot;multi-producer single-consumer.&quot; That matches up with what we just described, right?</p>\n<p>OK, a job queue is kind of similar. We want to have a single task that reads lines from the file and feeds them into a channel. Then, we want multiple workers to read values from the channel. This is &quot;single-producer multi-consumer.&quot; Unfortunately, <code>tokio</code> doesn't provide such a channel out of the box. After I asked on Twitter, I <a href=\"https://twitter.com/gallabytes/status/1300193419084460033?s=20\">was recommended</a> to use <a href=\"https://crates.io/crates/async-channel\">async-channel</a>, which provides a &quot;multi-producer multi-consumer.&quot; That works for us!</p>\n<p>Thanks to our work before with the <code>Workers</code> <code>struct</code> refactor, this is now pretty easy. Let's have a look at the modified <code>main</code> function:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">#[</span><span style=\"color:#268bd2;\">tokio</span><span style=\"color:#657b83;\">::</span><span style=\"color:#268bd2;\">main</span><span style=\"color:#657b83;\">]\nasync </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), AppError&gt; {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = std::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">urls.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> buffile = std::io::BufReader::new(file);\n\n    </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">URL,Status</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#93a1a1;\">// Feel free to define to any numnber (&gt; 0) you want\n    // At a value of 4, this could comfortably fit in OS threads\n    // But tasks are certainly up to the challenge, and will scale\n    // up more nicely for large numbers and more complex applications\n    </span><span style=\"color:#268bd2;\">const </span><span style=\"color:#cb4b16;\">WORKERS</span><span style=\"color:#657b83;\">: </span><span style=\"color:#268bd2;\">usize </span><span style=\"color:#657b83;\">= </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = reqwest::Client::new();\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> workers = workers::Workers::new();\n    </span><span style=\"color:#93a1a1;\">// Buffers double the size of the number of workers are common\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#657b83;\">(tx, rx) = async_channel::bounded(</span><span style=\"color:#cb4b16;\">WORKERS </span><span style=\"color:#657b83;\">* </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">);\n\n    </span><span style=\"color:#93a1a1;\">// Spawn the task to fill up the queue\n</span><span style=\"color:#657b83;\">    workers.</span><span style=\"color:#859900;\">spawn</span><span style=\"color:#657b83;\">(async </span><span style=\"color:#586e75;\">move </span><span style=\"color:#657b83;\">{\n        </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> line </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> buffile.</span><span style=\"color:#859900;\">lines</span><span style=\"color:#657b83;\">() {\n            </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> line = line</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n            tx.</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">(line).await.</span><span style=\"color:#859900;\">unwrap</span><span style=\"color:#657b83;\">();\n        }\n        </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())\n    });\n\n    </span><span style=\"color:#93a1a1;\">// Spawn off the individual workers\n    </span><span style=\"color:#859900;\">for _ in </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#859900;\">..</span><span style=\"color:#cb4b16;\">WORKERS </span><span style=\"color:#657b83;\">{\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> client = client.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">();\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> rx = rx.</span><span style=\"color:#859900;\">clone</span><span style=\"color:#657b83;\">();\n        workers.</span><span style=\"color:#859900;\">spawn</span><span style=\"color:#657b83;\">(async </span><span style=\"color:#586e75;\">move </span><span style=\"color:#657b83;\">{\n            </span><span style=\"color:#859900;\">loop </span><span style=\"color:#657b83;\">{\n                </span><span style=\"color:#859900;\">match</span><span style=\"color:#657b83;\"> rx.</span><span style=\"color:#859900;\">recv</span><span style=\"color:#657b83;\">().await {\n                    </span><span style=\"color:#93a1a1;\">// uses Err to represent a closed channel due to tx being dropped\n                    </span><span style=\"color:#859900;\">Err</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">_</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">=&gt; break Ok</span><span style=\"color:#657b83;\">(()),\n                    </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(line) </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#657b83;\">{\n                        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> resp = client.</span><span style=\"color:#859900;\">get</span><span style=\"color:#657b83;\">(</span><span style=\"color:#859900;\">&amp;</span><span style=\"color:#657b83;\">line).</span><span style=\"color:#859900;\">send</span><span style=\"color:#657b83;\">().await</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n                        </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#2aa198;\">,</span><span style=\"color:#cb4b16;\">{}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line, resp.</span><span style=\"color:#859900;\">status</span><span style=\"color:#657b83;\">().</span><span style=\"color:#859900;\">as_u16</span><span style=\"color:#657b83;\">());\n                    }\n                }\n            }\n        })\n    }\n\n    </span><span style=\"color:#93a1a1;\">// Wait for the workers to complete\n</span><span style=\"color:#657b83;\">    workers.</span><span style=\"color:#859900;\">run</span><span style=\"color:#657b83;\">().await\n}\n</span></code></pre>\n<p>And just like that, we have a concurrent job queue! It's everything we could have wanted!</p>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>I'll admit, when I wrote the post last week, I didn't think I'd be going this deep into the topic. But once I started playing with solutions, I decided I wanted to implement a full job queue for this.</p>\n<p>I hope you found this topic interesting! If you want more Rust content, please <a href=\"https://twitter.com/snoyberg\">hit me up on Twitter</a>. Also, feel free to check out some of our other Rust content:</p>\n<ul>\n<li><a href=\"https://www.fpcomplete.com/rust/\">Rust homepage</a></li>\n<li><a href=\"https://www.fpcomplete.com/rust/pid1/\">Implementing pid1 with Rust and async/await</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/2018/10/is-rust-functional/\">Is Rust functional?</a></li>\n</ul>\n",
        "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/",
        "slug": "http-status-codes-async-rust",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "HTTP status codes with async Rust",
        "description": "Following up a on previous blog post, we'll examine different levels of async code while building a semi-useful program that checks web links",
        "updated": null,
        "date": "2020-09-02",
        "year": 2020,
        "month": 9,
        "day": 2,
        "taxonomies": {
          "tags": [
            "rust"
          ],
          "categories": [
            "functional programming"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/http-status-codes-async-rust/",
        "components": [
          "blog",
          "http-status-codes-async-rust"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "fully-blocking",
            "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/#fully-blocking",
            "title": "Fully blocking",
            "children": []
          },
          {
            "level": 2,
            "id": "ditching-the-blocking-api",
            "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/#ditching-the-blocking-api",
            "title": "Ditching the blocking API",
            "children": []
          },
          {
            "level": 2,
            "id": "where-blocking-is-fine",
            "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/#where-blocking-is-fine",
            "title": "Where blocking is fine",
            "children": []
          },
          {
            "level": 2,
            "id": "concurrent-requests",
            "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/#concurrent-requests",
            "title": "Concurrent requests",
            "children": []
          },
          {
            "level": 2,
            "id": "wait-for-me",
            "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/#wait-for-me",
            "title": "Wait for me!",
            "children": []
          },
          {
            "level": 2,
            "id": "error-handling",
            "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/#error-handling",
            "title": "Error handling",
            "children": []
          },
          {
            "level": 2,
            "id": "workers",
            "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/#workers",
            "title": "Workers",
            "children": []
          },
          {
            "level": 2,
            "id": "job-queue",
            "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/#job-queue",
            "title": "Job queue",
            "children": []
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/http-status-codes-async-rust/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 3824,
        "reading_time": 20,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/different-levels-async-rust.md",
        "content": "<p>First there was cooperative multiprocessing. Then there were processes. An operating system could run multiple processes, each performing a series of sequential, blocking actions. Then came threads. A single processes could spawn off multiple threads, each performing its own series of sequential, blocking actions. (And really, the story starts earlier, with hardware interrupts and the like, but hopefully you'll forgive a little simplification.)</p>\n<p>Sitting around and waiting for stuff? Ain't nobody got time for that. Spawning threads at the operating system level? That's too costly for a lot of what we do these days.</p>\n<p>Perhaps the first foray into asynchronous programming that really hit the mainstream was the Nginx web server, which boasted a huge increase in throughput by natively using asynchronous I/O system calls. Some programming languages, like Go, Erlang, and Haskell, built runtimes systems that support spawning cheap green threads, and handle the muck of asynchronous system calls for you under the surface. Other languages, such as Javascript and more recently Rust, provide explicit asynchronous support in the language.</p>\n<p>Given how everyone seems to be bending over backwards to make it easy for you to make your code async-friendly, it would be fair to assume that all code at all times should be async. And it would also be fair to guess that, if you stick the word <code>async</code> on a function, it's completely asynchronous. Unfortunately, neither of these assumptions are true. This post is intended to dive into this topic, in Rust, using a simple bit of code for motivation.</p>\n<p><strong>Prior knowledge</strong> This post will assume that you are familiar with the Rust programming language, as well as its <code>async/.await</code> syntax. If you'd like to brush up on either language basics or async code, I'd recommend checking out <a href=\"https://www.fpcomplete.com/rust/crash-course/\">FP Complete's Rust Crash Course</a>.</p>\n<p><strong>Update</strong> I've published a follow up to this post <a href=\"https://www.fpcomplete.com/blog/http-status-codes-async-rust/\">covering a more sophisticated HTTP client example</a>.</p>\n<h2 id=\"blocking-vs-non-blocking-calls\">Blocking vs non-blocking calls</h2>\n<p>Just to make sure we're on the same page, I'm going to define here what a blocking versus non-blocking call is and explain how this impacts async vs sync. If you're already highly experienced with async programming, you can probably skip this section.</p>\n<p>For the most part, &quot;async code&quot; means code that relies on non-blocking, rather than blocking, system calls for performing I/O. By contrast, sync (or synchronous) code relies on blocking system calls. As a simple example, consider a web server that has 20 open sockets from web clients, and needs to read data from all of them. One approach would be to use the blocking <code>recv</code> system call. This will:</p>\n<ul>\n<li>Turn control of the current operating system thread over to the kernel</li>\n<li>Wait in the kernel until either new data is available, the connection dies, or some error occurs</li>\n<li>Wake up the operating system thread and let the program continue executing</li>\n</ul>\n<p>If you follow this approach, and you have the aforementioned 20 connections, you essentially have two choices:</p>\n<ol>\n<li>Have a single thread handle each of the connections one at a time</li>\n<li>Spawn 20 separate operating system threads, and let each of them handle a single connection</li>\n</ol>\n<p>(1) would be an abysmal client experience. If a slow client gets in line with a connection, you could easily end up waiting a <strong>long</strong> time to make your request. (Imagine going to a supermarket with a single checkout line, no self checkout, and the person at the front of the line is paying in coins.) (2) is much better, but spawning off those operating system threads is a relatively costly activity, ultimately.</p>\n<p>Both of these approaches are synchronous. By contrast, an asynchronous approach could handle all 20 connections in a single operating system thread, with a basic approach of:</p>\n<ul>\n<li>Register a callback function to be triggered on new data availability</li>\n<li>Register all 20 of the connections to trigger that callback</li>\n<li>Within that function, check each socket to see if data is available, and if so, handle it</li>\n</ul>\n<p>Writing code like this manually can be fairly complicated, which is why many languages have added either <code>async</code> syntax or some kind of green thread based runtime. And it seems overall that this simply makes your program better. But let's test those ideas out in practice.</p>\n<h2 id=\"count-by-lines\">Count by lines</h2>\n<p>Let's write a simple, synchronous, single threaded, blocking program in Rust. It will take all of the lines in a file (hard-coded to <code>input.txt</code>), and print to standard output the number of characters on each line. It will exit the program on any errors. The program is pretty straightforward:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#93a1a1;\">// We want to use the lines method from this trait\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">std::io::BufRead;\n\n</span><span style=\"color:#93a1a1;\">// Let&#39;s us use ? for simple error handling\n</span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), std::io::Error&gt; {\n    </span><span style=\"color:#93a1a1;\">// Try to open the file\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = std::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">input.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#93a1a1;\">// Create a buffered version of the file so we can use lines\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> buffered = std::io::BufReader::new(file);\n\n    </span><span style=\"color:#93a1a1;\">// Iterate through each line in the file\n    </span><span style=\"color:#859900;\">for</span><span style=\"color:#657b83;\"> line </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\"> buffered.</span><span style=\"color:#859900;\">lines</span><span style=\"color:#657b83;\">() {\n        </span><span style=\"color:#93a1a1;\">// But we get a Result each time, get rid of the errors\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> line = line</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n        </span><span style=\"color:#93a1a1;\">// And print out the line length and content\n        </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{} {}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line.</span><span style=\"color:#859900;\">len</span><span style=\"color:#657b83;\">(), line);\n    }\n\n    </span><span style=\"color:#93a1a1;\">// Everything went fine, so we return Ok\n    </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())\n}\n</span></code></pre>\n<p><strong>Recommendation</strong> I encourage readers to start playing along with this code themselves. Assuming you <a href=\"https://www.rust-lang.org/learn/get-started\">have installed Rust</a>, you can run <code>cargo new asyncpost</code> and then copy-paste the code above into <code>src/main.rs</code>. Then add in the <code>input.txt</code> file here:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">Hello world\nHope you have a great day!\nGoodbye\n</span></code></pre>\n<p>And if you run <code>cargo run</code>, you should get the expected output of:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">11 Hello world\n26 Hope you have a great day!\n7 Goodbye\n</span></code></pre>\n<p>I won't comment on running the code yourself any more, but I recommend you keep updating the code and running <code>cargo run</code> throughout reading this post.</p>\n<p>Anyway, back to async code. The code above is completely synchronous. Every I/O action will fully block the main (and only) thread in our program. To be crystal clear, let's see all the places this is relevant (ignoring error cases):</p>\n<ol>\n<li>Opening the file makes a blocking <code>open</code> system call</li>\n<li>As we iterate through the lines, the <code>BufRead</code> trait will implicitly be triggering multiple <code>read</code> system calls, which block waiting for data to be available from the file descriptor</li>\n<li>The <code>println!</code> macro will make <code>write</code> system calls on the <code>stdout</code> file descriptor, each of which is a blocking call</li>\n<li>Finally, when the <code>file</code> is dropped, the <code>close</code> system call to close the file descriptor</li>\n</ol>\n<p><strong>NOTE</strong> I'm using POSIX system call terms here, things may be slightly different on some operating systems, and radically different on Windows. That shouldn't take away from the main thrust of the message here.</p>\n<h2 id=\"make-it-async\">Make it async!</h2>\n<p>The most straightforward way to write asynchronous programs in Rust is to use <code>async/await</code> syntax. Let's naively try simply converting our <code>main</code> function into something <code>async</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">async </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), std::io::Error&gt; {\n    </span><span style=\"color:#93a1a1;\">// code inside is unchanged\n</span><span style=\"color:#657b83;\">}\n</span></code></pre>\n<p>That's going to fail (I <em>did</em> say naively). The reason is that you can't simply run an <code>async</code> function like <code>main</code>. Instead, you need to provide an executor that knows how to handle all of the work there. The most popular library for this is <code>tokio</code>, and it provides a nice convenience macro to make this really easy. First, let's modify our <code>Cargo.toml</code> file to add the <code>tokio</code> dependency, together with all optional features turned on (it will be convenient later):</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">[</span><span style=\"color:#b58900;\">dependencies</span><span style=\"color:#657b83;\">]\n</span><span style=\"color:#268bd2;\">tokio </span><span style=\"color:#657b83;\">= { </span><span style=\"color:#268bd2;\">version </span><span style=\"color:#657b83;\">= </span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">0.2.22</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">features </span><span style=\"color:#657b83;\">= [</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">full</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">] }\n</span></code></pre>\n<p>And then we stick the appropriate macro in front of our <code>main</code> function:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">#[</span><span style=\"color:#268bd2;\">tokio</span><span style=\"color:#657b83;\">::</span><span style=\"color:#268bd2;\">main</span><span style=\"color:#657b83;\">]\nasync </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), std::io::Error&gt; {\n    </span><span style=\"color:#93a1a1;\">// unchanged\n</span><span style=\"color:#657b83;\">}\n</span></code></pre>\n<p>And just like that, we have an asynchronous version of our program! Thank you very much everyone, have a great day, I'll see you later.</p>\n<h2 id=\"but-wait\">&quot;But wait!&quot;</h2>\n<p>&quot;But wait!&quot; you may be saying. &quot;How does Rust automatically know to rewrite all of those blocking, synchronous system calls into asynchronous, non-blocking ones just by sticking the <code>async</code> keyword on there???&quot;</p>\n<p>Answer: it doesn't. The cake is a lie.</p>\n<p>This is the first message I want to bring home. The <code>async</code> keyword allows for some special syntax (we'll see it in a little bit). It makes it much easier to write asynchronous programs. It relies on having some kind of an executor like <code>tokio</code> around to actually run things. But that's all it does. It does <em>not</em> provide any kind of asynchronous system call support. You've got to do that on your own.</p>\n<p>Fortunately for us, <code>tokio</code> <em>does</em> bring this to the table. Instead of using the <code>std</code> crate's versions of I/O functions, we'll instead lean on <code>tokio</code>'s implementation. I'm now going to follow one of my favorite development techniques, &quot;change the code and ask the compiler for help.&quot; Let's dive in!</p>\n<p>The first synchronous call we make is to open the file with <code>std::fs::File::open</code>. Fortunately for us, <code>tokio</code> provides a replacement for this method via its replacement <code>File</code> struct. We can simply swap out <code>std</code> with <code>tokio</code> and get the line:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = tokio::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">input.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">)</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n</span></code></pre>\n<p>Unfortunately, that's not going to compile, not even a little bit:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">error[E0277]: the `?` operator can only be applied to values that implement `std::ops::Try`\n --&gt; src\\main.rs:8:16\n  |\n8 |     let file = tokio::fs::File::open(&quot;input.txt&quot;)?;\n  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  |                |\n  |                the `?` operator cannot be applied to type `impl std::future::Future`\n  |                help: consider using `.await` here: `tokio::fs::File::open(&quot;input.txt&quot;).await?`\n  |\n  = help: the trait `std::ops::Try` is not implemented for `impl std::future::Future`\n  = note: required by `std::ops::Try::into_result`\n</span></code></pre>\n<p>(To the observant among you: yes, I'm compiling this on Windows.)</p>\n<p>So what exactly does this mean? The <code>open</code> method from <code>std</code> returned a <code>Result</code> to represent &quot;this may have had an error.&quot; And we stuck a <code>?</code> after the <code>open</code> call to say &quot;hey, if there was an error, please exit this function with that error.&quot; But <code>tokio</code>'s <code>open</code> isn't returning a <code>Result</code>. Instead, it's returning a <code>Future</code>. This value represents a promise that, at some point in the future, we'll get back a <code>Result</code>.</p>\n<p>But I want the <code>Result</code> now! How do I force my program to wait for it? Easy: <code>.await</code>. The closer-to-compiling version of our code is:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = tokio::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">input.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">).await</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n</span></code></pre>\n<p>Now we're saying to the compiler:</p>\n<ol>\n<li>Open the file</li>\n<li>Wait for the <code>Future</code> to complete to tell me the file result is ready (via <code>.await</code>)</li>\n<li>Then, if there was an error, exit this function (via <code>?</code>)</li>\n</ol>\n<p>And we end up with the <code>file</code> variable holding a <code>tokio::fs::File</code> struct. Awesome!</p>\n<p>At this point, our code still doesn't compile, since there's a mismatch between the <code>std</code> and <code>tokio</code> sets of traits. If you want to have some fun, try to fix the code yourself. But I'll just show you the completed version here:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#93a1a1;\">// Replaces std::io::BufRead trait\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">tokio::io::AsyncBufReadExt;\n</span><span style=\"color:#93a1a1;\">// We can&#39;t generally use normal Iterators with async code\n// Instead, we use Streams\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">tokio::stream::StreamExt;\n\n#[</span><span style=\"color:#268bd2;\">tokio</span><span style=\"color:#657b83;\">::</span><span style=\"color:#268bd2;\">main</span><span style=\"color:#657b83;\">]\nasync </span><span style=\"color:#268bd2;\">fn </span><span style=\"color:#b58900;\">main</span><span style=\"color:#657b83;\">() -&gt; </span><span style=\"color:#859900;\">Result</span><span style=\"color:#657b83;\">&lt;(), std::io::Error&gt; {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> file = tokio::fs::File::open(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#2aa198;\">input.txt</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">).await</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> buffered = tokio::io::BufReader::new(file);\n\n    </span><span style=\"color:#93a1a1;\">// Since we can&#39;t use a for loop, we&#39;ll manually\n    // create our Stream of lines\n    </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> lines = buffered.</span><span style=\"color:#859900;\">lines</span><span style=\"color:#657b83;\">();\n\n    </span><span style=\"color:#93a1a1;\">// Now keep popping off another, waiting for each\n    // I/O action to complete via .await\n    </span><span style=\"color:#859900;\">while </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#859900;\">Some</span><span style=\"color:#657b83;\">(line) = lines.</span><span style=\"color:#859900;\">next</span><span style=\"color:#657b83;\">().await {\n        </span><span style=\"color:#93a1a1;\">// Error handling again\n        </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> line = line</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n        </span><span style=\"color:#93a1a1;\">// And print out the line length and content\n        </span><span style=\"color:#859900;\">println!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{} {}</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line.</span><span style=\"color:#859900;\">len</span><span style=\"color:#657b83;\">(), line);\n    }\n\n    </span><span style=\"color:#859900;\">Ok</span><span style=\"color:#657b83;\">(())\n}\n</span></code></pre>\n<p>It's wordier, but it gets the job done. And now we have a fully asynchronous version of our program... right?</p>\n<p>Well, no, not really. I mentioned four blocking I/O calls above: <code>open</code>, <code>read</code> (from the file), <code>write</code> (to <code>stdout</code>), and <code>close</code>. By switching to <code>tokio</code>, <code>open</code> and <code>read</code> are now using async versions of these system calls. <code>close</code> is a bit more complicated, since it happens implicitly when dropping a value. Ultimately, this code falls back to the same <code>close</code> system call, since it uses <code>std</code>'s implementation of <code>File</code> under the surface. But for our purposes, let's pretend like this doesn't actually block.</p>\n<p>No, the more interesting thing remaining is the <code>println!</code> macro's usage of <code>write</code>. This is still fully blocking I/O. And what I find informative is that it's sitting in the middle of a fully asynchronous <code>while</code> loop leveraging <code>.await</code>! This hopefully drives home another one of the core messages I mentioned above: being async isn't a binary on/off. You can have programs that are more or less asynchronous, depending on how many of the system calls get replaced.</p>\n<p>We'll get to replacing <code>println!</code> at the end of this post, but I want to point out something somewhat striking first. If you look at the examples in the <code>tokio</code> docs (such as the <a href=\"https://docs.rs/tokio/0.2.22/tokio/io/index.html\"><code>tokio::io</code> module</a>), you'll notice that they're using <code>println!</code> themselves. Why would a library for async I/O use blocking calls?!?</p>\n<h2 id=\"it-s-not-always-worth-it\">It's not always worth it</h2>\n<p>Let's return to our web server with 20 connections. Let's pretend we wrote a half-async version of that web server. It uses non-blocking I/O for reading data from the clients. That means our web server will not need 20 independent threads, and it will know when data is available. However, like our program above, it's going to produce output (data sent back to the clients) using blocking I/O calls. How will this affect our web server's behavior?</p>\n<p>Well, it's better than the worst case we described above. We won't block the entire server because one client is sending a really slow request. We'll be able to wait until a client fully sends its request before we put together our response and send it back. However, since we're using a blocking call to send the data back, if that same slow client is also slow at <em>receiving</em> data, we'll be back to square one with a laggy server. And it won't just slow down sending of responses. We'll end up blocking <em>all</em> I/O, such as accepting new connections and receiving on existing connections.</p>\n<p>But let's pretend, just for a moment, that we know that each and every one of these clients has a super fast receive rate. The blocking <code>send</code> calls we make will always complete in something insanely fast like a nanosecond. Would we care that they're blocking calls? No, probably not. I don't mind blocking a system thread for one nanosecond. I care about having long and possibly indeterminate blocking I/O calls.</p>\n<p>The situation with <code>stdout</code> is closer to this. It's generally a safe assumption that outputting data to <code>stdout</code> is only going to block for a short duration of time. And therefore, most people think using <code>println!</code> is a fine thing to do in async code, and it's not worth rewriting to something more complex. Taking this a step further: many things we don't think of as blocking may, in fact block. For example, reading and writing memory that is memory mapped to files (via <code>mmap</code>) may involve blocking I/O. Generally, it's impossible to expunge all traces of blocking behavior in a program.</p>\n<p>But this &quot;it's not always worth it&quot; goes much deeper. Let's review our program above. With the new async I/O calls, our program is going to:</p>\n<ol>\n<li>Make a non-blocking system call to open a file descriptor</li>\n<li>Block (via <code>.await</code>) until the file is open</li>\n<li>In a loop:\n<ol>\n<li>Read data from the descriptor with a non-blocking system call</li>\n<li>Block (via <code>.await</code>) for a complete line to be read</li>\n<li>Make a blocking call to <code>write</code> to output data to <code>stdout</code></li>\n</ol>\n</li>\n<li>Make a blocking <code>close</code> system call</li>\n</ol>\n<p>Did moving from blocking to non-blocking calls help us at all? Absolutely not! Our program is inherently single threaded and sequential. We were previously blocking our main thread inside <code>open</code> and <code>read</code> system calls. Now we're blocking that same thread waiting for the non-blocking equivalents to complete.</p>\n<img style=\"max-width:100%\" alt=\"Blocking I/O with more steps\" src=\"/images/blog/blocking-more-steps.jpg\">\n<p>So just because we <em>can</em> make something asynchronous, doesn't mean it's always better code. By changing our program above, we've made it significantly more complex, added extra dependencies, and almost certainly slower to run.</p>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>To sum up what we covered in this post:</p>\n<ul>\n<li>Async code can be hard to write manually</li>\n<li>Rust (and other languages) provide async syntax to make it easier</li>\n<li>You need an executor to run async code</li>\n<li>But that's not enough: you need to replace blocking I/O calls with non-blocking calls</li>\n<li>Libraries like <code>tokio</code> provide both the executor and the non-blocking functions</li>\n<li>You can partially asyncify a program</li>\n<li>And it's not always worth converting to async</li>\n</ul>\n<p>I wanted to cover a more complex example of async in this post, but it's already on the long side. Instead, I'll follow up in a later post with a program that makes HTTP requests and checks their status codes. I'll update this post with a link when it's ready. Stay tuned!</p>\n<p><strong>Update</strong> And here's that updated blog post! <a href=\"https://www.fpcomplete.com/blog/http-status-codes-async-rust/\">HTTP status codes with async Rust</a></p>\n<p>And finally...</p>\n<h2 id=\"appendix-non-blocking-output\">Appendix: non-blocking output</h2>\n<p>I promised you I'd end with an example of replacing <code>println!</code> with non-blocking I/O. Remember, this isn't something I'm generally recommending you do. But it's informative to see it in action.</p>\n<p>The simplest way to do this is to use <code>format!</code> to create a <code>String</code> with the content you want to output, and then use <code>tokio::io::stdout()</code>. (Note that this forces a heap allocation of a <code>String</code>, something that doesn't occur with <code>println!</code> usage.) This looks like:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#93a1a1;\">// Use the trait\n</span><span style=\"color:#859900;\">use </span><span style=\"color:#657b83;\">tokio::io::AsyncWriteExt;\n\n</span><span style=\"color:#93a1a1;\">// same code as before\n\n</span><span style=\"color:#268bd2;\">let </span><span style=\"color:#586e75;\">mut</span><span style=\"color:#657b83;\"> stdout = tokio::io::stdout();\n</span><span style=\"color:#859900;\">while </span><span style=\"color:#268bd2;\">let </span><span style=\"color:#859900;\">Some</span><span style=\"color:#657b83;\">(line) = lines.</span><span style=\"color:#859900;\">next</span><span style=\"color:#657b83;\">().await {\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> line = line</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n    </span><span style=\"color:#93a1a1;\">// Note the extra newline character!\n    </span><span style=\"color:#268bd2;\">let</span><span style=\"color:#657b83;\"> s = </span><span style=\"color:#859900;\">format!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#cb4b16;\">{} {}</span><span style=\"color:#dc322f;\">\\n</span><span style=\"color:#839496;\">&quot;</span><span style=\"color:#657b83;\">, line.</span><span style=\"color:#859900;\">len</span><span style=\"color:#657b83;\">(), line);\n    stdout.</span><span style=\"color:#859900;\">write_all</span><span style=\"color:#657b83;\">(s.</span><span style=\"color:#859900;\">as_bytes</span><span style=\"color:#657b83;\">()).await</span><span style=\"color:#859900;\">?</span><span style=\"color:#657b83;\">;\n}\n</span></code></pre>\n<p>This overall looks safe, and in our program will perform correctly. However, generally, there are problems with this code. From the <a href=\"https://docs.rs/tokio/0.2.22/tokio/io/fn.stdout.html\">docs on <code>stdout</code></a></p>\n<blockquote>\n<p>In particular you should be aware that writes using <code>write_all</code> are not guaranteed to occur as a single write, so multiple threads writing data with <code>write_all</code> may result in interleaved output.</p>\n</blockquote>\n<p>Since our program is sequential anyway, we don't need to worry about multiple threads creating interleaved output. But in general, that would be a concern. One approach to address that would be to create a channel of messages to be sent to <code>stdout</code>.</p>\n<p>But the most popular solution would be to simply use <code>println!</code> and similar macros.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/different-levels-async-rust/",
        "slug": "different-levels-async-rust",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Different levels of async in Rust",
        "description": "Often, developers today look at asynchronous I/O as a good thing, and as a binary choice. The reality is more nuanced. In this post, we'll explore the situation with a simple Rust example.",
        "updated": null,
        "date": "2020-08-24",
        "year": 2020,
        "month": 8,
        "day": 24,
        "taxonomies": {
          "categories": [
            "functional programming"
          ],
          "tags": [
            "rust"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "image": "images/blog/blocking-more-steps.jpg",
          "blogimage": "/images/blog/functional.jpg"
        },
        "path": "blog/different-levels-async-rust/",
        "components": [
          "blog",
          "different-levels-async-rust"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "blocking-vs-non-blocking-calls",
            "permalink": "https://www.fpcomplete.com/blog/different-levels-async-rust/#blocking-vs-non-blocking-calls",
            "title": "Blocking vs non-blocking calls",
            "children": []
          },
          {
            "level": 2,
            "id": "count-by-lines",
            "permalink": "https://www.fpcomplete.com/blog/different-levels-async-rust/#count-by-lines",
            "title": "Count by lines",
            "children": []
          },
          {
            "level": 2,
            "id": "make-it-async",
            "permalink": "https://www.fpcomplete.com/blog/different-levels-async-rust/#make-it-async",
            "title": "Make it async!",
            "children": []
          },
          {
            "level": 2,
            "id": "but-wait",
            "permalink": "https://www.fpcomplete.com/blog/different-levels-async-rust/#but-wait",
            "title": "\"But wait!\"",
            "children": []
          },
          {
            "level": 2,
            "id": "it-s-not-always-worth-it",
            "permalink": "https://www.fpcomplete.com/blog/different-levels-async-rust/#it-s-not-always-worth-it",
            "title": "It's not always worth it",
            "children": []
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/different-levels-async-rust/#conclusion",
            "title": "Conclusion",
            "children": []
          },
          {
            "level": 2,
            "id": "appendix-non-blocking-output",
            "permalink": "https://www.fpcomplete.com/blog/different-levels-async-rust/#appendix-non-blocking-output",
            "title": "Appendix: non-blocking output",
            "children": []
          }
        ],
        "word_count": 3140,
        "reading_time": 16,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/devops-for-developers.md",
        "content": "<p>In this post, I describe my personal journey as a developer skeptical\nof the seemingly ever-growing, ever more complex, array of &quot;ops&quot;\ntools. I move towards adopting some of these practices, ideas and\ntools. I write about how this journey helps me to write software\nbetter and understand discussions with the ops team at work.</p>\n<div style=\"border:1px solid black;background-color:#f8f8f8;margin-bottom:1em;padding: 0.5em 0.5em 0 0.5em;\">\n<p><strong>Table of Contents</strong></p>\n<ul>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#on-being-skeptical\">On being skeptical</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#the-humble-app\">The humble app</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#disk-failures-arent-that-common\">Disk failures aren't that common</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#auto-deployment-is-better-than-manual\">Auto-deployment is better than manual</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#backups-become-worth-it\">Backups become worth it</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#deployment-staging\">Deployment staging</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#packaging-with-docker-is-good\">Packaging with Docker is good</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#custodiansmultiple-processes-are-useful\">Custodians/multiple processes are useful</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#kubernetes-provides-exactly-that\">Kubernetes provides exactly that</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#declarative-is-good-vendor-lock-in-is-bad\">Declarative is good, vendor lock-in is bad</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#more-advanced-rollout\">More advanced rollout</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#relationship-between-code-and-deployed-state\">Relationship between code and deployed state</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#argocd\">ArgoCD</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#infra-as-code\">Infra-as-code</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#where-the-dev-meets-the-ops\">Where the dev meets the ops</a></li>\n<li><a href=\"https://www.fpcomplete.com/blog/devops-for-developers/#what-we-do\">What we do</a></li>\n</ul>\n</div>\n<h2 id=\"on-being-skeptical\">On being skeptical</h2>\n<p>I would characterise my attitudes to adopting technology in two\nstages:</p>\n<ul>\n<li>Firstly, I am conservative and dismissive, in that I will usually\ndisregard any popular new technology as a bandwagon or trend. I'm a\nslow adopter.</li>\n<li>Secondly, when I actually encounter a situation where I've suffered,\nI'll then circle back to that technology and give it a try, and if I\ncan really find the nugget of technical truth in there, then I'll\nadopt it.</li>\n</ul>\n<p>Here are some things that I disregarded for a year or more before\ntrying: Emacs, Haskell, Git, Docker, Kubernetes, Kafka. The whole\nNoSQL trend came, wrecked havoc, and went, while I had my back turned,\nbut I am considering using Redis for a cache at the moment.</p>\n<h2 id=\"the-humble-app\">The humble app</h2>\n<p>If you’re a developer like me, you’re probably used to writing your\nsoftware, spending most of your time developing, and then finally\ndeploying your software by simply creating a machine, either a\ndedicated machine or a virtual machine, and then uploading a binary of\nyour software (or source code if it’s interpreted), and then running\nit with the copy pasted config of systemd or simply running the\nsoftware inside GNU screen. It's a secret shame that I've done this,\nbut it's the reality.</p>\n<p>You might use nginx to reverse-proxy to the service. Maybe you set up\na PostgreSQL database or MySQL database on that machine. And then you\nwalk away and test out the system, and later you realise you need some\nslight changes to the system configuration. So you SSH into the system\nand makes the small tweaks necessary, such as port settings, encoding\nsettings, or an additional package you forgot to add. Sound familiar?</p>\n<p>But on the whole, your work here is done and for most services this is\npretty much fine. There are plenty of services running that you have\nseen in the past 30 years that have been running like this.</p>\n<h2 id=\"disk-failures-aren-t-that-common\">Disk failures aren't that common</h2>\n<p>Rhetoric about processes going down due to a hardware failure are\nprobably overblown. Hard drives don’t crash very often. They don’t\nreally wear out as quickly as they used to, and you can be running a\nsystem for years before anything even remotely concerning happens.</p>\n<h2 id=\"auto-deployment-is-better-than-manual\">Auto-deployment is better than manual</h2>\n<p>When you start to iterate a little bit quicker, you get bored of\nmanually building and copying and restarting the binary on the\nsystem. This is especially noticeable if you forget the steps later\non.</p>\n<!-- Implementing Auto-Deployment -->\n<p>If you’re a little bit more advanced you might have some special\nscripts or post-merge git hooks, so that when you push to your repo it\nwould apply to the same machine and you have some associated token on\nyour CI machine that is capable of uploading a binary and running a\ncommand like copy and restart (e.g. SSH key or API\nkey). Alternatively, you might implement a polling system on the\nactual production system which will check if any updates have occurred\nin get and if so pull down a new binary. This is how we were doing\nthings in e.g. 2013.</p>\n<h2 id=\"backups-become-worth-it\">Backups become worth it</h2>\n<p>Eventually, if you're lucky, your service starts to become slightly\nmore important; maybe it’s used in business and people actually are\nusing it and storing valuable things in the database. You start to\nthink that back-ups are a good idea and worth the investment.</p>\n<!-- Redundancy of DB -->\n<p>You probably also have a script to back up the database, or replicate\nit on a separate machine, for redundancy.</p>\n<h2 id=\"deployment-staging\">Deployment staging</h2>\n<p>Eventually, you might have a staged deployment strategy. So you might\nhave a developer testing machine, you might have a QA machine, a\nstaging machine, and finally a production machine. All of these are\nconfigured in pretty much the same way, but they are deployed at\ndifferent times and probably the system administrator is the only one\nwith access to deploy to production.</p>\n<!-- Continuum -->\n<p>It’s clear by this point that I’m describing a continuum from &quot;hobby\nproject&quot; to &quot;enterprise serious business synergy solutions&quot;.</p>\n<h2 id=\"packaging-with-docker-is-good\">Packaging with Docker is good</h2>\n<p>Docker effectively leads to collapsing all of your system dependencies\nfor your binary to run into one contained package. This is good,\nbecause dependency management is hell. It's also highly wasteful,\nbecause its level of granularity is very wide. But this is a trade-off\nwe accept for the benefits.</p>\n<h2 id=\"custodians-multiple-processes-are-useful\">Custodians/multiple processes are useful</h2>\n<p>Docker doesn’t have much to say about starting and restarting\nservices. I’ve explored using CoreOS with the hosting provider Digital\nOcean, and simply running a fresh virtual machine, with the given\nDocker image.</p>\n<p>However, you quickly run into the problem of starting up and tearing\ndown:</p>\n<ul>\n<li>When you start the service, you need certain liveness checks\nand health checks, so if the service fails to start then you should\nnot stop the existing service from running, for example. You should\nkeep the existing ones running.</li>\n<li>If the process fails at any time during running then you should also\nrestart the process. I thought about this point a lot, and came to the\nconclusion that it’s better to have your process be restarted than to\nassume that the reason it failed was so dangerous that the process\nshouldn’t start again. Probably it’s more likely that there is an\nexception or memory issue that happened in a pathological case which\nyou can investigate in your logging system. But it doesn’t mean that\nyour users should suffer by having downtime.</li>\n<li>The natural progression of this functionality is to support\ndifferent rollout strategies. Do you want to switch everything to the\nnew system in one go, do you want it to be deployed piece-by-piece?</li>\n</ul>\n<!-- Summary: You Realise Worth Of Ops Tools -->\n<p>It’s hard to fully appreciate the added value of ops systems like\nKubernetes, Istio/Linkerd, Argo CD, Prometheus, Terraform, etc. until\nyou decide to design a complete architecture yourself, from scratch,\nthe way you want it to work in the long term.</p>\n<h2 id=\"kubernetes-provides-exactly-that\">Kubernetes provides exactly that</h2>\n<p>What system happens to accept Docker images, provide custodianship,\nroll out strategies, and trivial redeploy? Kubernetes.</p>\n<p>It provides this classical monitoring and custodian responsibilities\nthat plenty of other systems have done in the past. However, unlike\nsimply running a process and testing if it’s fine and then turning off\nanother process, Kubernetes buys into Docker all the way.  Processes\nare isolated from each other, in both the network on the file\nsystem. Therefore, you can very reliably start and stop the services\non the same machine. Nothing about a process's machine state is\npersistent, therefore you are forced to design your programs in a way\nthat state is explicitly stored either ephemerally, or elsewhere.</p>\n<!-- Cloud Managed Databases Make This Practical -->\n<p>In the past it might be a little bit scarier to have your database\nrunning in such system, what if it automatically wipes out the\ndatabase process? With today’s cloud base deployments, it's more\ncommon to use a managed database such as that provided by Amazon,\nDigital Ocean, Google or Azure. The whole problem of updating and\nbacking up your database can pretty much be put to one\nside. Therefore, you are free to mess with the configuration or\ntopology of your cluster as much as you like without affecting your\ndatabase.</p>\n<h2 id=\"declarative-is-good-vendor-lock-in-is-bad\">Declarative is good, vendor lock-in is bad</h2>\n<p>A very appealing feature of a deployment system like Kubernetes is\nthat everything is automatic and declarative. You stick all of your\nconfiguration in simple YAML files (which is also a curse because YAML\nhas its own warts and it's not common to find formal schemas for it).\nThis is also known as &quot;infrastructure as code&quot;.</p>\n<p>Ideally, you should have as much as possible about your infrastructure\nin code checked in to a repo so that you can reproduce it and track\nit.</p>\n<p>There is also a much more straight-forward path to migrate from one\nservice provider to another service provider. Kubernetes is supported\non all the major service providers (Google, Amazon, Azure), therefore\nyou are less vulnerable to vendor lock-in. They also all provide\nmanaged databases that are standard (PostgreSQL, for example) with\ntheir normal wire protocols. If you were using the vendor-specific\nAPIs to achieve some of this, you'd be stuck on one vendor. I, for\nexample, am not sure whether to go with Amazon or Azure on a big\npersonal project right now. If I use Kubernetes, I am mitigating risk.</p>\n<p>With something like Terraform you can go one step further, in which\nyou write code that can create your cluster completely from\nscratch. This is also more vendor independent/mitigated.</p>\n<h2 id=\"more-advanced-rollout\">More advanced rollout</h2>\n<p>Your load balancer and your DNS can also be in code. Typically a load\nbalancer that does the job is nginx. However, for more advanced\ndeployments such as A/B or green/blue deployments, you may need\nsomething more advanced like Istio or Linkerd.</p>\n<p>Do I really want to deploy a new feature to all of my users? Maybe,\nthat might be easier. Do I want to deploy a different way of marketing\nmy product on the website to all users at once? If I do that, then I\ndon’t exactly know how effective it is. So, I could perhaps do a\ndeployment in which half of my users see one page and half of the\nusers see another page. These kinds of deployments are\nstraight-forwardly achieved with Istio/Linkerd-type service meshes,\nwithout having to change any code in your app.</p>\n<h2 id=\"relationship-between-code-and-deployed-state\">Relationship between code and deployed state</h2>\n<p>Let's think further than this.</p>\n<p>You've set up your cluster with your provider, or Terraform. You've\nset up your Kubernetes deployments and services. You've set up your CI\nto build your project, produce a Docker image, and upload the images\nto your registry. So far so good.</p>\n<p>Suddenly, you’re wondering, how do I actually deploy this? How do I\ncall Kubernetes, with the correct credentials, to apply this new\nDoctor image to the appropriate deployment?</p>\n<p>Actually, this is still an ongoing area of innovation. An obvious way\nto do it is: you put some details on your CI system that has access to\nrun kubectl, then set the image with the image name and that will try\nto do a deployment. Maybe the deployment fails, you can look at that\nresult in your CI dashboard.</p>\n<p>However, the question comes up as what is currently actually deployed\non production? Do we really have infrastructure as code here?</p>\n<p>It’s not that I edited the file and that update suddenly got\nreflected. There’s no file anywhere in Git that contains what the\ncurrent image is. Head scratcher.</p>\n<p>Ideally, you would have a repository somewhere which states exactly\nwhich image should be deployed right now. And if you change it in a\ncommit, and then later revert that commit, you should expect the\nproduction is also reverted to reflect the code, right?</p>\n<h2 id=\"argocd\">ArgoCD</h2>\n<p>One system which attempts to address this is ArgoCD. They implement\nwhat they call &quot;GitOps&quot;. All state of the system is reflected in a Git\nrepo somewhere. In Argo CD, after your GitHub/Gitlab/Jenkins/Travis CI\nsystem has pushed your Docker image to the Docker repository, it makes\na gRPC call to Argo, which becomes aware of the new image. As an\nadmin, you can now trivially look in the UI and click &quot;Refresh&quot; to\nredeploy the new version.</p>\n<h2 id=\"infra-as-code\">Infra-as-code</h2>\n<p>The common running theme in all of this is\ninfrastructure-as-code. It’s immutability. It’s declarative. It’s\nremoving the number of steps that the human has to do or care\nabout. It’s about being able to rewind. It’s about redundancy. And\nit’s about scaling easily.</p>\n<!-- Circling Back -->\n<p>When you really try to architect your own system, and your business\nwill lose money in the case of ops mistakes, then you start to think\nthat all of these advantages of infrastructure as code start looking\nreally attractive.</p>\n<p>But before you really sit down and think about this stuff, however, it\nis pretty hard to empathise or sympathise with the kind of concerns\nthat people using these systems have.</p>\n<!-- Downsides/Tax -->\n<p>There are some downsides to these tools, as with any:</p>\n<ul>\n<li>Docker is quite wasteful of time and space</li>\n<li>Kubernetes is undoubtedly complex, and leans heavily on YAML</li>\n<li><a href=\"https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/\">All abstractions are leaky</a>,\ntherefore tools like this all leak</li>\n</ul>\n<h2 id=\"where-the-dev-meets-the-ops\">Where the dev meets the ops</h2>\n<p>Now that I’ve started looking into these things and appreciating their\nuse, I interact a lot more with the ops side of our DevOps team at work,\nand I can also be way more helpful in assisting them with the\ninformation that they need, and also writing apps which anticipate the\nkind of deployment that is going to happen. The most difficult\nchallenge typically is metrics and logging, for run-of-the-mill apps,\nI’m not talking about high-performance apps.</p>\n<!-- An Exercise -->\n<p>One way way to bridge the gap between your ops team and dev team,\ntherefore, might be an exercise meeting in which you do have a dev\nperson literally sit down and design an app architecture and\ninfrastructure, from the ground up using the existing tools that we\nhave that they are aware of and then your ops team can point out the\nadvantages and disadvantages of their proposed solution. Certainly,\nI think I would have benefited from such a mentorship, even for an\nhour or two.</p>\n<!-- Head-In-The-Sand Also Works -->\n<p>It may be that your dev team and your ops team are completely separate\nand everybody’s happy. The devs write code, they push it, and then it\nmagically works in production and nobody has any issues. That’s\ncompletely fine. If anything it would show that you have a very good\nprocess. In fact, that’s pretty much how I’ve worked for the past\neight years at this company.</p>\n<p>However, you could derive some benefit if your teams are having\ndifficulty communicating.</p>\n<p>Finally, the tools in the ops world aren't perfect, and they're made\nby us devs. If you have a hunch that you can do better than these\ntools, you should learn more about them, and you might be right.</p>\n<h2 id=\"what-we-do\">What we do</h2>\n<p>FP Complete are using a great number of these tools, and we're writing\nour own, too. If you'd like to know more, email use at\n<a href=\"mailto:sales@fpcomplete.com\">sales@fpcomplete.com</a>.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/",
        "slug": "devops-for-developers",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "DevOps for (Skeptical) Developers",
        "description": null,
        "updated": null,
        "date": "2020-08-16",
        "year": 2020,
        "month": 8,
        "day": 16,
        "taxonomies": {
          "categories": [
            "functional programming",
            "devops"
          ]
        },
        "extra": {
          "author": "Chris Done",
          "blogimage": "/images/blog/devops.jpg"
        },
        "path": "blog/devops-for-developers/",
        "components": [
          "blog",
          "devops-for-developers"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "on-being-skeptical",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#on-being-skeptical",
            "title": "On being skeptical",
            "children": []
          },
          {
            "level": 2,
            "id": "the-humble-app",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#the-humble-app",
            "title": "The humble app",
            "children": []
          },
          {
            "level": 2,
            "id": "disk-failures-aren-t-that-common",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#disk-failures-aren-t-that-common",
            "title": "Disk failures aren't that common",
            "children": []
          },
          {
            "level": 2,
            "id": "auto-deployment-is-better-than-manual",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#auto-deployment-is-better-than-manual",
            "title": "Auto-deployment is better than manual",
            "children": []
          },
          {
            "level": 2,
            "id": "backups-become-worth-it",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#backups-become-worth-it",
            "title": "Backups become worth it",
            "children": []
          },
          {
            "level": 2,
            "id": "deployment-staging",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#deployment-staging",
            "title": "Deployment staging",
            "children": []
          },
          {
            "level": 2,
            "id": "packaging-with-docker-is-good",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#packaging-with-docker-is-good",
            "title": "Packaging with Docker is good",
            "children": []
          },
          {
            "level": 2,
            "id": "custodians-multiple-processes-are-useful",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#custodians-multiple-processes-are-useful",
            "title": "Custodians/multiple processes are useful",
            "children": []
          },
          {
            "level": 2,
            "id": "kubernetes-provides-exactly-that",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#kubernetes-provides-exactly-that",
            "title": "Kubernetes provides exactly that",
            "children": []
          },
          {
            "level": 2,
            "id": "declarative-is-good-vendor-lock-in-is-bad",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#declarative-is-good-vendor-lock-in-is-bad",
            "title": "Declarative is good, vendor lock-in is bad",
            "children": []
          },
          {
            "level": 2,
            "id": "more-advanced-rollout",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#more-advanced-rollout",
            "title": "More advanced rollout",
            "children": []
          },
          {
            "level": 2,
            "id": "relationship-between-code-and-deployed-state",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#relationship-between-code-and-deployed-state",
            "title": "Relationship between code and deployed state",
            "children": []
          },
          {
            "level": 2,
            "id": "argocd",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#argocd",
            "title": "ArgoCD",
            "children": []
          },
          {
            "level": 2,
            "id": "infra-as-code",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#infra-as-code",
            "title": "Infra-as-code",
            "children": []
          },
          {
            "level": 2,
            "id": "where-the-dev-meets-the-ops",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#where-the-dev-meets-the-ops",
            "title": "Where the dev meets the ops",
            "children": []
          },
          {
            "level": 2,
            "id": "what-we-do",
            "permalink": "https://www.fpcomplete.com/blog/devops-for-developers/#what-we-do",
            "title": "What we do",
            "children": []
          }
        ],
        "word_count": 2614,
        "reading_time": 14,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/rust-at-fpco-2020.md",
        "content": "<p>At FP Complete, we have long spoken about the three pillars of a software development language: productivity, robustness, and performance. Often times, these three pillars are in conflict with each other, or at least appear to be. Getting to market quickly (productivity) often involves skimping on quality assurance (robustness), or writing inefficient code (performance). Or you can write simple code which is easy to test and validate (productivity and robustness), but end up with a slow algorithm (performance). Optimizing the code takes time and may introduce new bugs.</p>\n<p>For the entire history of our company, our contention has been that while some level of trade-off here is inevitable, we can leverage better tools, languages, and methodologies to improve our standing on all of these pillars. We initially focused on Haskell, a functional programming language that uses a strong type system and offers decent performance. We still love and continue to use Haskell. However, realizing that code was only half the battle, we then began adopting DevOps methodologies and tools.</p>\n<p>We've watched with great interest as the Rust programming language has developed, matured, and been adopted in industry. Virtually all major technology companies are now putting significant effort behind Rust. Most recently, Microsoft has been <a href=\"https://youtu.be/NQBVUjdkLAA\">quite publicly embracing Rust</a>.</p>\n<p>In this post, I wanted to share some thoughts on why we're thrilled to see Rust's adoption in industry, what we're using Rust for at FP Complete, and give some advice to interested companies in how they can begin adopting this language.</p>\n<h2 id=\"why-rust\">Why Rust?</h2>\n<p>We're big believers in using the computer itself to help us write better code. Some of this can be done with methodologies like test-driven development (TDD). But there are two weak links in the chain of techniques like TDD:</p>\n<ul>\n<li>It requires active effort to think through what needs to be tested</li>\n<li>It's possible to ignore these test failures and ship broken code </li>\n</ul>\n<p>The latter might sound contrived, but we've seen it happen in industry. The limitations of testing are well known, and we've <a href=\"https://www.fpcomplete.com/blog/2016/11/devops-best-practices-multifaceted-testing/\">previously blogged about recommended testing strategies</a>. And don't get me wrong: testing is an absolutely vital part of software development, and you should be doing more of it!</p>\n<p>But industry experience has shown us that many bugs slip through testing. Perhaps the most common and dangerous class of bug is memory safety issues. These include buffer overruns, use-after-free and double-free. What is especially worrying about these classes of bugs is that, typically, the best case scenario is your program crashing. Worst case scenario includes major security and privacy breaches.</p>\n<p>The industry standard approach has been to bypass these bugs by using managed languages. Managed languages bypass explicit memory management and instead rely on garbage collection. This introduces some downsides, latency being the biggest one. Typically, garbage collected languages are more memory hungry as well. This is the typical efficiency-vs-correctness trade-off mentioned above. We've been quite happy to make that trade-off ourselves, using languages like Haskell and accepting some level of performance hit.</p>\n<p>Rust took a different approach, one we admire deeply. By introducing concepts around ownership and borrowing, Rust seeks to drastically reduce the presence of memory safety errors, without introducing the overhead of garbage collection. This fits completely with FP Complete's mindset of using better tools when possible.</p>\n<p>The downside to this is complexity. Understanding ownership can be a challenge. But see below for information on how to get started with Rust. This is an area where FP Complete as a company, and I personally, have taken a lot of interest.</p>\n<p>Going beyond memory safety issues, however, is the rest of the Rust language design. As a relatively new language, Rust has the opportunity to learn from many other languages on the market already. And in our opinion, it has selected some of the best features available from other languages, especially our beloved Haskell. Some of these features include:</p>\n<ul>\n<li>Strong typing</li>\n<li>Sum types (aka enums) and pattern matching</li>\n<li>Explicit error handling, but with a beautiful syntax</li>\n<li><a href=\"https://www.fpcomplete.com/rust/pid1/\">Async syntax</a></li>\n<li>Functional style via closures and <a href=\"https://www.fpcomplete.com/blog/2017/07/iterators-streams-rust-haskell/\"><code>Iterator</code> pipelines</a></li>\n</ul>\n<p>In other words: Rust has fully embraced the concepts of using better approaches to solve problems, and to steal great ideas that have been tried and tested. We believe Rust has the potential to drastically improve software quality in the world, and lead to more maintainable solutions. We think Rust can be instrumental in <a href=\"https://www.fpcomplete.com/blog/2012/12/solving_the_software_crisis/\">solving the global software crisis</a>.</p>\n<h2 id=\"rust-at-fp-complete\">Rust at FP Complete</h2>\n<p>We've taken a three-pronged approach to Rust at FP Complete until now. This has included:</p>\n<ul>\n<li>Producing educational material for both internal and external audiences</li>\n<li>Using Rust for internal tooling</li>\n<li>Writing product code with Rust</li>\n</ul>\n<p>The primary educational offering we've created is our Rust Crash Course, which we'll provide at the end of this post. This course has been honed to address the most common pitfalls we've seen developers hit when onboarding with Rust.</p>\n<p>Also, as a personal project, I decided to see if Rust could be taught as a first programming language, and <a href=\"https://www.beginrust.com/\">I think it can</a>.</p>\n<p>For internal tooling and product code, we always have the debate: should we use Rust or Haskell. We've been giving our engineers more freedom to make that decision themselves in the past year. Personally, I'm still more comfortable with Haskell, which isn't really surprising: I've been using Haskell professionally longer than Rust has existed. But the progress we're seeing in Rust—both in the library ecosystem and the language itself—means that Rust becomes more competitive on an almost monthly basis.</p>\n<p>At this point, we have some specific times when Rust is a clear winner:</p>\n<ul>\n<li>When performance is critical, we prefer Rust. Haskell is usually fast enough, but microoptimizing Haskell code ends up taking more time than writing it in Rust.</li>\n<li>For client-side code (e.g., command line tooling) we've been leaning towards Rust. Overall, it has better cross-OS support than Haskell.</li>\n<li>There are some domains that have much better library coverage in Rust than in Haskell, and then we'll gravitate towards them. (The same applies in the other direction too.)</li>\n<li>And as we're engineers who like playing with shiny tools, if someone wants to have extra fun, Rust is usually it. In most places in the world, Haskell would probably be considered the shiny toy. FP Complete is pretty exceptional there.</li>\n</ul>\n<p>We're beginning to expand to a fourth area of Rust at FP Complete: consulting services. The market for Rust has been steadily growing over the past few years. We believe at this point Rust is ready for much broader adoption, and we're eager to help companies adopt this wonderful language. If you're interested in learning more, please <a href=\"mailto:consulting@fpcomplete.com\">contact our consulting team for more information</a>.</p>\n<h2 id=\"getting-started\">Getting started</h2>\n<p>How do you get started with a language like Rust? Fortunately, the tooling and documentation for Rust is top notch. We can strongly recommend checking out <a href=\"https://www.rust-lang.org/\">the Rust homepage</a> for guidance on installing Rust and getting started. The freely available <a href=\"https://doc.rust-lang.org/book/\">Rust book</a> is great too, covering many aspects of the language.</p>\n<p>That said, my recommendation is to check out our Rust Crash Course eBook (linked below). We've tried to focus this book on answering the most common questions about Rust first, and get you up and running quickly.</p>\n<p>If you're interested in getting your team started with Rust, you may also want to reach out to us for <a href=\"/services/#managedit\">information on our training programs</a>.</p>\n<p>Want to read more about Rust? Check out the <a href=\"https://www.fpcomplete.com/rust/\">FP Complete Rust homepage</a>.</p>\n<p>Want to learn more about FP Complete offerings? Please <a href=\"https://www.fpcomplete.com/contact-us/\">reach out to us any time</a>.</p>\n<!--HubSpot Call-to-Action Code --><span class=\"hs-cta-wrapper\" id=\"hs-cta-wrapper-0e10c2f9-a802-42ef-b462-81162079de11\"><span class=\"hs-cta-node hs-cta-0e10c2f9-a802-42ef-b462-81162079de11\" id=\"hs-cta-0e10c2f9-a802-42ef-b462-81162079de11\"><!--[if lte IE 8]><div id=\"hs-cta-ie-element\"></div><![endif]--><a href=\"https://cta-redirect.hubspot.com/cta/redirect/2814979/0e10c2f9-a802-42ef-b462-81162079de11\" ><img class=\"hs-cta-img\" id=\"hs-cta-img-0e10c2f9-a802-42ef-b462-81162079de11\" style=\"border-width:0px;width:1000px;max-width:100%;height:auto\" src=\"https://no-cache.hubspot.com/cta/default/2814979/0e10c2f9-a802-42ef-b462-81162079de11.png\"  alt=\"Rust-Crash-Course\"/></a></span><script charset=\"utf-8\" src=\"https://js.hscta.net/cta/current.js\"></script><script type=\"text/javascript\"> hbspt.cta.load(2814979, '0e10c2f9-a802-42ef-b462-81162079de11', {}); </script></span><!-- end HubSpot Call-to-Action Code -->\n",
        "permalink": "https://www.fpcomplete.com/blog/rust-at-fpco-2020/",
        "slug": "rust-at-fpco-2020",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Rust at FP Complete, 2020 update",
        "description": "FP Complete has been using Rust for the past few years, and recently we have increased our focus on it. Read about why we think Rust is such an important piece of software going forward.",
        "updated": null,
        "date": "2020-06-29",
        "year": 2020,
        "month": 6,
        "day": 29,
        "taxonomies": {
          "tags": [
            "rust",
            "insights"
          ],
          "categories": [
            "functional programming"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/rust-at-fpco-2020/",
        "components": [
          "blog",
          "rust-at-fpco-2020"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "why-rust",
            "permalink": "https://www.fpcomplete.com/blog/rust-at-fpco-2020/#why-rust",
            "title": "Why Rust?",
            "children": []
          },
          {
            "level": 2,
            "id": "rust-at-fp-complete",
            "permalink": "https://www.fpcomplete.com/blog/rust-at-fpco-2020/#rust-at-fp-complete",
            "title": "Rust at FP Complete",
            "children": []
          },
          {
            "level": 2,
            "id": "getting-started",
            "permalink": "https://www.fpcomplete.com/blog/rust-at-fpco-2020/#getting-started",
            "title": "Getting started",
            "children": []
          }
        ],
        "word_count": 1435,
        "reading_time": 8,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/reducing-maintenance-costs-with-functional-programming.md",
        "content": "<p>Most of the discussions we have in the\nsoftware industry today revolve around developer\nproductivity: How do we make it possible for\nfewer developers to produce more software in less\ntime? Reducing the upfront costs and delivering\nquickly is essentially the mantra of the startup\nworld: Move fast and break things.</p>\n<p>However, the vast majority of time in software\ndevelopment is not spent in the initial\ndevelopment phase. For any successful project, an\noverwhelming amount of time is spent on\nmaintaining that software. To keep your customers\nhappy, it’s vital to continue improving the\nsoftware, fixing bugs and enhancing performance.\nIf you are hamstrung on your ability to innovate,\nconstantly fighting bugs and delivering an\ninferior user experience, your competitors will\nbe able to outmaneuver you in the\nmarketplace.</p>\n<p>Functional programming is a significant\nparadigm shift in the software world over the\npast 10 years. Slowly but surely, it has moved\nfrom a niche feature of a few uncommonly used\nlanguages to a mainstay of even the most\nestablished languages. Unlike preceding\nparadigms, functional programming makes a focus\nof assisting in not just the productivity of\ndevelopers, but of long-term software\nmaintenance.</p>\n<h2 id=\"features-of-functional-programming\">Features of Functional Programming</h2>\n<p>Functional programming is a broad\nterm. Some languages describe themselves\nas functional, such as F#, Haskell and\nSwift. However, functional features are\nmaking their way into other languages.\nJavascript has multiple libraries\nimplementing functional paradigms. Rust,\na relative newcomer in the systems\nprogramming world, boasts many functional\nfeatures. C++ and Java have been adding\nlambdas and other functional features for\nyears. Many of the features below can be\nimplemented regardless of the language\nbeing used by your team.</p>\n<h2 id=\"immutable-data\">Immutable Data</h2>\n<p>The bane of many programs, especially\nconcurrent and network programs, is the\nfact that data changes in unexpected\nways. Functional programming advocates\nkeeping most of your data immutable. Once\ncreated, the data does not change. You\ncan share this data with other parts of\nyour program without fear of it being\nchanged or invalidated.</p>\n<p>Languages like Haskell and Rust make\nthis a cornerstone of their\nimplementation. C++ offers the ability to\nopt-in to immutability. Many Java coding\nguidelines recommend defaulting to\nimmutable data when possible.</p>\n<h2 id=\"declarative-programming\">Declarative Programming</h2>\n<p>Classic programming involves\ninstructing the computer which steps to\ntake to solve a problem. For example, to\nadd up the numbers in a list, an\nimperative programming approach might\nbe:</p>\n<ul>\n<li>Create a temporary variable to hold the sum</li>\n<li>Create a temporary variable to hold the current index</li>\n<li>Loop the index from 0 to the length of the list</li>\n<li>Add the value in the list at the index’s position to the sum</li>\n</ul>\n<p>This kind of imperative approach\nworks, but doesn’t scale particularly\nwell. As problems become more complex,\nthe imperative approach requires ever\nmore complicated solutions. It’s\ndifficult to separate logical components\ninto multiple separate loops without\nsacrificing performance. And in the era\nof multicore programming, creating a\nmultithreaded solution requires\nsignificant expertise with safe thread\nhandling.</p>\n<p>In functional programming, the\npreference is a declarative approach.\nSumming up a list is typically done\nas:</p>\n<ul>\n<li>Write a function to add two values together</li>\n<li>Fold over the list using the add function and 0 as an initial value</li>\n</ul>\n<p>This approach naturally translates\ninto a multicore solution. Instead of\neach loop needing to handle the\ncomplexities of thread management, a\nlibrary author can write a parallel fold\nonce. The caller can then replace their\nnon-parallel fold with a parallel fold\nand immediately gain the benefits of\nmulticore.</p>\n<p>By combining this approach with other\ndeclarative programming methods, like\nmapping, functional programming can\nexpress complex data pipeline operations\nas a composition of many individual,\nsimpler components. This forms the core\nof such well-known systems as Google’s\nMapReduce.</p>\n<h2 id=\"strong-typing\">Strong Typing</h2>\n<p>For years, the industry debate around\ntyped languages was usually between the\nC++ and Java families versus the Python\nand Ruby families. The former introduced\nsome sanity checks at compile time in\nexchange for lots of ceremony with\nexplicit type annotations. This improved\ncode maintenance somewhat, at the cost of\nsignificant developer productivity.\nPython and Ruby, by contrast, skipped the\ntype annotations entirely, leaving them\nas a runtime concern. This boosted\nproductivity, at the cost of\nmaintainability.</p>\n<p>The functional world went a different\nway: strong, expressive type systems with\ntype inference. Type inference avoided\nmuch of the boilerplate introduced by the\nC++-style of type systems, allowing\nproductivity on a par with Python and\nRuby. The strong type systems in\nfunctional languages allowed even more\nguarantees to be expressed in types,\nimproving maintainability beyond the\nlevels of C++ and Java.</p>\n<p>These days, even dynamically typed\nlanguages like Python are beginning to\nintroduce type systems due to the massive\ngains they are demonstrating. New\nlanguages like Rust are borrowing some of\nthe most popular type system features\nfrom functional languages like Haskell\nand O’Caml: sum types, traits and\nmore.</p>\n<h2 id=\"introducing-functional-programming\">Introducing Functional Programming</h2>\n<p>It’s important to note that you do not\nneed to completely rewrite all of your\nsoftware in a functional programming\nlanguage to reap many of the benefits of\nfunctional programming. You can begin\nrolling out functional features in your\nexisting software today with improvements\nto your internal coding guidelines.\nFocusing on some of the features above,\nand many of the other inspirations from\nfunctional programming, is a great\nstart.</p>\n<p>One option is to train your team on\nfunctional programming techniques with an\nintensive training program in a\nfunctional programming language. Once\nyour team knows the concepts, it’s much\neasier to incorporate them in your Java,\nJavascript, C# and other codebases.</p>\n<p>With the rise of microservices\narchitectures, a hybrid deployment model\nmay make a lot of sense. Oftentimes,\noffloading a particularly critical piece\nof business logic to a separate,\nwell-tested functional programming\ncodebase, connected via network APIs, can\nreduce the burden on the rest of your\nteam and increase the stability of your\nsoftware.</p>\n<p><a href=\"https://www.forbes.com/sites/forbestechcouncil/2020/04/10/reducing-maintenance-costs-with-functional-programming/\"><em>Original articles on Forbes</em></a></p>\n",
        "permalink": "https://www.fpcomplete.com/blog/reducing-maintenance-costs-with-functional-programming/",
        "slug": "reducing-maintenance-costs-with-functional-programming",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Reducing Maintenance Costs With Functional Programming",
        "description": "Most of the discussions we have in the software industry today revolve around developer productivity",
        "updated": null,
        "date": "2020-04-10",
        "year": 2020,
        "month": 4,
        "day": 10,
        "taxonomies": {
          "tags": [
            "haskell",
            "insights"
          ],
          "categories": [
            "functional programming"
          ]
        },
        "extra": {
          "author": "Wesley Crook",
          "blogimage": "/images/blog/functional.jpg"
        },
        "path": "blog/reducing-maintenance-costs-with-functional-programming/",
        "components": [
          "blog",
          "reducing-maintenance-costs-with-functional-programming"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "features-of-functional-programming",
            "permalink": "https://www.fpcomplete.com/blog/reducing-maintenance-costs-with-functional-programming/#features-of-functional-programming",
            "title": "Features of Functional Programming",
            "children": []
          },
          {
            "level": 2,
            "id": "immutable-data",
            "permalink": "https://www.fpcomplete.com/blog/reducing-maintenance-costs-with-functional-programming/#immutable-data",
            "title": "Immutable Data",
            "children": []
          },
          {
            "level": 2,
            "id": "declarative-programming",
            "permalink": "https://www.fpcomplete.com/blog/reducing-maintenance-costs-with-functional-programming/#declarative-programming",
            "title": "Declarative Programming",
            "children": []
          },
          {
            "level": 2,
            "id": "strong-typing",
            "permalink": "https://www.fpcomplete.com/blog/reducing-maintenance-costs-with-functional-programming/#strong-typing",
            "title": "Strong Typing",
            "children": []
          },
          {
            "level": 2,
            "id": "introducing-functional-programming",
            "permalink": "https://www.fpcomplete.com/blog/reducing-maintenance-costs-with-functional-programming/#introducing-functional-programming",
            "title": "Introducing Functional Programming",
            "children": []
          }
        ],
        "word_count": 992,
        "reading_time": 5,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/base-on-stackage.md",
        "content": "<h2 id=\"preface-for-unaware\">Preface for unaware</h2>\n<p>When you install a particular version of GHC on your machine it comes with a collection of\n&quot;boot&quot; libraries. What does it mean to be a &quot;boot&quot; library? Quite simply, a library must\nbe used for implementation of GHC and other core components. Two such notable libraries are\n<a href=\"https://www.stackage.org/package/base\"><code>base</code></a> and\n<a href=\"https://www.stackage.org/package/ghc\"><code>ghc</code></a>. All the matching package names and\ntheir versions for a particular GHC release can be found in <a href=\"https://gitlab.haskell.org/ghc/ghc/wikis/commentary/libraries/version-history\">this\ntable</a></p>\n<p>The fact that a library comes wired-in with GHC means that there is never a need to\ndownload sources for the particular version from Hackage or elsewhere. In fact, there is\nreally no need to upload the sources on Hackage even for the purpose of building the\nHaddock for each individual package, since those are conveniently hosted on\n<a href=\"https://downloads.haskell.org/%7Eghc/latest/docs/html/libraries/index.html\">haskell.org</a></p>\n<p>That being said, Hackage has always been a central place for releasing a Haskell package\nand historically Hackage trustees would upload the exact version of almost every &quot;boot&quot;\npackage on Hackage. That is why, for example, we have\n<a href=\"https://hackage.haskell.org/package/bytestring-0.10.8.2\"><code>bytestring-0.10.8.2</code></a> available\non Hackage, despite that it comes with versions of GHC from <code>ghc-8.2.1</code> to <code>ghc-8.6.5</code> inclusive.</p>\n<p>Such an upload makes total sense. Any Haskeller using a core package as a dependency for\ntheir own package in a cabal file has a central place to look for available versions and\ndocumentation for those versions. In fact some people have become so accustomed to this\nprocess that it has been discussed on\n<a href=\"https://mail.haskell.org/pipermail/haskell-cafe/2019-October/131618.html\">Haskell-Cafe</a>\nand a few other places when such package was never uploaded:</p>\n<blockquote>\n<p>It's a crisis that the standard library is unavailable on Hackage...</p>\n</blockquote>\n<h2 id=\"the-problem\">The problem</h2>\n<p>A bit over a half a year ago <code>ghc-8.8.1</code> was released, with current latest one being\n<code>ghc-8.8.3</code>. If you carefully inspect the <a href=\"https://gitlab.haskell.org/ghc/ghc/wikis/commentary/libraries/version-history\">table of core\npackages</a>\nand try to match to available versions on Hackage for those libraries, you will quickly\nnotice that a few of them are missing. I personally don't know the exact reasoning\nbehind this is, but from what I've heard it has something to do with the fact that\n<code>ghc-8.8.1</code> now depends on <code>Cabal-3.0</code>.</p>\n<p>The problem for us is that it also affects Stackage's web interface. Let's see how and why.</p>\n<h2 id=\"the-how\">The &quot;how&quot;</h2>\n<p>The &quot;how&quot; is very simple. Until recently, if a package was missing from Hackage, it would\nnot have been listed on Stackage either. This means that if you tried to follow a\ndependency of any package on <code>base-4.13.0.0</code> in nightly snapshots starting September of\nlast year you would not find it. As I noted before, not only was <code>base</code> missing, but a few\nothers as well.</p>\n<p>This problem also depicted itself in a funny looking bug on Stackage.  For every package\nin a list of dependencies the count was always off by at least 1 when compared with the\nactual links in the list\n(eg. <a href=\"https://www.stackage.org/nightly-2020-01-17/package/primitive-0.7.0.0#dependencies\">primtive</a>). This\nhad me puzzled at first. It was later that I realized that <code>base</code> was missing and since\nalmost every every package depends on it, it was counted, but not listed, causing a\nmismatch.</p>\n<h2 id=\"the-why\">The &quot;why&quot;</h2>\n<p>Stackage was structured in such a way that it always used Hackage as true source of\navailable packages, except for the core packages, since those would always come bundled\nwith GHC. For example if you look at the specification of a latest <a href=\"https://github.com/commercialhaskell/stackage-snapshots/blob/master/lts/15/3.yaml\">LTS-15.3\nsnapshot</a>\nyou will not find any of the core packages listed there, for they are decided by the GHC version, which\nin turn is specified in the snapshot.</p>\n<p>There are a few stages, tools and actual people involved in making a Stackage snapshot\nhappen. Here are some of the steps in the pipeline:</p>\n<ul>\n<li>\n<p>a <a href=\"https://github.com/commercialhaskell/stackage/blob/master/README.md\">curated list of\npackages</a> that involves\npackage maintainers and sometimes Stackage curators.</p>\n</li>\n<li>\n<p>a <a href=\"https://github.com/commercialhaskell/curator/\">curator tool</a> that is used to\nconstruct the actual snapshot, build packages, run test suites and generate Haddocks.</p>\n</li>\n<li>\n<p>a\n<a href=\"https://github.com/fpco/stackage-server/blob/master/app/stackage-server-cron.hs\">stackage-server-cron</a>\ntool that runs at some interval and updates the\n<a href=\"https://www.stackage.org/\">stackage.org</a> database to reflect all of the above work in a\nform of package relations and their respective documentation.</p>\n</li>\n</ul>\n<p>The last step is of the most interest to us because\n<a href=\"https://www.stackage.org/\">stackage.org</a> is the place where we had stuff missing. Let's\nlook at some pieces of information the tool needs in order for <code>stackage-server</code> to create\na page for a package:</p>\n<ul>\n<li>Package name, its version and <a href=\"https://www.fpcomplete.com/blog/2018/07/pantry-part-2-trees-keys\">Pantry\nkeys</a> (cryptographic\nkeys that uniquely identify the contents of source distribution)</li>\n<li>Previously generated haddocks and hoogle files for each package</li>\n<li>Cabal file, so we can extract useful information about the package, such as description,\nlicense, maintainers, module names etc.</li>\n<li>Optionally Readme and Changelog files from the source distribution can be served on a\npackage page as well.</li>\n</ul>\n<p>Information from the latter two bullet points is only available in the source\ndistribution tarballs. Packages that are defined in the snapshot do not pose a problem\nfor us, because by definition their sources are available from Hackage or any of <a href=\"https://hackage.haskell.org/mirrors.json\">its\nmirrors</a>. Core packages on the other hand are\ndifferent, in a sense that they are always available in a build environment, so\ninformation about them is present when we build a package:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">$ stack --resolver lts-15.0 exec -- ghc-pkg describe base\nname:                 base\nversion:              4.13.0.0\nvisibility:           public\n...\n</span></code></pre>\n<p>The problem is that <code>stackage-server-cron</code> tool is just an executable that is running\nsomewhere in a cloud and it doesn't have such environment. Therefore, until recently, we\nhad no means of getting the cabal files for core packages except by checking on\nHackage. With more and more core packages missing from Hackage, especially such critical\nones as <code>base</code> and <code>bytestring</code>, we had to come up with solution.</p>\n<h2 id=\"solution\">Solution</h2>\n<p>Solving this problem should be simple, because all we really need is cabal files. Haddock\nfor missing packages has been generated and was always available, it is the extra little\nbit of the meta information that was needed in order to generate the appropriate links and\nthe package home page.</p>\n<p>The first place to look for cabal files was the GHC git repository. The whole GHC bundle though is\nquite different from all other packages that we are normally used to:</p>\n<ul>\n<li>Libraries that GHC depends on do not come from Hackage, as we already know, instead they\nare pinned as git submodules.</li>\n<li>Most of the packages that are defined in the <a href=\"https://gitlab.haskell.org/ghc/ghc\">GHC\nrepository</a> do not have cabal files. Instead they have\ntemplates that are used for generating cabal files for a particular architecture during\nthe build process.</li>\n</ul>\n<p>This means that the repository is not a good source for grabbing cabal files. Building GHC\nfrom source is a time consuming process and we don't want to be doing that for every\nrelease, just to get cabal files we need. A better alternative is to simply <a href=\"https://www.haskell.org/ghc/download.html\">download a\ndistribution package</a> for a common operating\nsystem and extract the missing cabal files from there. We used Linux x86_64 for Debian,\nbut the choice of the OS shouldn't really matter, since we only really need high level\ninformation from those cabal files.</p>\n<p>That was it. The only thing we really needed to do in order to get missing core files on\nStackage was to collect all missing cabal files and make them available to the\n<code>stackage-server-cron</code> tool</p>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>Going back to the origin of Stackage it turns out that there was quite a few of such core\npackages missing, one most common and most notable one was <code>ghc</code> itself. Only a handful of\nofficially released versions were ever uploaded to Hackage.</p>\n<p>From now on we have a special repository\n<a href=\"https://github.com/commercialhaskell/core-cabal-files\">commercialhaskell/core-cabal-files</a>\nwhere we can place cabal files for missing core packages, which <code>stackage-server-cron</code>\ntool will pick up automatically. As it usually goes with public repositories\nanyone from the community is encouraged to submit pull requests, whenever they notice\nthat a core package is not being listed on Stackage for a newly created snapshot.</p>\n<p>For the past few weeks the very first such missing core package from Hackage\n<a href=\"https://www.stackage.org/lts-15.3/package/base-4.13.0.0\"><code>base-4.13.0.0</code></a> was being\nincluded on Stackage. With recent notable additions being <code>bytestring-0.10.9.0</code>,\n<code>ghc-8.8.x</code> and <code>Cabal-3.0.1.0</code>.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/base-on-stackage/",
        "slug": "base-on-stackage",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Get base onto stackage.org",
        "description": "Solving the problem of missing core packages from Stackage's website.",
        "updated": null,
        "date": "2020-03-11T13:45:00Z",
        "year": 2020,
        "month": 3,
        "day": 11,
        "taxonomies": {
          "tags": [
            "haskell"
          ],
          "categories": [
            "functional programming"
          ]
        },
        "extra": {
          "author": "Alexey Kuleshevich",
          "blogimage": "/images/blog/functional.jpg"
        },
        "path": "blog/base-on-stackage/",
        "components": [
          "blog",
          "base-on-stackage"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "preface-for-unaware",
            "permalink": "https://www.fpcomplete.com/blog/base-on-stackage/#preface-for-unaware",
            "title": "Preface for unaware",
            "children": []
          },
          {
            "level": 2,
            "id": "the-problem",
            "permalink": "https://www.fpcomplete.com/blog/base-on-stackage/#the-problem",
            "title": "The problem",
            "children": []
          },
          {
            "level": 2,
            "id": "the-how",
            "permalink": "https://www.fpcomplete.com/blog/base-on-stackage/#the-how",
            "title": "The \"how\"",
            "children": []
          },
          {
            "level": 2,
            "id": "the-why",
            "permalink": "https://www.fpcomplete.com/blog/base-on-stackage/#the-why",
            "title": "The \"why\"",
            "children": []
          },
          {
            "level": 2,
            "id": "solution",
            "permalink": "https://www.fpcomplete.com/blog/base-on-stackage/#solution",
            "title": "Solution",
            "children": []
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/base-on-stackage/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 1479,
        "reading_time": 8,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/storing-generated-cabal-files.md",
        "content": "<p>tl;dr: I'm moving towards recommending that hpack-using projects store their generated cabal files in their repos, and modifying Stack and Pantry to more strongly recommend this practice. This is a reversal of previous recommendations. <a href=\"https://github.com/commercialhaskell/stack/issues/5210\">Vote and comment on this proposal</a>.</p>\n<h2 id=\"backstory\">Backstory</h2>\n<p>Stack 2.0 switched over to using the Pantry library to manage dependencies. Pantry does a number of things, but at its core it focuses heavily on reproducibility. The idea is that, with a fully qualified package specification, you should always get the same source code. As an example, <code>https://example.com/foo.tar.gz</code> would not be a fully qualified package specification, because the content in that tarball could silently change without being detected. Instead, with Pantry, you would specify something like:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#268bd2;\">size</span><span style=\"color:#657b83;\">: </span><span style=\"color:#6c71c4;\">9526\n</span><span style=\"color:#268bd2;\">url</span><span style=\"color:#657b83;\">: </span><span style=\"color:#268bd2;\">https://github.com/snoyberg/filelock/archive/97e83ecc133cd60a99df8e1fa5a3c2739ad007dc.tar.gz\ncabal-file</span><span style=\"color:#657b83;\">:\n  </span><span style=\"color:#268bd2;\">size</span><span style=\"color:#657b83;\">: </span><span style=\"color:#6c71c4;\">1571\n  </span><span style=\"color:#268bd2;\">sha256</span><span style=\"color:#657b83;\">: </span><span style=\"color:#268bd2;\">d97c2ee2b4f0c72b35cbaf04ad37cda2e9e6a2eb1e162b5c6ab084acb94f4634\nname</span><span style=\"color:#657b83;\">: </span><span style=\"color:#268bd2;\">filelock\nversion</span><span style=\"color:#657b83;\">: </span><span style=\"color:#6c71c4;\">0.1.1.2\n</span><span style=\"color:#268bd2;\">sha256</span><span style=\"color:#657b83;\">: </span><span style=\"color:#268bd2;\">78332e0d964cb2f24fdbb6b07c2a6a84a029c4fe540a0435993c85ad58eab051\npantry-tree</span><span style=\"color:#657b83;\">:\n  </span><span style=\"color:#268bd2;\">size</span><span style=\"color:#657b83;\">: </span><span style=\"color:#6c71c4;\">584\n  </span><span style=\"color:#268bd2;\">sha256</span><span style=\"color:#657b83;\">: </span><span style=\"color:#2aa198;\">19914e8fb09ffe2116cebb8b9d19ab51452594940f1e3770e01357b874c65767\n</span></code></pre>\n<p>Of course, writing these out by hand is tedious and annoying, so Stack uses Pantry to generate these values for you and put them in a lock file.</p>\n<p>Separately: Stack has long supported the ability to include hpack's <code>package.yaml</code> files in your source code, and to automate the generation of a <code>.cabal</code> file. There are two quirks we need to pay attention to with hpack:</p>\n<ul>\n<li>The cabal files it generates change from one version to the next. Some of those changes may be semantically meaningful. At the very least, each new version will stamp a different hpack version in the comments of the cabal file.</li>\n<li>hpack generation is a highly I/O-focused activity, looking at all of the files in a package. Furthermore, as <a href=\"https://github.com/commercialhaskell/stack/issues/4906\">I was recently reminded</a> it can refer to files outside of the specific package you're trying to build but inside the same Git repository or tarball.</li>\n</ul>\n<p>Finally, Stack and Pantry make a stark distinction between two different kinds of packages. <em>Immutable</em> packages are things which we can assume never change. These would be one of the following:</p>\n<ul>\n<li>A package on Hackage, specified by a name, version number, and information on the Hackage revision</li>\n<li>A tarball or ZIP file given by a file path or URL. While these absolutely <em>can</em> change over time, Pantry makes an explicit recommendation that only immutable packages should be used. And the hashes and file size stored in the lock file provide protection against changes.</li>\n<li>A Git or Mercurial repository, specified by a commit.</li>\n</ul>\n<p>On the other hand, mutable packages are packages stored as files on the file system. These are the packages that you are working on in your local project. Reproducibility is far less important here. We allow Stack to regularly check the timestamps and hashes of all of these files and determine when things need to be rebuilt.</p>\n<h2 id=\"the-conflict\">The conflict</h2>\n<p>There's been a debate for a while around how to manage your packages with Stack and hpack. The question is simple: do you store the generated <code>cabal</code> files in the repo? There are solid arguments in both directions:</p>\n<ul>\n<li>You shouldn't store the file, because generated files should in general not be stored in repositories. This can lead to unnecessary diffs, and when people are using different hpack versions, &quot;commit battles&quot; of the file jumping back and forth between different generated content.</li>\n<li>You should store the file, since for maximum reproducibility we want to ensure that we have identical cabal files as input to the build. Also, for people using build tools without built in support for hpack, it's more convenient to have a cabal file present.</li>\n</ul>\n<p>I've had this discussion off and on over the years with many different people, and before Stack 2 had personally settled on the first approach: not storing the cabal files. Then I started working on Pantry.</p>\n<h2 id=\"early-pantry\">Early Pantry</h2>\n<p>Earlier in the development of Pantry, I made a decision to focus on reproducibility. I quickly ran into a problem with hpack: I needed to be able to tell the package name and version of a package easily, but the only code path I had for that was parsing the cabal file. In order to support hpack files for this, I would need to write the entire package contents to the filesystem, run hpack on the resulting directory, and then parse the generated file.</p>\n<p>(I probably could have whipped up something hacky around parsing the hpack YAML file directly, but that felt like a can of worms.)</p>\n<p>Performing these steps each time Stack or Pantry needed to know a package name/version would have been prohibitively expensive, so I dismissed the option. I also considered caching the generated cabal file, but since the generated file contents would change version by version, I didn't follow that path, since it would violate reproducibility.</p>\n<h2 id=\"current-pantry\">Current Pantry</h2>\n<p>An early beta tester of Stack 2.0 complained about this change. While hpack worked perfectly for mutable, local packages, it no longer worked for immutable packages. If you had a Git repository with a package, that repo didn't include the generated cabal file, <em>and</em> you wanted to use that repo as an extra-dep, things would fail. This didn't fail with Stack 1, so this was viewed (correctly) as a regression in functionality.</p>\n<p>However, Stack 2 was aiming for caching and reproducibility goals that Stack 1 hadn't achieved. If anyone remembers, Stack 1 had a bad tendency to reclone Git repos far more often than you would think it should need to. Pantry's caching ultimately solved that problem, and did so by relying on reproducibility.</p>\n<p>My initial recommendation was to require changing all Git repos used as extra-deps to include the generated cabal files. However, after further discussion with beta testers, we ended up changing Pantry instead. We added the ability to cache the generated cabal files (keyed on the version of hpack used). I was uneasy about this, but ultimately it seemed to work fine, and let us keep the functionality we wanted. So we shipped this in Pantry, in Stack 2, and continued recommending people not include generated cabal files.</p>\n<h2 id=\"the-problems-arise\">The problems arise</h2>\n<p>Unfortunately, things were far from rosey. There are now at least three problems I'm aware of with this situation:</p>\n<ul>\n<li>Continuing from before: people using build tools without hpack support are still out of luck with these repos.</li>\n<li>As raised in <a href=\"https://github.com/commercialhaskell/stack/issues/4906\">issue #4906</a>, due to how Pantry handles subdirectories in megarepos, cabal file generation will fail for extra-deps in some cases.</li>\n<li>Lock files have regularly become corrupted by changing generated cabal files. If you use a new version of Stack using a different version of hpack, it will generate a different cabal file, which will change the hashes associated with a package in the lock file. This can cause a lot of frustration between teams, and undermines the whole purpose of lock files in the first place.</li>\n</ul>\n<p>There are probably solutions to the second and third problem. But there's definitely no solution to the first short of including the cabal files again.</p>\n<h2 id=\"changes\">Changes</h2>\n<p>Based on all of this, I'm recommending that we make the following changes:</p>\n<ul>\n<li>Starting immediately: update docs and Stack templates to recommend checking in generated cabal files. This would involve a few minor doc improvements, and removing <code>*.cabal</code> from a few <code>.gitignore</code> files.</li>\n<li>For the next releases of Pantry and Stack, add a warning any time an immutable package does not include a <code>.cabal</code> file. Reference potentially this blog post, and warn that lock files may be broken by doing this.</li>\n<li>Personally: I'll start including the generated cabal files in my repos. Since I have a bunch of them, I'll appreciate people sending PRs to modify my .gitignore files and adding the generated files, as you discover them.</li>\n</ul>\n<p>For those who are truly set against including generated cabal files, all is not lost. For those cases, my recommendation would be pretty simple: keep the generated file out of your repository, and then generate a source tarball with <code>stack sdist</code> to be used as an extra-dep. This will essentially mirror the <code>stack upload</code> step you would follow to upload a package to Hackage.</p>\n<h2 id=\"next-steps\">Next steps</h2>\n<p>The changes necessary to make this a reality are small, and I'm happy to make the changes myself. I'm opening up a short discussion period for this topic, probably around a week, depending on how the discussion goes. If you have an opinion, please jump over to <a href=\"https://github.com/commercialhaskell/stack/issues/5210\">issue #5210</a> and either leave an emoji reaction or a comment.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/storing-generated-cabal-files/",
        "slug": "storing-generated-cabal-files",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Storing generated cabal files",
        "description": "A discussion of some trade-offs in Stack and Pantry's design and user recommendations",
        "updated": null,
        "date": "2020-03-04T08:26:00Z",
        "year": 2020,
        "month": 3,
        "day": 4,
        "taxonomies": {
          "tags": [
            "haskell"
          ],
          "categories": [
            "functional programming"
          ]
        },
        "extra": {
          "author": "Michael Snoyman",
          "blogimage": "/images/blog/functional.jpg"
        },
        "path": "blog/storing-generated-cabal-files/",
        "components": [
          "blog",
          "storing-generated-cabal-files"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "backstory",
            "permalink": "https://www.fpcomplete.com/blog/storing-generated-cabal-files/#backstory",
            "title": "Backstory",
            "children": []
          },
          {
            "level": 2,
            "id": "the-conflict",
            "permalink": "https://www.fpcomplete.com/blog/storing-generated-cabal-files/#the-conflict",
            "title": "The conflict",
            "children": []
          },
          {
            "level": 2,
            "id": "early-pantry",
            "permalink": "https://www.fpcomplete.com/blog/storing-generated-cabal-files/#early-pantry",
            "title": "Early Pantry",
            "children": []
          },
          {
            "level": 2,
            "id": "current-pantry",
            "permalink": "https://www.fpcomplete.com/blog/storing-generated-cabal-files/#current-pantry",
            "title": "Current Pantry",
            "children": []
          },
          {
            "level": 2,
            "id": "the-problems-arise",
            "permalink": "https://www.fpcomplete.com/blog/storing-generated-cabal-files/#the-problems-arise",
            "title": "The problems arise",
            "children": []
          },
          {
            "level": 2,
            "id": "changes",
            "permalink": "https://www.fpcomplete.com/blog/storing-generated-cabal-files/#changes",
            "title": "Changes",
            "children": []
          },
          {
            "level": 2,
            "id": "next-steps",
            "permalink": "https://www.fpcomplete.com/blog/storing-generated-cabal-files/#next-steps",
            "title": "Next steps",
            "children": []
          }
        ],
        "word_count": 1425,
        "reading_time": 8,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/safe-decimal-right-on-the-money.md",
        "content": "<p>Fixed point decimal numbers are used for representing all kinds of data: percentages, temperatures, distances, mass, and many others. I would like to share an approach for safely and efficiently representing currency data in Haskell with <code>safe-decimal</code>.</p>\n<h2 id=\"problems-we-want-to-solve\">Problems we want to solve</h2>\n<h3 id=\"floating-point\">Floating point</h3>\n<p>I wonder how much money gets misplaced because programmers choose a floating point type for representing money. I will not attempt to convince you that using <code>Double</code> or <code>Float</code> for monetary values is unacceptable, it is a known fact. Values like <code>NaN</code>, <code>+/-Infinity</code> and <code>+/-0</code> have no meaning in handling money. In addition, the inability to represent most decimal values exactly should be enough reason to avoid floating point.</p>\n<h3 id=\"fixed-point-decimal\">Fixed point decimal</h3>\n<p>Floating point types make sense when numerical approximation acceptable and you care primarily about performance rather than correctness. This is most common in numerical analysis, signal processing and other areas alike. In many other circumstances a type capable of representing decimal numbers exactly should be used instead. Unlike floating point in a <code>Decimal</code> type we manually restrict how many digits after the decimal point we can have. This is called <a href=\"https://en.wikipedia.org/wiki/Fixed-point_arithmetic\">fixed-point number representation</a>. We use fixed-point numbers on a daily basis when paying in the store with cash or card, tracking distance with an odometer, and reading values off of a digital hydrometer or thermometer.</p>\n<p>We can represent fixed-point decimal numbers in Haskell by using an integral type\nfor the actual value, which is called a precision, and a scale parameter, which is used\nfor keeping track of how far from the right the decimal point is. In <code>safe-decimal</code> we\ndefine a <code>Decimal</code> type that allows us to choose a precision (<code>p</code>) and supply our <code>s</code> scale\nparameter with the type level natural number:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">newtype </span><span style=\"color:#cb4b16;\">Decimal</span><span style=\"color:#657b83;\"> r (s </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Nat</span><span style=\"color:#657b83;\">) p </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Decimal</span><span style=\"color:#657b83;\"> p\n  </span><span style=\"color:#859900;\">deriving</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Ord</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Eq</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">NFData</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Functor</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Generic</span><span style=\"color:#657b83;\">)\n</span></code></pre>\n<p>Unlike floating point numbers we cannot move our decimal point without changing the scaling parameter and sometimes the precision as well. This means that when we use operations like multiplication or division we might have to do some rounding. The rounding strategy is selected at the type level with the <code>r</code> type variable. At time of writing the most common rounding strategies have been implemented: <code>RoundHalfEven</code>, <code>RoundHalfUp</code>, <code>RoundHalfDown</code>, <code>RoundDown</code> and <code>RoundToZero</code>. There is a plan to add more in the future.</p>\n<h3 id=\"precision\">Precision</h3>\n<p>It is common to use a type like <code>Integer</code> for decimal representation, for\nstraightforward reasons:</p>\n<ul>\n<li><code>Integer</code> is easy to use</li>\n<li><code>Integer</code> can represent any number in the universe, if you have enough memory</li>\n</ul>\n<p>Let's look at an example which starts with enabling an extension in Haskell. We need to\nturn on <code>DataKinds</code> so that we can use type level natural numbers.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; :</span><span style=\"color:#657b83;\">set </span><span style=\"color:#859900;\">-</span><span style=\"color:#cb4b16;\">XDataKinds\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Decimal </span><span style=\"color:#6c71c4;\">12345 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Integer\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> x\n</span><span style=\"color:#6c71c4;\">1.2345\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> x * </span><span style=\"color:#6c71c4;\">5\n6.1725\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> roundDecimal (x * </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">3 </span><span style=\"color:#cb4b16;\">Integer\n</span><span style=\"color:#6c71c4;\">6.173\n</span></code></pre>\n<p>The concrete <code>Decimal</code> type backed by <code>Integer</code> has a <code>Num</code> instance. That is why we\nwere able to use literal <code>5</code> and GHC converted it to a <code>Decimal</code> for us. This is how the same numbers multiplied together look as <code>Double</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">1.2345</span><span style=\"color:#657b83;\"> * </span><span style=\"color:#6c71c4;\">5 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Double\n</span><span style=\"color:#6c71c4;\">6.172499999999999\n</span></code></pre><h3 id=\"storage-and-performance\">Storage and Performance</h3>\n<p><code>Integer</code> is nice, but in some applications <code>Integer</code> isn't an acceptable representation of our data. We might need to store decimal values in database, transmit them over the network, or improve performance by storing numbers in an unboxed instead of boxed array. It is faster to store a 64-bit integer value in a database rather than converting a number to a sequence of bytes in a blob as is necessary with <code>Integer</code>. Transmission over a network is another limitation that comes to mind. Having a 508 byte limit on a UDP packet can quickly become a problem for <code>Integer</code> based values.</p>\n<p>The best way to solve this is to use fixed width integer types such as <code>Int64</code>, <code>Int32</code>, <code>Word64</code>, etc. If precision of more than 64 bits is desired there are packages that provide 128-bit, 256-bit, and other variants of signed/unsigned integers. All of them can be used\nwith <code>safe-decimal</code>, eg:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#cb4b16;\">import </span><span style=\"color:#859900;\">Data.Int </span><span style=\"color:#657b83;\">(</span><span style=\"color:#268bd2;\">Int8</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Int64</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#cb4b16;\">Decimal </span><span style=\"color:#6c71c4;\">12345 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">6 </span><span style=\"color:#cb4b16;\">Int64\n</span><span style=\"color:#6c71c4;\">0.012345\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#cb4b16;\">Decimal </span><span style=\"color:#6c71c4;\">123 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">6 </span><span style=\"color:#cb4b16;\">Int8\n</span><span style=\"color:#6c71c4;\">0.000123\n</span></code></pre><h3 id=\"bounds\">Bounds</h3>\n<p>Even discarding the desire for better performance and ignoring the memory constraints imposed on us, there are often types that have domain-specific bounds anyway. The most common example is when people use signed types like <code>Int</code> to represent values that have no sensible negative value. Use unsigned types like <code>Word</code> for representing values that should have no negative value.</p>\n<p>Some values that can be represented by a decimal number have a lower and upper bound that we estimate. Percentages go from 0% to a 100%, the total circulation of US dollars is about 14 trillion, and the surface temperature of a star is somewhere in a range of 225-40000K. If we use our domain specific knowledge we can come up with some safe bounds, instead of blindly assuming that we need infinitely large values.</p>\n<p>Beware though, that using integral types with bounds come with real danger: integer <a href=\"https://cwe.mitre.org/data/definitions/190.html\">overflow</a> and <a href=\"https://cwe.mitre.org/data/definitions/191.html\">underflow</a>. These are common reasons for bugs in software that lead to a whole variety of exploits. This is the area where protection in <code>safe-decimal</code> really shines, and here is an example of how it protects you:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">123 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Int8\n</span><span style=\"color:#6c71c4;\">127\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">123 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">5 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Int8\n</span><span style=\"color:#859900;\">-</span><span style=\"color:#6c71c4;\">128\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Decimal </span><span style=\"color:#6c71c4;\">123 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">6 </span><span style=\"color:#cb4b16;\">Int8\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> x\n</span><span style=\"color:#6c71c4;\">0.000123\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> plusDecimalBounded x (</span><span style=\"color:#cb4b16;\">Decimal </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Maybe</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">6 </span><span style=\"color:#cb4b16;\">Int8</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#cb4b16;\">Just </span><span style=\"color:#6c71c4;\">0.000127\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> plusDecimalBounded x (</span><span style=\"color:#cb4b16;\">Decimal </span><span style=\"color:#6c71c4;\">5</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Maybe</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">6 </span><span style=\"color:#cb4b16;\">Int8</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#cb4b16;\">Nothing\n</span></code></pre><h3 id=\"runtime-exceptions\">Runtime exceptions</h3>\n<p>We know that division by zero will result in <code>DivideByZero</code> exception:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">`div` </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Int</span><span style=\"color:#657b83;\">\n*** </span><span style=\"color:#cb4b16;\">Exception</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\"> divide by zero\n</span></code></pre>\n<p>Less well known is that while some integral operations result in silent overflows, others will cause runtime exceptions:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; -</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> * minBound </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Int\n</span><span style=\"color:#859900;\">-</span><span style=\"color:#6c71c4;\">9223372036854775808\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">`div`</span><span style=\"color:#657b83;\"> minBound </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Int\n</span><span style=\"color:#859900;\">-</span><span style=\"color:#6c71c4;\">1\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> minBound </span><span style=\"color:#859900;\">`div`</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#859900;\">-</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Int</span><span style=\"color:#657b83;\">\n*** </span><span style=\"color:#cb4b16;\">Exception</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\"> arithmetic overflow\n</span></code></pre>\n<p>Floating point values also have a sad story for division by zero. You'd be surprised how often you can stumble upon those values online:</p>\n<img src=\"/images/blog/safe-decimal.jpeg\" style=\"max-width:95%\">\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">/ </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Double\nNaN\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">/ </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Double\nInfinity\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; -</span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">/ </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Double\n</span><span style=\"color:#859900;\">-</span><span style=\"color:#cb4b16;\">Infinity\n</span></code></pre>\n<p>Long story short we want to be able to prevent all these issues from within pure code.\nWhich is exactly what <code>safe-decimal</code> will do for you:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; -</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> * pure minBound </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Int</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#cb4b16;\">ArithError</span><span style=\"color:#657b83;\"> arithmetic overflow\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> pure minBound </span><span style=\"color:#859900;\">/</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#859900;\">-</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Int</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#cb4b16;\">ArithError</span><span style=\"color:#657b83;\"> arithmetic overflow\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">/ </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Int</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#cb4b16;\">ArithError</span><span style=\"color:#657b83;\"> divide by zero\n</span></code></pre>\n<p><code>Arith</code> is a monad defined in <code>safe-decimal</code> and is used for working with arithmetic\noperations that can fail for any particular reason. It is isomorphic to <code>Either SomeException</code>, which means there is straightforward conversion from <code>Arith</code> monad to\nothers that have <code>MonadThrow</code> instance with <code>arithM</code> and a few other helper functions:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> arithM (</span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">/ </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Int</span><span style=\"color:#657b83;\">))\n*** </span><span style=\"color:#cb4b16;\">Exception</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\"> divide by zero\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> arithMaybe (</span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">/ </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Int</span><span style=\"color:#657b83;\">))\n</span><span style=\"color:#cb4b16;\">Nothing\n</span></code></pre><h2 id=\"decimal-for-crypto\">Decimal for crypto</h2>\n<p>At the beginning of the post I mentioned that we will implement a currency. Everyone seems to be implementing cryptocurrencies nowadays, so why don't we do the same?</p>\n<p>The most popular cryptocurrency at time of writing is Bitcoin, so we'll use it for this\nexample. A few assumptions we are going to make before we start:</p>\n<ul>\n<li>The maximum amount is 21M BTC</li>\n<li>No negative amounts are allowed</li>\n<li>Precision is up to 8 decimal places</li>\n<li>Smallest expressible value is 0.00000001 BTC, which is one Satoshi. It is named after the pseudonymous Satoshi Nakamoto who published the seminal Bitcoin paper.</li>\n</ul>\n<h3 id=\"definition\">Definition</h3>\n<p>Here we'll demonstrate how we can represent Bitcoin with <code>safe-decimal</code> and in case if you\nwould like to follow along here is <a href=\"https://gist.github.com/lehins/98d835b51c15270c2a600135b64d474d\">the\ngist</a> with all of the\ncode presented in this blogpost. First, we declare the raw amount <code>Satoshi</code> that will be\nused, so we can specify its bounds. Following that is the <code>Bitcoin</code> wrapper around the\n<code>Decimal</code> that specifies all we need to know in order to operate on this currency:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">{-# </span><span style=\"color:#859900;\">LANGUAGE</span><span style=\"color:#b58900;\"> DataKinds #-}\n{-# </span><span style=\"color:#859900;\">LANGUAGE</span><span style=\"color:#b58900;\"> NumericUnderscores #-}\n{-# </span><span style=\"color:#859900;\">LANGUAGE</span><span style=\"color:#b58900;\"> GeneralizedNewtypeDeriving #-}\n\n</span><span style=\"color:#859900;\">module Bitcoin </span><span style=\"color:#657b83;\">(</span><span style=\"color:#268bd2;\">Bitcoin</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">where\n\n</span><span style=\"color:#cb4b16;\">import </span><span style=\"color:#859900;\">Data.Word\n</span><span style=\"color:#cb4b16;\">import </span><span style=\"color:#859900;\">Numeric.Decimal\n</span><span style=\"color:#cb4b16;\">import </span><span style=\"color:#859900;\">Data.Coerce\n\nnewtype </span><span style=\"color:#cb4b16;\">Satoshi </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Satoshi Word64 </span><span style=\"color:#859900;\">deriving</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Show</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Eq</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Ord</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Enum</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Num</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Real</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Integral</span><span style=\"color:#657b83;\">)\n\n</span><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">Bounded Satoshi </span><span style=\"color:#859900;\">where</span><span style=\"color:#657b83;\">\n  minBound </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Satoshi </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\">\n  maxBound </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Satoshi</span><span style=\"color:#657b83;\"> 21_000_000_00000000\n\n</span><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">NoRounding\n\n</span><span style=\"color:#859900;\">type </span><span style=\"color:#cb4b16;\">BitcoinDecimal </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Decimal NoRounding </span><span style=\"color:#6c71c4;\">8 </span><span style=\"color:#cb4b16;\">Satoshi\n\n</span><span style=\"color:#859900;\">newtype </span><span style=\"color:#cb4b16;\">Bitcoin </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Bitcoin BitcoinDecimal </span><span style=\"color:#859900;\">deriving</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Eq</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Ord</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Bounded</span><span style=\"color:#657b83;\">)\n\n</span><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">Show Bitcoin </span><span style=\"color:#859900;\">where</span><span style=\"color:#657b83;\">\n  show (</span><span style=\"color:#cb4b16;\">Bitcoin</span><span style=\"color:#657b83;\"> b) </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> show b\n</span></code></pre>\n<p>Important parts of these definitions are:</p>\n<ul>\n<li>We are using a newtype wrapper around <code>Word64</code> with custom bounds, so that the library\ncan protect us from creating an invalid value. Using <code>Int64</code> would have not made a\ndifference in this case, but using another type with less available bits would not be\nenough to hold large values.</li>\n<li>We define no rounding strategy to make sure that at no point rounding could cause money\nto appear or disappear.</li>\n<li>We do not export the constructor for <code>Bitcoin</code> type to ensure that invalid values cannot\nbe constructed manually. Smart constructors will follow below, which can be exported if\nneeded.</li>\n</ul>\n<h3 id=\"construction-and-arithmetic\">Construction and arithmetic</h3>\n<p>Helper functions that do zero cost coercions from <code>Data.Coerce</code> will be used to go between\ntypes without making us repeat their signatures.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">toBitcoin </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">BitcoinDecimal </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">Bitcoin\n</span><span style=\"color:#657b83;\">toBitcoin </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> coerce\n\n</span><span style=\"color:#b58900;\">fromBitcoin </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">Bitcoin </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">BitcoinDecimal\n</span><span style=\"color:#657b83;\">fromBitcoin </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> coerce\n\n</span><span style=\"color:#b58900;\">mkBitcoin </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">MonadThrow m </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#268bd2;\">Rational </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">m Bitcoin\n</span><span style=\"color:#657b83;\">mkBitcoin r </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Bitcoin </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> fromRationalDecimalBoundedWithoutLoss r\n\n</span><span style=\"color:#b58900;\">plusBitcoins </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">MonadThrow m </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#268bd2;\">Bitcoin </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">Bitcoin </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">m Bitcoin\n</span><span style=\"color:#657b83;\">plusBitcoins b1 b2 </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> toBitcoin </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> (fromBitcoin b1 </span><span style=\"color:#859900;\">`plusDecimalBounded`</span><span style=\"color:#657b83;\"> fromBitcoin b2)\n\n</span><span style=\"color:#b58900;\">minusBitcoins </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">MonadThrow m </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#268bd2;\">Bitcoin </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">Bitcoin </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">m Bitcoin\n</span><span style=\"color:#657b83;\">minusBitcoins b1 b2 </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> toBitcoin </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> (fromBitcoin b1 </span><span style=\"color:#859900;\">`minusDecimalBounded`</span><span style=\"color:#657b83;\"> fromBitcoin b2)\n</span></code></pre>\n<p><code>mkBitcoin</code> gives us a way to construct new values, while giving us a freedom to choose\nthe monad in which we want to fail by restricting to <code>MonadThrow</code>, for simplicity we'll\nstick to <code>IO</code>, but it could just as well be <code>Maybe</code>, <code>Either</code>, <code>Arith</code> and many others.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">&gt;&gt;&gt; mkBitcoin 1.23\n1.23000000\n&gt;&gt;&gt; mkBitcoin (-1.23)\n*** Exception: arithmetic underflow\n</span></code></pre>\n<p>Examples below make it obvious that we are guarded from constructing invalid values from\n<code>Rational</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; :</span><span style=\"color:#657b83;\">set </span><span style=\"color:#859900;\">-</span><span style=\"color:#cb4b16;\">XNumericUnderscores\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> mkBitcoin 21_000_000</span><span style=\"color:#859900;\">.</span><span style=\"color:#6c71c4;\">00000000\n21000000.00000000\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> mkBitcoin 21_000_000</span><span style=\"color:#859900;\">.</span><span style=\"color:#6c71c4;\">00000001</span><span style=\"color:#657b83;\">\n*** </span><span style=\"color:#cb4b16;\">Exception</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\"> arithmetic overflow\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> mkBitcoin </span><span style=\"color:#6c71c4;\">0.123456789</span><span style=\"color:#657b83;\">\n*** </span><span style=\"color:#cb4b16;\">Exception</span><span style=\"color:#859900;\">: </span><span style=\"color:#cb4b16;\">PrecisionLoss</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#6c71c4;\">123456789 </span><span style=\"color:#859900;\">% </span><span style=\"color:#6c71c4;\">1000000000</span><span style=\"color:#657b83;\">) to </span><span style=\"color:#6c71c4;\">8</span><span style=\"color:#657b83;\"> decimal spaces\n</span></code></pre>\n<p>Same logic goes for operating on <code>Bitcoin</code> values. Nothing gets past, any operation that\ncould produce an invalid value will result in a failure.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> balance </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> mkBitcoin </span><span style=\"color:#6c71c4;\">10.05\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> receiveAmount </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> mkBitcoin </span><span style=\"color:#6c71c4;\">2.345\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> plusBitcoins balance receiveAmount\n</span><span style=\"color:#6c71c4;\">12.39500000\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> maliciousReceiveBitcoin </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> mkBitcoin </span><span style=\"color:#6c71c4;\">20999990.0\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> plusBitcoins balance maliciousReceiveBitcoin\n*** </span><span style=\"color:#cb4b16;\">Exception</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\"> arithmetic overflow\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> arithEither </span><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\"> plusBitcoins balance maliciousReceiveBitcoin\n</span><span style=\"color:#cb4b16;\">Left</span><span style=\"color:#657b83;\"> arithmetic overflow\n</span></code></pre>\n<p>Subtracting values is handled in the same fashion. Note that going below a lower bound\nwill be reported as underflow, which, contrary to popular belief, is a real term not only\nfor floating points, but for integers as well.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> balance </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> mkBitcoin </span><span style=\"color:#6c71c4;\">10.05\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> sendAmount </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> mkBitcoin </span><span style=\"color:#6c71c4;\">1.01\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> balance </span><span style=\"color:#859900;\">`minusBitcoins`</span><span style=\"color:#657b83;\"> sendAmount\n</span><span style=\"color:#6c71c4;\">9.04000000\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> sendAmountTooMuch </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> mkBitcoin </span><span style=\"color:#6c71c4;\">11.01\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> balance </span><span style=\"color:#859900;\">`minusBitcoins`</span><span style=\"color:#657b83;\"> sendAmountTooMuch\n*** </span><span style=\"color:#cb4b16;\">Exception</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\"> arithmetic underflow\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> sendAmountMalicious </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> mkBitcoin </span><span style=\"color:#6c71c4;\">184467440737.09551616</span><span style=\"color:#657b83;\">\n*** </span><span style=\"color:#cb4b16;\">Exception</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\"> arithmetic overflow\n</span></code></pre>\n<p>I would like to emphasize in the example above the fact that we did not have to check if\n<code>balance</code> was sufficient enough for the amounts to be fully deducted from it. This means\nwe are automatically protected from incorrect transactions as well as <a href=\"https://consensys.github.io/smart-contract-best-practices/known_attacks/#integer-overflow-and-underflow\">very common attack\nvectors</a>,\nsome of which <a href=\"https://medium.com/@jeancvllr/the-value-overflow-incident-in-the-bitcoin-blockchain-15th-august-2010-a59a516e03db\">really did happen with\nBitcoin</a>\nand other cryptocurrencies.</p>\n<h3 id=\"num-and-fractional\">Num and Fractional</h3>\n<p>Using a special smart constructor is cool and all, but it would be cooler if we could use\nour regular math operators to work with <code>Bitcoin</code> values and utilize GHC desugarer to\nautomatically convert numeric literal values too. For this we need instances of <code>Num</code>\nand <code>Fractional</code>. We can't create instances like that:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">Num Bitcoin </span><span style=\"color:#859900;\">where\n...\ninstance </span><span style=\"color:#268bd2;\">Fractional Bitcoin </span><span style=\"color:#859900;\">where\n...\n</span></code></pre>\n<p>because then we would have to use partial functions for failures, which is exactly what we\nwant to avoid. Moreover some functions simply do no make sense for monetary\nvalues. Multiplying or dividing Bitcoins together, is simply undefined. We'll\nhave to represent a special type of failure through an exception. This is a bit unfortunate, but we'll go with it anyways:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">UnsupportedOperation </span><span style=\"color:#859900;\">=\n  </span><span style=\"color:#cb4b16;\">UnsupportedMultiplication </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">UnsupportedDivision\n  </span><span style=\"color:#859900;\">deriving </span><span style=\"color:#cb4b16;\">Show\n\n</span><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">Exception UnsupportedOperation\n\n</span><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">Num</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Arith Bitcoin</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">where\n  </span><span style=\"color:#b58900;\">(+) </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> bindM2 plusBitcoins\n  </span><span style=\"color:#b58900;\">(-) </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> bindM2 minusBitcoins\n  </span><span style=\"color:#b58900;\">(*) </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> bindM2 (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">_ _ </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> throwM </span><span style=\"color:#cb4b16;\">UnsupportedMultiplication</span><span style=\"color:#657b83;\">)\n  abs </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> id\n  signum mb </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> fmap toBitcoin </span><span style=\"color:#859900;\">.</span><span style=\"color:#657b83;\"> signumDecimalBounded </span><span style=\"color:#859900;\">.</span><span style=\"color:#657b83;\"> fromBitcoin </span><span style=\"color:#859900;\">=&lt;&lt;</span><span style=\"color:#657b83;\"> mb\n  fromInteger i </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> toBitcoin </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> fromIntegerDecimalBoundedIntegral i\n\n</span><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">Fractional</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Arith Bitcoin</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">where\n  </span><span style=\"color:#b58900;\">(/) </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> bindM2 (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">_ _ </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> throwM </span><span style=\"color:#cb4b16;\">UnsupportedDivision</span><span style=\"color:#657b83;\">)\n  fromRational </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> mkBitcoin\n</span></code></pre>\n<p>It is important to note that defining the instances above is strictly optional and exporting helper functions that perform the same operations is preferable. We have the instances now so we can demonstrate their use:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">7.8 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">10 </span><span style=\"color:#859900;\">- </span><span style=\"color:#6c71c4;\">0.4 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith Bitcoin\nArith </span><span style=\"color:#6c71c4;\">17.40000000\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">7.8 </span><span style=\"color:#859900;\">- </span><span style=\"color:#6c71c4;\">10 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">0.4 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith Bitcoin\nArithError</span><span style=\"color:#657b83;\"> arithmetic underflow\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">7.8</span><span style=\"color:#657b83;\"> * </span><span style=\"color:#6c71c4;\">10 </span><span style=\"color:#859900;\">/ </span><span style=\"color:#6c71c4;\">0.4 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith Bitcoin\nArithError UnsupportedMultiplication\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">7.8 </span><span style=\"color:#859900;\">/ </span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\"> * </span><span style=\"color:#6c71c4;\">0.4 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith Bitcoin\nArithError UnsupportedDivision\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">7.8 </span><span style=\"color:#859900;\">- </span><span style=\"color:#6c71c4;\">7.7 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">0.4 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith Bitcoin\nArith </span><span style=\"color:#6c71c4;\">0.50000000\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">0.4 </span><span style=\"color:#859900;\">- </span><span style=\"color:#6c71c4;\">7.7 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">7.8 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Arith Bitcoin\nArithError</span><span style=\"color:#657b83;\"> arithmetic underflow\n</span></code></pre>\n<p>The order of operations can play tricks on you, which probably serves as another reason to stick to exporting functions: <code>mkBitcoin</code>, <code>plusBitcoins</code>, <code>minusBitcoins</code> and whatever other operations we might need.</p>\n<p>Let's take a look at a more realistic example where the amount sent is supplied to us as a <code>Scientific</code> value likely from some JSON object and we want to update the balance of our account. For simplicity's sake I will use a <code>State</code> monad, but same approach will work just as well with whatever stateful setup you have.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">newtype </span><span style=\"color:#cb4b16;\">Balance </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Balance Bitcoin </span><span style=\"color:#859900;\">deriving </span><span style=\"color:#cb4b16;\">Show\n\n</span><span style=\"color:#b58900;\">sendBitcoin </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">MonadThrow m </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#268bd2;\">Balance </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">Scientific </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">m</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Bitcoin</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">Balance</span><span style=\"color:#657b83;\">)\nsendBitcoin startingBalance rawAmount </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\">\n  flip runStateT startingBalance </span><span style=\"color:#859900;\">$ do</span><span style=\"color:#657b83;\">\n    amount </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> toBitcoin </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> fromScientificDecimalBounded rawAmount\n    </span><span style=\"color:#cb4b16;\">Balance</span><span style=\"color:#657b83;\"> balance </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> get\n    newBalance </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> minusBitcoins balance amount\n    put </span><span style=\"color:#859900;\">$ </span><span style=\"color:#cb4b16;\">Balance</span><span style=\"color:#657b83;\"> newBalance\n    pure amount\n</span></code></pre>\n<p>Usage of this simple function will demonstrate us the power of the approach taken in the\nlibrary as well as its limitations:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> balance </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> mkBitcoin </span><span style=\"color:#6c71c4;\">10.05\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> sendBitcoin (</span><span style=\"color:#cb4b16;\">Balance</span><span style=\"color:#657b83;\"> balance) </span><span style=\"color:#6c71c4;\">0.5</span><span style=\"color:#657b83;\">\n(</span><span style=\"color:#6c71c4;\">0.50000000</span><span style=\"color:#657b83;\">,</span><span style=\"color:#cb4b16;\">Balance </span><span style=\"color:#6c71c4;\">9.55000000</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> sendBitcoin (</span><span style=\"color:#cb4b16;\">Balance</span><span style=\"color:#657b83;\"> balance) </span><span style=\"color:#6c71c4;\">1e-6</span><span style=\"color:#657b83;\">\n(</span><span style=\"color:#6c71c4;\">0.00000100</span><span style=\"color:#657b83;\">,</span><span style=\"color:#cb4b16;\">Balance </span><span style=\"color:#6c71c4;\">10.04999900</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> sendBitcoin (</span><span style=\"color:#cb4b16;\">Balance</span><span style=\"color:#657b83;\"> balance) </span><span style=\"color:#6c71c4;\">1e+6</span><span style=\"color:#657b83;\">\n*** </span><span style=\"color:#cb4b16;\">Exception</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\"> arithmetic underflow\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> arithEither </span><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\"> sendBitcoin (</span><span style=\"color:#cb4b16;\">Balance</span><span style=\"color:#657b83;\"> balance) (</span><span style=\"color:#859900;\">-</span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#cb4b16;\">Left</span><span style=\"color:#657b83;\"> arithmetic underflow\n</span></code></pre>\n<p>We witness <code>Overflow</code>/<code>Underflow</code> errors as expected, but we get almost no information on where exactly the problem occurred and which value was responsible for it. This is something that can be fixed with customized exceptions, but for now we do achieve the most important goal, namely protecting our calculations from all the dangerous problems without doing any explicit checking.</p>\n<p>Nowhere in <code>sendBitcoin</code> did we have to validate our input, output, or intermediate values. Not a single <code>if then else</code> statement. This is because all of the information needed to determine the validity of the above operations was encoded into the type and the library enforces that validity for the programmer.</p>\n<h3 id=\"mixing-decimal-types\">Mixing Decimal types</h3>\n<p>Although multiplying two <code>Bitcoin</code> values makes no sense, computing the product of an amount and a percentage makes perfect sense. So, how do we go about multiplying different decimals together?</p>\n<p>While demonstrating interoperability of different decimal types we'd like to also show how higher precision integrals can be used with <code>Decimal</code>. In this example we'll use a <code>Word128</code> backed <code>Decimal</code> for computing <a href=\"https://en.wikipedia.org/wiki/Future_value\">future value</a>. There are a couple of packages that provide 128-bit integral types and it doesn't matter which one it comes from.</p>\n<p>Our goal is to compute the savings account balance at 1.9% APY (Annual Percentage Yield) in 30 days if you start with 10,000 BTC and add 10 BTC each day.</p>\n<p>We will start by defining the rounding strategy implementation for the <code>Word128</code> type and\nspecifying the <code>Decimal</code> type we will be using for computation:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">Round RoundHalfUp Word128 </span><span style=\"color:#859900;\">where</span><span style=\"color:#657b83;\">\n  roundDecimal </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> roundHalfUp\n\n</span><span style=\"color:#859900;\">type </span><span style=\"color:#cb4b16;\">CDecimal </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Decimal RoundHalfUp </span><span style=\"color:#6c71c4;\">33 </span><span style=\"color:#cb4b16;\">Word128\n</span></code></pre>\n<p>This is not the implementation of <code>FV</code> (Future Value) function as it is known in finance. It is a direct translation of how we think the accrual of interest works. In plain English we can say that to compute balance of the account tomorrow, we take balance we have today, multiply it by the daily interest rate and add it to the today's balance together with the amount we promised to top up daily.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">futureValue </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">MonadThrow m </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#268bd2;\">CDecimal </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">CDecimal </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">CDecimal </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">Int </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">m CDecimal\n</span><span style=\"color:#657b83;\">futureValue startBalance dailyRefill apy days </span><span style=\"color:#859900;\">= do</span><span style=\"color:#657b83;\">\n  dailyScale </span><span style=\"color:#859900;\">&lt;- </span><span style=\"color:#93a1a1;\">-- apy is in % and the year of 2020 is a leap year\n</span><span style=\"color:#657b83;\">    fromIntegralDecimalBounded (</span><span style=\"color:#6c71c4;\">100</span><span style=\"color:#657b83;\"> * </span><span style=\"color:#6c71c4;\">366</span><span style=\"color:#657b83;\">)\n  dailyRate </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> divideDecimalBoundedWithRounding apy dailyScale\n  </span><span style=\"color:#859900;\">let</span><span style=\"color:#657b83;\"> go curBalance day\n        </span><span style=\"color:#859900;\">|</span><span style=\"color:#657b83;\"> day </span><span style=\"color:#859900;\">&lt;</span><span style=\"color:#657b83;\"> days </span><span style=\"color:#859900;\">= do</span><span style=\"color:#657b83;\">\n          accruedDaily </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> timesDecimalBoundedWithRounding curBalance dailyRate\n          nextDayBalance </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> sumDecimalBounded [curBalance, accruedDaily, dailyRefill]\n          go nextDayBalance (day </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">)\n        </span><span style=\"color:#859900;\">|</span><span style=\"color:#657b83;\"> otherwise </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> pure curBalance\n  go startBalance </span><span style=\"color:#6c71c4;\">0\n</span></code></pre>\n<p>The above implementation works on the <code>CDecimal</code> type. What we need to calculate is <code>Bitcoin</code>. This means we have to do some type conversions and scaling in order to match up the types of <code>futureValue</code> function. Then we do some rounding and conversion again to reduce precision to obtain the new <code>Balance</code>:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">futureValueBitcoin </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">MonadThrow m </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#268bd2;\">Balance </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">Bitcoin </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">Rational </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">Int </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">m</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Balance</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">CDecimal</span><span style=\"color:#657b83;\">)\nfutureValueBitcoin (</span><span style=\"color:#cb4b16;\">Balance</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Bitcoin</span><span style=\"color:#657b83;\"> balance)) (</span><span style=\"color:#cb4b16;\">Bitcoin</span><span style=\"color:#657b83;\"> dailyRefill) apy days </span><span style=\"color:#859900;\">= do</span><span style=\"color:#657b83;\">\n  balance&#39; </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> scaleUpBounded (fromIntegral </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> castRounding balance)\n  dailyRefill&#39; </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> scaleUpBounded (fromIntegral </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> castRounding dailyRefill)\n  apy&#39; </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> fromRationalDecimalBoundedWithoutLoss apy\n  endBalance </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> futureValue balance&#39; dailyRefill&#39; apy&#39; days\n  endBalanceRounded </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> integralDecimalToDecimalBounded (roundDecimal endBalance)\n  pure (</span><span style=\"color:#cb4b16;\">Balance </span><span style=\"color:#859900;\">$ </span><span style=\"color:#cb4b16;\">Bitcoin </span><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\"> castRounding endBalanceRounded, endBalance)\n</span></code></pre>\n<p>Now we can compute what our balance will be in 30 days:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">computeBalance </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">Arith</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Balance</span><span style=\"color:#657b83;\">, </span><span style=\"color:#268bd2;\">CDecimal</span><span style=\"color:#657b83;\">)\ncomputeBalance </span><span style=\"color:#859900;\">= do</span><span style=\"color:#657b83;\">\n  balance </span><span style=\"color:#859900;\">&lt;- </span><span style=\"color:#cb4b16;\">Balance </span><span style=\"color:#859900;\">&lt;$&gt; </span><span style=\"color:#6c71c4;\">10000</span><span style=\"color:#657b83;\">\n  topup </span><span style=\"color:#859900;\">&lt;- </span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\">\n  futureValueBitcoin balance topup </span><span style=\"color:#6c71c4;\">1.9 30\n</span></code></pre>\n<p>Let's see what values we get and how they compares to the actual <code>FV</code> function that works on <code>Double</code> (for the curious here is one possible implementation <a href=\"https://github.com/numpy/numpy/blob/e94ed84010c60961f82860d146681d3fd607de4e/numpy/lib/financial.py#L36\">numpy.fv</a>)</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> fst </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> arithM computeBalance\n</span><span style=\"color:#cb4b16;\">Balance </span><span style=\"color:#6c71c4;\">10315.81142818\n</span><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> fv (</span><span style=\"color:#6c71c4;\">1.9 </span><span style=\"color:#859900;\">/ </span><span style=\"color:#6c71c4;\">36600</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">30</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#859900;\">-</span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\">) (</span><span style=\"color:#859900;\">-</span><span style=\"color:#6c71c4;\">10000</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#6c71c4;\">10315.811428177167\n</span></code></pre>\n<p>That's pretty good. We get the accurately rounded result of our new balance. But how\naccurate is the computed result before the rounding is applied? As accurate as 128 bits can\ndo in presence of rounding:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt;</span><span style=\"color:#657b83;\"> snd </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> arithM computeBalance\n</span><span style=\"color:#6c71c4;\">10315.811428176906130029412612348658890\n</span></code></pre>\n<p>We get much better accuracy here than we could with <code>Double</code>. This isn't surprising, since we have more bits at our disposal, but accuracy is not the only benefit of this calculation. The result is also deterministic! This is practically impossible to guarantee with floating point number calculations across different platforms and architectures.</p>\n<h2 id=\"available-solutions\">Available solutions</h2>\n<p>A very common question people usually ask when a new library is being announced: &quot;What is wrong with currently available solutions?&quot;. That is a perfectly reasonable question, which hopefully we have a compelling answer for.</p>\n<p>We had a strong requirement for safety, correctness, and performance. Which is the combination that none of the available libraries in Haskell ecosystem could provide.</p>\n<p>I will use <code>Data.Fixed</code> from <code>base</code> as an example and list some of limitations that prevented us from using it:</p>\n<ul>\n<li>\n<p>Backed by <code>Integer</code>, which makes it slower than it should be for common cases.</p>\n</li>\n<li>\n<p>Truncation instead of some more useful rounding strategies.</p>\n</li>\n</ul>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">5.39 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Fixed E1\n</span><span style=\"color:#6c71c4;\">5.3\n</span><span style=\"color:#859900;\">&gt;&gt;&gt; </span><span style=\"color:#6c71c4;\">5.499999999999 </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Fixed E1\n</span><span style=\"color:#6c71c4;\">5.4\n</span></code></pre>\n<ul>\n<li>No built-in protection against runtime exceptions:</li>\n</ul>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">&gt;&gt;&gt; f = 5.49 :: Fixed E1\n&gt;&gt;&gt; f / 0\n*** Exception: divide by zero\n</span></code></pre>\n<ul>\n<li>\n<p>There is a limited number of scaling types: <code>E0</code>, <code>E1</code>, <code>E2</code>, <code>E3</code>, <code>E6</code>, <code>E9</code> and <code>E12</code>. It is possible to add new ones with <code>HasResolution</code>, but it is a bit inconvenient.</p>\n</li>\n<li>\n<p>No built-in ability to specify bounds. This means that there is no protection against things like negative values or going outside of artificially imposed limits.</p>\n</li>\n</ul>\n<p>Similar arguments can be applied to other libraries. Especially the objection <a href=\"https://gist.github.com/lehins/727a86c71ff32e18a51b42d3dc0736fe\">regarding performance</a>. This objection is not unfounded: our benchmarks have revealed performance issues of practical relevance with existing implementations.</p>\n<h2 id=\"conclusion\">Conclusion</h2>\n<p>I encourage everyone who writes software for finance, blockchain and other areas that require exact precision and safety of calculations, to seriously consider all implications of choosing the wrong data type for representing their numeric values.</p>\n<p>Haskell is a very safe language out of the box, but as you saw in this post, it does not offer the desired level of safety when it comes to operations on numeric values. Hopefully we were able to convince you, that, at least for decimal numbers, such safety can be achieved with <code>safe-decimal</code> library.</p>\n<p>If you feel like this post describes problems that are familiar to you and you are looking for a solution, please reach out to us and we will be glad to help.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/",
        "slug": "safe-decimal-right-on-the-money",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Decimal Safety Right on The Money",
        "description": "We developed a library for operating on decimal numbers that guards you from errors. Come learn how it works and how you can use it",
        "updated": null,
        "date": "2020-02-12T17:42:00Z",
        "year": 2020,
        "month": 2,
        "day": 12,
        "taxonomies": {
          "tags": [
            "haskell",
            "blockchain"
          ],
          "categories": [
            "functional programming"
          ]
        },
        "extra": {
          "author": "Alexey Kuleshevich",
          "blogimage": "/images/blog/distributed-ledge.jpg"
        },
        "path": "blog/safe-decimal-right-on-the-money/",
        "components": [
          "blog",
          "safe-decimal-right-on-the-money"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "problems-we-want-to-solve",
            "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#problems-we-want-to-solve",
            "title": "Problems we want to solve",
            "children": [
              {
                "level": 3,
                "id": "floating-point",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#floating-point",
                "title": "Floating point",
                "children": []
              },
              {
                "level": 3,
                "id": "fixed-point-decimal",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#fixed-point-decimal",
                "title": "Fixed point decimal",
                "children": []
              },
              {
                "level": 3,
                "id": "precision",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#precision",
                "title": "Precision",
                "children": []
              },
              {
                "level": 3,
                "id": "storage-and-performance",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#storage-and-performance",
                "title": "Storage and Performance",
                "children": []
              },
              {
                "level": 3,
                "id": "bounds",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#bounds",
                "title": "Bounds",
                "children": []
              },
              {
                "level": 3,
                "id": "runtime-exceptions",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#runtime-exceptions",
                "title": "Runtime exceptions",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "decimal-for-crypto",
            "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#decimal-for-crypto",
            "title": "Decimal for crypto",
            "children": [
              {
                "level": 3,
                "id": "definition",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#definition",
                "title": "Definition",
                "children": []
              },
              {
                "level": 3,
                "id": "construction-and-arithmetic",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#construction-and-arithmetic",
                "title": "Construction and arithmetic",
                "children": []
              },
              {
                "level": 3,
                "id": "num-and-fractional",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#num-and-fractional",
                "title": "Num and Fractional",
                "children": []
              },
              {
                "level": 3,
                "id": "mixing-decimal-types",
                "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#mixing-decimal-types",
                "title": "Mixing Decimal types",
                "children": []
              }
            ]
          },
          {
            "level": 2,
            "id": "available-solutions",
            "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#available-solutions",
            "title": "Available solutions",
            "children": []
          },
          {
            "level": 2,
            "id": "conclusion",
            "permalink": "https://www.fpcomplete.com/blog/safe-decimal-right-on-the-money/#conclusion",
            "title": "Conclusion",
            "children": []
          }
        ],
        "word_count": 3370,
        "reading_time": 17,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/rust-devops.md",
        "content": "<p>On February 2, 2020, one of FP Complete's Lead Software Engineers—Mike McGirr—presented a webinar on using Rust for creating DevOps tooling.</p>\n<h2 id=\"webinar-outline\">Webinar Outline</h2>\n<p>FP Complete is hosting a functional programming\nwebinar on, “Learn Rapid Rust with DevOps Success\nStrategies.” A beginner’s guide including sample Rust\ndemonstration on writing your DevOps tools with Rust\nover Haskell. An introduction to Rust, with basic DevOps\nuse cases, and the library ecosystem, airing on\nFebruary 5th, 2020.</p>\n<p>The webinar will be hosted by Mike McGirr, a DevOps\nSoftware Engineer at FP Complete which will provide an\nabundance of Rust information with respect to\nfunctional programming and DevOps, featuring (safety,\nspeed and accuracy) that make it unique and contributes\nto its popularity, and its possible preference as a\nlanguage of choice for operating systems over Haskell,\nweb browsers and device drivers among others. The\nwebinar offers an interesting opportunity to learn and\nuse Rust in developing real world projects aside from\nHaskell or other functional programming languages\navailable today.</p>\n<h2 id=\"topics-covered\">Topics covered</h2>\n<p>During the webinar we will cover the following\ntopics:</p>\n<ul>\n<li>A quick intro and background into the Rust programming language</li>\n<li>Some scenarios and reasons why you would want to use Rust for writing your DevOps tooling (and some reasons why you wouldn’t)</li>\n<li>A small example of using the existing AWS libraries to create   a basic DevOps tool</li>\n<li>How to Integrate FP into your Organization</li>\n</ul>\n<p>Mike Mcgirr, a Lead Software Engineer at FP\nComplete,will help us understand reasoning that\nsupports using Rust over other functional programming\nlanguages offered in the market today.</p>\n<h2 id=\"more-about-your-host\">More about your host</h2>\n<p>The webinar will be hosted by Mike McGirr, a veteran\nDevOps Software Engineer at FP Complete. With years of\nexperience in DevOps software development, Mike will\nwalk us through a first in a series of Rust webinars\ndiscussing why we would, and how we could utilize Rust\nas a functional programming language to build DevOps\nover other functional programming languages available\nin the market today. Mike will also share with us a\nsmall example script written in Rust showing how Rust\nmay be used.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/rust-devops/",
        "slug": "rust-devops",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Rust with DevOps Success Strategies",
        "description": "Wednesday Feb 5th, 2020, at 10:00 AM PST. Webinar Outline: FP Complete is hosting a functional programming webinar on, “Learn Rapid Rust with DevOps Success Strategies.” A beginner’s guide including sample Rust demonstration on writing your DevOps tools with Rust over Hasell. An introduction to Rust, with basic DevOps use cases, and the library ecosystem, […]",
        "updated": null,
        "date": "2020-02-05",
        "year": 2020,
        "month": 2,
        "day": 5,
        "taxonomies": {
          "categories": [
            "functional programming",
            "devops"
          ],
          "tags": [
            "devops",
            "rust",
            "insights"
          ]
        },
        "extra": {
          "author": "Mike McGirr",
          "blogimage": "/images/blog/rust.jpg"
        },
        "path": "blog/rust-devops/",
        "components": [
          "blog",
          "rust-devops"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "webinar-outline",
            "permalink": "https://www.fpcomplete.com/blog/rust-devops/#webinar-outline",
            "title": "Webinar Outline",
            "children": []
          },
          {
            "level": 2,
            "id": "topics-covered",
            "permalink": "https://www.fpcomplete.com/blog/rust-devops/#topics-covered",
            "title": "Topics covered",
            "children": []
          },
          {
            "level": 2,
            "id": "more-about-your-host",
            "permalink": "https://www.fpcomplete.com/blog/rust-devops/#more-about-your-host",
            "title": "More about your host",
            "children": []
          }
        ],
        "word_count": 351,
        "reading_time": 2,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/transformations-on-applicative-concurrent-computations.md",
        "content": "<p>When deciding which language to use to solve challenges that require heavy concurrent algorithms, it's hard to not consider Haskell. Its immutable and persistent data structures reduce the introduction of accidental complexity, and the GHC runtime facilitates the creation of thousands of (green) threads without having to worry as much about the memory and performance costs.</p>\n<p>The epitome of Haskell's concurrent API is the <code>async</code> package, which provides higher-order functions (e.g. <a href=\"https://www.stackage.org/haddock/lts-12.24/async-2.2.1/Control-Concurrent-Async.html#v:race\"><code>race</code></a>, <a href=\"https://www.stackage.org/haddock/lts-12.24/async-2.2.1/Control-Concurrent-Async.html#v:mapConcurrently\"><code>mapConcurrently</code></a>, etc.) that allow us to run <code>IO</code> sub-routines and combine their results in various ways while executing concurrently. It also offers the type  <a href=\"https://www.stackage.org/haddock/lts-12.24/async-2.2.1/Control-Concurrent-Async.html#t:Concurrently\"><code>Concurrently</code></a> which allows developers to give normal sub-routines concurrent properties, and also provides <code>Applicative</code> and <code>Alternative</code>  instances that help in the creation of values from composing smaller sub-routines.</p>\n<p>In this blog post, we will discuss some of the drawbacks of using the <a href=\"https://www.stackage.org/haddock/lts-12.24/async-2.2.1/Control-Concurrent-Async.html#t:Concurrently\"><code>Concurrently</code></a> type when composing sub-routines. Then we will show how we can overcome these shortcomings by taking advantage of the structural nature of the <code>Applicative</code> and <code>Alternative</code> typeclasses; re-shaping and optimizing the execution of a tree of sub-routines.</p>\n<p>And, if you simply want to get these performance advantages in your Haskell code today, you can cut to the chase and begin using the new <a href=\"https://www.stackage.org/haddock/lts-12.24/unliftio-0.2.9.0/UnliftIO-Async.html#t:Conc\"><code>Conc</code></a> datatype we've introduced in <a href=\"https://www.stackage.org/lts-12.24/package/unliftio-0.2.9.0\"><code>unliftio</code> 0.2.9.0</a>.</p>\n<h2 id=\"the-drawbacks-of-concurrently\">The drawbacks of <code>Concurrently</code></h2>\n<p>Getting started with <code>Concurrently</code> is easy. We can wrap an <code>IO a</code> sub-routine with the <code>Concurrently</code> constructor, and then we can compose async values using the map (<code>&lt;$&gt;</code>), apply (<code>&lt;*&gt;</code>), and alternative (<code>&lt;|&gt;</code>) operators. An example might be:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">myPureFunction </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">String </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">String </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">String </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#268bd2;\">String\n</span><span style=\"color:#657b83;\">myPureFunction a b c </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">++ </span><span style=\"color:#839496;\">&quot; &quot; </span><span style=\"color:#859900;\">++</span><span style=\"color:#657b83;\"> b </span><span style=\"color:#859900;\">++ </span><span style=\"color:#839496;\">&quot; &quot; </span><span style=\"color:#859900;\">++</span><span style=\"color:#657b83;\"> c\n\n</span><span style=\"color:#b58900;\">myComputation </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">Concurrently String\n</span><span style=\"color:#657b83;\">myComputation </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\">\n  myPureFunction\n  </span><span style=\"color:#859900;\">&lt;$&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI1\n  </span><span style=\"color:#859900;\">&lt;</span><span style=\"color:#657b83;\">*</span><span style=\"color:#859900;\">&gt;</span><span style=\"color:#657b83;\"> (    </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region1\n       </span><span style=\"color:#859900;\">&lt;|&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region2\n       </span><span style=\"color:#859900;\">&lt;|&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region3\n       </span><span style=\"color:#859900;\">&lt;|&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region4)\n  </span><span style=\"color:#859900;\">&lt;</span><span style=\"color:#657b83;\">*</span><span style=\"color:#859900;\">&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI3\n</span></code></pre>\n<p>Let's talk a bit on the drawbacks of this approach. How many threads do you think we need to make sure all these calls execute concurrently? Try to come up with a number and an explanation and then continue reading.</p>\n<p>I am guessing you are expecting this code to spawn six (6) threads, correct? One for each <code>IO</code> sub-routine that we are using. However, with the existing implementation of <code>Applicative</code> and <code>Alternative</code> in <code>Concurrently</code>, we will spawn at least ten (10) threads. Let's explore these instances to have a better understanding of what is going on:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">Applicative Concurrently </span><span style=\"color:#859900;\">where</span><span style=\"color:#657b83;\">\n  pure </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Concurrently </span><span style=\"color:#859900;\">.</span><span style=\"color:#657b83;\"> return\n  </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fs </span><span style=\"color:#859900;\">&lt;</span><span style=\"color:#657b83;\">*</span><span style=\"color:#859900;\">&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> as </span><span style=\"color:#859900;\">=\n    </span><span style=\"color:#cb4b16;\">Concurrently </span><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">(f, a) </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> f a) </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> concurrently fs as\n\n</span><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">Alternative Concurrently </span><span style=\"color:#859900;\">where\n  </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> as </span><span style=\"color:#859900;\">&lt;|&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> bs </span><span style=\"color:#859900;\">=\n    </span><span style=\"color:#cb4b16;\">Concurrently </span><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\"> either id id </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> race as bs\n</span></code></pre>\n<p>First, let us expand the alternative calls in our example:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">    </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region1\n</span><span style=\"color:#859900;\">&lt;|&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region2\n</span><span style=\"color:#859900;\">&lt;|&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region3\n</span><span style=\"color:#859900;\">&lt;|&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region4\n\n</span><span style=\"color:#93a1a1;\">--- is equivalent to\n</span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> (\n  either id id </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\">\n    race </span><span style=\"color:#93a1a1;\">{- 2 threads -}</span><span style=\"color:#657b83;\">\n      fetchStringFromAPI2_Region1\n      (either id id </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\">\n         race </span><span style=\"color:#93a1a1;\">{- 2 threads -}</span><span style=\"color:#657b83;\">\n           fetchStringFromAPI2_Region2\n           (either id id </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\">\n              race </span><span style=\"color:#93a1a1;\">{- 2 threads -}</span><span style=\"color:#657b83;\">\n                fetchStringFromAPI2_Region3\n                fetchStringFromAPI2_Region4))\n)\n</span></code></pre>\n<p>Next, let us expand the applicative calls:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">    </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> (myPureFunction </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> fetchStringFromAPI1)\n</span><span style=\"color:#859900;\">&lt;</span><span style=\"color:#657b83;\">*</span><span style=\"color:#859900;\">&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI2\n</span><span style=\"color:#859900;\">&lt;</span><span style=\"color:#657b83;\">*</span><span style=\"color:#859900;\">&gt; </span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> fetchStringFromAPI3\n\n</span><span style=\"color:#93a1a1;\">--- is equivalent to\n\n</span><span style=\"color:#cb4b16;\">Concurrently</span><span style=\"color:#657b83;\"> (\n  (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">(f, a) </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> f a) </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\">\n    concurrently </span><span style=\"color:#93a1a1;\">{- 2 threads -}</span><span style=\"color:#657b83;\">\n      ( (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">(f, a) </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> f a) </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\">\n         concurrently </span><span style=\"color:#93a1a1;\">{- 2 threads -}</span><span style=\"color:#657b83;\">\n           (myPureFunction </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> fetchStringFromAPI1)\n           fetchStringFromAPI2\n      )\n      fetchStringFromAPI3\n)\n</span></code></pre>\n<p>You may tell we are always spawning two threads for each pair of sub-routines. Suppose we have 7 sub-routines we want to compose via <code>Applicative</code> or <code>Alternative</code>. Using this implementation we would spawn at least 14 new threads when at most 8 should do the job. For each composition we do, an extra thread is going to be spawned to deal with bookkeeping.</p>\n<p>Another drawback to consider: what happens if one of the values in the call is a <code>pure</code> call? Given this code:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">pure foo </span><span style=\"color:#859900;\">&lt;|&gt;</span><span style=\"color:#657b83;\"> bar\n</span></code></pre>\n<p>We get to spawn a new thread (unnecessarily) to wait for <code>foo</code>, even though it has already been computed and it should always win. As we mentioned before, Haskell is an excellent choice for concurrency because it makes spawning threads cheap; however, these threads don't come for free, and we should strive to avoid redundant thread creation.</p>\n<h2 id=\"introducing-the-conc-type\">Introducing the <code>Conc</code> type</h2>\n<p>To address the issues mentioned above, we implemented a new type called <code>Conc</code> in our <code>unliftio</code> package. It has the same purpose as <code>Concurrently</code>, but it offers some extra guarantees:</p>\n<ul>\n<li>There is going to be only a single bookkeeping thread for all <code>Applicative</code> and <code>Alternative</code> compositions.</li>\n<li>If we have <code>pure</code> calls in an <code>Applicative</code> or an <code>Alternative</code> composition, we will not spawn a new thread.</li>\n<li>We will optimize the code for trivial cases. For example, not spawning a thread when evaluating a single <code>Conc</code> value (instead of a composition of <code>Conc</code> values).</li>\n<li>We can compose more than <code>IO</code> sub-routines. Any monadic type that implements <code>MonadUnliftIO</code> is accepted.</li>\n<li>Children threads are always launched in an unmasked state, not the inherited state of the parent thread.</li>\n</ul>\n<p>The <code>Conc</code> type is defined as follows:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m a </span><span style=\"color:#859900;\">where\n  </span><span style=\"color:#cb4b16;\">Action </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> m a </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m a\n  </span><span style=\"color:#cb4b16;\">Apply  </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m (v </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> a) </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m v </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m a\n  </span><span style=\"color:#cb4b16;\">LiftA2 </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> (x </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> a) </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m x </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m y </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m a\n  </span><span style=\"color:#cb4b16;\">Pure   </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m a\n  </span><span style=\"color:#cb4b16;\">Alt    </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m a </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m a </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m a\n  </span><span style=\"color:#cb4b16;\">Empty  </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Conc</span><span style=\"color:#657b83;\"> m a\n\n</span><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">MonadUnliftIO m </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#268bd2;\">Applicative</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Conc m</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">where</span><span style=\"color:#657b83;\">\n  pure   </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Pure\n  </span><span style=\"color:#b58900;\">(&lt;*&gt;)  </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Apply\n  </span><span style=\"color:#b58900;\">(*&gt;)   </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Then</span><span style=\"color:#657b83;\">\n  liftA2 </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">LiftA2\n\n</span><span style=\"color:#859900;\">instance </span><span style=\"color:#268bd2;\">MonadUnliftIO m </span><span style=\"color:#859900;\">=&gt; </span><span style=\"color:#268bd2;\">Alternative</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Conc m</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">where\n  </span><span style=\"color:#b58900;\">(&lt;|&gt;) </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Alt\n</span></code></pre>\n<p>If you are familiar with <code>Free</code> types, this will look eerily familiar. We are going to represent our concurrent computations as data so that we can later transform it or evaluate as we see fit. In this setting, our first example would look something like the following:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">myComputation </span><span style=\"color:#859900;\">:: </span><span style=\"color:#268bd2;\">Conc String\n</span><span style=\"color:#657b83;\">myComputation </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\">\n  myPureFunction\n  </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> conc fetchStringFromAPI1\n  </span><span style=\"color:#859900;\">&lt;</span><span style=\"color:#657b83;\">*</span><span style=\"color:#859900;\">&gt;</span><span style=\"color:#657b83;\"> (    conc fetchStringFromAPI2_Region1\n       </span><span style=\"color:#859900;\">&lt;|&gt;</span><span style=\"color:#657b83;\"> conc fetchStringFromAPI2_Region2\n       </span><span style=\"color:#859900;\">&lt;|&gt;</span><span style=\"color:#657b83;\"> conc fetchStringFromAPI2_Region3\n       </span><span style=\"color:#859900;\">&lt;|&gt;</span><span style=\"color:#657b83;\"> conc fetchStringFromAPI2_Region4)\n\n</span><span style=\"color:#93a1a1;\">--- is equivalent to\n\n</span><span style=\"color:#cb4b16;\">Apply</span><span style=\"color:#657b83;\"> (myPureFunction </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> fetchStringFromAPI1)\n      (</span><span style=\"color:#cb4b16;\">Alt</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Action</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region1)\n           (</span><span style=\"color:#cb4b16;\">Alt</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Action</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region2)\n                (</span><span style=\"color:#cb4b16;\">Alt</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Action</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region3)\n                     (</span><span style=\"color:#cb4b16;\">Action</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region4))))\n\n</span></code></pre>\n<p>You may notice we keep the tree structure of the <code>Concurrently</code> implementation. However, given we are dealing with a pure data structure, we can modify our <code>Conc</code> value to something that is easier to evaluate. Indeed, thanks to the <code>Applicative</code>  interface, we don't need to evaluate any of the <code>IO</code> sub-routines to do transformations (magic!).</p>\n<p>We have additional (internal) types that flatten all our alternatives and applicative values:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">Flat</span><span style=\"color:#657b83;\"> a\n  </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">FlatApp </span><span style=\"color:#859900;\">!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> a)\n  </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">FlatAlt </span><span style=\"color:#859900;\">!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> a) </span><span style=\"color:#859900;\">!</span><span style=\"color:#657b83;\">(</span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> a) </span><span style=\"color:#859900;\">!</span><span style=\"color:#657b83;\">[</span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> a]\n\n</span><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">where\n  </span><span style=\"color:#cb4b16;\">FlatPure   </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> a\n  </span><span style=\"color:#cb4b16;\">FlatAction </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">IO</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> a\n  </span><span style=\"color:#cb4b16;\">FlatApply  </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Flat</span><span style=\"color:#657b83;\"> (v </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> a) </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Flat</span><span style=\"color:#657b83;\"> v </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> a\n  </span><span style=\"color:#cb4b16;\">FlatLiftA2 </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> (x </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> a) </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Flat</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">Flat</span><span style=\"color:#657b83;\"> y </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> a\n</span></code></pre>\n<p>These types are equivalent to our <code>Conc</code> type, but they have a few differences from <code>Conc</code>:</p>\n<ul>\n<li>The <code>Flat</code> type separates <code>Conc</code> values created via <code>Applicative</code> from the ones created via <code>Alternative</code></li>\n<li>The <code>FlatAlt</code> constructor flattens an <code>Alternative</code> tree into a list (helping us spawn all of them at once and facilitating the usage of a single bookkeeping thread).\n<ul>\n<li>Note that we represent this as a &quot;at least two&quot; list, with a similar representation of a non empty list from the <code>semigroups</code> package.</li>\n</ul>\n</li>\n<li>The <code>Flat</code> and <code>FlatApp</code> records are not polymorphic on their monadic context given they rely directly on <code>IO</code>. We can transform the <code>m</code> parameter in our <code>Conc m a</code>  type to <code>IO</code> via the <code>MonadUnliftIO</code>  constraint.</li>\n</ul>\n<p>The first example of our blog post, when flattened, would look something like the following:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\">\n  (</span><span style=\"color:#cb4b16;\">FlatApply</span><span style=\"color:#657b83;\">\n    (</span><span style=\"color:#cb4b16;\">FlatApp</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">FlatAction</span><span style=\"color:#657b83;\"> (myPureFunction </span><span style=\"color:#859900;\">&lt;$&gt;</span><span style=\"color:#657b83;\"> fetchStringFromAPI1)))\n    (</span><span style=\"color:#cb4b16;\">FlatAlt</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">FlatAction</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region1)\n             (</span><span style=\"color:#cb4b16;\">FlatAction</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region2)\n             [ </span><span style=\"color:#cb4b16;\">FlatAction</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Region3\n             , </span><span style=\"color:#cb4b16;\">FlatAction</span><span style=\"color:#657b83;\"> fetchStringFromAPI2_Regoin4 ]))\n</span></code></pre>\n<p>Using a <a href=\"https://github.com/fpco/unliftio/blob/2c68959a5f498bf4ea457fc4bd673b4dfce5c512/unliftio/src/UnliftIO/Internals/Async.hs#L617\"><code>flatten</code></a> function that transforms a <code>Conc</code> value into a <code>Flat</code> value, we can <a href=\"https://github.com/fpco/unliftio/blob/2c68959a5f498bf4ea457fc4bd673b4dfce5c512/unliftio/src/UnliftIO/Internals/Async.hs#L662\">later evaluate</a> the concurrent sub-routine tree in a way that is optimal for our use case.</p>\n<h2 id=\"performance\">Performance</h2>\n<p>So given that the <code>Conc</code> API reduces the number of threads created via <code>Alternative</code>, our implementation should work best, correct? Sadly, it is not all peachy. To ensure that we get the result of the first thread that finishes on an <code>Alternative</code> composition, we make use of the STM API. This approach works great when we want to gather values from multiple concurrent threads. Sadly, the STM monad doesn't <a href=\"https://www.oreilly.com/library/view/parallel-and-concurrent/9781449335939/ch10.html#sec_stm-cost\">scale too well when composing lots of reads</a>, making this approach prohibitive if you are composing tens of thousands of <code>Conc</code> values.</p>\n<p>Considering this limitation, we only use <code>STM</code> when an <code>Alternative</code> function is involved; otherwise, we rely on <code>MVar</code>s for multiple thread result composition via <code>Applicative</code>. We can do this without sweating because we can change the evaluator of the sub-routine tree created by <code>Conc</code> on the fly.</p>\n<h2 id=\"conclusions\">Conclusions</h2>\n<p>We showcased how we can model the composition of computations using an <code>Applicative</code> and <code>Alternative</code> tree, and then, taking advantage of this APIs; we transformed this computation tree into something more approachable to execute concurrently. We also took advantage of this <em>sub-routines as data</em> approach to change the evaluator from <code>MVar</code> to <code>STM</code> compositions.</p>\n",
        "permalink": "https://www.fpcomplete.com/blog/transformations-on-applicative-concurrent-computations/",
        "slug": "transformations-on-applicative-concurrent-computations",
        "ancestors": [
          "_index.md",
          "blog/_index.md"
        ],
        "title": "Transformations on Applicative Concurrent Computations",
        "description": "We wrote a data type called Conc, which provides for more efficient concurrent computations. Come read how you can use this in your Haskell code today!",
        "updated": null,
        "date": "2020-01-27T04:28:32Z",
        "year": 2020,
        "month": 1,
        "day": 27,
        "taxonomies": {
          "categories": [
            "functional programming"
          ],
          "tags": [
            "haskell"
          ]
        },
        "extra": {
          "author": "Román González",
          "blogimage": "/images/blog/functional.jpg"
        },
        "path": "blog/transformations-on-applicative-concurrent-computations/",
        "components": [
          "blog",
          "transformations-on-applicative-concurrent-computations"
        ],
        "summary": null,
        "toc": [
          {
            "level": 2,
            "id": "the-drawbacks-of-concurrently",
            "permalink": "https://www.fpcomplete.com/blog/transformations-on-applicative-concurrent-computations/#the-drawbacks-of-concurrently",
            "title": "The drawbacks of Concurrently",
            "children": []
          },
          {
            "level": 2,
            "id": "introducing-the-conc-type",
            "permalink": "https://www.fpcomplete.com/blog/transformations-on-applicative-concurrent-computations/#introducing-the-conc-type",
            "title": "Introducing the Conc type",
            "children": []
          },
          {
            "level": 2,
            "id": "performance",
            "permalink": "https://www.fpcomplete.com/blog/transformations-on-applicative-concurrent-computations/#performance",
            "title": "Performance",
            "children": []
          },
          {
            "level": 2,
            "id": "conclusions",
            "permalink": "https://www.fpcomplete.com/blog/transformations-on-applicative-concurrent-computations/#conclusions",
            "title": "Conclusions",
            "children": []
          }
        ],
        "word_count": 1528,
        "reading_time": 8,
        "assets": [],
        "draft": false,
        "lang": "en",
        "lighter": null,
        "heavier": null,
        "earlier": null,
        "later": null,
        "translations": []
      },
      {
        "relative_path": "blog/teaching-haskell-with-duet.md",
        "content": "<h1 id=\"teaching-haskell-with-duet\">Teaching Haskell with Duet</h1>\n<p>Teaching Haskell to complete beginners is an enjoyable\nexperience. Haskell is foreign; many of its features are alien to\nother programmers. It's purely functional. It's non-strict. Its type\nsystem is among the more pervasive of other practical languages.</p>\n<h2 id=\"simple-at-the-core\">Simple at the core</h2>\n<p>Haskell's core language is simple, though. It shares this with Lisp.\n<a href=\"https://mitpress.mit.edu/sites/default/files/sicp/full-text/sicp/book/node10.html\">Structured and Interpretation of Computer Programs by MIT</a>\nteaches Lisp beginning with a substitution model for function\napplication. This turns out to work well for Haskell too. This is how\nI've been teaching Haskell to beginners at FP Complete for our\nclients.</p>\n<p>For example, in SICP, they use the example:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">(+ (square </span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\">) (square </span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\">))\n</span></code></pre>\n<p>which reduces the function <code>square</code> to</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">(+ (* </span><span style=\"color:#6c71c4;\">6 6</span><span style=\"color:#657b83;\">) (* </span><span style=\"color:#6c71c4;\">10 10</span><span style=\"color:#657b83;\">))\n</span></code></pre>\n<p>which reduces by multiplication to:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">(+ </span><span style=\"color:#6c71c4;\">36 100</span><span style=\"color:#657b83;\">)\n</span></code></pre>\n<p>and finally to</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">136\n</span></code></pre>\n<p>As they note in SICP,</p>\n<blockquote>\n<p>The purpose of the substitution is to help us think about procedure\napplication, not to provide a description of how the interpreter\nreally works. Typical interpreters do not evaluate procedure\napplications by manipulating the text of a procedure to substitute\nvalues for the formal parameters.</p>\n</blockquote>\n<p>That this, this is a <em>model</em>, it's not the real thing. In fact, if you\nreally eyeball the very first step, you might wonder which <code>(square ..)</code> argument is evaluated first between the two. Scheme doesn't\nspecify an argument order; it varies. An implementation may even\ninline the whole thing.</p>\n<p>Rather, if we think about programs in terms of a simple sequence of\nrewrites, we get a lot of bang for our buck in terms of reasoning and\nunderstanding.</p>\n<h2 id=\"the-right-language-to-model\">The right language to model</h2>\n<p>Motivated by this goal, I started thinking about automating this\nprocess, so that students could use this model more readily, and see\nthe shape of functions and algorithms visually. The solution I came up\nwith was a new language that is a subset of Haskell, which I'll cover\nin this post.</p>\n<p>The reason that it's not full Haskell is that Haskell has a lot of\nsurface-level syntactic sugar. Evaluating the real language is\ncomplicated and infeasible. The following contains too many things at\nonce to consider:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#b58900;\">quicksort1 </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#268bd2;\">Ord a</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">=&gt;</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#268bd2;\">a</span><span style=\"color:#657b83;\">] </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#268bd2;\">a</span><span style=\"color:#657b83;\">]\nquicksort1 </span><span style=\"color:#b58900;\">[] </span><span style=\"color:#859900;\">= </span><span style=\"color:#b58900;\">[]</span><span style=\"color:#657b83;\">\nquicksort1 (x</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\">xs) </span><span style=\"color:#859900;\">=\n  let</span><span style=\"color:#657b83;\"> smallerSorted </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> quicksort1 [a </span><span style=\"color:#859900;\">|</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> xs, a </span><span style=\"color:#859900;\">&lt;=</span><span style=\"color:#657b83;\"> x]\n      biggerSorted </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> quicksort1 [a </span><span style=\"color:#859900;\">|</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">&lt;-</span><span style=\"color:#657b83;\"> xs, a </span><span style=\"color:#859900;\">&gt;</span><span style=\"color:#657b83;\"> x]\n  </span><span style=\"color:#859900;\">in</span><span style=\"color:#657b83;\">  smallerSorted </span><span style=\"color:#859900;\">++</span><span style=\"color:#657b83;\"> [x] </span><span style=\"color:#859900;\">++</span><span style=\"color:#657b83;\"> biggerSorted\n</span></code></pre>\n<p>Pattern matches at the definition, list syntax, comprehensions,\nlets. There's a lot going on here that makes a newbie's eyes glaze\nover. We have to start simpler. But how simple?</p>\n<p>GHC Haskell has a tiny language called Core to which all Haskell\nprograms compile down to. Its AST looks roughly like this:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">Expr\n  </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">App Expr Expr\n  </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">Var Var\n  </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">Lam Name Type Expr\n  </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">Case Expr</span><span style=\"color:#657b83;\"> [</span><span style=\"color:#cb4b16;\">Alt</span><span style=\"color:#657b83;\">]\n  </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">Let Bind Expr\n  </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">Lit Literal\n</span></code></pre>\n<p>Evaluation of Core is simple. However, Core is also a little <em>too\nlow-level</em>, because it puts polymorphic types and type-class\ndictionaries as normal arguments, inlines a lot of things, looks\nunderneath boxed types like <code>Int</code> (into <code>I#</code>), and adds some extra\ncapabilities normal Haskell doesn't have that are only appropriate for\na compiler writer to see. The above function compiled to Core starts\nlike this:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">quicksort1\n  </span><span style=\"color:#859900;\">= \\</span><span style=\"color:#657b83;\"> (@ a_a1Zd) (</span><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\">dOrd_a1Zf </span><span style=\"color:#859900;\">:: </span><span style=\"color:#cb4b16;\">Ord</span><span style=\"color:#657b83;\"> a_a1Zd) (ds_d22B </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> [a_a1Zd]) </span><span style=\"color:#859900;\">-&gt;\n      case</span><span style=\"color:#657b83;\"> ds_d22B </span><span style=\"color:#859900;\">of</span><span style=\"color:#657b83;\"> {\n        </span><span style=\"color:#b58900;\">[] </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#cb4b16;\">GHC</span><span style=\"color:#859900;\">.</span><span style=\"color:#cb4b16;\">Types</span><span style=\"color:#859900;\">.</span><span style=\"color:#b58900;\">[]</span><span style=\"color:#657b83;\"> @ a_a1Zd;\n        </span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\"> x_a1sG xs_a1sH </span><span style=\"color:#859900;\">-&gt;\n          ++</span><span style=\"color:#657b83;\">\n            @ a_a1Zd\n            (quicksort1\n               @ a_a1Zd\n               </span><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\">dOrd_a1Zf\n               (letrec {\n                  ds1_d22C [</span><span style=\"color:#cb4b16;\">Occ</span><span style=\"color:#859900;\">=</span><span style=\"color:#cb4b16;\">LoopBreaker</span><span style=\"color:#657b83;\">] </span><span style=\"color:#859900;\">::</span><span style=\"color:#657b83;\"> [a_a1Zd] </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> [a_a1Zd]\n           </span><span style=\"color:#859900;\">...\n</span></code></pre>\n<p>We have to explain the special list syntax, module qualification,\npolymorphic types, dictionaries, etc. all in one go, besides the\nobvious challenge of the unreadable naming convention. Core is made\nfor compilers and compiler writers, not for humans.</p>\n<h2 id=\"duet\">Duet</h2>\n<p>Therefore I took a middle way. Last year I wrote\n<a href=\"https://github.com/chrisdone/duet\">a language called Duet</a>, which is\na Haskell subset made specifically for teaching at this period of\nlearning Haskell. Duet only has these language features: data types,\ntype-classes, top-level definitions, lambdas, case expressions, and\nsome literals (strings, integrals, rationals). Its main feature is\nsteppability; the ability to step through the code. Every step\nproduces a valid program.</p>\n<p>Returning to the SICP example with our new tool, here's the same\nprogram in Duet:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">square </span><span style=\"color:#859900;\">= \\</span><span style=\"color:#657b83;\">x </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x * x\nmain </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> square </span><span style=\"color:#6c71c4;\">6 </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> square </span><span style=\"color:#6c71c4;\">10\n</span></code></pre><pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">chris@precision</span><span style=\"color:#859900;\">:~/</span><span style=\"color:#cb4b16;\">Work</span><span style=\"color:#859900;\">/</span><span style=\"color:#657b83;\">duet</span><span style=\"color:#859900;\">-</span><span style=\"color:#657b83;\">lang</span><span style=\"color:#859900;\">/</span><span style=\"color:#657b83;\">duet</span><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\"> duet run examples</span><span style=\"color:#859900;\">/</span><span style=\"color:#657b83;\">sicp</span><span style=\"color:#859900;\">.</span><span style=\"color:#657b83;\">hs\n(square </span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> (square </span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\">)\n((</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x * x) </span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> (square </span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\">)\n(</span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\"> * </span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> (square </span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#6c71c4;\">36 </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> (square </span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#6c71c4;\">36 </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> ((</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x * x) </span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#6c71c4;\">36 </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\"> * </span><span style=\"color:#6c71c4;\">10</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#6c71c4;\">36 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">100\n136\n</span></code></pre>\n<p>Here we see a substitution model in action. Each line is a valid\nprogram! You can take any line from the output and run it from that\npoint.</p>\n<p>Unlike Scheme, Duet picks an argument order (left-to-right) for strict\nfunctions like integer operations.</p>\n<p><strong>Note</strong>: You can follow along at home by creating a file and running\nit using <a href=\"https://github.com/chrisdone/duet\">docker run</a>\non Linux, OS X or Windows.</p>\n<h2 id=\"folds\">Folds</h2>\n<p>Let's turn our attention to the teaching of folds, which is a classic\nhurdle to get newbies through, as it is a kind of forcing function for\na variety of topics.</p>\n<p>The right fold is clasically defined like this:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">foldr f z </span><span style=\"color:#b58900;\">[]     </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> z\nfoldr f z (x</span><span style=\"color:#859900;\">:</span><span style=\"color:#657b83;\">xs) </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> f x (foldr f z xs)\n</span></code></pre>\n<p>This is not a valid Duet program, because (1) it uses list syntax\n(lists aren't special), and (2) it uses case analysis at the\ndeclaration level. If you try substitution stepping these, you quickly\narrive at an awkward conversation about the difference between the\nseemingly three-argument function <code>foldr</code>, and lambdas, partial\napplication, currying, and pattern matching, and whether we're\ndefining two functions or one. Here is the same program in Duet:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">List</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> a (</span><span style=\"color:#cb4b16;\">List</span><span style=\"color:#657b83;\"> a)\nfoldr </span><span style=\"color:#859900;\">= \\</span><span style=\"color:#657b83;\">f </span><span style=\"color:#859900;\">-&gt; \\</span><span style=\"color:#657b83;\">z </span><span style=\"color:#859900;\">-&gt; \\</span><span style=\"color:#657b83;\">l </span><span style=\"color:#859900;\">-&gt;\n  case</span><span style=\"color:#657b83;\"> l </span><span style=\"color:#859900;\">of\n    </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> z\n    </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> x xs </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> f x (foldr f z xs)\n</span></code></pre>\n<p>At the end of teaching the substitution model, I cover that <code>\\x y z</code>\nis syntactic sugar for <code>\\x -&gt; \\y -&gt; \\z -&gt; ...</code>, but only after the\nintuition has been solidified that all Haskell functions take one\nargument. They may return other functions. So the updated program is:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">List</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> a (</span><span style=\"color:#cb4b16;\">List</span><span style=\"color:#657b83;\"> a)\nfoldr </span><span style=\"color:#859900;\">= \\</span><span style=\"color:#657b83;\">f z l </span><span style=\"color:#859900;\">-&gt;\n  case</span><span style=\"color:#657b83;\"> l </span><span style=\"color:#859900;\">of\n    </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> z\n    </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> x xs </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> f x (foldr f z xs)\n</span></code></pre>\n<p>Which is perfectly valid Haskell, and each part of it can be rewritten\npredictably.</p>\n<p>Let's look at comparing <code>foldr</code> with <code>foldl</code>.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">List</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> a (</span><span style=\"color:#cb4b16;\">List</span><span style=\"color:#657b83;\"> a)\nfoldr </span><span style=\"color:#859900;\">= \\</span><span style=\"color:#657b83;\">f z l </span><span style=\"color:#859900;\">-&gt;\n  case</span><span style=\"color:#657b83;\"> l </span><span style=\"color:#859900;\">of\n    </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> z\n    </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> x xs </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> f x (foldr f z xs)\nfoldl </span><span style=\"color:#859900;\">= \\</span><span style=\"color:#657b83;\">f z l </span><span style=\"color:#859900;\">-&gt;\n  case</span><span style=\"color:#657b83;\"> l </span><span style=\"color:#859900;\">of\n    </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> z\n    </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> x xs </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> foldl f (f z x) xs\nlist </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)\n</span></code></pre><h2 id=\"folds-at-a-glance\">Folds at a glance</h2>\n<p>For a <em>quick</em> summary, we can use holes like in normal Haskell\nindicated by <code>_</code> or <code>_foo</code>. In Duet, these are ignored by the type\nsystem and the stepper, letting you run the stepper with holes in\ntoo. They don't result in an error, so you can build up expressions\nwith them inside.</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">main_foldr </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> foldr _f _nil list\nmain_foldl </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> foldl _f _nil list\nlist </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">3 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)))\n</span></code></pre>\n<p>(I increased the size of the list for a longer more compelling\noutput.)</p>\n<p>We can pass <code>--concise</code> which is a convenience flag to literally\nfilter out intermediate steps (cases, lambdas) which helps us see the\n&quot;high-level&quot; recursion. This flag is still under evaluation (no pun\nintended), but is useful here. Full output is worth studying with\nstudents too, but is too long to fit in this blog post. I will include\na snippet from a non-concise example below.</p>\n<p>The output looks like this:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\"> duet run examples</span><span style=\"color:#859900;\">/</span><span style=\"color:#657b83;\">folds</span><span style=\"color:#859900;\">-</span><span style=\"color:#657b83;\">strictness</span><span style=\"color:#859900;\">.</span><span style=\"color:#657b83;\">hs </span><span style=\"color:#93a1a1;\">--main main_foldr --concise\n</span><span style=\"color:#657b83;\">foldr _f _nil list\n_f </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (foldr _f _nil (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">))))\n_f </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (_f </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\"> (foldr _f _nil (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">))))\n_f </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (_f </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\"> (_f </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (foldr _f _nil (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">))))\n_f </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (_f </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\"> (_f </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (_f </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\"> (foldr _f _nil </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">))))\n_f </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (_f </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\"> (_f </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (_f </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\"> _nil)))\n</span><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\"> duet run examples</span><span style=\"color:#859900;\">/</span><span style=\"color:#657b83;\">folds</span><span style=\"color:#859900;\">-</span><span style=\"color:#657b83;\">strictness</span><span style=\"color:#859900;\">.</span><span style=\"color:#657b83;\">hs </span><span style=\"color:#93a1a1;\">--main main_foldl --concise\n</span><span style=\"color:#657b83;\">foldl _f _nil list\nfoldl _f (_f _nil </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">) (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)))\nfoldl _f (_f (_f _nil </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">) (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">))\nfoldl _f (_f (_f (_f _nil </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">) (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)\nfoldl _f (_f (_f (_f (_f _nil </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">4</span><span style=\"color:#657b83;\">) </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">\n_f (_f (_f (_f _nil </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\">) </span><span style=\"color:#6c71c4;\">4\n</span></code></pre>\n<p>We can immediately see what the &quot;right&quot; part of <code>foldr</code>\nmeans. Experienced Haskellers can already see the teaching\nopportunities sprouting from the earth at this point. We're using O(n)\nspace here, building nested thunks, or using too much stack. Issues\nabound.</p>\n<p>Meanwhile, in <code>foldl</code>, we've shifted accumulation of the nested thunks\nto an argument of <code>foldr</code>, but at the end, we still have a nested\nthunk. Enter strict left fold!</p>\n<p>We also see the argument order come into play: <code>_f</code> is applied to <code>1</code>\nfirst in <code>foldr</code> (<code>_f 1 (foldr ...)</code>), but last in <code>foldl</code> (<code>_f (_f _nil 1) ...</code>, which is another important part of understanding the\ndistinction between the two.</p>\n<h2 id=\"strict-folds\">Strict folds</h2>\n<p>To see the low-level mechanics, and as a precursor to teaching strict\nfold, we ought to use an actual arithmetic operation (because you\ncan't strictly evaluate a <code>_</code> hole, by definition, it's missing):</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">main_foldr </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> foldr (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\"> list\nmain_foldl </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> foldl (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\"> list\n</span></code></pre>\n<p>Both folds eventually yield:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">2\n3\n</span></code></pre>\n<p>And:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#657b83;\">((</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">0 1</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">\n((</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">y </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\">\n(</span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">2\n1 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">2\n3\n</span></code></pre>\n<p>(Here you can also easily see where the <code>0</code> lies in the tree.)</p>\n<p>Which both have the built up thunk problem mentioned above.</p>\n<p>Duet has bang patterns, so we can define a strict fold like this:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">data </span><span style=\"color:#cb4b16;\">List</span><span style=\"color:#657b83;\"> a </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">| </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> a (</span><span style=\"color:#cb4b16;\">List</span><span style=\"color:#657b83;\"> a)\nfoldr </span><span style=\"color:#859900;\">= \\</span><span style=\"color:#657b83;\">f z l </span><span style=\"color:#859900;\">-&gt;\n  case</span><span style=\"color:#657b83;\"> l </span><span style=\"color:#859900;\">of\n    </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> z\n    </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> x xs </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> f x (foldr f z xs)\nfoldl </span><span style=\"color:#859900;\">= \\</span><span style=\"color:#657b83;\">f z l </span><span style=\"color:#859900;\">-&gt;\n  case</span><span style=\"color:#657b83;\"> l </span><span style=\"color:#859900;\">of\n    </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> z\n    </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> x xs </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> foldl f (f z x) xs\nfoldl_ </span><span style=\"color:#859900;\">= \\</span><span style=\"color:#657b83;\">f z l </span><span style=\"color:#859900;\">-&gt;\n  case</span><span style=\"color:#657b83;\"> l </span><span style=\"color:#859900;\">of\n    </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> z\n    </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> x xs </span><span style=\"color:#859900;\">-&gt;\n      case</span><span style=\"color:#657b83;\"> f z x </span><span style=\"color:#859900;\">of\n        !</span><span style=\"color:#657b83;\">z_ </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> foldl_ f z_ xs\nlist </span><span style=\"color:#859900;\">= </span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)\nmain_foldr </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> foldr (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\"> list\nmain_foldl </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> foldl (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\"> list\nmain_foldl_ </span><span style=\"color:#859900;\">=</span><span style=\"color:#657b83;\"> foldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\"> list\n</span></code></pre>\n<p>(We don't allow <code>'</code> as part of a variable name, as it's not really\nnecessary and is confusing for non-Haskeller beginners. An undercore\nsuffices.)</p>\n<p>Now, looking in detail without the <code>--concise</code> arg, just before the\nrecursion, we see the force of the addition:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">case </span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">) </span><span style=\"color:#859900;\">of\n  </span><span style=\"color:#cb4b16;\">Nil </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#6c71c4;\">0\n  </span><span style=\"color:#cb4b16;\">Cons</span><span style=\"color:#657b83;\"> x xs </span><span style=\"color:#859900;\">-&gt;\n    case</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">of\n      !</span><span style=\"color:#657b83;\">z_ </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> foldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) z_ xs\n</span><span style=\"color:#859900;\">case</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">0 1 </span><span style=\"color:#859900;\">of\n  !</span><span style=\"color:#657b83;\">z_ </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> foldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) z_ (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#859900;\">case</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">y </span><span style=\"color:#859900;\">-&gt; </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">of\n  !</span><span style=\"color:#657b83;\">z_ </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> foldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) z_ (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#859900;\">case </span><span style=\"color:#6c71c4;\">0 </span><span style=\"color:#859900;\">+ </span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">of\n  !</span><span style=\"color:#657b83;\">z_ </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> foldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) z_ (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)\n</span><span style=\"color:#859900;\">case </span><span style=\"color:#6c71c4;\">1 </span><span style=\"color:#859900;\">of\n  !</span><span style=\"color:#657b83;\">z_ </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> foldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) z_ (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)\nfoldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)\n</span></code></pre>\n<p>And finally, taking a glance with <code>--concise</code>, we see:</p>\n<pre style=\"background-color:#fdf6e3;\">\n<code><span style=\"color:#859900;\">$</span><span style=\"color:#657b83;\"> duet run examples</span><span style=\"color:#859900;\">/</span><span style=\"color:#657b83;\">folds</span><span style=\"color:#859900;\">-</span><span style=\"color:#657b83;\">strictness</span><span style=\"color:#859900;\">.</span><span style=\"color:#657b83;\">hs </span><span style=\"color:#93a1a1;\">--main main_foldl_ --concise\n</span><span style=\"color:#657b83;\">foldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">0</span><span style=\"color:#657b83;\"> list\nfoldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">1</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">2</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)))\nfoldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">3</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">))\nfoldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">6</span><span style=\"color:#657b83;\"> (</span><span style=\"color:#cb4b16;\">Cons </span><span style=\"color:#6c71c4;\">4 </span><span style=\"color:#cb4b16;\">Nil</span><span style=\"color:#657b83;\">)\nfoldl_ (</span><span style=\"color:#859900;\">\\</span><span style=\"color:#657b83;\">x y </span><span style=\"color:#859900;\">-&gt;</span><span style=\"color:#657b83;\"> x </span><span style=\"color:#859900;\">+</span><span style=\"color:#657b83;\"> y) </span><span style=\"color:#6c71c4;\">10 </span><span style=\"color:#cb4b16;\">Nil\n</span><span style=\"color:#6c71c4;\">10\n</span></code></pre>\n<p>Which spells out quite clearly that now we are: (1) doing direct\nrecursion, and (2) calculating the accumulator with each recursion\nstep (<code>0</code>, <code>1</code>, <code>3</code>, <code>6</code>, <code>10</code>).</p>\n<h2 id=\"concluding\">Concluding</h2>\n<p>This post serves as both knowledge sharing for our team and a public\npost to show the kind of detailed level of training that we're doing\nfor our clients.</p>\n<p>If you'd like Haskell training for your company,\n<a href=\"mailto:sales@fpcomplete.com\">contact us</a> to arrange a meeting.</p>\n<p>Want to read more about Haskell? Check out <a href=\"https://www.fpcomplete.com/blog/\">our blog</a> and our <a href=\"https://www.fpcomplete.com/haskell/\">Haskell\nhomepage</