Whisper of the Heartman

Bevy: getting started on NixOS

Sat, 28 Aug 2021 15:31:35 GMT

A little while ago, I received the good news that I had gotten two (TWO 🎉) of my talk proposals accepted by NDC Oslo this year: one on profiling performance issues and one on building Rust games for the browser 🥳 While this is great, it also means that I need to start doing some serious work, because these talks don't exist yet.

For the talk on building games with Rust, I knew immediately that I wanted to use the Bevy game engine. However, I've never used Bevy before, so I thought I'd start with the Bevy Book and [[https://bevyengine.org/learn/book/getting-started/][its /Getting Started/ section]]. For most cases, the docs should be sufficient, but if you run NixOS (I do 🙋), then you might run into some cases not covered by the guide. This post details all the extra steps I had to take to get through the tutorial. At the end, I'll put the whole shell.nix file that I ended up with as well as any other changes I had to make.

This was for Bevy 0.5 running on NixOS 21.05 on a Dell XPS 9570 using the NVIDIA graphics card. If your system is different, the steps in this post may or may not work for you. But if you try and follow the steps in this post, do let me know how it went (@ me on Twitter!)!

Oh, and before we get started: I'd like to extend a big thank you to everyone who helped me out in the Bevy discord as well as to the people behind the engine itself and the docs ❤️

Initial setup

Let's get a dev environment going with the dependencies we need. The Bevy GitHub repo has a [[https://github.com/bevyengine/bevy/blob/main/docs/linux_dependencies.md#nixos][linux setup file (with a specific section for NixOS!)]] that tells you what you need. As mentioned in the text, add this to a build.rs file (in your project root):

fn main() {
    if cfg!(target_os = "linux") {
        println!("cargo:rustc-link-lib=vulkan");
    }
}

It also gives you a Nix shell that you can use. It contains most dependencies, but as we'll see, we need to modify it slightly (or at least I had to). The version that's there is:

{ pkgs ? import <nixpkgs> { } }:
with pkgs;
mkShell {
  buildInputs = [
    cargo
    pkgconfig udev alsaLib lutris
    x11 xorg.libXcursor xorg.libXrandr xorg.libXi
    vulkan-tools vulkan-headers vulkan-loader vulkan-validation-layers
  ];
}

Problem 1: fast compilation

The first issues I ran into were in the /Setup/ section of the guide. Specifically around fast compilation.

The guide says to copy this Cargo config file into YOUR_WORKSPACE/.cargo/config.toml. This config file enables fast compilation by configuring the linker and specific Rust flags. The configuration for building on Linux looks like this:

[target.x86_64-unknown-linux-gnu]
linker = "/usr/bin/clang"
rustflags = ["-Clink-arg=-fuse-ld=lld", "-Zshare-generics=y"]

If you've used NixOS for a bit, you'll probably realize why this won't work: there's no /usr/bin directory on NixOS. Also, if you look at the shell.nix file from before, there's no clang or lld included.

We'll fix this by changing the linker to just clang and by adding clang and lld to the shell.nix build inputs. That means the .cargo/config.toml file should look like this:

[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-Clink-arg=-fuse-ld=lld", "-Zshare-generics=y"]

Problem 2: Apps

The next issue I ran into was in the next section, /Apps/. The basic "Hello World" program you get when starting a new project via Cargo built and ran just fine. However, it suddenly stopped working when I tried to import and use Bevy.

The error I got was:

target/debug/<project>: error while loading shared libraries:
libudev.so.1: cannot open shared object file: No such file or
directory

The solution was to modify the environment's LD_LIBRARY_PATH by adding udev. This took some searching to find out, but thanks to this GitHub comment and @LegendOfMiracles [[https://discordapp.com/channels/691052431525675048/742884593551802431/843481717318484018][post in the Bevy Discord's #help channel]], I got there in the end.

In addition to udev we also need to add alsaLib to the LD_LIBRARY_PATH. To achieve this, add this to your shell.nix:

shellHook = ''export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:${pkgs.lib.makeLibraryPath [
  pkgs.alsaLib
  pkgs.udev
]}"'';

Problem 3: Plugins

The third and final problem I ran into was with [[https://bevyengine.org/learn/book/getting-started/plugins/][the section on /Plugins/]]. When I added DefaultPlugins, the program immediately panicked with this message:

Finished dev [unoptimized + debuginfo] target(s) in 0.08s
 Running `target/debug/bevy-tutorial`
thread 'main' panicked at 'Unable to find a GPU! Make sure you have installed required drivers!', /home/thomas/.cargo/registry/src/github.com-1ecc6299db9ec823/bevy_wgpu-0.5.0/src/wgpu_renderer.rs:47:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Turns out the solution to this was to modify the LD_LIBRARY_PATH further and add vulkan-loader too. When I added the vulkan-loader everything ran and worked as expected.

Summary and full files

It took some extra work, but I got there in the end. Below are the files that I ended up with. In addition to these, remember to also add the build.rs file as specified above.

`shell.nix`

Here's what I use for the dev environment. I use fenix for managing the Rust toolchain. I've also added cargo-edit and cargo-watch because they're handy tools to have. Everything else is Bevy-related.

{ pkgs ? import <nixpkgs> {} }:

let

  fenix = import "${
  fetchTarball "https://github.com/nix-community/fenix/archive/main.tar.gz"
  }/packages.nix";

in

pkgs.mkShell {
  shellHook = ''export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:${pkgs.lib.makeLibraryPath [
    pkgs.alsaLib
    pkgs.udev
    pkgs.vulkan-loader
  ]}"'';

  buildInputs = with pkgs; [
    (
      with fenix;
      combine (
        with default; [
          cargo
          clippy-preview
          latest.rust-src
          rust-analyzer
          rust-std
          rustc
          rustfmt-preview
        ]
      )
    )
    cargo-edit
    cargo-watch

    lld
    clang

    # # bevy-specific deps (from https://github.com/bevyengine/bevy/blob/main/docs/linux_dependencies.md)
    pkgconfig
    udev
    alsaLib
    lutris
    x11
    xorg.libXcursor
    xorg.libXrandr
    xorg.libXi
    vulkan-tools
    vulkan-headers
    vulkan-loader
    vulkan-validation-layers
  ];

}

`.cargo/config.toml`

This is what the full config.toml file looks like after adjusting the Linux config for NixOS.

# Add the contents of this file to `config.toml` to enable "fast build" configuration. Please read the notes below.

# NOTE: For maximum performance, build using a nightly compiler
# If you are using rust stable, remove the "-Zshare-generics=y" below.

[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-Clink-arg=-fuse-ld=lld", "-Zshare-generics=y"]

# NOTE: you must manually install https://github.com/michaeleisel/zld on mac. you can easily do this with the "brew" package manager:
# `brew install michaeleisel/zld/zld`
[target.x86_64-apple-darwin]
rustflags = ["-C", "link-arg=-fuse-ld=/usr/local/bin/zld", "-Zshare-generics=y"]

[target.x86_64-pc-windows-msvc]
linker = "rust-lld.exe"
rustflags = ["-Zshare-generics=y"]

# Optional: Uncommenting the following improves compile times, but reduces the amount of debug info to 'line number tables only'
# In most cases the gains are negligible, but if you are on macos and have slow compile times you should see significant gains.
#[profile.dev]
#debug = 1

That's it for now. I hope this is useful to someone.

Until next time! 👋

2020 wrap-up

Sat, 09 Jan 2021 17:17:23 GMT

At the tail end of 2019, I published a post where I summarized what I'd done in 2019 and laid plans for 2020, including a list of goals. Now that 2020 is (finally) over, I thought it'd be a good time to review the year based on those goals.

I think that for the most of us, 2020 really didn't pan out the way we'd planned. It certainly didn't for me. But despite the (still) raging pandemic and all the other stuff that the world threw at us, I'm quite content with the year. It turned out very different to what I had imagined, but that's not necessarily a bad thing.

In particular, I have gained some very close friends this year, and my social life has, contrary to what you might expect, gotten much better. A better and more active social life means less time spent alone in front of a computer and thus less time for working on the goals I set out for the year.

I also started writing articles for LogRocket, which I consider a significant step forward. However, being paid to write elsewhere naturally means that I spend less time writing for my own blog, for better or worse.

2020: Review

I set myself a number of so-called SMART (Specific, Measurable, Achievable, Relevant, Time-bound) goals last year. Some I achieved, some I made fair progress on, and others I failed completely. Let's have a look!

Programming language theory

Goal: Read Pierce's Types and Programming Languages by July.

Status: Failed.

I'd almost forgotten about this one, but not for lack of interest. I started working on the book around March or so, but found that I was missing some basic mathematical background. So I followed the book's advice and picked up a number of other books to make sure I knew what was going on: Halmos' Naive Set Theory, Winskel's /The Formal Semantics of Programming Languages/, and Davey & Priestley's Introduction to Lattices and Order.

I made it through Naive Set Theory before embarking on Introduction to Lattices and Order, which I have not yet finished. It seems this could take a while longer than I had anticipated.

Community engagement

Goal: Give at least three talks/workshops at community events and apply to speak at at least three conferences by the end of the year.

Status: Failed ... but wait!

This is the goal that was impacted the most by the pandemic this year. It started well, but the cancellation of most in-person meetups got in the way of presenting at local events. I also only applied to two conferences this year, but one of those went through, and I spoke at NDC Oslo this summer about contributing to Open Source.

So while the exact targets specified in the goal weren't met, I'd say the impact of getting that first conference talk is big enough for me to think that maybe the minutiae don't matter that much, and that the essence of the goal was achieved.

Emacs

Goal: Read the Emacs manual by February 1^st.

Status: Achieved.

At least the year started off well! While the manual was much longer than I thought, I did get through it by the deadline ([[https://blog.thomasheartman.com/posts/tips-and-tricks-for-the-fledgling-emacs-user][and wrote about it shortly thereafter]]), so this is an easy pass. I also read the org mode manual a bit later and picked up some more tips and tricks.

As a corollary to the goal, I mentioned that I might try and configure Emacs from the ground up (instead of basing it on Spacemacs), but that that wasn't a necessity. I'm happy to report that I did do just that around August or so. As part of configuring Emacs from the ground up, I also decided I'd try and go without Vim key bindings. That went well for about six months until I accidentally (not really) configured Evil again just a few days ago. Whoops!

Nix

Goal: Write about build.nix, default.nix, and shell.nix by the end of the year.

Status: Failed. Thoroughly.

I'm not entirely sure why I didn't get around to this. I think I was planning on doing this later in the year, at which point the pandemic was very much happening and I was busy doing other things that felt more important at the time.

I'd still like to sit down and do this at some point, but I don't know when just yet.

OpenShift

Goal: Pass the [[https://www.redhat.com/en/services/training/ex288-red-hat-certified-specialist-openshift-application-development-exam][Red Hat Certified Specialist in OpenShift Application Development exam]] by the end of the year.

Status: Failed.

This is another goal that almost fell off my radar. Because I was doing this in relation to work and work got rather busy for a while, I never quite got around to working on this. Until September/October, when I spent a lot of time reading up on this and preparing.

I took the exam again on January 6^th (already too late for the goal, but close enough; I scheduled it within the deadline) and failed. Again. With the exact same score as last time. And while I did clear all the stuff that I didn't previously, I scored 0% on some of the stuff that I know really well. I blame running out of time and not reading the problem text properly.

Oh, well. At least I learned a lot, right?

Containers/infrastructure/Haskell/Nix

Goal: Manage a Kubernetes cluster by April 1^st, run a Haskell API (built with Nix) on it by June 1^st.

Status: Failed.

I made some progress on this and did have the Kubernetes cluster up and running (on DigitalOcean) in time for the first deadline. But then I just never got around to creating or setting up the Haskell API.

2021: New goals?

So what about 2021? Do I want to set any new goals? Much like [[https://daverupert.com/2020/12/twenty-twenty/][Dave Rupert]], I think I'll ease up on setting myself any goals for 2021. The pandemic is still very much present and will likely shape much of the coming year, even with vaccinations being rolled out at the moment. Based on the occurrences of the past week (/looking at you, America!/), I'm also not sure whether this year is going to be any easier than the last one in other regards. So for now, I think I'll just roll with the punches and take things one at a time. I'm working on a couple things, but I'll save those for later.

So a bit late, but happy new year to all of you! You survived! Now let's make the most of it.

Rust 2021

Sun, 27 Sep 2020 20:05:45 GMT

This is my response to the Rust Core Team's 2021 Roadmap call for blogs.

A year ago, stabilizing slice patterns was at the top of my wish list for Rust 2020. I'd been waiting for that for a long time. A few months later, it landed on the stable channel with release 1.42. I don't expect to be quite as lucky time around, but one can dream.

At the outset, I didn't really have any strong wishes for the next year. But when I sat down and thought about it, I came up with two things that I've run into lately that I know are being worked on and that I would love to see finalized.

Besides these two, there's const generics and GATs. High-profile features that get a lot of love in the community. The thing about these two, however, is that whenever I read up on them, they seem great and super useful, but I don't think I've actually run into any cases where I need them while programming. So while I hope the work on these continues, I want to turn my attention to two other points: nested OR-patterns and trait aliases.

Nested OR-patterns

This was mentioned as an upcoming feature in the post Recent and future pattern matching improvements from the Inside Rust Blog in March. In short, it allows you to nest patterns when pattern matching.

Imagine you have an Option<u8> and you want to check whether the possibly contained u8 is 4 or 13. Without using the or_patterns, you'd have to do something like this:

fn is_unlucky(n: Option<u8>) -> bool {
    match n {
        Some(4) => true,
        Some(13) => true,
        _ => false,
    }
}

Notice that double Some check? Using the or_patterns feature, we can simplify this:

// make sure you're on nightly!
#![feature(or_patterns)]

fn is_unlucky(n: Option<u8>) -> bool {
    match n {
        Some(4 | 13) => true,
        _ => false,
    }
}

As always, this is a simplified and contrived example, but it can simplify code considerably in some cases. And yes, I'm sure it can make some code even harder to read too, but that's a tradeoff I'm willing to make.

So where are we on this? There is an open tracking issue for it. But there seems to be some blocking issues at the moment, including one that makes the compiler panic, so I'm not sure what the status is. There hasn't been any activity on this since July, but I'm hopeful that we might see some progress on this in the coming year.

Trait aliases

Another feature that I've wished for at times is trait aliases (thanks to or Igor Aleksanov (popzxc) for mentioning this in their Rust 2021 post). This feature would allow you to refer to a collection of traits with an alias that you choose.

This is particularly nifty in situations where you use the same set of traits to define trait bounds over a number of functions. Instead of specifying the same combination over and over again, you can give the combination a name and use that name from there on.

Using Aleksanov's post as inspiration again, we could have something like this (without trait aliases):

trait MyTrait {}

fn f<T>(x: &T)
where
    T: MyTrait + Send + std::fmt::Debug,
{
    // do something with x here
}

fn g<T>(x: &T)
where
    T: MyTrait + Send + std::fmt::Debug,
{
    // do something else with x here
}

There are a few things about this code that could be improved with trait aliases. Most notably is the fact that the combination of the three traits we're using (MyTrait, Send, std::fmt::Debug) is repeated several times. It's not clear from the context whether these are intentionally or accidentally the same. If they are supposed to be the same, then when changing the combination one place, you'd have to remember to change it everywhere else. And while it's not necessarily bad, this combination is fairly verbose.

By using a trait alias, we can also give this combination of traits a more meaningful and descriptive name. This can help us better communicate to other developers (and to our future selves) why this combination is what it is.

Using the trait_alias feature on nightly, we can instead write it like this:

// again: you need to be on the nightly channel for this
#![feature(trait_alias)]

trait MyTrait {}

trait MyExtendedTrait = MyTrait + Send + std::fmt::Debug;

fn f<T>(x: &T)
where
    T: MyExtendedTrait,
{
    // do something with x here
}

fn g<T>(x: &T)
where
    T: MyExtendedTrait,
{
    // do something else with x here
}

At first, it might only look like we've added a line or two. But by grouping the trait combination under a new alias, we've ensured that the functions will change their trait bounds in lockstep. We have also given the trait combination a more meaningful name, though, admittedly, MyExtendedTrait isn't my strongest effort.

Much like with nested OR-patterns, there is a tracking issue issue for this. There seems to have been some activity back in March, but not much since.

That's it for now!

It's easy to just sit back and wish for new features when you're not the one implementing them. I'm very grateful to all the Rust teams for all their efforts, and I trust them wholeheartedly on what features to focus on in the coming year, whether they align with my wish list or not.

Rust continues to be one of the most exciting programming languages around and with one of the most welcoming communities to boot! As always: To everyone involved with the language, the ecosystem, and the community (and all the other parts that I'm forgetting): thank you very much. I'm looking forward to another year (and many more) with you!

Understanding lifetimes

Sun, 20 Sep 2020 09:50:42 GMT

This article was originally published by LogRocket on their blog under
the title /Understanding lifetimes in Rust/. You can find their
version [[https://blog.logrocket.com/understanding-lifetimes-in-rust/][here]].

You sit down for another crack at this Rust-thing. Last time went pretty smoothly except for some minor hiccups with the borrow checker. But you got through it and gained a slightly better understanding of how it works in the process. Maybe it's all worth it in the end?

Today, though, you've got some grand plans and you're not going to let the borrow checker stop you! You can practically feel the energy coursing through your veins as you imprint your thoughts on the keyboard and translate them into pure Rust. This must be that sweet feeling you've heard so much about. You save your project, start the compilation process, and ...

error[E0597]: `x` does not live long enough

You sigh. Not this again.

You take a deep breath, lower your shoulders, and read the error message one more time. 'Does not live long enough'? What does that even mean?

Introducing lifetimes

You've run into another one of Rust's peculiarities: lifetimes.

Lifetimes are how the Rust compiler keeps track of how long references are valid for. If you remember the article on understanding the borrow checker, you'll know that checking references is one of the borrow checker's main responsibilities. Lifetimes help the borrow checker ensure that you never have invalid references.

Lifetime annotations, then, let you tell the borrow checker how long references are valid for. In many cases, the borrow checker can infer the correct lifetimes and take care of everything on its own. But often it needs your help to figure it out.

In this post, we'll go over the basics of lifetimes and annotations and how to work with them. We'll also look at some common scenarios you might run into and how you can solve them with lifetimes. I'm expecting a basic grasp of Rust and some if its concepts (such as the borrow checker), but nothing particularly deep.

Lifetime annotations

Before we go any further, just a short note on the notation of lifetimes, as it's a bit different from what you get in a lot of other languages.

Lifetimes are annotated by a leading apostrophe followed by a variable name. When talking about generic lifetimes, we often use single, lowercase letters, starting from ~'a~, ~'b~, etc. However, there is nothing stopping you from using longer, more explanatory names if that suits you better.

Why we need lifetimes

If they're such a weird feature, then why do we need lifetimes? The answer, my friend, lies in Rust's ownership model. The borrow checker takes care of allocating and freeing memory and also of making sure that no references point to memory that has been freed. Like borrows, lifetimes are checked at compile-time, which means that your program can't compile if the borrow checker deems the references invalid.

In particular, lifetimes are important to keep in mind when returning references from functions and when creating structs with references. These are both common situations, and it's easy to get lost if you don't understand what's going on.

The explanatory example

Ultimately, lifetimes are a matter of scope. Values get dropped when they go out of scope and any references to them after they have been dropped are invalid.

The simplest way to demonstrate lifetimes is something like the following example, shamelessly stolen/adapted from the Book's chapter on lifetimes.

// this code sample does *not* compile
{
    let x;
    {                           // create new scope
        let y = 42;
        x = &y;
    }                           // y is dropped

    println!("The value of 'x' is {}.", x);
}

This little piece of code has two distinct scopes. When the inner scope closes, y is dropped. At that point, even if x is still available in the outer scope, the reference is invalid because the value it pointed to is dropped: The value that x points to 'does not live long enough'.

In lifetime jargon, we can say that the outer scope has the lifetime ~'outer~ and the inner scope the lifetime ~'inner~. ~'outer~ clearly outlives ~'inner~ in this case. When ~'inner~ ends, all values with that lifetime are invalidated.

Lifetime elision

When writing functions that accept references as arguments, the compiler can infer the correct lifetimes in many cases, saving us the trouble of writing them out by hand. When lifetime annotations are implicit, we call this lifetime elision.

The compiler uses three rules to figure out whether lifetime annotations can be elided or not. The section on lifetime elision talks about these rules in detail, but the short form is that you can elide lifetime annotations in functions if one of the following is true:

The function doesn't return a reference.
There is exactly one reference input parameter.
The function is a method, taking &self or &mut self as the

first parameter.

Examples and common problems

Lifetimes are a tricky thing to wrap your head around, and it's unlikely that a wall of text will make you really understand how they work. The best way to get a proper understanding is of course to play around with them yourself and solve problems, but a couple of examples can go a long way.

Returning references from functions

You can't return a reference from a function without also passing in a reference. If you try, you'll find that the reference is invalid as soon as the function returns and your program won't compile^[1].

If your function takes exactly one reference parameter, then you'll be fine without annotations. All output references will be given the same lifetime as the input parameter. As such, this simple function will compile just fine, even if there are no explicit lifetime annotations:

fn f(s: &str) -> &str {
    s
}

However, if we add another input string parameter (even if we don't use it), we'll suddenly not be able to compile this:

// this code sample does *not* compile
fn f(s: &str, t: &str) -> &str {
    if s.len() > 5 { s } else { t }
}

This is because of how the automatic lifetime annotation works. When a function accepts multiple references, they're each given their own lifetime. We know that the returned reference must be one of the references we received as an input argument, but we don't know which one. What goes in place of the ~'???~ below?

// this code sample does *not* compile
fn f<'a, 'b>(s: &'a str, t: &'b str) -> &'??? str {
    if s.len() > 5 { s } else { t }
}

Imagine that we want to use the returned value outside of this function. What lifetime would we assign to it? The only thing we can guarantee is that the reference we return is valid for at least as long as the shortest-lived reference we pass into the function. That tells the compiler that these two references are definitely valid for the shorter lifetime. We cannot give any guarantees outside of that.

The way we achieve this, is by giving both input parameters the same lifetime annotation. It's how we tell the compiler that 'as long as both of these input parameters are valid, so is the returned value'.

fn f<'a>(s: &'a str, t: &'a str) -> &'a str {
    if s.len() > 5 { s } else { t }
}

If you're returning a reference from a function that takes multiple input lifetime parameters, but you know exactly which one it is you're returning, you can annotate that specific lifetime. That way, the relationship between the lifetimes doesn't matter anymore.

fn f<'a, 'b>(s: &'a str, _t: &'b str) -> &'a str {
    s
}

Structs with references

References in structs can be a real hassle. You're often better off avoiding them and using owned values instead. That way, you don't need to worry about references being invalidated and lifetimes not lasting long enough. In my experience, it's usually also what you want.

However, there are certain cases where structs with references are exactly what you want. In particular: if you want to create a 'view' into something else. Using structs with references is a great way of organizing some data into a package that's easier to handle, without moving or copying data. This means that the original data source can still be referenced elsewhere and that we save on the extra work it would be to clone the data.

Here's an example. Imagine that we want to find the first and the last sentence of a paragraph and keep them in a struct S. Because we don't want to copy the data, we need to use references and give them lifetime annotations.

struct S<'a> {
    first: &'a str,
    last: &'a str,
}

We could use a function like this to populate the struct. For simplicity's sake, we'll assume that a full stop is the only sentence-ending punctuation mark in use. If the paragraph is empty, we'll return None, and if there is only a single sentence, we'll use that as both the first and the last sentence:

fn try_create(paragraph: &str) -> Option<S> {
    let mut sentences = paragraph.split('.').filter(|s| !s.is_empty());
    match (sentences.next(), sentences.next_back()) {
        (Some(first), Some(last)) => Some(S { first, last }),
        (Some(first), None) => Some(S { first, last: first }),
        _ => None,
    }
}

Notice how we don't need to annotate lifetimes in the function signature because the compiler can figure it out for us. In a case like this, there is really only one choice: the lifetime of the input string. Pretty neat, huh?

Summary and further reading

This has been a cursory glance at lifetimes and lifetime annotations. We have glossed over a lot of the finer and more intricate details of how lifetimes work, but we've covered enough ground that you should be able to reason about them when you run into an issue.

Because lifetimes are such an important part of Rust, I encourage you to have a look at the Validating References with Lifetimes chapter of The Book if you want a more comprehensive introduction.

Furthermore, if you feel like you've got a decent grasp on lifetimes but want to dive a bit deeper, check out Jon Gjengset's excellent video Crust of Rust: Lifetime Annotations, where he explores a case that needs multiple explicit lifetime annotations. He also gives a great introduction to lifetime annotations in general, so it's well worth a watch just for that.

Footnotes

[1]

Well, you can use the ~'static~ lifetime, but that's probably not what you want. It's also outside the scope of this article, so let's forget about that for now.

[1]: Well, you can use the ~'static~ lifetime, but that's probably not what you want. It's also outside the scope of this article, so let's forget about that for now.

Meet the borrow checker

Sun, 16 Aug 2020 13:04:31 GMT

This article was originally published by LogRocket on their blog under
the title /Understanding the Rust borrow checker/. You can find their
version [[https://blog.logrocket.com/introducing-the-rust-borrow-checker/][here]].

You've heard a lot about it, you've bought into the hype, and the day has finally come. It's time for you to start writing Rust!

So you sit down---hands on the keyboard, heart giddy with anticipation---and write a few lines of code. You run the cargo run command, excited to see whether the program works as expected. You've heard Rust is one of those 'once it compiles, it works' languages and want to test it for yourself. The compiler starts up, you follow the output, when suddenly:

error[E0382]: borrow of moved value

Uh-oh. Seems like you've run into ... (puts on scary voice) the /borrow checker/! Dun, dun, DUUUUUUN!

The ... borrow checker?

That's right. The borrow checker is an essential part of the Rust language and part of what makes Rust Rust. The borrow checker helps you (or forces you to) manage ownership. As chapter 4 (/Understanding Ownership/) of [[https://doc.rust-lang.org/book/][the Rust Programming Language]] puts it: "Ownership is Rust's most unique feature, and it enables Rust to make memory safety guarantees without needing a garbage collector."

Ownership, borrow checker, and garbage collectors: There's a lot to unpack in the above paragraph, so let's break it down a bit. We'll look at what the borrow checker does for us (and what it stops us from doing), what guarantees it gives us, and how it compares to other forms of memory management. I'll assume that you have some experience with writing code in higher level languages such as Python, JavaScript, or C#, but not necessarily that you're familiar with how computer memory works.

Garbage collection vs manual memory allocation vs the borrow checker

Let's talk about memory and memory management for a minute. In most popular programming languages, you don't need to think about where your variables are stored. You simply declare them and the language runtime takes care of the rest via a garbage collector. This abstracts away how the computer memory actually works and makes it easier and more uniform to work with. This is a good thing.

However: we need to peel back a layer to talk about how this compares to the borrow checker. We'll start by looking at the stack and the heap.

The stack and the heap

Your programs have access to two kinds of memory where it can store values: the stack and the heap. These differ in a number of ways, but for our sake, the most important difference is that data that is stored on the stack must have a known, fixed size. Data on the heap can be of any arbitrary size.

What do I mean by size? The size is how many bytes it takes to store the data. In broad terms, certain data types, such as booleans, characters, and integers, have a fixed size. These are easy to put on the stack. On the other hand, data types such as strings, lists, and other collections, can be of any arbitrary size. As such, they cannot be stored on the stack, and we must instead use the heap.

Because data of arbitrary size can be stored on the heap, the computer needs to find a chunk of memory large enough to fit whatever we are looking to store. This is time-consuming, and the program also doesn't have direct access to the data as with the stack, but is instead left with a pointer to where the data is stored.

A pointer is pretty much what it says on the tin. It points to some memory address on the heap where the data you're looking for can be found. There's any number of pointer tutorials out there on the web, and which one works for you may depend on your background. But for a quick primer, [[https://dev.to/codemouse92/comment/28fl][here's one I found that explains C pointers pretty well]] (by Jason C. McDonald).

What is the point of having these two different memory stores? Because of the way the stack works, data access on the stack is very fast and easy, but requires the data to conform to certain standards. The heap is slower, but more versatile, and is thus useful for when you can't use the stack.

Garbage collection

In garbage collected languages, you don't need to worry about what goes on the stack and what goes on the heap. Data that goes on the stack gets dropped once it goes out of scope. Data that lives on the heap is taken care of by the garbage collector once it's no longer needed.

In languages like C, on the other hand, you need to manage memory yourself. Where you might simply initialize a list in higher level languages, you need to manually allocate memory on the heap in C. And when you've allocated memory, you should also free the memory once you're done with it to avoid memory leaks. But take care: Memory should only be freed once.

This process of manual allocation and freeing is error-prone. In fact, a Microsoft representative has said that [[https://thenewstack.io/microsoft-rust-is-the-industrys-best-chance-at-safe-systems-programming/][70% of all of Microsoft's vulnerabilities and exploits are memory-related]]! So why would you use manual memory management? Because it allows for more control and can often give better performance characteristics than garbage collection. The program doesn't need to stop what it's doing and spend time finding out what it needs to clean up before cleaning it up.

Rust's ownership model feels like something in between. By keeping track of where data is used throughout the program and by following a set of rules, the borrow checker is able to determine where data needs to be initialized and where it needs to be freed (or dropped, in Rust terms). It's like it auto-inserts memory allocations and frees for you, giving you the convenience of a garbage collector, but with the speed and efficiency of manual management.

The way this comes out in practice, is that when passing variables around you can do one of three things. You can move the data itself and give up ownership in the process. You can create a copy of the data and pass that along. Or you can pass a reference to the data and retain ownership, letting the recipient borrow it for a while. Which one is more appropriate depends entirely on the situation.

Other borrow checker superpowers: paralyzed or parallelized?

In addition to handling memory allocation and freeing for the programmer, the borrow checker also prevents data races (though not general race conditions) through its set of sharing rules.

These same borrowing rules also help you work with concurrent and parallel code without having to worry about memory safety, enabling Rust's [[https://doc.rust-lang.org/book/ch16-00-concurrency.html][fearless concurrency]].

Drawbacks

But as with all good things in life, Rust's ownership system comes with it's set of drawbacks. Indeed, without any drawbacks, this article probably wouldn't exist. The borrow checker can be tricky to understand and work with. So much so that it's pretty common for newcomers to the Rust community to get stuck 'fighting the borrow checker' (and yes, I've personally lost many hours of my life to that struggle).

For instance, sharing data can suddenly become a problem, especially if you need to mutate it at the same time. Certain data structures that are super easy to create from scratch in other languages are very hard to get right in Rust. For a good example of the latter, check out the book [[https://rust-unofficial.github.io/too-many-lists/index.html][Learn Rust With Entirely Too Many Linked Lists]]. It goes through a number of ways to implement a linked list Rust and details all the issues the author ran into on the way there. It's both informative and very entertaining, so it's well worth a look.

But once you get on board with the borrow checker, things start to improve. I quite like Reddit user dnkndnts' explanation from this comment:

[The borrow checker] operates by a few simple rules. If you don't understand or at least have some intuition for what those rules are, then it's going to be about as useful as using a spell checker to help you write in a language you don't even know: it'll just reject everything you say. Once you know the rules the borrow checker is based on, you'll find it useful rather than oppressive and annoying, just like a spell checker.

And what are these rules? Here are the two most important ones to remember concerning variables that are stored on the heap:

When passing a variable (instead of a reference to the variable) to

another function, you are giving up ownership. The other function is now the owner of this variable and you can't use it anymore.

When passing references to a variable (lending it out), you can have

either as many immutable borrows as you want or a single mutable borrow. Once you start borrowing mutably, there can be only one.

In practice

With some understanding of what the borrow checker is and how it works, let's examine how it affects us in practice. We'll be working with the Vec<T> type, which is Rust's version of a growable list (analogous to Python's lists or JavaScript's arrays). Because it doesn't have a fixed size, a Vec needs to be heap-allocated.

The example may be contrived, but it demonstrates the basic principles. We'll create a vector, call a function that simply accepts it as an argument, and then try and see what's inside later.

Note: this code sample does not compile.

fn hold_my_vec<T>(_: Vec<T>) {}

fn main() {
    let v = vec![2, 3, 5, 7, 11, 13, 17];
    hold_my_vec(v);
    let element = v.get(3);

    println!("I got this element from the vector: {:?}", element);
}

When trying to run this, you'll get the following compiler error:

error[E0382]: borrow of moved value: `v`
--> src/main.rs:6:19
          |
        4 |     let v = vec![2, 3, 5, 7, 11, 13, 17];
          |         - move occurs because `v` has type `std::vec::Vec<i32>`, which does not implement the `Copy` trait
        5 |     hold_my_vec(v);
          |                 - value moved here
        6 |     let element = v.get(3);
          |                   ^ value borrowed here after move

The above message tells us that Vec<i32> doesn't implement the ~Copy~ trait, and as such must be moved (or borrowed). The Copy trait is only implementable by data types that can be put on the stack, and because Vec must go on the heap, it cannot implement Copy. We need to find another way around this.

Attack of the clones

Even though a Vec can't implement the Copy trait, it can (and does) implement the ~Clone~ trait. In Rust, cloning is another way to make duplicates of data. But while copying can only be done on stack-based values and is always very cheap, cloning also works on heap-based values and can be very expensive.

So if the function takes ownership of the value, why don't we just give it a clone of our vector? That'll make it happy, right? Indeed, the below code works just fine.

fn hold_my_vec<T>(_: Vec<T>) {}

fn main() {
    let v = vec![2, 3, 5, 7, 11, 13, 17];
    hold_my_vec(v.clone());
    let element = v.get(3);

    println!("I got this element from the vector: {:?}", element);
}

However, we have now done a lot of extra work for nothing! The hold_my_vec function doesn't even use the vector for anything; it just takes ownership of it. In this case, our vector (v) is pretty small, so it's not a big deal to clone it, and in the just-getting-things-to-work-stage of development this may be the quickest and easiest way to see results. However, there is a better, more idiomatic way to do this. Let's have a look.

References

As mentioned previously, rather than giving away our variable to the other function, we can lend it to them. To do this, we need to change the signature of hold_my_vec to instead accept a reference by changing the type of the incoming parameter from Vec<T> to &Vec<T>.

We also need to change how we call the function and let Rust know that we're only giving the function a reference: a borrowed value. This way, we let the function borrow the vector for a little bit, but make sure that we get it back before continuing the program:

fn hold_my_vec<T>(_: &Vec<T>) {}

fn main() {
    let v = vec![2, 3, 5, 7, 11, 13, 17];
    hold_my_vec(&v);
    let element = v.get(3);

    println!("I got this element from the vector: {:?}", element);
}

Summary

It's worth noting that this is only a very brief overview of the borrow checker, what it does, and why it does it. A lot of the finer details have been left out to make this article as easy to digest as possible.

Often, as your programs grow, you'll find more intricate problems that require more thinking and fiddling with ownership and borrows. Oftentimes you'll even have to rethink how you've structured your program to make it work with Rust's borrow checker. It's a learning curve, for sure, but if you stick around and make your way to the top, you're sure to have learned a thing or two about memory along the way.

New domain: blog.thomasheartman.com

Sun, 12 Jul 2020 18:47:25 GMT

This is a heads-up to anyone subscribed to thomashartmann.dev's RSS feed: I recently acquired the 'thomasheartman.com' domain and have moved my blog over to blog.thomasheartman.com. The new RSS feed can be found at blog.thomasheartman.com/rss.xml.

If you have been reading my posts on my website recently, you should have been silently redirected to the new domain. With this post, this behavior is now made explicit. If you're reading this on the web (as opposed to in some sort of feed reader), you should also find that the URL says blog.thomasheartman.com, regardless of which site you accessed.

I'll leave thomashartmann.dev up and running to catch and redirect any potential traffic to blog.thomasheartman.com, but once this post is up, nothing else will get published to the former.

Notmuch: (un)tag outgoing mail

Sun, 12 Jul 2020 13:58:48 GMT

I'm here to save past me from spending hours searching the internet for information on how to make notmuch (the Emacs client, specifically) not apply default tags to outgoing mail.

The problem

So you've managed to set up OfflineIMAP to fetch mail from your server and you've set up msmtp to be able to send mail. It's taken hours (many more than you care to admit), but you've finally got a functioning email client directly embedded in Emacs! This is a good day! You celebrate by firing off an email to another email account you've got set up and ... what's this? You've got an unread email from yourself in your inbox?

On closer inspection, you realize it's the one you just sent off. But why is it in your inbox? And why is it tagged 'unread'?

Notmuch is a tag-based email system. Rather than folders, everything is done
through tagging, filtering, and searching. As such, your 'inbox' is just all
mail that's tagged 'inbox'.

You check your .notmuch-config and find that you've told notmuch to automatically tag new mail (pulled from the server) with 'inbox' and 'unread'. This makes sense, you think to yourself. After all, you'd like to get new mail into your inbox and you'd like it to be marked as unread. But why would notmuch apply the same tags to outgoing mail?

The solution

After spending hours trying to make notmuch's post-insert hook work (and still not quite understanding that one), you decide to check out where sent mail is stored, thinking you could use the location to filter the email that gets processed by the [new] section of your config. You remember reading that it relates to the 'Fcc' field, so you check out the description for the variable notmuch-fcc-dirs and find this nugget:

[...] the header should be of the form "folder +tag1 -tag2" where folder is the folder (relative to the notmuch mailstore) to store the message in, and tag1 and tag2 are tag changes to apply to the stored message.

Can it really be that simple? Is that really it?! All these hours for that? You set it to sent -unread -inbox and fire off an email.

...

Yup. That's it. Your message template now looks like this and behaves the way you want it to, both removing 'unread' and 'inbox' tags and adding a 'sent' tag so you can easily see all your sent mail:

From: Your Name &lt;yourname@provider.tld&gt;
To:
Subject:
Fcc: sent +sent -unread -inbox
--text follows this line--

org-babel-load-file doesn't tangle correctly

Thu, 21 May 2020 19:52:19 GMT

I recently started using Org mode for configuring Emacs and I really like it. However, I did run into an issue where, for some reason, org-babel-load-file didn't seem to update the tangled version of the file.

I thought perhaps it was an issue with caching at first, because if I manually tangled the file, everything would be just as I expected. However, if I started another instance of Emacs, it would re-evaluate the org-babel-load-file function, and I would end up with the wrong version of the file again.

I couldn't quite understand what was going on, but found the answer in a stack overflow post that I have sadly lost the link to: In Org mode, the official code for Emacs Lisp is emacs-lisp, but for code blocks, it also accepts elisp. From what I understand, elisp is set up as an alias for emacs-lisp. This seems to apply to manual tangling too, but not to org-babel-load-file.

And sure enough, I'd started using elisp instead of emacs-lisp partway through my config and those blocks wouldn't get tangled by org-babel-load-file. Going through and updating the blocks to use emacs-lisp instead of elisp was enough to get everything working again.

While the fix turned out to be simple, it was hard to find. So I'll leave this here, hoping it saves someone some time and effort in the future.

Remember: use #+begin_src emacs-lisp and not #+begin_src elisp.

70,000 words

Mon, 18 May 2020 20:41:10 GMT

Today marks a personal achievement of mine. A year ago, on May 18th 2019, I published the first post in a year-long chain of weekly content.

The decision to put something out every week was motivated in part by the words of Scott Hanselman, who says the most important thing is regularity. It doesn't matter if it's not stellar every time, as long as you're regular. As long as people can expect new content from you at regular intervals, you'll build a following.

Another important factor was this: if you force yourself to create and release content every week for a year, you will get better at it. Not only will you get better at writing and expressing yourself, but you will also learn to accept that things are never perfect and that you can (and must) release something nonetheless. You can't hold on to something until it is perfect, because it never will be. Learning to accept this is a key part of being a creator.

The ups and downs

As with everything else, committing to a weekly writing schedule has its share upsides and downsides. While finding a topic to write about every week is surprisingly easy, doing the research and putting words on the page is surprisingly hard and time-consuming.

The Good

The most obvious positive is that I've gotten a lot of writing practice. I've also gained some exposure within certain programming communities, and I've learned a lot about a wide variety of topics.

The most valuable thing, however, may be that I now have a body of work available on the internet. It may sound silly, but the satisfaction I get from looking back at how far I've come and how much I've produced is not to be understated.

The Bad

Time. Writing takes time. Writing takes a lot of time. Writing takes even more time when you are a bit of perfectionist and want to make sure everything is just right ™.

I estimate that I've spent, on average, about a work day on each post. That makes over four hundred hours in the past year and is equivalent to a little over 10 weeks worth of work, assuming a work week of about 40 hours.

This is a pretty serious cost. I can't remember the last time I had full weekend off. Sometimes you just want to relax, but you can't because you know you have to write that next post. It hasn't always been fun. I haven't always wanted to do this, but I've pushed through.

I've also set myself an increasingly high bar and specialized in a few key areas. I've created a system where it feels as if every post has to be of a certain length and about a certain topic. This has kept me from writing shorter posts and from exploring other topics, such as CSS and the .NET ecosystem, for instance.

Tangible outcomes

What do you get out of pouring all these hours and all this effort into a blog like this? Personal satisfaction and a feeling of accomplishment is one thing, but have I gotten anything tangible in return for my efforts?

Let's see:

Newsletters and reading lists: I've had posts featured on a number of weekly newsletters, including This Week in Rust, Haskell Weekly, and NixOS Weekly. As an avid reader of all of these, it's been a joy to see my name on the lists. I've also often seen links from other reading lists, such as Read Rust.
Rust release notes: And speaking of Rust, my post on advanced slice patterns got a special mention in the release notes for 1.42. Big, proud crab moment 🦀
Freelancer agreements: Off the back of the post mentioned above, I was recently contacted by (and subsequently signed a contract with) an external company to write content for their blog. By definition, I'm now a professional writer.
Being able to look up my own knowledge: More than a few times, I've found myself trying to recall how to use a specific git command or just how awk's syntax works. At times like these, knowing I've written about it previously makes it very easy to quickly find what I've forgotten. This also matches how I've often heard other content creators say that they end up finding their own content when looking up how to do something.
Conference talks: I'm speaking at NDC Oslo this year. I don't know whether the blog played any part in me landing the spot or not, but I like to think that it might have. At the very least, I don't think it hurt.

Stats

In addition to the outcomes mentioned above, I thought it'd be interesting to present some statistics on what I've published so far (before this post).

Total number of published words

71,554

Total number of posts

Average word count

1,350

Longest post

Let's Read Haskell Programming from First Principles, part IV: Basic data types (2,866 words)

Shortest post

Rebasing off a repo root (277 words)

Number of unique tags

Most used tags ::

Haskell (15 posts)
Git (10 posts)
Rust (7 posts)

Time spent

400+ hours

Traffic generators

I use Netlify's analytics to get a rough overview over how many page views I get and to see what content gets the most traffic. Based entirely on anecdotal experience from checking my numbers every now and then, I can tell you that Rust is far and away what generates the most views on my blog. At the time of writing, the four most visited blog posts are all Rust posts.

The post on Magit Forge also performed very well and got some pretty good circulation (in relative terms) on Twitter.

Now what?

Now that I've reached my goal and completed the year of blogging, what's next for the blog? Do I just drop it? Do I keep it up? Do I put even more time into it?

I have no intentions of letting the blog go any time soon. However, I do intend to change up the format a bit. I'd like experiment more with shorter posts and different kinds of content.

I also expect not to write much in the next month, as I'd like to focus much more on the talk I'll be giving at NDC Oslo in June. I'd say a little time off is well deserved.

Was it worth it?

Chris Coyier has said that the best way to create a name for yourself online is to write. Write, write, and then write some more. Content is valuable, and you can always offer something new and unique.

Words like these motivate me, and when I look back upon what I've written so far, I'm even more motivated to keep going. I've got so much more to learn and so much more to say.

So yes, it was worth it.

Now excuse me while I go play some video games.

The results of splitting empty strings

Mon, 11 May 2020 07:28:25 GMT

What should a function splitting an empty string return? What about splitting a string on a separator that does not exist within the string? This was the topic of some debate within my team recently, when one of my teammates had assumed that splitting an empty string on a separator would return an empty list. It does not^[1].

Splitting on a substring that doesn't appear in the string

When you split a string an arbitrary number of times, it's reasonable to expect to find yourself with a list of all the pieces. If you were to join all these pieces back together using the same separator that you used to split the original string, you should arrive back at where you started. In other words, if sep is an arbitrary separator, then the following bit of pseudocode should be true: originalString == originalString.split(sep).join(sep).

The algorithmic approach

Given a string and a separator, we can describe a string splitting algorithm in two, simple steps:

Take everything up until the separator and append it to the list of substrings. If you don't find the separator, add the whole string to the list.
Take what's left of the string after the separator, and go back to step one, using the remaining part of the string as input.

When there is no more string to split, you've completed the substring list and can return that to the caller.

Adding the whole string to the list if you don't find the separator is a crucial part of step one. Without it, you would never get the final piece of a string. If you split "I fell asleep" on spaces, you'd expect to get ["I", "fell", "asleep"] in return. But if you discard a string if it doesn't have the separator, you'd end up with only ~["I", "fell"]~; a very different outcome.

If the separator doesn't appear in the string at all, we simply add the whole string as the first element of the list and consider our work done. This way, the function is internally consistent. As you go through the string, you will always eventually reach a substring that doesn't contain the separator. That substring is also part of the result.

The user experience approach

There's another angle to come at this from: as the end user. What would you, the developer using the function, expect it to return? This is a subjective thing, so I can only really speak for myself, but if I know that if a function always returns data of the same shape (such as a list), I find it much easier to work with.

What if the string doesn't contain the separator at all? Well, then it can't be split. Imagine we're cooking together. I give you some carrots and ask you to cut the tops off. What do you do if one of the carrots has already had the top cut off? You'd probably realize it doesn't need anything done to it, and put it in the done-pile with the others.

Similarly, a string that has no separator (no top) need not be split, and you can just return it as-is.

Splitting an empty string

Splitting an empty string isn't any different than what we've already explored. In fact, it's just a more specialized version of the above problem. It can also come up if the string you're splitting ends with the separator. For instance "juice," split on commas, would be ["juice", ""]. An empty string is still a string.

The empty string as separator

What if the separator is an empty string? This varies from language to language. For instance, in JavaScript, an empty string as separator means 'split the string into a list of characters', while Python throws an exception. Rust's split splits the string into a list of one-character strings (like JavaScript does), but inserts an empty string at the start and at the end (playground link).

What does it even mean to use an empty string as a separator? There's no clear answer here, so it's at your discretion as the function author. For instance, a Haskell-implementation I quickly threw together while writing this post^[2] returns an infinite list of empty strings. I'm inclined to say that that's reasonable, but it's probably not very useful.

Summary

There's any number of ways to slice a string. Most languages seem to follow the idea that splitting a string gives you back a list of substrings, even if the string is empty or the separator doesn't appear in the string at all. The disagreement is about what to do with an empty separator, but at least they all agree on it being a special case.

If you're feeling the urge come over you after all this, then how about taking a stab at implementing a string splitting algorithm yourself? It's quite the fun, little exercise if you're looking for something to occupy your mind for a bit.

Footnotes

[1]

This particular case was in JavaScript, where splitting using an empty string as the separator will actually return an empty list given an empty string (~''.split('')~ returns []), but this is a special case. Most languages (and their standard functions), that I'm aware of, require you to opt in to remove empty strings from the output, though there are outliers.

[2]

The implementation is included here for the particularly interested reader:

split :: String -> String -> [String]
split sep input =
  go "" (Just input)
  where go before remaining =
          case remaining of
            Nothing -> [before]
            Just(s) ->
              if take (length sep) s == sep
              then before : (go "" $ Just $ drop (length sep) s)
              else case s of
                "" -> go before Nothing
                (c:cs) -> go (before ++ [c]) $ Just cs

[1]

[2]

The implementation is included here for the particularly interested reader:

split :: String -> String -> [String]
split sep input =
  go "" (Just input)
  where go before remaining =
          case remaining of
            Nothing -> [before]
            Just(s) ->
              if take (length sep) s == sep
              then before : (go "" $ Just $ drop (length sep) s)
              else case s of
                "" -> go before Nothing
                (c:cs) -> go (before ++ [c]) $ Just cs

Save on typing and improve legibility with Rust's macros

Mon, 04 May 2020 07:54:28 GMT

I found some spare time this past week and sat down with a nice brew and Jon Gjengset's excellent Crust of Rust video on declarative macros. For the longest time, macros have felt 'that last part of Rust that I haven't gotten around to checking out'. I've had a vague notion of what they are, but have never quite gotten to exploring them. However, Gjengset's video served as a perfect introduction to declarative macros, and was just enough to get me started.

One thing that was mentioned in the video that I have never thought about before, is that one of the simplest things to do with macros is simple substitution. In fact, that's all a declarative macro can do: given some input, it'll expand to a block of code. That suddenly gave me an idea for writing my own first macro.

Post purpose

This post is intended to be a very brief and basic introduction to declarative macros based on what have I found in the past week. For more comprehensive material, see the 'Further Reading' section at the end.

The post describes one very simple use case for macros, and that's all it's intended to do. In particular, this post will not disccuss

macro syntax: There will be no talk of macro syntax, of capture kinds and patterns, or of clever ways to count.
other use cases: This is not an exploration of all the ways in which you can use declarative macros or where they shine. This is a description of one case that solved a problem that's been irking me.
proc macros: Proc macros are a different subject, and is something I don't know much (or anything, really) about. They're both a form of metaprogramming, but from what I understand, proc macros are quite a bit more complex than declarative macros (and thus more powerful), so it's best left for later.

I assume a basic level of familiarity with Rust, but a deep understanding is not required.

My Little Macro 🦄

I was very excited when advanced slice patterns were stabilized in Rust 1.42. Among the things I'd been looking forward to was the ability to match on strings as if they were a slice of characters, similar to what you might do in Haskell or Elm. It wasn't immediately obvious how to do it, but I figured something out in the end^[1]:

fn f(s: &str) {
    match &s.chars().collect::<Vec<char>>() as &[char] {
        ['💘', .., '🦄'] => println!("<3 and horse"),
        ['💘', snd, .., '😪'] => println!("Love, {}, and sleeps", snd),
        ['💘', ..] => println!("Just <3"),
        _ => {}
    }
}

However, it's not immediately obvious what's happening on line 2: what exactly does &s.chars().collect::<Vec<char>>() as &[char] mean? Sure, I can tell you that it turns the string into a char slice for matching, but as it stands, it's quite the mouthful. Let's write a macro to make this cleaner and clearer!

Replacement as a form of abstraction

Because a declarative macro is nothing but text substitution^[2], we should be able to simply abstract away the pesky line from above. Instead, we want to write something like this:

fn f(s: &str) {
    match chars!(s) {
        ['💘', .., '🦄'] => println!("<3 and horse"),
        ['💘', snd, .., '😪'] => println!("Love, {}, and sleeps", snd),
        ['💘', ..] => println!("Just <3"),
        _ => {}
    }
}

To do this, we write a very simple macro with a single pattern:

macro_rules! chars {
    ($s:expr) => {
        &$s.chars().collect::<Vec<char>>() as &[char]
    };
}

All it does is replace the macro call (chars!(s)) with the long incantation (&s.chars().collect::<Vec<char>>() as &[char]) when compiling. It's incredibly simple, but it's also incredibly powerful and it's made the code look less cluttered and read better at the same time.

Footnotes

[1]

Iterating over a string and collecting it into a vector of characters is probably not the most efficient way to work with strings, but it's the best way I've found to turn a string into a slice of (UTF-8) characters that can be matched on. If you've got a better solution for this, let me know! I've been looking for a while.

[2]

Well, saying it's just text substitution might be misleading: Because the expanded macro is parsed into the program's abstract syntax tree, it has to be valid Rust. See the first chapter of the Little Book of Rust Macros for more information.

[1]: Iterating over a string and collecting it into a vector of characters is probably not the most efficient way to work with strings, but it's the best way I've found to turn a string into a slice of (UTF-8) characters that can be matched on. If you've got a better solution for this, let me know! I've been looking for a while.
[2]: Well, saying it's just text substitution might be misleading: Because the expanded macro is parsed into the program's abstract syntax tree, it has to be valid Rust. See the first chapter of the Little Book of Rust Macros for more information.

Configuring Emacs with Org mode and literate programming

Mon, 27 Apr 2020 07:51:14 GMT

Even if you have the source code in front of you, there are limits to what a human reader can absorb from thousands of lines of text designed primarily to function, not to convey meaning. --- Ellen Ullman (from quotes about software aging on literateprogramming.com)

I've been intrigued by literate programming for a while now, but never quite found the right opportunity to try it out. All the business with tangling and detangling, calling code from other snippets, and understanding how or even if you can import literate code into non-literate code, made it seem like you'd need to understand a lot just to get started. Fortunately, I found something that doesn't require any of the above: extending your Emacs config.

Using Org mode and literate programming to configure Emacs is something I've heard a lot about, but I found it difficult to find out exactly how to get started. Luckily it turned out to be pretty simple.

Below, I'll show you how to start configuring Emacs with Org mode and what I've learned about it so far, but if you're really eager to just get started, the trick is to use the function org-babel-load-file and give it the path to your Org file.

*Disclaimer*: I am /not/ an expert at literate programming, or even particularly knowledgeable; these are my first steps into this brave new world. If I've made any mistakes or if you've got tips: don't hesitate to reach out.

Who is this for?

This post is aimed at anyone interested in literate programming with Org mode, but who does not know where or how to get started. It assumes some familiarity with Emacs and Org mode, but no further knowledge of programming is required. Further, I assume no prior knowledge of literate programming.

What is literate programming?

I believe that the time is ripe for significantly better documentation of programs, and that we can best achieve this by considering programs to be works of literature. Hence, my title: "Literate Programming."

Those are the words of Donald Knuth, the creator of literate programming. Knuth calls for a form of programming where we shift our focus from instructing the computer what to do, to explaining to another person what we want the computer to do. This forms the basis for literate programming.

A literate source code file inverts the typical notion of a source code file: rather than being source code with comments strewn around, is a text with source code blocks inserted. The exact file format and medium of the file doesn't matter. The most well-known literate programming tool^[1] is probably Jupyter Notebook.

Org mode

With Emacs, Org mode is arguably the most readily available way to do literate programming. Babel, which has been included in Org mode since version 7.0, enhances Org mode's source blocks by providing (as described in the introductory tutorial)

interactive and on-export execution of code blocks
code blocks as functions that can be parameterized, that can refer to other code blocks, and that can be called remotely
export to files for literate programming

In short, this enables you to write Org documents with source code blocks, where the source code blocks can be interacted with and used to generate pure source code.

Configuring Emacs

It turns out that if you're on a semi-recent version of Emacs, it's really very simple. Babel provides a function called org-babel-load-file which 'Load[s] Emacs Lisp source code blocks in the Org [file]'. As such, all you need to do is to call this function from your Emacs configuration with the path to the Org file you want to load.

Here's what I've got in my configuration:

(org-babel-load-file
 (expand-file-name
  "config.org"
  user-emacs-directory))

This snippet assumes that the file you're loading is located in a file called config.org in your user-emacs-directory (which defaults to ~/.emacs.d).

This was tested with vanilla Emacs 26.3, so any recent version of Emacs shouldn't need any more configuration than this, but if you want a more detailed guide, check out the Emacs Initialization with Babel section of the Babel introduction.

Benefits of using Org mode for configuration

So far, I've found a number of benefits to using literate programming for my configuration, including (but not limited to):

I'm able to express /why/ I'm making certain configurations: Not that you can't describe what you're doing or why in pure elisp, but I certainly find it harder to be clear about it. Explaining the reasoning behind making certain configuration decisions is not only useful to other people reading my configuration, but also to myself when I return to something after weeks or months (or years) after having last touched it. 'Why did I configure it this way again? Oh, yeah: that's it!"

Furthermore, being able to use a text-based mode to write text not only makes a lot of sense, but it's also much more ergonomic, and offers much more expressibility than code comments.

Formatting: Another bonus of using a text-based mode is that you can take advantage of text formatting. For instance, being able to include links that only display a link text and not a full URL is a nice bonus. Lists (numbered, unnumbered, and definition lists) are also readily available.
Expanding and collapsing regions: If you want to get a quick overview over your configuration, it's easy to quickly toggle the headings in the entire file to find exactly what you're looking for. It's also super easy to narrow your editor view to the section you're focusing on right now.
Using tags: Even better than being able to fold your config and scan through the headings is using tags. Org mode's support for tags makes it a breeze to show only sections that relate to whatever you're looking for. For instance: one of the tags I've defined is keybinding. By tagging every section that deals with key bindings with this, I can quickly whip up a view that shows all the relevant sections by using org-sparse-tree.

Summary

That sums up my journey into literate programming thus far. I still have a lot to learn and have probably only truly understood a fraction of the power that lies within Org mode, but I've gotten over that first step. It's not (as) scary anymore, and I'm looking forward to learning more about it.

If you came here looking for how to use Org mode to configure Emacs or to do literate programming, I hope you found what you were looking for. And if you came here wondering what this was all about, I hope your curiosity was rewarded and that you're still at least as curious or intrigued as before.

Footnotes

[1]

Or at least the only one that I hear mentioned at semi-regular intervals.

[1]: Or at least the only one that I hear mentioned at semi-regular intervals.

Building a request inspector

Mon, 20 Apr 2020 17:32:32 GMT

I've been quite involved with distributed tracing and have spent a fair amount of time looking at the W3C recommendation for dealing with trace context at work lately. As a result of this, I have found myself wanting to inspect the headers on outbound requests to ensure that the framework and libraries we use handle tracing correctly.

But how do you do that? I couldn't find any services or command line utilities that did this (at least not simply), so I set out to build one myself. It's been a while since I did anything in Rust, and this sounded like a fun, little weekend project. Spoiler alert: It wasn't.

I wanted this to be a short and simple tutorial on how to build such a server in Rust, but things didn't go quite as I planned, and it took much more time and effort than I expected. Rather than hiding it and pretending it never happened, though, I'm going to take it and run with it. I'm sure that if I'm running into these issues, I'm not the only one.

Intended audience

This post is intended for people who have at least some experience with Rust, including familiarity with the type system and the borrow checker (at least as concepts). You should also have some passing knowledge of HTTP requests.

This post is not intended to be a thorough tutorial or a list of best practices, but rather to serve as a way of demonstrate how I work. Yes, the code here works and runs as expected, and clippy doesn't complain, but it's not good.

There are code samples below, and you can also check out the repo on GitLab. However, this should not be considered a final version, and there will likely be further updates to the repo later on. This is also not an in-depth analysis of the code, but a short tour of it. In short: it works, but it's far from perfect. In a lot of ways, writing this code felt a lot like how the Oatmeal describes projects coming together in his fantastic comic 'Erasers are Wonderful': full of twists, turns, and toilet fires, and in the end you have something that's good enough.

The goal

I set out to make a simple web server that:

would accept requests at any endpoint
would accept requests with any method
would respond with a JSON object containing data about the request's:

headers
method
path
query string

I also wanted to add the request body (if there was one) to the response, but it wasn't the most important issue. Other additional features, such as reading data from environment variables, command line options, logging, etc., could be added later.

How (or: 'show me the code')

When building a web server in Rust, there's a number of frameworks to choose from. I first went with Actix, but after not reading the docs, ended up working directly with Hyper because it was easier to create a function that would handle any request at any route with any method. Or at least it was covered in the initial tutorial.

In addition to Hyper, I'm also pulling in anyhow and serde-json for dealing with errors and working with JSON.

Below, we'll break the program up into functions and look at them one at a time.

Dependencies and imports

Let's get the boring (but very important) bits out of the way. Here's the dependencies section of the Cargo.toml file, as well as the program imports.

Dependencies:

[dependencies]
hyper = "0.13"
tokio = { version = "0.2", features = ["full"] }
serde_json = "1.0"
anyhow = "1.0"

Imports:

use anyhow::Result;
use hyper::service::{make_service_fn, service_fn};
use hyper::{Body, HeaderMap, Request, Response, Server};
use serde_json::json;
use std::collections::HashMap;
use std::convert::Infallible;

The `main` function

#[tokio::main]
async fn main() -> Result<(), hyper::error::Error> {
    let make_svc = make_service_fn(|_| async { Ok::<_, Infallible>(service_fn(handle_requests)) });

    let addr = ([127, 0, 0, 1], 8080).into();

    let server = Server::bind(&addr).serve(make_svc);

    println!("Server started. Listening on http://{}", addr);

    server.await
}

There's a few things happening here, but it's rather self-explanatory. We declare a handler and an address for the server; start the server with the aforementioned handler and address, and wait for it to finish (which happens on termination).

The request handler

async fn handle_requests(req: Request<Body>) -> Result<Response<Body>> {
    let response_data = json!({
        "headers": to_string_map(req.headers()),
        "path": req.uri().path(),
        "queryString": req.uri().query(),
        "method": req.method().as_str(),
        "version": format!("{:?}", req.version()),
    }).to_string();

    println!("Received request: {:?}", response_data);

    Ok(Response::builder()
       .header("content-type", "application/json")
       .body(Body::from(response_data))?)
}

This is the meat of the program and really what it's all about: extracting data from the request and returning it to the caller. As this is a very rough proof of concept, I'm mapping the the data into a completely arbitrary JSON structure rather than into a struct.

After mapping, I print the result of the mapping, and return the response with an appropriate content-type.

Serializing the `HeaderMap`

Serde takes care of serializing most of the data very well, but doesn't like the HeaderMap which contains the request's headers. ~HeaderMap~ is a multimap---a map structure that can associate multiple values with a single key---and as such, doesn't easily serialize to JSON.

To solve this, I decided to turn the HeaderMap into a HashMap<String, String>, simply creating a comma-separated string for headers that have multiple values. Not the most elegant or robust solution, but hey, it works.

Also, because the header_value.to_str function 'yields a &str slice if the HeaderValue only contains visible ASCII chars' (according to the docs), I put ~"Non-ASCII header value"~ to handle cases where it contains non-ASCII characters.. Again: it works.

fn to_string_map(headers: &HeaderMap) -> HashMap<String, String> {
    let mut map = HashMap::new();
    for (header_name, header_value) in headers.iter() {
        let k = header_name.as_str();
        let v = header_value
            .to_str()
            .unwrap_or("Non-ASCII header value")
            .into();

        match map.get_mut(k) {
            None => {
                map.insert(k.into(), v);
            }
            Some(old_val) => *old_val = format!("{}, {}", old_val, v),
        }
    }

    map
}

The unexpected challenges

So what made this so difficult? Why didn't it work out as I expected? Well, here's some of the issues I ran into:

The request body: As briefly mentioned up top, I originally wanted to also include the request body in the response. I spent too much time on trying to make this work before realizing that I should leave it out for now.

This turned out to be difficult because of how I couldn't easily parse the body as a String and include that in the output JSON. But after a bit of thought, I realized that I can't just assume that the body is JSON (or even a string), so it's more work than I expected.

Converting between different types: Related to the issues with the request body is conversion between different data types, and especially between types that are and aren't serializable by Serde. It felt like there was a lot of juggling types around just to please the compiler.
Manually serializing data types: While most of the data types can easily be represented as strings in this case, the header map needed some work. While it wasn't a very difficult exercise, it took more time than expected, especially because there wasn't an obvious way to perform an upsert-like action into a HashMap.
Lack of examples: This could be me or it could be the documentation, but I found it difficult to do what I wanted. I'd expected there to be more information on getting data from a request, but it's quite possible that I just didn't read far enough.
I'm ... /rusty/: It's been a while since I last worked with Rust, and the borrow-checker was stricter than I remember.
Working directly with Hyper?: I don't know whether this was much of an issue or not. It gave me quick and easy access to the endpoint setup I wanted, but it might have introduced other complications. That said, it looks as if Actix simply re-exports a lot of Hyper's data types, so I don't know how much of a difference that would have made.

Wrapping up

Even if things didn't go exactly as planned, it was a fun, and at times very frustrating, little project. Having looked a little bit more at the Actix docs, I have found a few sections that make me think it could be quite suitable after all, so I'll probably rewrite the project some fourteen times in the coming week.

Next time I'll hopefully have something a bit more polished to show off.

Peace.

On null and undefined

Mon, 13 Apr 2020 14:53:21 GMT

I came across a tweet the other day (no link, sorry), where the author said they'd stopped using null in JavaScript, instead using only undefined. The author was encouraging the readers to do the same. From what I can remember, the reason for this was that mixing null and undefined can be confusing and lead to weird bugs in your application.

I've also heard a number of dev friends express negative emotions towards the fact that JavaScript has both null and undefined, saying it makes things harder.

However, I'm in the camp where I think having both null and undefined is a good thing because they are separate concepts. In this post, I want to discuss and put forward the way I view them, both in JavaScript itself, and also as part of a JSON API.

Note: this is largely my own opinion, and I do not have a significant amount of literature to back it up. I welcome opinions and discussions on this!

JavaScript

In JavaScript, both null and undefined represent an absence of value. However, the kind of absence they represent is quite different from one another.

undefined values are values that have simply not been defined. MDN says that 'undefined is a primitive value automatically assigned to variables that have just been declared, or to formal arguments for which there are no actual arguments.':

let x // x is `undefined`

var y // y is `undefined`

const f = (x) =>
  console.log('x is', x)

f() // prints 'x is undefined'

Additionally, the return value of a function that doesn't return anything is undefined:

// returns `undefined`
const g = () => {}

On the other hand, null is never implicitly assigned. It must always be assigned explicitly by the programmer. This makes null well suited for cases when you want to represent a value that can either be present or not, similar to Haskell's Maybe and Rust's Option.

In short

An undefined value is something that hasn't been assigned, while a null value has been assigned that specific value.

Does that matter in your application code? I'm inclined to think that it's not much of a big deal whether you use only one or both. In my experience, most cases will be testing the 'truthiness' of a value anyway, so null and undefined can intermingle freely.

Where it does matter, though, is in areas where these two concepts carry different meanings, such as when communicating via JSON, which is what we'll be looking at in the next section.

Personally, I'm 'Team Both'. I'll use undefined for optional parameters to React components or optional keys of object arguments that I expect, and I'll set a value to null if I intend it to carry a semantic meaning.

What do I think you should do? Whatever fits your use case. If you have no need to differentiate between these two, then it really doesn't matter much what you do. If you'd prefer to only use the one, that's perfectly fine by me.

JSON and APIs

While null and undefined may carry largely the same meaning in JavaScript, they have quite distinct meanings in JSON^[1] and when communicating with an API. This is especially true when making PATCH requests.

The RFC for the JSON Merge Patch document format (7396) states that when patching, an undefined value should be left as is and a null value should be removed. This means that the two values have wildly different meanings and that mixing them up can cause some very unwanted side effects.

The API story

However, even if JSON works as described above and JavaScript has both undefined and null, a lot of server-side languages don't have this distinction. And they're also often statically typed languages that need to populate a model. This can lead to problems.

Where I work, most of the backend systems are written in C#, where we must have defined classes for JSON payloads. C# doesn't have undefined, so depending on how you configure your system, it can either mark non-existing fields as null or throw an error because it didn't get all the data it needed. When you're working with PATCH as described above, neither of those are ideal.

I don't know of any libraries (in any statically typed languages) that have an elegant solution for this, but I'd be very interested in hearing about it if you do. It may not be applicable to any languages I work with, but just seeing a solution would be very interesting.

So those are my thoughts. I think there is a clear difference between null and undefined, and I support having both. In some cases, they might be interchangeable, but if nothing else, I maintain that they express intent differently, so I think it's worth it just for that.

Footnotes

[1]

To be clear: JSON doesn't have an explicit undefined value, but you can leave out a field, which serves the same purpose:

// JS object
{
  a: null,
  b: undefined,
  c: 1
}

// JSON object, converted
{
  "a": null,
  "c": 1
}

[1]

To be clear: JSON doesn't have an explicit undefined value, but you can leave out a field, which serves the same purpose:

// JS object
{
  a: null,
  b: undefined,
  c: 1
}

// JSON object, converted
{
  "a": null,
  "c": 1
}

Haskell's Maybe and Either types

Mon, 06 Apr 2020 08:23:42 GMT

If you don't have exceptions and you don't have null: How do you handle errors and invalid inputs? Based on your background, this can either be trivial or mind-bending. Coming into the Haskell (actually: Elm and Rust) world from C#, JavaScript, and Python, it definitely wasn't obvious to me at first.

Luckily, the Maybe and Either types are there for you, and they are fairly easy to get started with. In this post, I want to give a quick introduction to both of these data types and how they're used.

To any returning readers: This post is based chapter 12 of /Haskell Programming
from First Principles/, which deals with basic error handling in Haskell, using
the ~Maybe~ and ~Either~ types. The chapter doesn't actually introduce much else
of interest, so I wanted to approach this entry slightly differently.

What we will (and won't) be covering

This post is intended to be a brief introduction to the Maybe and Either data types in Haskell; suitable for someone who has no previous experience with these. Some understanding of Haskell syntax would be beneficial, but is not required. The post should give the reader a basic understanding of what Maybe and Either are and how they can be used for modeling data and responses.

This post is not intended to be a thorough examination of these types. It will not discuss anything related to typeclasses, mapping, or binding. There will also not be any extensive code samples. As such, please note that there is a lot more to these data types than we will look at here, but that most of it is out of scope for this post.

For further reading, please consult the section at the end of the article.

`Maybe`

Maybe represents the potential absence of a value. It is used when some data can be in one of two states: defined/present, or undefined/absent. Because Maybe represents a value that may or may not be present, we also need to specify the type of the potentially contained item: Maybe String, Maybe Integer, or Maybe a for instance.

In many ways, Maybe is similar to null in C#, JavaScript, etc., None in Python, and related concepts (like nil) in a lot of other languages. However, in these other languages, the potential lack of a value is (almost) always implicit, meaning the programmer can never be sure whether the value is there or not. In Haskell, this is always explicit. The Maybe type is the same as the Option type in languages like Rust and F#.

The data declaration looks a little something like this:

data Maybe a = Nothing | Just a

As such, Maybe has two data constructors, Nothing and Just, where the latter takes a value to wrap.

Basic usage

While there's any number of ways you can use the Maybe type in your programs, I find the two most obvious ways to be

to model data where some parts are optional
to return something from a function if the input is invalid

Being able to indicate that a value may or may not be present is a simple, but very powerful tool of modeling data. Imagine you've created a dating app for Haskellers---let's call it Hinder---where you want to display a list of users. The list should display only their profile picture and their name, but there's no requirement to have set a profile picture to be listed, so we'll need to account for that. A simplistic way to model that would be by using a list of records like this:

data User = User
  { name :: String
  , picture :: Maybe String
  }

Where the picture value is Nothing if there is no profile picture set. If they have a profile picture, it's Just <picture id>.

The other way to use Maybe is as the return value of a function. As much as possible, you should use the type system to constrain the set of allowed arguments to a function, but that'll only get you so far. The built-in div function throws an exception if you pass 0 as the second argument, so let's fix that by using Maybe. If the divisor is 0, return Nothing. Otherwise, return Just <result>.

safeDiv :: Integral a => a -> a -> Maybe a
safeDiv _ 0 = Nothing
safeDiv x y = Just $ x `div` y

`Either`

The Either type is similar to the Maybe type, but comes with some added functionality. It's also used to model things that can fail, but it provides a way to report what went wrong, not just that something failed.

Either represents a piece of data that can be one of two things. That is, it can be either 'this' or 'that', whatever 'this' and 'that' may be. Either is parameterized by two types (so 'this' and 'that' can be of different types), meaning you'll usually see Either a b or something like Either String Int.

Where Maybe has a very easy analog in the concept of a null value, Either doesn't enjoy the same luxury. However, it is conceptually equivalent to the Result type in Rust and F#, even if the order of the arguments are flipped.

The data declaration (simplified) looks like this:

data Either a b = Left a | Right b

Either has two data constructors, Left and Right, each of which take a single value. By convention, the Left value is used for when something goes wrong, and the Right value for when something goes, well, right.

Basic usage

Either is usually used for error handling and validation, so it's ideal as a return type of functions that can 'fail' or that are otherwise not able to create a valid result based on the arguments given.

Let's return to our safeDiv function from before. This time, however, we'll return an error message instead of Nothing:

safeDiv :: Integral a => a -> a -> Either String a
safeDiv _ 0 = Left "You cannot divide by zero."
safeDiv x y = Right $ x `div` y

The error may be obvious in this case, but for more complex functions (or chains of them), it gets ever more useful.

For a slightly more advanced example, imagine you're creating a Person record. In this case, a Person has a name of type String and an Age of type Integer. But with these constraints, we can easily pass in invalid values, so we'll have to validate that the name isn't an empty string and that the age is non-negative:

data PersonError = InvalidName | InvalidAge

data Person = Person
  { name :: String
  , age :: Integer
  }

makePerson :: String -> Integer -> Either PersonError Person
makePerson name age = undefined -- exercise for the reader ;)

The actual implementation of the validation isn't important in this case, so I've left that as an exercise for the reader. Oh, and what if there are multiple errors? Well, we can deal with that too, but that's not for now.

On Generics and Associated Types

Mon, 30 Mar 2020 08:05:57 GMT

Compared to other languages I've learned, Rust has a fair few concepts that can be a bit tricky to get your head around. Borrowing, ownership, and the borrow-checker are common enough to have spawned a range of memes on their own, and I've personally spent hours on lifetime issues only to give up and rewrite something using cloning.

Associated types, though not something that'll have you banging your head on your desk for hours, is something that it took me quite a few tries to finally understand---or at least think I understand.

What really never stuck was how it was different from generics, and why you'd need (or event want) associated types. So what's a good way to learn and internalize a topic like this? Well, write something down and release it to the internet, of course! They'll let you know if you're wrong.^[1]

tl;dr:

The quick and dirty answer to when to use generics and when to use associated types is: Use generics if it makes sense to have multiple implementations of a trait for a specific type (such as the From<T> trait). Otherwise, use associated types (like Iterator and Deref).

Goals and constraints of this post

This post is intended to demonstrate and explain the differences and similarities between associated types and generic types. It will deal specifically with traits, as this is the only place where associated types come into play.

Furthermore, even if we're talking about associated types, we will not venture into the dark forest of generic associated types. However, if you're curious about this and haven't kept up to date, I encourage you to look through the RFC.

If, after reading this post, you're still not sure what I'm on about, check out the /Advanced Traits/ chapter of the Book, and specifically the section on associated types.

Finally, this post assumes you have some familiarity with programming (and with Rust specifically), and with some forms of generic programming. For a primer on generic types in Rust, check out Chapter 10.1, /Generic Data Types/, of the book

Definitions

To make sure we're all on the same page, let's have some quick definitions to start us off, shall we?

Generic types

In the context of traits, generic types, also known as type parameters, are a way of deferring the specific types that your trait uses until the trait is implemented. Generic types can be completely open, such that any type would work, or they can be constrained to types that implement some trait.

Take, for instance, Rust's ~std::convert::From<T>~ trait. The T in the type signature says that this type is generic over any type T ^[2], meaning you can convert any type T into the type you're implementing the trait for.

Constrained (or bounded) generic types are more often seen in generic functions than in generic traits, but what they do is allow the author of trait X to say that 'only types which implement some other trait ~Y~ can be used for this trait'. We'll see examples of this later, so don't worry if it's a bit fuzzy right now.

Associated types

Associated types are, as the name implies, types that are associated with a trait. When you define the trait, the type is still unspecified. Much like with generics, you can put constraints on the type if you want to, or you can choose not to.

One of the most prominent examples of a trait with associated types is the ~Iterator~ trait. The Iterator trait has an associated type Item and a function next. The next function returns an Option<Self::Item>. You could have done the same thing with generic types, but, as we'll see later, using associated types offer some benefits in certain situations.

Syntax

Before we go any further, let's just quickly review the syntax for these concepts. If for no other reason, then just to make everything a bit less abstract. We'll define two traits, Generic and Associated, which use generics and associated types respectively, and we'll look at using bounds and default types as well.

Basic traits

A type-parameterized trait can look a little something like this:

trait Generic<T> {
    fn get(&self) -> T;
}

Similarly, a similar trait with an associated type looks like this:

trait Associated {
    type T;
    fn get(&self) -> Self::T;
}

Note how the type gets moved from the type signature and into the trait definition itself, and how, when referencing it later, we need to use Self::T, instead of just T.

With constraints

If we want to set constraints on the associated type or type parameter, we use the same syntax as Rust uses everywhere else for bounds: the : operator.

For instance, say we want to constrain our types to only types that implement the core::fmt::Display trait:

trait Generic<T: Display> {
    fn get(&self) -> T;
}

// or using the `where` keyword
trait Generic<T>
where
    T: Display,
{
    fn get(&self) -> T;
}

With associated types, the syntax is much the same:

trait Associated {
    type T: Display;
    fn get(&self) -> Self::T;
}

With default types

Rust has a cool feature for generic types where you can set the default type, which will be assumed if no type is specified. This can be useful if, for most use cases, you want to use a specific type, but want to be able to override it sometimes. See the section on default generic types in the Book for more information.

They look like this:

// basic trait, no constraint
trait Generic<T = String> {
    // ...
}

// with constraint
trait Generic<T: Display = String> {
    // ...
}

// or using the `where` clause
trait Generic<T = String>
where
    T: Display,
{
    // ...
}

From what I tried, it seems you cannot put the default type (= String) in the where clause of the trait.

For associated types, there is no such thing as default types on stable rust today. However, if you're on nightly, you can use the #![feature(associated_type_defaults)] flag, which enables this. Judging from the GitHub tracking issue, this is being worked on but I've not seen anything about stabilization of this yet.

But let's see what it'd look like:

#![feature(associated_type_defaults)]

// simple
trait Associated {
    type T = String;
    // ...
}

// with constraint
trait Associated {
    type T: Display = String;
    // ...
}

It's pretty neat and pretty similar. You could even do something like this:

trait Associated {
    type T: Display = String;
    type U = Self::T;
    // ...
}

Don't know when it'd be useful, but it's nifty 🤷

Commonalities

Now that we've covered what they are and what the syntax looks like, let's continue by looking at what they have in common.

The most important thing is that both generics and associated types allow you to defer the decision of what type to use for the trait implementation. Even if the notation is a bit different, anywhere you have an associated type, you can replace it with a generic type instead (though the opposite does not hold true). As the the RFC puts it: "associated types do not increase the expressiveness of traits per se, because you can always use extra type parameters to a trait instead". However, they do offer other benefits.

The fact that you can always use generics instead of associated types is why it took me so long to understand what associated types exist for. As we move into the next section, we'll examine what makes them different, and why you'd want to choose one over the other.

Differences

As we've seen, generics and associated types cover a lot of the same use cases, but there are some reasons why you might choose one over the other.

Generics allow you to implement the same trait numerous times for the same type by changing the type parameter. The From<T> trait was mentioned earlier, and it's a great example of this. Because From<T> uses a generic type, we can implement it for any number of type parameters.

For instance, say you have a type MyNumeric. You can implement From<u8>, From<u16>, From<u32>, and so on. This makes generics very useful if it makes sense to have multiple trait implementations varying only in type parameters.

Associated types, on the other hand, only allow a single implementation. Because a type can only implement a trait once, this can be used to constrain the number of implementations.

The ~Deref~ trait comes to mind. Deref has an associated type Target, which it can be dereferenced to. It would get really confusing if a type could implement Deref into an arbitrary set of other types (and probably very tricky for type inference!).

Because a trait can only be implemented once per type, associated types also offer some notational benefits. Using associated types means you don't have to add type annotations for all the extra types. This is touted as an engineering benefit in the RFC.

Summary and further reading

In short, use generics when you want to a type A to be able to implement a trait any number of times for different type parameters, such as in the case of the From<T> trait.

Use associated types if it makes sense for a type to only implement the trait once, such as with Iterator and Deref.

If you want to know more about associated types and what problems they solve, I recommend starting with the RFC that introduced them and the section on associated types in the Book. The section on the ~Add~ trait, which uses both generics (with defaults) and associated types, is also well worth a read. As an alternative resource, this Stack Overflow response also contains a well-worded explanation, with examples, of when you'd use one over the other.

Footnotes

[1]

This is mostly said in jest. I haven't actually gotten very many corrections on statements I've made at all; and the ones I have had, have always been very nice. I like you, reader. You're my friend.

[2]

Well, actually, it says that this type is generic over any type T that implements the Sized trait. We'll gloss over this in this article, but for more information, check out this section from chapter 19.4, /Advanced Functions and Closures/, of the Book.

[1]: This is mostly said in jest. I haven't actually gotten very many corrections on statements I've made at all; and the ones I have had, have always been very nice. I like you, reader. You're my friend.
[2]: Well, actually, it says that this type is generic over any type T that implements the Sized trait. We'll gloss over this in this article, but for more information, check out this section from chapter 19.4, /Advanced Functions and Closures/, of the Book.

Let's read: Haskell Programming from First Principles, pt XI

Mon, 23 Mar 2020 08:20:58 GMT

Let's get this out of the way before we go any further: This is a massive chapter. There is a so much content in here, and there is no way that I'll be able to summarize it all in one neatly packaged post. Instead, I'll be focusing on the things that struck me as particularly important: kinds and cardinality.

That means that in exchange for covering a few topics more thoroughly, I will not be devoting many words to /newtypes/, arity, what sum and product types are (see chapter 4 and chapter 9), or constructing and deconstructing data types (see chapter 7a).

Kinds

This section has several references to the data type known as ~Maybe~. We have not covered this data type previously in this series, so a short introduction is in order if you're not already familiar with it.

~Maybe~ is used to model data that you may or may not have. It has two data constructors: ~Just~ and ~Nothing~, takes one type parameter, and is in some ways similar to how a lot of other languages have a ~null~ value, though more robust. It is similar to the ~Option~ type in other languages such as Rust, OCaml, and F#.

The data declaration is as follows:
#+begin_src haskell
  data Maybe a = Just a | Nothing
      deriving (Eq, Ord)
#+end_src

For more information, please see the [[https://wiki.haskell.org/Maybe][Haskell wiki article]].

Kinds are to types what types are to terms. Or, put differently, a kind can be seen as a type constructor's type or as a type one level up. It sounds convoluted, but bear with me.

Keep in mind that, as covered in chapter 4, a type constructor is one of two constructor types in Haskell (the other being data constructors). It is the name of the type. Type constructors are used only at the type level, in type signatures, and in typeclass declarations and instances.

We represent kinds with the * symbol. If you have a fully applied, concrete type constructor, such as Bool or [Int], then the kind is *. If you have a type constructor that is still waiting for a type parameter---such as Maybe---then that has the kind * -> *. A kind is not a type until it is fully applied. In other words, Maybe isn't a type, but Maybe String is.

Second, we only talk about kinds on the type level. That is, Bool is a type and has kind *. False is a term (or value) and does not have a kind. For data constructors that share names with their type constructors (e.g. []), this can be confusing, but remember that kinds only operate on type level constructors.

To make this clearer, let's turn to the ever faithful GHCi. To print the kind of a type constructor, we use the :k (or :kind) command followed by the constructor we want information on. Let's have a couple of examples and see what we can learn.

Prelude> :k Bool
Bool :: *

Bool is a fully applied type in and of itself. it takes no type parameters and as such has kind *.

Prelude> :k Maybe
Maybe :: * -> *

Maybe is our first higher-kinded type (more on that in a bit), which means it's a type constructor that requires another type before it's complete. In other words, you couldn't simply put Maybe in a type signature without saying what type of maybe it is.

Prelude> :k Maybe Integer
Maybe Integer :: *

Maybe Integer, on the other hand, is a concrete type. As opposed to just Maybe above, we also know what the type is (Integer). As such, it is fully applied and of kind *.

Prelude> :k (,)
(,) :: * -> * -> *

This one is quite interesting. This is the type constructor for a tuple. Without any type parameters, it's of kind * -> * -> * because it needs two types to be fully applied.

Higher-kinded types

According to the book, there are only 'a few' kinds, and the default (and the only one covered in this chapter) is *. Kind signatures work the same as type signatures, i.e. using the same :: and -> syntax.

Higher-kinded types are types that require type parameters to be fully applied, to become real types---that is, their kind is at least * -> *. Lists, tuples, and Maybe are good examples of higher-kinded types.

Cardinality

The cardinality of a type is the number of different values it defines. A value defined by a type is also known as an inhabitant of said type. Types' cardinalities can be anywhere from as small as 0 (empty types) and as large as infinite.

To demonstrate this concept we'll use these arbitrary sum types, modeled after the starters in the mainline Pokémon games (just pretend we're in 1999 for simplicity's sake):

data StarterType
  = Grass
  | Fire
  | Water

data Starter
  = GenI StarterType
  | GenII StarterType

The cardinality of StarterType is 3 because it has only three possible inhabitants. The cardinality of Starter is 6, because it has two data constructors, each of which also require a StarterType. As such, we get 2 \cdot 3 (or 3 + 3), which is 6.

Above, the Starter type demonstrates why it's called a sum type: We can calculate the cardinality of the type by summing the number of possible values of each of it's data constructors.

If you've got a friend with a link cable, then maybe you've traded some pocket monsters and ended up with two starters---lucky you! In this case, you now have a tuple of starters, i.e. (Starter, Starter). A tuple is a product type, so that means we find the cardinality by multiplying the cardinality of each of its contained types. As such, the cardinality of (Starter, Starter) is 6 \cdot 6 = 36.

This is the basic concept of cardinality in types. It's pretty straightforward, and it's a good thing to keep in mind when designing systems.

The function type is exponential

We've seen that sum types are the addition operator and that product types are the multiplication operator when it comes to calculating inhabitants of types. However, we haven't mentioned function types, which act as the exponent operator: given a function a -> b, the formula for the number of inhabitants is b^a. So if we have a function StarterType -> Bool, for instance, the number of possible implementations is 2^3 = 8. This relationship continues as the number of parameters increases, so for a -> b -> c it's equal to c^{ba} (or (c ^ b) ^ a).

If this is a bit confusing: don't worry, I'm right there with ya. But let's give it a go. Let's write out all possible permutations of the function StarterType -> Bool.

acceptStarter1 :: StarterType -> Bool
acceptStarter1 Grass = True
acceptStarter1 Fire = True
acceptStarter1 Water = True

acceptStarter2 :: StarterType -> Bool
acceptStarter2 Grass = True
acceptStarter2 Fire = True
acceptStarter2 Water = False

acceptStarter3 :: StarterType -> Bool
acceptStarter3 Grass = True
acceptStarter3 Fire = False
acceptStarter3 Water = True

acceptStarter4 :: StarterType -> Bool
acceptStarter4 Grass = False
acceptStarter4 Fire = True
acceptStarter4 Water = True

acceptStarter5 :: StarterType -> Bool
acceptStarter5 Grass = False
acceptStarter5 Fire = False
acceptStarter5 Water = True

acceptStarter6 :: StarterType -> Bool
acceptStarter6 Grass = False
acceptStarter6 Fire = True
acceptStarter6 Water = False

acceptStarter7 :: StarterType -> Bool
acceptStarter7 Grass = True
acceptStarter7 Fire = False
acceptStarter7 Water = False

acceptStarter8 :: StarterType -> Bool
acceptStarter8 Grass = False
acceptStarter8 Fire = False
acceptStarter8 Water = False

Whew. And that's with only two possible result values and three possible inputs. Yeah, this gets wild pretty quickly.

Closing up

While the format is a bit different from previous posts in the series, I hope you found something useful here too. The rest of what the chapter covers is definitely interesting, but not eye-opening in the way that kinds and cardinality are. The next chapter is on handling errors and includes Maybe, Either, more higher-kindedness, and anamorphisms (if that sounds interesting, why not check out this post on corecursion and anamorphisms in the meantime?). Stay tuned, and it should land at some unspecified future date.

Until next time: stay safe.

Improve your workflow with Forge

Mon, 16 Mar 2020 08:20:30 GMT

Some tools are so good that once you start using them, you can't imagine not using them. Magit is one such tool. If you're also working with GitHub or GitLab, you can use Forge to make life even better.

In this post, I'll be giving a brief overview of what Forge is, how to get started (well, at least what's not in the manual), and the problems I ran into.

Magit and Forge

Magit is a 'Git porcelain inside Emacs'. In other words, it's a Git UI inside Emacs, providing access to repo status and a number of different commands. Magit is so good, that even if I used another editor for developing (hah, as if!), I'd still switch back to Emacs to handle my Git business.

Forge is a Magit extension that allows deeper integration with certain /forges/^[1], such as GitHub and GitLab. For instance, when using GitLab, it allows you to create and comment on issues and pull requests, assign tasks, add labels (with auto-complete), open and close issues and more; all from within Emacs.

Motivation and early impressions

After posting this article to Reddit, I was asked about my motivation for using Forge over the GitLab UI. This section attempts to outline why I wanted to switch and what my impressions are after having used it actively for work for a couple of days.

Why not the GitLab UI?

I use GitLab for work, and while I think it's great in a lot of respects, it also has its share of issues, especially with the user interface. In general, navigating between projects, issue boards, and merge requests is very clunky and takes far too many clicks for it to be a good experience.

Because of this, I have bookmarked various projects' merge request overviews and issue boards, and use the bookmarks for navigating. It's not ideal, but it works tolerably well.

What doesn't work very well, however, is creating issues or merge requests. Not only do you need to write fairly long-form content, but you will also frequently want to add 'labels' or assign someone to a task. GitLab does have so-called /quick actions/, but they don't seem to work when used in the web UI.

This means writing and editing text in plain HTML text areas (without your favorite key bindings) and then having to use your mouse to navigate a UI, find multiple searchable lists of elements and filter and find the items you're looking for.

Overall: not a great experience.

Other options

As was pointed out in the Reddit thread, there are a number of ways to deal with the web UI issue. Notable contenders are the /Edit with Emacs/ extension for Chrome and Firefox, Emacs Application Framework (EAF) and it's browser, and Vimium and Surfingkeys.

I'm already a heavy user of Surfingkeys (though I realize I still have a lot of key bindings to internalize), and it's great for navigating your average website, but GitLab has so many things you can click on and interact with that it gets quite messy.

I tried Edit with Emacs, and while it's a really neat way to interact with text, it wasn't for me (for GitLab). I still had to activate it using my mouse, and it didn't relieve me from using the mouse to interact with the label and assignment menus either.

As for Emacs Application Framework, I'd never heard of it before. It looks super cool, but I'm a bit scared that if I go down that road, I'll never make it out alive. Further, this post by u/DarkNightened has some really good points on why the EAF might not be the right fit for this use case (as well as some pretty 🔥 Surfingkeys tips).

Early impressions

After having used Forge for a few more days, my impression is overwhelmingly positive. There are a few things it doesn't do, but I imagine some of that may be limited by the API that GitLab exposes. It also seems like GitHub receives more attention than GitLab, so it's quite possible that that integration has more features.

In the following sections, I'll be referring to issues and merge requests (MRs) collectively as topics, which is what Forge calls them.

The good parts

Issues and MRs in the Magit status buffer ::

This is the most basic thing that Forge gives you. In addition to just displaying these topics, it also displays labels (correctly color-coded) and whether a topic is open or not. If you expand an MR it'll list the commits it is made of. Viewing one of these items (by pressing <RET>) will also display assignees, all related comments and actions that have been taken.

Easy creation of issues and MRs from within Emacs ::

Of all the things that Forge does, this is what I wanted more than anything. I'm pleased to say that it works incredibly well. There are a few things that you can't do from Emacs (see the next section), but these are minor issues.

Autocomplete on typing when referencing issues and MRs, assigning users and labels ::

This one took me by surprise, but when you start typing out a topic ID (prefixed with # for issues or ! for MRs) you get auto-completion on topics in the project. The completion candidates show not only the ID, but also the topic title. This is super handy if you're adding related topics or otherwise want to reference them. This applies to seemingly all Magit buffers, including buffers for topic creation and for commit messages.

You'll also gain tab completion when assigning users to a topic or adding labels.

Quick copy a link to any topic, or open the topic in your browser ::

Opening a topic in your browser comes with a default key binding, and there is a function for adding the URL of the topic at point to your kill ring (forge-copy-url-at-point-as-kill). The latter wasn't bound to a key by default, but it's a very useful command when you need to share the link to a topic with a coworker or the like.

Viewing and checking out MRs is a breeze ::

It is incredibly easy to check out and create a branch from a merge request. Not much more to say about this, really. See the section on branching in the manual for more information.

The not-so-good parts

Certain values can't be set from Emacs ::

When creating merge requests, I haven't been able to specify that a merge request requires approval before being merged or that the source branch should be deleted after the merge. It's a minor thing, but it's something we do all the time on the team I'm on, so it'd be a nice feature to have.

Merge requests show diff between local branches ::

Say you make a few commits to the master branch (without pushing) before branching off, making a few more commits, and creating a merge request. Forge shows this MR as only containing the commits you made after branching off from master, even if the first commits you made to master haven't been pushed to the remote.

May (or may not) affect Magit performance ::

This is very uncertain and probably inaccurate. Magit has felt a bit slow over the past couple of days, but my machine is under quite heavy load, so I don't actually think it's been any slower than usual. All Forge updates should be performed asynchronously, so it shouldn't have a noticeable impact. However, when the minibuffer says 'pulling <repo>', my mind tends to automatically assume that Emacs isn't ready before the pull is successful. It does however relate to the next thing:

Updates to topics aren't 'optimistic' ::

When you make a change in Emacs, Forge goes ahead and tries to sync this with the remote right away, but until the sync is complete, the data you see in Emacs is still the same that it was before you edited it. This gives the impression that you must wait for a sync to finish before it's complete and that Emacs is busy updating, when it actually isn't. I'd love to see the Magit buffers update with your changes instantaneously, and then rather revert back (with an error message) if syncing with the remote failed.

Getting started

To get started, I'd recommend following the manual, which outlines most of what you need to know and shows you all the keybindings you need. That said, I did run into some issues (both on macOS and on NixOS), and there was some prerequisite knowledge I did not have, so I've decided to document that below.

Authentication: using `.authinfo`

The manual does have a section on getting started that includes some writing on setting up authentication, but, at least for GitLab, it requires some steps not mentioned in the manual.

Emacs uses ~/.authinfo files to handle credentials. I struggled to find much information about the format and what is required, but I can confirm that this entry format works:

machine gitlab.com/api/v4 login username^forge password api-token

Simply replace username and api-token with your username and your API-token (with 'api' scope), put it in ~/.authinfo, and you're off to the races. Admittedly, I don't know what the ^forge bit after the username means, but I assume it somehow relates it to Forge.

For more information on secrets and authentication, see this article on /Mastering Emacs/, which goes into quite a bit of detail of how to deal with encryption and credentials in Emacs.

macOS issues

Can't fetch topics

When setting it up on macOS, the biggest issue I ran into was that I couldn't successfully fetch topics (issues and pull requests). This appears to be a well-known bug with the ghub package that Forge uses, and this GitHub issue has a couple of workarounds for it. Personally, I had to force a workaround, by evaluating the following snippet:

(setq ghub-use-workaround-for-emacs-bug 'force)

However, I didn't put it in my config, and it seems to have worked successfully without setting this on subsequent uses, so it may or may not be necessary.

Recursive load

I also ran into a problem with window-purpose during the setup, causing an error about 'recursive load'. There is an issue on GitHub that describes it. One of the comments said to put

(require 'window-purpose)

in your .spacemacs file, which seemed to fix the issue for me.

This has not been an issue on NixOS.

NixOS issues: 'no EmacSQL SQLite binary available'

NixOS required much less fiddling to get set up, but there was an issue with my Emacs missing a binary, causing it to fail on startup. Specifically, I was told that there was 'no EmacSQL SQLite binary available', much like this issue about a missing C compiler. There may be a number of ways around it, but the solution I found was to change my installation of Emacs to one that includes the required emacsql-sqlite package. To do that, I replaced my Emacs listing in .configuration.nix with the following:

(emacsWithPackages (epkgs: [ epkgs.emacsql-sqlite ]))

I've only just started using Forge, but I've been very positively surprised by how much power it gives me directly from my favorite editor. Being able to check out and create branches from merge requests with a single command? Yes, please. Not having to take my hands off the keyboard to set issue labels? Perfect. Editing text and creating issues in an actual text editor and not having to copy it over into a browser window? You had me at Emacs.

I suspect there is still much to discover, though, so let's get going!

Footnotes

[1]

According to wikipedia, a forge is 'a web-based collaborative software platform'

[1]: According to wikipedia, a forge is 'a web-based collaborative software platform'

Org mode: tasty tricks

Mon, 09 Mar 2020 07:35:46 GMT

Intro

Org mode (Wikipedia, official website) is a powerful tool. On the surface, it looks like a lightweight markup language akin to Markdown, but if you look closer, you'll find that the Emacs major mode is so much more. It's a scheduler, a task manager, a spreadsheet editor, an organizing tool, and a way to do literate programming. I've been using the basic functionality of Org for a long time, but I recently took some time to explore the manual and pick up some new tips and tricks.

Custom TODOs

One of the easiest features to get started with is the TODO-system. Being able to tag a headline as TODO or DONE with a keypress makes it a super easy to get started with task management. I initially used it for logging work at my old job because keeping track of JIRA tickets through a web browser became a bit tedious.

The basic two-state system works well in a lot of cases, but sometimes you want a little more control. For that, you can create your own set of keywords. The paragraphs below give some examples of how I use it and some basic tips. For more information, consult the manual.

Creating and configuring keywords

One simple, but very powerful feature of Org is that you can define your own set of keywords to use as TODO-states. You can do this on a global level or on the file level. While doing it on the global level might be useful if you always want access to a specific set of keywords, defining your keywords at the file level allows you to work with a custom set of keywords for every file you operate on.

To define keywords for a file, use the #+TODO: keyword and list your desired states, using | as a separator between states that need work and states that that are considered closed. For instance, here's the set of keywords that I'm using for writing this very post. The set indicates whether a heading needs more work, is ready to be reviewed, is done, or if it should be cut:

#+TODO: TODO REVIEW | DONE CUT

Emacs offers you a number of ways to switch the state of an item. By default C-c C-t cycles through the list. You can also use S-<right> and S-<left> (that's S as in super), to change to the previous and next states, respectively.

When you have more than two states, switching to a specific state can start to get tricky if the flow isn't necessarily unidirectional. To make it easier to jump to a state, Org also allows you to add a 'hotkey' that you can use to set that state. This is done by putting the desired key in parentheses after the keyword:

#+TODO: TODO(t) REVIEW(r) | DONE(d) CUT(c)

Now, when you invoke the org-todo command (C-c C-t), rather than cycling through states, Emacs asks you to enter one of the assigned keys. Thus, if an item is currently marked as TODO, but you want to move it to REVIEW, the whole command would be C-c C-t r.

If you want more information assigned to each state, you can also tell Org to log the date and time of when a TODO-state was assigned as well as prompt you to add a note when that happens. As an example, here's the setup I'm experimenting with for recording and tracking blog post ideas:

#+TODO: IDEA(i!) DRAFT(d!) UPDATE(u@) READY(r!) | PUBLISHED(p!) ONHOLD(o@)

The ! after a letter tells Org mode to log the time when this state was entered. The @ tells Org mode that in addition to logging the time, I also want it to prompt me for a little note. With the set above, I want to track the times of all state changes, and when setting the state to UPDATE or ONHOLD, I also want to log a little note. This helps me remember why I put it in that state: Was there a paragraph that needed rephrasing? Have I got a better way to do things that I want to update a post with? Did I no longer see the post as relevant?

The ability to log all of these details is great, but it can quickly get messy. For instance, here's what a heading might end up looking like if we add some timestamps and notes to it:

** CUT Drawers
- State "CUT"        from "REVIEW"     [2020-03-08 Sun 20:17] \\
This is out of scope for now. Consider extracting this into a separate post.
- State "REVIEW"     from "TODO"       [2020-03-08 Sun 20:16]

This can add up fast, so we can make use of what Org calls drawers to keep our file tidy. These are delimited blocks of content that can contain anything but headlines and other drawers. Content put into drawers will be hidden by default (Org collapses it for you), and it will not be expanded by regular visibility cycling. If you want to cycle the visibility of a drawer, place your cursor on the drawer to cycle it.

Logging to a drawer would change the above example to this:

** CUT Drawers
:LOGBOOK:
- State "CUT"        from "REVIEW"     [2020-03-08 Sun 20:17] \\
This is out of scope for now. Consider extracting this into a separate post.
- State "REVIEW"     from "TODO"       [2020-03-08 Sun 20:16]
:END:

Now, no matter how many more entries you add to the log, Org will just show you a collapsed :LOGBOOK: line (that you can expand if you want to). To make Org log into drawers, use the following line:

#+PROPERTY: LOG_INTO_DRAWER t

Org-agenda

Another part of Org that has great synergy with TODO-items is the agenda view. To get started with agenda, you can add a file to the agenda list with org-agenda-file-to-front (C-c [). Then, when you invoke Org Agenda, you can choose between a number of different ways to view data found in all your agenda list files.

I'm still getting into it, but I've found immediate benefit from being able to quickly get an overview of all my open TODO items, and being able to easily toggle their states from a unified view, no matter which file they're located in.

For more information, see the chapter on agenda views in the Org Mode manual.

Org capture templates

Capture templates allow you to quickly and easily record notes and ideas into predefined files using a format you specify yourself. This is great for when you get little ideas that you just need to jot down somewhere and store for later.

I use it to record post ideas and to track the state of certain processes at work. The template allows you to automatically add plenty of context and to prompt the user for specific keywords, making the recording process as painless as you want to. Again, the manual is the best place to go for more information.

Literate programming

One aspect of Org mode that I'm really excited about but haven't really gotten into yet is /literate programming/. In short, literate programming is a way of intermingling source code blocks and natural language to be able to express intent and explain choices taken along the way.

There is some documentation on it on the org mode website, and I have done some very limited experiments with tangling and weaving. Writing an Emacs config from the ground up should be a perfect use case for it, as should any upcoming posts that contain code examples that can be compiled into a standalone program.

Closing thoughts

This is just scratching the surface of what Org mode can do. There is so much I haven't covered here, including working with tables, org-babel, tags, exporting files, and so forth; but even with just this, Org mode is already a force to be reckoned with.

I can see a bright future where my life is nothing but plain text. 🦄

Nix: override packages with overlays

Mon, 02 Mar 2020 08:26:30 GMT

In general, I am exceedingly happy with the Nix package manager, but one issue that I occasionally run into is that some packages aren't quite up to date. Usually, this isn't a big deal, and I've just had to learn that I can't always live on the bleeding edge. However, when you require functionality that was introduced after the Nix package was last updated, you're suddenly out of luck.

Luckily, though, Nix provides something called overlays, which you can use "to extend and change nixpkgs" according to the NixOS documentation.

A brief summary of the problem

To make this easier to follow, I'll make the problem more concrete. One of my goals for myself when working with Kubernetes, is to have absolutely all configuration stored as code. Regardless of how you feel about YAML, there's no denying that it can easily get tedious and repetitive with no templating or programming language features baked in.

For a while now, I've been wanting to look into the Dhall language for configuration, and this seemed a perfect opportunity, considering it also has Kubernetes bindings. But here's the problem: the Dhall version in nixpkgs is 1.24.0, but the Kubernetes bindings require at least 1.27.0 to work.

What are overlays?

For a more thorough explanation, consult the NixOS wiki or watch /Nixpkgs Overlays --- A place for all excluded packages/ by Nicolas Pierron. In short, overlays are functions that transform package sets by adding or overriding keys. Notably, Mozilla has their own set of Nix packages, Mozilla nixpkgs, which contains a Rust overlay, allowing Nix users to stay up to date with the quick cadence of Rust releases. In our case, it allows us to replace the packages that are out of date with newer updated versions.

There's quite a bit of content out there on how to use overlays: the aforementioned resources are a great place to start, and blog posts like the DOs and DON'Ts of overlays by Flying Circus offer further insights. That said, I couldn't find anything that really fit my use case. Thankfully, the helpful people over at the NixOS Discourse forums pointed me in the right direction.

The solution

Now that we know what they are and how they work, we can take a stab at creating our own overlays.

Let's assume our folder structure looks like the following:

├── shell.nix
└── nix-files
    └── overlay.nix

We can then write a very simple shell.nix file that just sets us up with the dhall package.

{ pkgs ? import <nixpkgs> { overlays = [ (import ./nix-files/overlay.nix) ]; }
}:
stdenv.mkDerivation {
  name = "dhall";
  buildInputs = [ pkgs.dhall ];
}

There's nothing terribly exciting going on here, but notice that when importing nixpkgs, we specify a list of overlays. In our case, it's only one, but we can use as many as we want.

With this all set up, let's have a look at the overlay:

self: super: {
  dhall = builtins.fetchTarball
    "https://github.com/dhall-lang/dhall-haskell/releases/download/1.30.0/dhall-1.30.0-x86_64-linux.tar.bz2";
}

The first line lists the expression's parameters. By convention, they're called self and super. In our case, we needn't concern ourselves with them, because we're replacing one of the keys entirely. The actual value of the expression is just a set with a single key, dhall, mapped to the result of fetching a tarball. This replaces that value in the original package set.

Some additional notes on the overlay: We're using the plain version of the builtins.fetchTarball function in this example. There's also a version that takes an attribute set with a URL and a hash, which will make sure that the you get the same version every time. Furthermore, we're specifying which exact version we want (1.30.0). That's perfectly fine for a little example like this, and will probably be good enough for a little dev environment. It would be ideal, though, if we could make sure we're always up to date with the latest release. The aforementioned forum help thread has a suggestion for how to do this, but it seemed to have some unexpected issues, so I’ll leave that as an exercise for the reader.

Now, when loading the shell.nix file, we should see the following:

$ dhall version
1.30.0

Boom.

Kubernetes first steps

Sun, 23 Feb 2020 20:20:58 GMT

One of my stated goals for this year, was to have a Kubernetes cluster running somewhere. As stated in the goals post, the deadline for this was April 1^{st}. As of Saturday night, I've got one.

I ended up using Digital Ocean for this, taking advantage of the Changelog podcast's sponsored signup offer, giving me two months to spend a $100 credit. While I've heard a lot of good things about Digital Ocean being very affordable, their managed Kubernetes offering can easily rack up some costs if you don't take care.

But now that I've got the cluster up and running, I'm at a bit of a loss. Just where do I go from here? What's the next thing to do? What should I prioritize? And why didn't I just start off with Minikube?

Why didn't I just use Minikube for now?

Actually, let's start with the last question: Why didn't I just use Minikube? Honestly, it was mostly a moment of weakness. I was actually busy procrastinating when I thought, 'hmm, I wonder how long it'd take to get set up with Digital Ocean'.

In hindsight: sure, Minikube would have been cheaper (at least once my credit is up), but having the actual cluster feels more 'real'. Like it's something I need to take care of. It's also available from anywhere and doesn't use my machine's processing power.

What do I want to get out of this?

So I got the cluster. Now what? Well, why did I want to do this?

While the option to just spawn little demo applications and have them available from wherever is great, I'm more interested in looking at Kubernetes for the overarching architecture and operational side of things. Specifically, I'd like to use this as a way to familiarize myself with:

service meshes and API gateways
monitoring and telemetry
security best practices

Each of these topics require a great deal of time and practice to master, but I'm not expecting to be a wizened monk any time soon. These are areas that I find fascinating and that I want to explore, but I cannot yet lay out any specific goals, as I need more time to research them.

Wait, didn't I have another goal for Kubernetes?

Sure did! In addition to just getting a cluster ready, my second Kubernetes goal was to expose a Haskell app with an API. This is still on the cards, but isn't due any time soon. This leans more towards the Nix and Haskell side of things, and less directly towards Kubernetes, so I can take some time to figure out how I want to do it.

Now what?

So where does this leave me? I think the biggest issue I have identified is my lack of knowledge. I've been working with RedHat's OpenShift for the past year, and feel like I have a pretty good grasp of how that works, at least from the dev side. But when faced with that clean cluster, I froze. I didn't know where to go, didn't know what commands to run, or where to turn to for advice. So I need to formulate a plan.

I think I would like to familiarize myself with Kubernetes more or less from the ground up. That means reading documentation, doing tutorials, and making sure my understanding is correct. Second: I would like to have a look at basic security measures and best practices. At the very least, I want to enable and configure RBAC. When I get this far, I think it would be an appropriate time to take a step back and reevaluate where I'm headed and recalculate.

Corecursion and anamorphisms

Mon, 17 Feb 2020 06:37:13 GMT

You know that feeling where you hear about something and you immediately need to look into it? I had that while listening to the most recent episode of Adam Gordon Bell's /Corecursive/ podcast today, where they were talking about where the name of the podcast came from. Up until then I had just assumed that corecursive meant /mutually recursive/^[1], but boy, was I wrong!

The C-word

According to Wikipedia, 'corecursion is type of operation that is a dual to recursion'. This quickly gets very theoretical, but the long and short of it is that corecursion can be seen as a kind of opposite of recursion: Where recursion allows you to operate on arbitrarily complex data as long as you can reduce it down to a set of base cases, corecursion allows you to generate arbitrarily complex data when given a base case.

Err ... what?

Think of it like this: Given a list of numbers (arbitrarily complex data), you can define a recursive function to sum all the numbers using simple base cases: is the list empty or are there more elements to add? Your language of choice may well have a sum function that does just this. If not, you can implement it with a fold or reduce.

However, given a number, can you create a list that when summed up would equal this number? This would be a form of corecursion, where we take simple data (a number), and generate arbitrarily complex data based on the input (the resulting list of numbers).

Let's talk about fold functions specifically. As we talked about in a previous post on folding, fold functions are catamorphisms. They take a data structure and reduce it to a 'lower' form. The opposite of a catamorphism is an /anamorphism/^[2], and the opposite of a fold is an unfold (at least in Haskell). An anamorphism generates a sequence by repeatedly applying a function onto its previous result.

According to Wikipedia, 'the anamorphism of a coinductive type denotes the assignment of a coalgebra to its unique morphism to the final coalgebra of an endofunctor.' Don't worry: you needn't understand that to understand unfolding and corecursion (I sure don't). Instead, let's try and get a feel of what we might use corecursion for.

Returning to our previous example of destructuring a number into a list of terms, let's look at a couple of ways to do it using unfold.

First, let's look at the unfoldr function itself. It is defined in the Data.List module, and its type signature is:

unfoldr :: (b -> Maybe (a, b)) -> b -> [a]

Given a function from b to Maybe (a, b) and a b, it will produce a list of a. If the function (let's call it f) returns Just (x, y), x will be added to the result, and f will be called again with y. This continues until f returns Nothing, at which point unfoldr terminates, returning the list it has created.

With this, we can create a function that takes an integral value and returns a list that when summed, is equal to the value we passed in. We'll start with an easy variant that just destructures the input into ones. For simplicity's sake, we're ignoring negative numbers.

module Unfold where

import Data.List (unfoldr)

unroll :: Integral a => a -> [a]
unroll = unfoldr f
  where
    f n
      | n <= 0 = Nothing
      | otherwise = Just (1, n - 1)

Easy enough. The function passed to unfoldr returns nothing if there is no more to sum. Otherwise, add $1$ to the list, and call again with n-1.

unroll 5 -- [1,1,1,1,1]
unroll 0 -- []

But we can have some more fun with this. How about we try and destructure a number into a list of the pieces we'd need to create a binary representation of it?

binary :: Integral a => a -> [a]
binary = unfoldr f
  where
    f n
      | n <= 0 = Nothing
      | otherwise =
        let powerOfTwo = 2 ^ floor (logBase 2 $ fromIntegral n)
         in Just (powerOfTwo, n - powerOfTwo)

This is a bit more complicated, but only because we need to map the input value to a power of two. Luckily, we can use logBase 2 to get the exponent you'd need to get n, and then floor it to get the greatest integral exponent. This becomes the next entry to the list. What's left gets passed in to the next application of the function.

binary 0 -- []
binary 5 -- [4,1]
binary 255 -- [128,64,32,16,8,4,2,1]
binary 256 -- [256]

Pretty neat, huh? What if we take it a step further and convert the number to its binary representation instead, as if it was base 2?

base2 :: Integral a => a -> a
base2 = sum . unfoldr f
  where
    f n
      | n <= 0 = Nothing
      | otherwise =
        let exponent = floor (logBase 2 $ fromIntegral n)
         in Just (10 ^ exponent, n - 2 ^ exponent)

As you'd expect:

base2 0 -- 0
base2 5 -- 101
base2 10 -- 1010

Not too shabby at all.

Alright, I think we have had enough fun with corecursion for now. It's been a very unexpected, but very insightful little journey, and I thank you for taking it with me. Until next time!

Footnotes

[1]

I'm probably not the only one to do this. Wikipedia has a note under disambiguation on its article on corecursion that says 'not to be confused with mutual recursion'

[2]

Similar to how your body can be in either anabolic or catabolic states, for all you fitness people out there.

[1]: I'm probably not the only one to do this. Wikipedia has a note under disambiguation on its article on corecursion that says 'not to be confused with mutual recursion'
[2]: Similar to how your body can be in either anabolic or catabolic states, for all you fitness people out there.

My first Emacs Lisp

Mon, 10 Feb 2020 08:04:46 GMT

In my previous post on reading the Emacs manual, I mentioned that there were a couple of things that I was missing from my editing workflow when using regular Emacs bindings. The most notable of which was the ability to kill up until a search hit. I knew that it'd be possible to write a function to do it, but I didn't really know where to start, so I figured I'd just do it later.

In the Reddit thread about the post, user e17i gave me a little snippet to get me started for writing such a function. Turns out that was all I needed to get started. I put aside my fear of Lisp and went to work.

I ended up with three functions for Vim-like search movement: just exiting a search at a result, killing up to a search result, and copying up to a search result. Not particularly complicated, but I was so proud of myself when I got it working. The functions are included below for your viewing pleasure^[1]:

(defun isearch-vim-style-exit ()
  "Move point to the start of the matched string, regardless
  of search direction."
  (interactive)
  (when (eq isearch-forward t)
    (goto-char isearch-other-end))
  (isearch-exit))

(defun isearch-vim-style-kill ()
  "Kill up to the search match when searching forward. When
searching backward, kill to the beginning of the match."
  (interactive)
  (isearch-vim-style-exit)
  (call-interactively 'kill-region))

(defun isearch-vim-style-copy ()
  "Copy up to the search match when searching forward. When
  searching backward, copy to the start of the search match."
  (interactive)
  (isearch-vim-style-exit)
  (call-interactively 'kill-ring-save)
  (exchange-point-and-mark))

I've mapped the functions to three separate key bindings in isearch-mode-map to make them easily accessible while searching:

(define-key isearch-mode-map
  (kbd "<C-return>") 'isearch-vim-style-exit)

(define-key isearch-mode-map
  (kbd "<M-return>") 'isearch-vim-style-kill)

(define-key isearch-mode-map
  (kbd "<C-M-return>") 'isearch-vim-style-copy)

Justification and motivation

In Emacs, when searching with isearch, when you 'accept' a match and move point there, Emacs will put you at the end of your matched text. Sometimes this is exactly what you want. Often, though, I find that I'd rather have point move to just before where the search string matches. This is how it works in Vim, and I have convinced myself that it's also the standard way of moving cursors to searches in other text editors. In addition to just moving to a search result, I want the same pattern to apply for killing and for copying the text between your original cursor position and the search match. In short, these functions act on everything between your original cursor position and the start of the selected match, regardless of whether you search forwards or backwards.

This functionality is the same as regular isearch when searching backward, but when searching forward it's the same as adding an extra C-r (isearch-backward) after picking a match.

The Reddit snippet

The tip I got on Reddit gave me a little code snippet to start me off. At first I thought it was just what I wanted, but I realized later that it wasn't quite what I was looking for. The original snippet is very symmetrical in that it goes to the end of a match when searching backward and to the start of a match when searching forward. However, I have found that I always want to move to the start of a match, no matter what side I come at it from. This may seem asymmetrical, but one nice thing about it is that it'll work the same on any match, regardless of whether you're searching forward or backward. This is especially useful if your search has wrapped.

The original snippet was:

(define-key isearch-mode-map (kbd "<C-return>")
  (lambda () (interactive)
    (isearch-repeat(if (eq isearch-forward nil)
                      'forward
                    'backward))
    (isearch-exit)))

So even if it wasn't quite what I needed, it gave me the tools necessary to start working on my own implementation, namely the isearch-repeat and isearch-exit functions and the isearch-forward variable. With this I had all I needed to start playing around with the functionality myself.

Elisp crash course

If you've never encountered Lisp before, here's a short (and very incomplete) introduction to Emacs Lisp. Do bear in mind that this is the first Emacs Lisp code I've written myself, so I'm probably missing a lot of context and nuance. If you find any errors, please do tell me; I'll be very grateful. For a more complete introduction to the language, check out the Emacs Lisp manual.

defun [name] (args) ::

The first line of the function contains the keyword defun, signifying that we're defining a function, the name of the function, and a list of parameters. In all the above functions, the parameter list is empty, so it's just a set of empty parentheses (()).

Comments ::

After the first line of each function, I've added a string describing what the function does. This works as documentation. When looking up the function in Emacs (C-h f <name of function>), this text gets displayed. It's not a requirement to put into a function, but it's nice to have when you need to look things up.

Furthermore, lines starting with ; are standard code comments, such as the one about setting current~prefix-arg.

(interactive) and call-interactively ::

In Emacs Lisp, (interactive) turns a Lisp function into a command. In short, this means that you can assign it to a key sequence and call it from anywhere in Emacs by using M-x. Similarly, call-interactively is used to call interactive commands that take arguments. These commands can either be given explicit arguments and called like normal functions, or we can use call-interactively. When doing the latter, certain parameters can be passed implicitly. For instance kill-region needs two arguments BEG and END to know what region to operate on. When called interactively, BEG and END get the values of point and mark, so we don't need to pass them explicitly. For more information about the commands and interactive, check out the Emacs Lisp manual, chapter 21.2.

What's with the quotes? ::

Again, the manual (chapter 10.3) has info on what the single quotes do on a deeper level, but in our case, what we want is simply to pass the quoted function as an argument, and not the result of evaluating that function. In short, it's passing a function pointer, rather than the result of a function.

when ::

when is the Lisp way to only evaluate some code when a condition holds true. It's similar to an if expression, but an if expression needs an else-clause to run if the provided condition doesn't hold true. In languages like Python, Rust, JavaScript, etc., it's the equivalent of just using an if expression/statement without an else-clause.

let and current-prefix-arg ::

There's not a whole lot of variable binding going on in these functions. The one place it's done is in the isearch-vim-style-copy function. As you might expect, the let keyword assigns a value to a variable (in this case (4) to current-prefix-arg). In Emacs Lisp, variables can have global scope, so the let binding ensures that the variable is only bound to this value within this scope. The variable current-prefix-arg is used to augment set-mark-command. We do this to move point back to where the search started after copying the region.

And that's the story of how I got started writing Lisp. It's dead simple, but I've got a taste for it now, and I think I like it. ... yeah. I think I like it.

Footnotes

[1]

The code above has been modified from its original published state after feedback from Reddit and working with it some more. The original snippet looked like this:

(defun isearch-vim-style-exit ()
  "Move point to the start of the matched string, regardless
    of search direction."
  (interactive)
  (when (eq isearch-forward t)
    (isearch-repeat 'backward))
  (isearch-exit))

(defun isearch-vim-style-kill ()
  "Kill up to the search match when searching forward. When
    searching backward, kill to the beginning of the match."
  (interactive)
  (isearch-vim-style-exit)
  (call-interactively 'kill-region))

(defun isearch-vim-style-copy ()
  "Copy up to the search match when searching forward. When
    searching backward, copy to the start of the search match."
  (interactive)
  (isearch-vim-style-exit)
  (call-interactively 'kill-ring-save)
  ;; set prefix arg to move point back to where search started
  (let ((current-prefix-arg '(4)))
    (call-interactively 'set-mark-command)))

[1]

The code above has been modified from its original published state after feedback from Reddit and working with it some more. The original snippet looked like this:

(defun isearch-vim-style-exit ()
  "Move point to the start of the matched string, regardless
    of search direction."
  (interactive)
  (when (eq isearch-forward t)
    (isearch-repeat 'backward))
  (isearch-exit))

(defun isearch-vim-style-kill ()
  "Kill up to the search match when searching forward. When
    searching backward, kill to the beginning of the match."
  (interactive)
  (isearch-vim-style-exit)
  (call-interactively 'kill-region))

(defun isearch-vim-style-copy ()
  "Copy up to the search match when searching forward. When
    searching backward, copy to the start of the search match."
  (interactive)
  (isearch-vim-style-exit)
  (call-interactively 'kill-ring-save)
  ;; set prefix arg to move point back to where search started
  (let ((current-prefix-arg '(4)))
    (call-interactively 'set-mark-command)))

Tips and tricks for the fledgling Emacs user

Mon, 03 Feb 2020 07:31:30 GMT

Whenever I have gone looking for the Emacs equivalent of Drew Neil's amazing Practical Vim/[fn:1], I have always found people recommending reading the Emacs manual, saying how good it is. For the longest time I put it off, and I was on the verge of ordering Mickey Petersen's /Mastering Emacs, which I also often see mentioned, when I decided that I should just bite the bullet and at least start with the manual. It can't be that much of an undertaking, right? ... right?

Wrong.

Coming into 2020, I decided that I'd get through the manual by the end of January. I had no idea how much content was in there. Throughout the first 30 days of 2020 I sunk hour upon hour upon hour into this collection of text documents, assembled through many decades, until I finally found my way out, clothes tattered and torn, beard long and graying^[2], but my mind opened and enlightened.

What I want to share today is not the entire voyage, but rather just the high points that come to mind when I think back on my journey. Undoubtedly there will be things I've forgotten that should be here and things that are here that I should have forgotten, but such is the nature of human experiences. The best we can do is to just sit back and enjoy what we have.

I will be using the same notation for key bindings as Emacs does in the manual. Most characters are just represented as themselves, but some are special. Chords (or modified keys) are represented as modifiers connected to letters by a hyphen. A list of the modifiers and special keys used in this document is presented below. More information on characters and notation can be found in [[https://www.gnu.org/software/emacs/manual/html_node/emacs/User-Input.html#User-Input][chapter 2]] of the Emacs manual.
  - ~C~ and ~M~ :: These upper case letters represent ~Control~ (your trusty control key) and ~Meta~ (usually alt), respectively. When used with other characters, they're connected by a hyphen: ~C-a~ and ~C-M-a~, for instance.
  - <RET> :: The return (or enter) key
  - <ESC> :: The escape key

In addition, note that ~M-x~ opens the ~execute-extended-command~ menu, wherein you can type in the name of a command. Thus, when you see something such as ~M-x other-window~, that means that the text after ~M-x~ is the name of the command. To execute the command, press return.

The Emacs key bindings

Among the reasons to go through the manual, learning the 'proper' Emacs key bindings was actually pretty high up. It might seem trivial, but there's been a number of times where I've ended up in a state that I just can't navigate and don't know how to get out of.

As someone who fell in love with the Vim way of editing, I came to Emacs through Spacemacs, which uses Evil mode to provide a very comfortable editing experience for Vim users. As such, I've already spent a considerable amount of time learning one very specific set of key bindings (and I love it every day). Why would I bother learning another?

For some time, I've been using 'hybrid-mode', giving me Vim key bindings in every mode except insert mode, where I switch to using Emacs key bindings. What I have discovered is that they both have their uses, and they shine in different ways. When navigating text, I feel much safer and much more agile in Vim's normal mode, able to take leaps and bounds through the buffer, working on text as objects, confident that I won't accidentally insert some text by a stray key press. However, when typing out text, having to exit insert mode and then type a command before entering insert mode again can be pretty tedious, and being able to make certain movements from that state just makes a lot of sense. A good example would be needing to go back to an earlier part of a line, delete some text, and then keep inserting at the end. In Vim, assuming we're already in insert-mode:

<ESC> ? <search term> <RET> d a w A

And in Emacs

C-r <search term> <RET> M-d C-e

If my counting is correct, the Vim version has three more key presses than the Emacs version. Unless you're into golfing, this probably doesn't mean much to you (especially if d a w feels like a single movement to you), but to me, it feels (subjective, I know) like more work.

A surprising key binding that I've never really considered, but which Emacs has bound to C-M-v by default: scroll-other-window. If you have multiple windows open in a single frame, it'll scroll one of the other windows, but lets you keep your focus (and your cursor) in your current window. There may be a way to target a specific window, but I found it most useful when having only two windows open. If you have more than two windows open, it'll scroll the next window as defined by Emacs buffer ordering. From what I can tell, this is left to right, top to bottom. It's the same order as when going through the windows by C-x o (other-window).

Navigating text (chapter 25)

Naturally, coming from Vim, I think that the Vim way is the way for navigating text on a screen. Or at least I did. In fact, I was surprised to learn that Emacs has all the same facilities for navigating a buffer as Vim does, and it turns out that all the same movement patterns that I rely on in Vim are available in Emacs; they're just bound to different keys.

There are a couple of editing patterns that I haven't found the equivalent of yet, though (send me tips if you've got them!):

The equivalent of [[https://github.com/tpope/vim-surround][~vim-surround~]]: Yup, Evil has a function for it, and I'm sure you can write a function that will contain text, change surrounding delimiters to something else, or delete surrounding delimiters, but it's not immediately available. That said, even in Vim, it's a plugin, so maybe this shouldn't count.
Delete to next search hit: In Vim (normal mode), a common way for me to operate on some text is to d / <search string> <RET> to delete up until the first occurrence of the search string. I haven't found a way to do that in barebones Emacs. Emacs comes with the default binding of M-z as zap-to-char, which is almost what I want. However, rather than being able to input just a single character, I want to be able to input a search string and kill up to that point. It'd be similar to C-<SPC> C-s <search string> <RET> <DEL>, but without killing the actual search string (kill up to, not including), and it should be a single key chord. If I were to override the M-z binding, it'd be M-z <search string> <RET>.

Overall, though, I am surprisingly happy with Emacs' movement keys, and while I don't think I'll be switching over completely just yet (or ever?), being able to navigate comfortably in either mode can only make things better.

The kill ring (chapter 12.2.1)

Of all the things I've learned, this is probably the one that's given me the most bang for my buck. Whatever your preferred way of getting text into your kill ring (also known as the clipboard), once you've got it there, Emacs doesn't just store the last entry, but rather your last 60 (by default) entries. The reason it's called a ring is that you can cycle through your entries, and once you reach the end, you loop back around.

So when you paste (or yank) something into a buffer with C-y, you can then follow that up by using M-y to cycle through the kill ring. You can view the entire contents of the kill ring with C-h v kill-ring, though be warned: the kill ring is displayed as lisp code, so it may not read as plainly as you'd expect.

Side note: I wish Vim and Emacs could at least agree on the meaning of the word 'yank'. Whether it means to copy something from the buffer (Vim) or to paste something into the buffer (Emacs), I don't care. Just pick one.

The undo-stack (chapter 16.1)

Spacemacs uses the undo-tree-mode (see the Emacs wiki entry) to store buffer states in a change tree. This is really intuitive and lets you visualize your changes in a graphical model that maps pretty well onto my view of a set of changes. What I wasn't aware of was that plain, ol', vanilla Emacs comes with a pretty uncommon, but super-powerful undo history out of the box: instead of taking the view that history branchces after undoing a change, the default undo is more like a strictly linear timeline. When you undo a change, Emacs simply pushes this onto the history 'stack' as a new change. Thus, Emacs doesn't have a traditional redo as such: it's just another undo.

It's a bit tricky to wrap your head around, but is really worth checking out. For context, here is an excerpt from the relevant chapter in the manual (link to web version):

To begin to undo, type ~‘C-/’~ (or its aliases, ~‘C-_’~ or ~‘C-x u’~). This undoes the most recent change in the buffer, and moves point back to where it was before that change. Consecutive repetitions of ~‘C-/’~ (or its aliases) undo earlier and earlier changes in the current buffer. If all the recorded changes have already been undone, the undo command signals an error. Any command other than an undo command breaks the sequence of undo commands. Starting from that moment, the entire sequence of undo commands that you have just performed are themselves placed into the undo record. Therefore, to re-apply changes you have undone, type ~‘C-f’~ or any other command that harmlessly breaks the sequence of undoing; then type ~‘C-/’~ one or more times to undo some of the undo commands. Alternatively, if you want to resume undoing, without redoing previous undo commands, use ~‘M-x undo-only’~. This is like ‘undo’, but will not redo changes you have just undone.

Bookmarks (chapter 13.8)

When reading through a document in several sittings, it's useful to be able to jot down how far you'd gotten so that you can easily resume your reading. To this end, Emacs has a bookmarks system which was very useful when reading through the documentation. Similarly to how one might use bookmarks in a web browser, I've also bookmarked a number of documents to read later, so that I don't forget. In addition to being useful for reading, I can also imagine it being useful if you often need to refer back to a specific file when programming, though registers (chapter 13), specifically File Registers (chapter 13.6), may be better suited to this.

Amusements (chapter 47)

The Amusements chapter covers a couple of really neat, mostly useless, tricks that you can do with Emacs. It lists a couple of built-in games, including classics such as Snake, Pong, and Tetris (turns out I'm horrible at that, by the way); text-based adventure games (dunnet) and Conway's Game of Life simulation (life); ways to convert a region into morse code or NATO phonetic alphabet (morse-region and nato-region to encode, unmorse-region and denato-region to decode); ways to play with the display when idling (zone); and references to XKCD.

Glasses mode (chapter 26.9) and flipping tables

In addition to being able to encode a region as morse or the NATO phonetic alphabet, Emacs also comes with something known as glasses mode. This isn't actually an amusement and is originally supposed to help make camelCased words easier to read by turning them into snake_cased words instead (or using any other separator that you want). This is all just presentational, so even though Emacs might display the identifier as my_Favorite_Identifier, it'll get saved as myFavoriteIdentifier, and when you edit the text, the underlines aren't actually there for you to edit.

Because you can change the separator, you can also use glasses mode to separate your identifiers with an arbitrary text string. My coworker suggested the 'table flip', and sure enough, Emacs is happy to comply:

myFavoriteIdentifier = someOtherIdentifier

becomes

my(╯°□°）╯︵ ┻━┻Favorite(╯°□°）╯︵ ┻━┻Identifier = some(╯°□°）╯︵ ┻━┻Other(╯°□°）╯︵ ┻━┻Identifier

and for fun, here it is in morse

--/-.--/..-./.-/...-/---/.-./../-/./../-.././-./-/../..-./.././.-. -...- .../---/--/./---/-/...././.-./../-.././-./-/../..-./.././.-.

Debugging (chapter 27.6)

This is something I want to get into at a later stage, and the manual makes it seem rather impressive. I've not used a debugger at all since switching to Emacs, and while I usually manage just fine, I find myself longing for one every so often.

Compilation buffers and jumping to the next error (chapter 27.2)

Somewhat related to debugging is compiling your programs in your editor. While a lot of text editors have built-in terminals that you can run things in, I haven't often seen them also have the ability to jump to any errors that occurred during compilation, though I haven't actually gone looking. I am told, however, that this is fairly commonplace in applications known traditionally as IDEs. Emacs, eager not to disappoint, comes with support for this built in.

When using Emacs for compiling a program, it will usually open a compilation buffer. If you have compilation errors, you can then use one of a number of keyboard shortcuts (my favorite is `M-g n`) to jump to the next compilation error. The best part is: this works from any buffer, even one not in compilation mode. In other words, if you have errors, you can jump through them sequentially without ever going back to the compilation buffer.

Indentation and programming indentation (chapter 24 and chapter 26.3)

There's a number of cool indentation tricks listed in the indentation and the program indentation chapters, but I think the one I found that the most useful was for indenting languages derived from C. For a detailed explanation, check out this page on customizing indentation in C, C++, and Java, but the short version of it is that you can analyze the various syntactic parts of your program directly in your buffer, and then tell Emacs how it should indent a specific construct.

Org mode

What about org mode? You'd think that that was one of the best parts of Emacs, and that I'd have a ton of tips to share, right? Well, you're probably right, but org mode comes with its own manual (another pretty massive text document), and I haven't made my way through that one yet. Trust me, though: it's on my list of must-reads this year. I'll get to it later.

So that's the first item on my SMART goal list completed. I'm not quite ready to start configuring Emacs from scratch, but it doesn't feel that daunting anymore. Now: do I go to work on Kubernetes, or do I spend some extra time with org mode to get really comfy with Emacs?

Footnotes

[1]

Seriously, if you're into Vim and haven't read it: I highly recommend you take the time to go through it. For a beginner it's an absolute gem, and even for experienced vimmers, I'm sure there's a good few tricks in there that you didn't know about.

[2]

I have actually found a grey hair in my beard this month (sshhh!) and I attribute that wholly to Emacs.

[2]: I have actually found a grey hair in my beard this month (sshhh!) and I attribute that wholly to Emacs.

Use git to restore parts of a file

Mon, 27 Jan 2020 10:52:44 GMT

One of my favorite, little-known git tricks is using the --patch option (-p for short) to affect only parts of a file when you're adding, committing, or---as I recently found out---/restoring/.

Now some of you may be thinking 'hold up! Restoring a file? What's that?' If you haven't been following git development for a while (while a solid project, it's not exactly the JS framework du jour), you'd be excused for not knowing about the two new git commands switch and restore. Introduced in git 2.23.0 (released in August 2019), these two commands were introduced to offload the checkout command for switching branches and restoring files. As you may expect, switch allows you to switch branches (and create new ones in the same way as checkout -b by using switch -c), and restore takes over for restoring files from a previous commit.

`git restore -p`

While there is lots to be said about the new commands and all the options they accept, I advise you to go check out the official docs for ~switch~ and ~restore~ for that. This post is about using restore with the -p option, specifically.

Sometimes (read: quite often) when making changes to a file, you have some changes that you want to keep and some that you want to discard. In situations like this, there's a number of ways to go about it. If the changes are ready to be staged or committed, you can use the -p option to pick the parts you want and then just restore the file afterwards. However, in the event that the changes you want aren't quite ready just yet, and you don't want to stage, restore, and unstage, you might want to consider using restore -p.

By default, restore -p brings up an interactive prompt that lists the differences between your working tree and the last commit, asking you in turn whether you want to discard each hunk. This way, if you have some changes in one part of the file that you don't want to keep, you can discard those, but keep the rest. As per usual with the --patch option, you can also split hunks or edit them manually if you need to.

When not given any paths to act on, git restore -p will ask you about all your unstaged changes in all your files. To limit it to a specific file, supply a pathspec: git restore -p myfile.txt. The documentation says that '--patch can accept no pathspec and will prompt to restore all modified paths.'^[1], but when testing it out using git version 2.23.1, using pathspecs was no problem at all.

In addition to the default patch restoration functionality, some of the other notable options are:

~-s <tree>~ / ~--source=<tree>~: Use this option if you want to restore a file to a different commit than HEAD. What you pick as your source can be another branch, a commit hash, or something like HEAD~2. Without the --patch flag, this will change the entire file to reflect the state at that source.
~-S~ / ~--staged~: Not to be confused with the lowercase -s above, the -S option allows you to restore changes from the index (your staging area or 'added files'). By default, restore only works on files in your work tree. Using this option will make it work only on files in your staging area. To act on both at once, supply both the -S and the -W (--worktree) options: git restore -SW.

Footnotes

^[1]https://git-scm.com/docs/git-restore#Documentation/git-restore.txt---patch

feature(slice_patterns)

Mon, 20 Jan 2020 07:56:09 GMT

About a week ago, there was an item in This Week in Rust (issue 320) that caught my eye under the Tracking Issues & PRs heading:

[disposition: merge] Stabilize ~#![feature(slice_patterns)]~ in 1.42.0.

Aww, yeah! I've been waiting for this for literal years, so you better believe that was a good day! I mentioned this in my Rust 2020 post as one of the things I'm the most excited to see this year, so now that it's getting stabilized soon (2020-03-12), let's make sure we're prepared!

About slice patterns

Most of the information contained in this section can also be found in the [[https://github.com/rust-lang/rust/pull/67712][RFC]] on GitHub. The RFC is very thorough and gives lots of examples, so go give it a read if you want to know more about the feature!

We've had some form of slice matching on stable Rust for a while now, but without this feature, the form of matching you can do is rather limited. With an array of a known length, you can destructure and match as you please, but for slices of an unknown length, you must provide a fallback because there is no way to cover all the potential cases in a match expression. Also, quite importantly: there is no way to bind variables to subslices. This feature finally opens the gates to subslice and subarray matching, mitigating both the above issues, and making slice patterns immensely more powerful.

Two flavors

There are two syntactic flavors for the new subslice patterns: one for when you want to bind a subslice to a variable, and one for when you just want to denote that there are elided elements. Both flavors use the .. pattern (referred to as a rest pattern) to match a variable number of elements. The number of elements matched depend on the length of the array or slice and the number of matching elements before and after in the match.

Matching without binding

Looking at the first flavor, matching without binding, we get introduced to the .. pattern straight away:

fn f<T>(xs: &[T])
where
    T: std::fmt::Debug,
{
    match xs {
        // the slice has at least two variables.
        // we bind the first and last items of
        // the slice to `x` and `y`, respectively
        [x, .., y] => {
            println!("First and last: {:?} and {:?}.", x, y)
        }

        // the slice has a single item: `x`
        [x] => {
          println!("the slice has a single item: {:?}.", x)
        }

        // the slice is empty
        [] => {
            println!("Got an empty slice.")
        }
    }
}

Remember that .. can match any number of elements, including 0. This means that the first pattern matches anything that has at least two items. You can also use the pattern without 'delimiting' it on both ends, such as if you wanted to implement these two functions for getting the first and last element of a slice:

fn first<T>(xs: &[T]) -> Option<&T> {
    match xs {
        [x, ..] => Some(x),
        [] => None,
    }
}

fn last<T>(xs: &[T]) -> Option<&T> {
    match xs {
        [.., x] => Some(x),
        [] => None,
    }
}

Notice how both of these functions pick out a single element of the slice (first and last respectively) and ignore the rest. Because .. matches 0 or more elements, the first pattern in both functions matches slices with one or more elements.

Matching and binding a subslice

The other flavor lets you bind a subslice to a value, which takes slice patterns and cranks the power level up another notch or two. The binding is done using the @ operator.

Imagine for instance that we want to write a sum function. That could be done as such:

fn sum(xs: &[i32]) -> i32 {
    match xs {
        [] => 0,
        [x, xs @ ..] => x + sum(xs),
    }
}

In the example above, if the slice is not empty, we take the first element, x, and add it to the result of summing the rest of the list, xs. With Rust already having a sum method on iterators, this function is pretty redundant, but it makes a good example of how to bind and use subslices.

Another example would be to get the middle element of a slice if the slice has an odd number of elements. If the slice is empty or has an even number of elements, return None:

fn middle<T>(xs: &[T]) -> Option<&T> {
    match xs {
        // ignore the first and last element.
        // recurse with what's in between.
        [_, inner @ .., _] => middle(inner),

        // one element! got it!
        [x] => Some(x),

        // oops! there's nothing here.
        [] => None,
    }
}

Here we iterate through the slice from both sides, continuously picking off one element at the start and one element at the end. Whatever is left in the middle (if there are at least two elements) gets assigned to xs and used as input to another step through the function. Once we have either one or zero elements left, we have our answer.

Why this is a big deal

It might come across as somewhat strange that I'm so enthused about a feature that may seem rather small, but it's one of those quality of life things that I find myself running into all the time. Being used to Haskell and their pattern matching behavior, I always forget how cumbersome it is to match on an arbitrary slice in Rust. Up until now, we've had the ~split_first~ method (and split_at) on slices, which I can never remember the name of, which returns an Option, and which doesn't let you do arbitrary match-stuff (such as using match guards, for instance). The new slice_patterns feature is a major step up in that regard.

The other thing that I'm super jazzed about? Being able to match on the end of a slice. Not only can you pick off items from either end of the slice, but you can also make sure that the slice ends in a certain value or series of values.

In short, I think this is an amazing addition to stable Rust. Hats off to all the people that have made it possible. Now go read the RFC and look out for all the other cool stuff they're talking about (arbitrarily nested OR patterns? Oh, my!).

Let's read: Haskell Programming from First Principles, pt X

Mon, 13 Jan 2020 10:23:48 GMT

A whole chapter dedicated to folding lists? I thought it would be quick and easy, but it turns out there's a lot of things to uncover in regards to folding. In this post we'll be looking at what a fold is and how folding left differs from folding right.

A brief explanation of folding

Before we can talk about folding left or right, it's important to know what we're talking about. In short, a fold is a way to reduce a data structure (lists in our case) to some other type by working through the data structure. If that sounds unclear, a more concrete example could be a function that sums all the numbers in a list or a function which tells you whether any element of a list satisfies a predicate.

Let's look at foldr, short for 'fold right'. We'll talk about foldl (fold left) in a bit. The signature for foldr is as follows as taken from GHCi, version 8.6.5:

class Foldable (t :: * -> *) where
  foldr :: (a -> b -> b) -> b -> t a -> b

We haven't talked about kinds yet (that's in the next chapter), but the t :: * -> * part means that t is a type that is parameterized by another type. In our instance, we can substitute it with a list instead:

foldr :: (a -> b -> b) -> b -> [a] -> b

So foldr is a higher-order function that relies on two parametrically polymorphic types, a and b. It takes a function from a and b to b, an initial b value, a list (or Foldable) of a, and returns a b.

This snippet is taken from the Haskell Wiki article on folding (which I recommend you take a look at if you're not quite familiar with it), and shows the equations defining foldr:

-- if the list is empty, the result is the initial value z; else
-- apply f to the first element and the result of folding the rest
foldr f z []     = z
foldr f z (x:xs) = f x (foldr f z xs)

In the above snippet f is the (a -> b -> b), z is the b, and the [a] is either destructured into [] or (x:xs).

Folds are catamorphisms, a way of deconstructing data, reducing a structure to a completely different result.

Why direction matters: `foldr` vs `foldl`

Building on the basic definition of a fold, let's explore the differences between folding left and folding right and what impacts that has on your programs.

Let's revisit the definition of foldr, but this time put foldl just below it. Again, the snippet is taken from the Haskell Wiki article on folding, but I've added the function types for clarity.

-- if the list is empty, the result is the initial value z; else
-- apply f to the first element and the result of folding the rest
foldr :: (a -> b -> b) -> b -> [a] -> b
foldr f z []     = z
foldr f z (x:xs) = f x (foldr f z xs)

-- if the list is empty, the result is the initial value; else
-- we recurse immediately, making the new initial value the result
-- of combining the old initial value with the first element.
foldl :: (b -> a -> b) -> b -> [a] -> b
foldl f z []     = z
foldl f z (x:xs) = foldl f (f z x) xs

These two functions are nearly identical, but they flip the order of the parameters of the first function and they behave quite differently. Notice how foldr applies f to x and then to the result of folding the rest of the list, while foldl immediately recurses before applying f to z and x. This may seem unimportant, but it has big ramifications in a lazy language like Haskell.

With foldr, if you have a function that lazily evaluates its arguments, you could potentially short-circuit the recursion long before reaching the end of the list, but with foldl, because the function recurses immediately, you force the evaluation of the entire list. This means that if you have an infinite list, foldl will never finish, while foldr might.

Let's see an example. If we want to use foldr to find whether any number in an infinite list is even, we could do it like this:

foldr (\x b -> even x || b) False [ 1 .. ]
-- returns True

Even if the list is infinite, the function we pass to foldr doesn't evaluate its second parameter (b) if even x evaluates to True. In the example above, the evaluation would work a little something like this:

-- the list is not empty, so apply f to the head of the list
-- and set up potential recursion
even 1 || foldr (\x b -> even x || b) False [ 2 .. ]
-- even 1 is False, so we take another step and recurse.

even 2 || foldr (\x b -> even x || b) False [ 3 .. ]
-- even 2 is True, so we don't need to evaluate anymore.
-- we can happily return the result with only two steps.

In this example, the function can exit after only two steps through the list because our function (\x b -> even x || b) doesn't evaluate b unless even x is False. If we look at the same example with foldl, however, the result is quite different:

-- note the flipped parameter order for f
foldl (\b x -> even x || b) False [ 1 .. ]
-- ... ... ... ...
-- never terminates

Because foldl recurses unconditionally (foldl f (f z x) xs), the function will start to evaluate, but because it needs to get to the end of the list before it can evaluate the function and because the list is infinite, the function can never terminate.

foldr, on the other hand, doesn't recurse immediately, and flips back and forth between evaluating f and taking another step. This means that, because of lazy evaluation, foldr can stop recursing through the list after any number iterations, depending on the function.

Let's relate back to what we talked about last time and talk about spines. Folding happens in two stages: traversal and folding. Traversal is the stage in which the fold recurses over the spine. Folding is the evaluation of the folding function applied to the values. All folds recurse over the spine in the same direction. However, the difference lies in the association of the folding function and direction the folding proceeds. This sounds tricky, but we can demonstrate it by parenthesizing:

foldr (+) 0 [ 1 .. 5 ]
-- evaluates as the following
-- (1+(2+(3+(4+(5+0)))))

foldl (+) 0 [ 1 .. 5 ]
-- evaluates as the following
-- (((((0+1)+2)+3)+4)+5)

Notice how the order in which the expressions are evaluated is reversed, as is the end at which the zero element is applied. If we use a non-commutative operation instead of summing, we'll see the last point more clearly:

foldr (++) "0" ["a", "b", "c"]
-- evaluates to "abc0"

foldl (++) "0" ["a", "b", "c"]
-- evaluates to "0abc"

Notice how the 0 ends up at opposite ends of the result because of the evaluation order.

Folding summary

In short, the book recommends to always use foldr over foldl and provides a helpful summary:

foldr ::

The recursive invocation of foldr is an argument to the folding function. It doesn't directly self-call as a tail call the way foldl does. In a way, you can think of foldr as alternating between applications of the folding function f and foldr, where the next invocation of foldr is dependent on f asking for it.
Is right-associative.
Works with infinite lists.
Is a good default choice for transforming data structures.

foldl ::

Self-calls (tail call) through the list, starts to produce values after reaching the very end.
Is left-associative.
Cannot be used with infinite lists.
Is 'nearly useless' and should almost always be replaced with foldl'~. We haven't talked about ~foldl'~, but it works the same as ~foldl except it is strict, meaning it evaluates the values inside the cons-cells as it traverses the list, rather than leaving them as unevaluated expressions. We'll talk more about this when we get to the chapter on efficient code.

Definitions

Fold: A higher-order function that takes an accumulating function and a recursive data structure and returns the value that results from applying the function to the elements in the structure.
Catamorphism: A generalization of folds to arbitrary data types. A morphism is a transformation, and cata means 'down' or 'against', so a catamorphism is a transformation to a 'lower' form.
Tail call: A tail call is the final result of a function, i.e. the last function that gets called in a function. In the below example---assuming g and h are functions---~h~ is the tail call.

f x y = h (g x y)

Tail recursion: a function whose tail calls are recursive invocations of itself. In our definitions of foldr and foldl, we see that foldl is tail recursive, while foldr is not, as the latter alternates between calling f and foldr.

Summary

In addition to going deep on folding, this chapter also touches briefly on the scan functions scanl and scanr. In the interest of time I won't cover them here (unless we need them later), but if you're interested you can go check out Hackage for more information.

Next chapter is on Algebraic Data Types, and it's a big one, so stay tuned.

Let's read: Haskell Programming from First Principles, pt IX

Mon, 06 Jan 2020 22:55:25 GMT

Ah, another data type deep dive! This time we're looking at lists and how they are created and evaluated. Not much more of an introduction needed here; let's just dive right in!

List creation

There's a number of ways to construct lists in Haskell. This chapter covers using ranges and list comprehensions, both of which are quite neat and something I get deep personal satisfaction out of (hey, I'm not weird!).

We covered ranges when looking at the Enum typeclass in Chapter 6, so let's just do a quick refresher: In Haskell you can create lists out of anything that implements the Enum typeclass (such as Int and Char and Bool) by using the syntactic range sugar:

-- expression and what they evaluate to
[ 1 .. 5 ] -- [1, 2, 3, 4, 5]
[ 1 .. ] -- an infinite list of integers, each entry increasing by 1
[ 1, 3 .. ] -- an infinite list of integers, each entry increasing by 2
[ 1, 5 .. 10 ] -- [1, 5, 9] (step size is 5-1 = 4)

Neat, huh? Just keep in mind that the range must be in increasing order. If you try something like [5 .. 1], you'll get an empty list.

In addition to ranges, there's also this thing called list comprehension. List comprehensions are something I normally associate with Python, but I think Haskell's version is much cooler. The syntax looks a lot like set comprehension does in mathematics (which is also where the concept comes from). A list comprehension requires at least one list (called a generator) that provides an input for the comprehension. Here's a simple comprehension:

[ x^2 | x <- [1 .. 5] ] -- [1, 4, 9, 16, 25]

The first bit before the pipe, x^2, is the expression that goes into every cell. The variable x is assigned from the generator [1 .. 5]. Indeed, what this list comprehension does is simply to square each number of the generator. The same result could be achieved by a simple map: map (^2) [1 .. 5].

Where list comprehensions really shine, however, is when you start to add more variables and optional filters. When adding more variables, the comprehension will create a list with every possible permutation of those values. For instance: say you want to create a list of all the positions in a 3x3 grid around an origin ((-1, -1) to (1, 1)). Using list comprehensions, that's trivial:

[ (x, y) | x <- [-1 .. 1], y <- [-1 .. 1] ]

In addition to variables, you can also add filters after the variables. Say you want the same list as above, but you don't want to include the origin ((0, 0)):

[ (x, y) | x <- [-1 .. 1], y <- [-1 .. 1], (x, y) /= (0, 0) ]

And not just one filter; you can add multiple! What if we want a bigger list, but only want the tiles where the Manhattan distance to the origin is divisible by 7 (and excluding the origin itself)?

[ (x, y) | x <- [-4 .. 4], y <- [-4 .. 4], (x, y) /= (0, 0), (abs x + abs y) `mod` 7 == 0 ]

It may be a farfetched example, but I think it adequately shows just how cool list comprehensions can be in Haskell.

Spines and non-strict evaluation

To talk about list evaluation, we need to talk about how lists are represented. To talk about how lists are represented, we need to talk about spines. But let's take a step back. It's not all that scary.

The list data type is defined as such:

data [] a = [] | a : [a]

It's a recursive data structure. It's either an empty list, or an element prepended to another list. Syntactically, when we write lists or print them, they're usually presented like this: [1, 2, 3], but can also be thought of as 1 : 2 : 3 : []---that is 1 before 2 before 3 before the empty list.

The : operator is known as cons. The list construction in the last paragraph can be thought of as a series of cons cells (a value consed onto a list). It could also be written like this, which may make the relation to the list data constructor clearer: 1 : (2 : (3 : [])).

One problem with this representation, though, is that it makes it seem like 1 exists 'outside' of the list, when it is actually contained by it (and by its cons cell). Because of how lists are evaluated in Haskell, we'll see that because the value is contained, we can evaluate cons cells without evaluating their contents.

Let's talk about spines. You see, the list above can also be represented---conceptually---like this:

:
/ \
1   :
 / \
2   :
   / \
  3  []

In this representation, the cons operators line up to form what is known as the spine, which is a fancy word for the connective structure that makes up a data structure such as a list or a tree. In the case of lists, it's just a series of cons operators.

Crucially, spines are evaluated independently of values. What does this mean? Well, functions that only require evaluating the spine and not the values, such as length, can walk down a list without evaluating the contents. So, returning to our list from before, if we were to apply the length function on it (and the values hadn't already been evaluated), you could think of it like this, where the underscores represent values that haven't been evaluated.

:
/ \
_   :
 / \
_   :
   / \
  _  []

Another way to get this point across is to look at what happens when we add bottom values to a list. Run this in the REPL to try it out.

let xs = [1, undefined, 3]
length xs -- 3
take 1 xs -- [1]
take 2 xs -- Exception: Prelude.undefined

Taking the length of the list works just fine because we don't need to evaluate the items in the list. Taking the first element is no problem because it's a regular value and the rest of the list has not been evaluated. When we take two values, however, this forces the evaluation of the second item in the list, undefined, which causes the program to crash.

There is loads more to be said about list evaluation and the impacts it has on what we can and can't do in code, but these are the most important things covered in this chapter.

Chapter definitions

Product type ::

A product type is a type that contains several other types at the same time. The canonical example is a tuple: You can have a tuple of a Bool and an Int, and for every instance you have of that type, you will have both a Bool and an Int. The reason its called a product type is that the number of possible permutations is the product of the number of possible values for each type.

Say you have a data type: data Direction = Up | Down | Left | Right. If you have a (Bool, Direction) tuple, you now have $8$ ($4 \cdot 2$) different possible permutations of that tuple.

Sum type ::

A sum type is a type that is only one of a set number of types at any time. Relating this back to the example above with Bool and Int: If you have a data type that is either a Bool or an Int (such as Either Bool Int), then that's a sum type. The number of possible permutations is the sum of the number of possible values for each type.

Using the Direction type from above, Either Bool Direction has $6$ ($4 + 2$) possible values.

Cons ::

The : operator. Used to prepend a value onto a list. Can also be used as a verb (e.g. consing a value onto a list).

Cons cell ::

A data constructor and a product of the types a and [a] as defined in the list data type definition.

Spine ::

A structure that strings a collection of values together, the scaffolding of a data structure. For the list datatype, it's formed by the recursive nesting of cons cells.

Goodbye 2019, Hello 2020

Mon, 30 Dec 2019 18:47:30 GMT

It's the end of the year and there's round-ups and predictions everywhere, so I think it's an appropriate time to take a step back and look at what I've achieved this year and what I have in mind for next year.

2019

Let's move in a chronological fashion. In 2019, my list of goals was just a slightly-longer-than-usual Slack message to my friends. They were pretty vague, but still hold some value as a set of overarching goals.

Goals

Containers ::

Before 2019, I'd hardly ever touched a container. I'd heard lots about them on various podcasts and was really intrigued. I wanted to 'really familiarize myself with the whole thing' (how's that for a vague goal?).

A year later, do I think I've succeeded? Yes. It's stated in a very unclear fashion and can be taken to mean a lot of things, but since then, I've been working pretty closely with containers and Kubernetes pretty much the entire time. Yes, there's still some things that are unclear to me, but overall, I think I understand containers well enough to have cleared that goal.

NixOS ::

I installed NixOS on my personal computer pretty much exactly a year ago. It was (and still is) my first real foray into the Linux world. I'd used Ubuntu for some university assignments previously, but that's it. It took me a good few weeks to get my external monitors up and working, but these days, most everything is running pretty smoothly (except for that NVidia GPU I've got ...).

I said that I'd 'just got it set up over Christmas, but there is still a lot left to be done and a lot of things to discover.' A year later, I'm fairly comfortable in NixOS, I use Nix for a lot of work related stuff, and I love it. I'd say I've finished the NixOS setup and discovered much, though there is still a lot left to be done. What I have achieved is: I have a stable operating system that works as I expect and with a window manager / desktop environment that I enjoy. I'm happy with the OS state of things for now. Achieved.

Give a talk ::

Said at the time: 'This is something I've been wanting to do for a while, and I think this year is the right time, once I'm out of uni and have a bit more time (and experience) on my hands.'

This is probably the one goal I set for myself that is the easiest to measure. I spoke internally at my company about Algebraic Data Types, I held a workshop (still counts) on re-implementing some basic data types and creating opaque types at Oslo Elm Meetup, and I held a talk about the new async/.await features in Rust at Rust Oslo. Done.

Non-goals

In addition to the above goals, I also achieved a few other things that I'd like to highlight in this post. These are items that I didn't put on my to-do list, but that I consider significant enough to mention.

Kubernetes/OpenShift ::

I started working in a team where OpenShift is an important part of the workflow, and interacting with it became a natural part of my day-to-day work.

Started blagging ::

Took me a while, but I finally got going. It's been a lot of work, but very rewarding.

Became a co-organizer of the Rust Oslo Meetup group ::

I'm excited to see what we can do to further engage the Rust community in the vicinity and what we'll be doing in the coming year.

2020

Looking forward to 2020, what do I want to achieve? What skills do I want to develop? After some thought, I've come to the conclusion that my 'theme' for 2020's goals is mastery. I've found tools that I really like and that align with my way of thinking. However, I don't feel like I've fully mastered any of them just yet. Most of my goals will revolve around improving my skill with tools that I already know and love.

Goals

Here's a loose definition of my goals for the coming year.

General goals

These are the general goals that I'm setting for myself. These are the ones I will actively be pursuing and working towards.

More infrastructure work ::

Building on the 'containers' goal of this past year, I want to keep working on infrastructure: Kubernetes, CI/CD pipelines, containers, the whole deal. I went through a Red Hat training course this year, working towards a certification as a 'Specialist in OpenShift Application Development'. I failed the exam, but learned a lot in the process. Let's see if I can use that to pass! Apart from that one specific goal, this post is more about making myself familiar with the tools and getting really comfortable with them. You know, mastering them.

More Nix: It's been a year of NixOS, and it's been great. Now I want to get really comfortable with the difference between build.nix, default.nix, shell.nix, how nix-build works, etc. Specifically, I'd like to be comfortable with setting up a Haskell dev env with Nix and Cabal.
More Emacs: Emacs has been my go-to editor for a year and a half now, and I still get stuck in weird states sometimes. This coming year, I want to get more comfortable with it on a deeper level. Step one: Read the built-in Emacs manual.
More community engagement: I gave some talks and workshops, both internally and at local meetups this past year. I want to do more of that in 2020. Also: apply for more conference talks. I applied to a few this past year, but didn't get any of them.
Programming Language Theory / Compilers: I've had an interest in type systems and PLT for a while. While writing this post, I realized I should do something about it and ordered Types and Programming Languages by Benjamin C. Pierce. I don't know enough about it to set any clear goals just yet, however.

Languages

While I'm not going into this coming year with any clear language goals in mind, I'd like this to serve as a recording of what languages have me the most curious at the moment. I'm not making any clear commitments to learning any of these, but I think it's important to keep an open mind and keep exploring new languages and new concepts. As Scott Wlaschin talks about in his talk [[https://fsharpforfunandprofit.com/fourfromforty/][Four Languages from Forty Years Ago]], you should look for languages that expand your mind and help you think about problems in a new way. He awards languages that achieve this a 'Galaxy Brain Seal of Approval'. There is an Alan Perlis quote in the slides saying that 'A language that doesn't affect the way you think about programming, is not worth knowing'.

Haskell: Yeah, I've been doing a bit of this, but not enough. I've got the basics down, but I need to put it into practice. I want to build something; a simple Web API or command line interface should be enough.
Lisp: Doesn't matter whether it's common lisp, scheme, (typed) racket, or clojure: I want to get comfortable with lisp. If nothing else, at least it'll help me configure Emacs.
Elixir: This is the wildcard on the list. I have not really looked much at it at all, but I hear great things. Should offer some nice Galaxy Brain moments.

SMART goals

The goals listed above are generally quite vague, and it'd be hard to say for sure whether I've achieved them or not. To make them more actionable, let's create a set of SMART goals that I can tick. What are SMART goals? Of course, Wikipedia has an article on it, but the short version is that SMART is an acronym that stands for 'Specific, Measurable, Achievable, Relevant, Time-bound'. Organized by topic, this is my list:

Programming Language Theory ::

I don't have enough insight into this to make any clear goals for it, but I've had a brief look at the Types and Programming Languages book. It's about 650 pages long. I'd say I should get through it by July at latest. In short: Read Types and Programming Languages by July.

Community engagement ::

Give at least three talks/workshops at community events.
Apply as a speaker to at least three conferences.

Emacs ::

By February 1^st, I want to have gotten through the Emacs manual. At this point, I may decide to try and (at least start to) configure Emacs from the ground up (rather than relying on Spacemacs), but that is not a requirement.

Nix ::

Write about what the various .nix files are and what their intended use is. Should discuss at least build.nix, default.nix, and shell.nix.

OpenShift ::

Pass the Red Hat Certified Specialist in OpenShift Application Development exam.

Containers / Infrastructure / Haskell / Nix ::

This is a set of goals that apply to broad range of topics. I want to manage a Kubernetes cluster somewhere and create a Haskell API that serves requests. The Haskell project should use Nix for as much as practically possible. What sort of data the API serves doesn't matter.

Deadlines:

April 1^st: Have Kubernetes cluster up and running.
June 1^st: Have the Haskell API up and running.

And that's what I'm planning to look into in the coming year. As always: plans change, life happens, and things inevitably don't go as expected, but it's nice to have a little something to look back on, at least. What do you think? Do you have any clear goals?

Let's read: Haskell Programming from First Principles, pt VIII

Mon, 23 Dec 2019 08:37:31 GMT

Taking a step back from a pretty dense chapter on more functional patterns, this time we're looking specifically at one thing: recursion. Chapter 8 of the book covers this in pretty good detail, giving a fairly wide range of examples, but I'll be trying to distill it down to its essence in this post. Let's start at the very beginning, why don't we?

What is recursion?

A recursive function is---in its simplest terms---a function which calls itself. As the book describes it: "recursion is defining a function in terms of itself via self-referential expressions." This means that until some condition is met, the function will keep calling itself. If this condition is never met, the function never terminates. Recursion is an important concept in Haskell, because it gives us a way to express indefinite computations.

Base cases

To reiterate: given that it terminates, a recursive function is a function that will keep calling itself until some condition to stop recursing is met. This condition is often known as the base case. For instance, let's look at writing a factorial function, which is a function that multiplies all positive integers (${1..n}$) up to and including the number we're calculating for. In maths, it's commonly written using an exclamation point, such as $4!$.

So we're all on the same page about what the function should do, $4!$ is evaluated as $4 \cdot 3 \cdot 2 \cdot 1$, which is equal to $24$.

To make this a recursive function, we know that for any number $n$ greater than one, it is the result of the factorial of $n-1$ multiplied by $n$. In other words, for $n$ greater than one, $n! = n \cdot (n-1)!$. If $n$ is $1$, then $n! = 1$. In our case, the function is not defined for integers less than one.

Based on the little analysis above, we now know that the value of $n!$ is recursively dependent on the value of $(n-1)!$ and so on. We have also identified our base case: $1$. So how would we go about writing this out? How about something like this:

factorial :: Integral a => a -> a
factorial 1 = 1
factorial n = n * factorial (n - 1)

Note that this function does not terminate if applied to a value less than $1$. This is an issue that we'll come back to shortly.

Recursion as indefinite function composition

One thing that the book mentions that I found quite interesting, is that you can think of recursion as indefinite function composition. When composing functions, all we do is pipe the output of one function into another function. Recursion is the same thing, except it's always the same function. We can use this to create functions that will get applied an arbitrary number of times. Indeed, taking it a step further, we can write a function that takes a function, an argument, and the number of times to apply it, and then applies the function the set number of times:

applyNTimes :: (Integral a) => a -> (b -> b) -> b -> b
applyNTimes 0 _ b = b
applyNTimes n f b = f . applyNTimes (n - 1) f $ b

Using this, we can create an increment function:

increment :: (Integral a, Num b) => a -> b -> b
increment times = applyNTimes times (+1)
-- increment 5 2 = 7

A contrived example, but it's a pretty cool effect!

Bottom

I mentioned when talking about the factorial function that it doesn't terminate for every possible value we can apply it to. A function that never terminates is one of the ways we can play with bottoms in Haskell (and no, it's not a kinky as it sounds!).

In Haskell, bottom (symbolically: ⊥) is used to refer to computations that do not result in a value. The two main varieties of bottom are computations that fail with an error (partial functions) and computations that fail to terminate (such as with infinite recursion and the factorial example above). When you can, stay away from bottom.

Definitons

This chapter only comes with a single definition: recursion. The funny thing would be to tell you to start from the top, but let's be boring and do it the proper way:

Recursion: Recursion is a means of computing results that may require an indefinite amount of work to obtain through the use of repeated function application. A recursive function will usually have at least one case that calls itself and a base case that stops the recursion.

Let's read: Haskell Programming from First Principles, pt VIIc

Mon, 16 Dec 2019 22:57:22 GMT

Hey, and welcome back to the third and final part of chapter 7 of Haskell Programming From First Principles! Today we'll be looking at function composition and pointfree style; two related topics that are very commonly seen in Haskell.

Function composition

Let's start of with function composition. Function composition allows us to create new functions by combining existing ones. The one criterion for composing two functions is that the return value (range) of the first function matches the input value (domain) of the second function. In Haskell, we use the dot operator (.) to compose two functions.

Let's inspect the type signature of the . operator and see if we can unpack it:

(.) :: (b -> c) -> (a -> b) -> a -> c

At first, this may seem a bit complicated, so let's break it down. The . operator goes between two functions (like this: f . g) and returns a function that goes from a to c. What it does is to simply 'glue' the two provided functions together.

I quite like Scott Wlaschin's explanation of it from his talk The Power of Composition (this link takes you to the point in the talk that talks about gluing together functions, but I highly recommend checking out the whole thing if you're interested), where he talks about gluing together a transformation from an apple to a banana, and a function from a banana to a cherry to create a function from an apple to a cherry.

The simplest way to explain composition may be to say that you perform an operation on a value, and then pass the result of that operation to the next function. That new function which puts those two together is the composed function. You might see an example like this:

(f . g) x = f (g x)

This tells us that applying f composed with g to x is the same as applying g to the argument x and then applying f to the result of that. Note that the . operator might initially appear to read backwards: It's the last function to get run that gets written out first. This is because of it's mathematical roots, where people use the '∘' symbol for composing functions (so our f . g would be $f ∘ g$). I suggest reading the symbol as 'after', which makes it 'f after g', for instance.

If this is all a bit abstract and jargon-y, why don't we look at a concrete example. Say we want a function that tells us whether a string is of even length or not. In that case, we can express the full function as a composition of two smaller functions (the signatures have been simplified/specialized for the sake of example; check the REPL for more): even :: Int -> Bool, and length :: String -> Int (so it'd be 'even after length'). Notice that the return type of length matches the input type of even, and that the type of isOfEvenLength matches the combination of the functions.

isOfEvenLength :: String -> Bool
isOfEvenLength = even . length

The astute reader (that's you!) might notice that the function we defined as the composition of the other two functions (isOfEvenLength) doesn't list any arguments in its definition. This might look strange but it takes us nicely into our next point: pointfree style.

Pointfree style

Pointfree style is a way of writing functions without specifying their arguments, most notably when working with composition. It's called 'pointfree' because the arguments are also known as 'points'. There are arguments both for and against writing pointfree style, and it's true that excessive use may cause code to become harder to understand, but used sensibly it can make code tidier, cleaner, and let's us put the focus on the transformations, the functions, rather than the data supplied to them.

Using one of the examples from before, we now get:

(f . g) x = f (g x)
-- the above now becomes
f . g = \x -> f (g x)

-- similarly, we can add more functions
f . g . h = \x -> f (g (h x))

How does this work? Well, it all goes back to the lambda calculus and currying. Let's take a step back and look at how you could write an alias for a function in Haskell:

f a b = -- ... implementation

g = f

In the above code, we have defined g as simply being the same as f. We'll still need to provide it with the same arguments (a and b), but it's just given a different name. If we want to, we can choose to define it as a specialized version of f, with the first argument already applied:

f a b = -- ... implementation

g = f 2

In this case, g is a partially applied version of f and is a function from one argument to a result (whatever f returns). We could also write the above as g x = f 2 x, but because of how partial application works, there's no need to do this. We've already said that g is the same as f applied to 2, and that naturally returns a function from one argument to the result.

Similarly, we might do it for something like map. Say we want a function that doubles all values in a list:

doubleVals = map (*2)
-- is the same as
doubleVals xs = map (*2) xs

The two definitions above are equivalent, they're just written out differently.

Whether you prefer pointfree or 'pointful' styles is an individual thing, and is probably based on experience and circumstance. As mentioned earlier, too much and it makes your code hard to read, too little and you're writing out way more than what you need to and taking focus away from the important parts. Like with so many things, try and apply the Goldilocks principle. As always, the Haskell wiki has more detailed information on the topic, so go give that a read if you fancy.

Definitions

As it's the last post for this chapter, let's go over the chapter definitions.

Anonymous function: Often known as a lambda. It is a function that isn't bound to an identifier and is instead just used as an argument to another function or the like. For instance, \x -> x is an anonymous version of the id function (id x = x).
Currying: The act of transforming a function which takes multiple arguments to a series of nested functions that each take a single argument and return the next function in line (or the result if we're at the end). In Haskell, every function is curried, and this is baked into the language so you don't have to worry about it.
Pattern matching: A way to deconstruct various data types to do something with---or based on---their contents.
Bottom: Bottom is a non-value used to denote that a program cannot return a value or a result. A simple example would be an infinitely looping function, but values that don't handle all their inputs and throw errors also apply here. We'll talk more about Bottom in the chapter on non-strictness.
Higher-order functions: Functions that take other functions as arguments or return functions themselves.
Composition: A way of gluing together multiple functions such that each each function is applied to the result of the next function. Creates a pipeline of transformations.
Pointfree: Also known as 'tacit programming'. Programming without mentioning arguments explicitly.

Let's read: Haskell Programming from First Principles, pt VIIb

Mon, 09 Dec 2019 10:30:01 GMT

Continuing our foray into the More Functional Patterns chapter, today we're looking at higher-order functions and guards.

Higher-order functions

First off, let's establish some common ground and agree on the definition. According to Wikipedia and the Haskell wiki, a higher-order function is a function that takes other functions as arguments or returns a function as a result. In essence, it's a function that operates on functions as values. Because functions are just any other value in haskell^[1], this comes very naturally to the language.

So to qualify as a higher-order function, a function must fulfill one of two criteria:

Accept a function as a parameter ::

map and sorting functions are great examples of this. For instance, map's signature is map :: (a -> b) -> [a] -> [b], where the first parameter is the function used to transform the values of type a in the list to values of type b. Sorting functions could take a comparison function (a -> a -> Ordering) and a list ([a]), sort the list for you and return the sorted list.

Return a function ::

This probably happens more than you realize. Any function that accepts more than one argument in Haskell is by default a higher-order function due to how the lambda calculus works. That said, you can also explicitly return a partially applied function based on the input arguments. For instance, this example takes a boolean value saying whether the function should return a partially applied multiplication or addition, and the first argument to apply to the calculation. It returns the partially applied function:

f :: Num a => Bool -> a -> (a -> a)
f multiply n =
  if multiply then
    (n*)
  else
    (n+)

It's a silly example, but it gets the point across.

Guards

Next up, the book returns to a form of pattern matching by introducing guards, which allow us to run code conditionally based on the truth of a statement. If you think that sounds a lot like an if-then-else expression, that's because it works pretty much the same but with some different syntax. We'll run through a couple of different examples to examine the various features that are introduced.

The basic syntax looks like this:

myAbs :: Integer -> Integer
myAbs x
  | x < 0     = (-x)
  | otherwise =   x

This function has two guards, each beginning with a pipe symbol (|). Note that we don't need an equals sign after the arguments in the first line of the definition; instead, the symbol comes after each respective guard. The first of the guards that evaluates to True will be executed.

In this function, we have two cases: One for when the value of x is negative (in which case we'll return the absolute value of x), and one for every other case. In the above example, we have used the word otherwise, which is a synonym for True, as a catch-all for every other case.

If we want to explicitly enumerate all options, we can do that too:

myAbs :: Integer -> Integer
myAbs x
  | x <  0 = (-x)
  | x >= 0 =   x

This will work the exact same as the previous version, but in this case we're handling all cases explicitly. When explicitly listing all options, make sure you have enabled warnings (or errors) for non-exhaustive patterns to avoid accidentally partial functions.

What if we want to share some values between the different guards? We can use where-statements for that! Imagine a function that takes the lengths of the two legs (catheti) of a right triangle and returns whether the triangle is big, small, or medium based on the length of the hypotenuse:

triangleSize :: (Floating a, Ord a) => a -> a -> String
triangleSize c1 c2
  | h >= 5 = "Big triangle"
  | h >= 2 = "Medium triangle"
  | h <  2 = "Small triangle"
  where h = sqrt (c1^2 + c2^2)

When declaring variables like this, they're in scope for all of the guards.

This last example also demonstrates the importance of order on the guards. Because only the first guard that evaluates to True is executed, the above function works as expected. If we switched the two guards for big and medium triangles, no triangle would ever be considered 'big'. What a sad world that would be.

Next up

That was all we had time for today, my dear reader. We still have a little bit left of this chapter: function composition and pointfree style. Two very interesting topics which I'm looking forward to covering next time. Until then: take care!

Footnotes

^[1] You may have heard the term 'first-class functions'. This is what that means. You can pass functions around like any other variable, store them in data structures, assign them to variables, and so forth. See MDN or Wikipedia for more.

Let's read: Haskell Programming from First Principles, pt VIIa

Mon, 02 Dec 2019 22:39:31 GMT

We've reached chapter 7: More Functional Patterns. This is a pretty big chapter that covers a lot of ground, so to make it more digestible to you (and more manageable to me), I'm going to break this chapter up into multiple pieces. In this post, we'll be looking at pattern matching and case expressions. Later posts will cover higher-order functions, function composition, and pointfree style, so there's lots to look forward to!

Pattern matching

"Pattern matching is an integral and ubiquitous feature of Haskell". Thus opens the sub-chapter on pattern matching. If you've seen any amount of Haskell code, you've probably come across it, but in case you haven't (and if you have: just to make sure we're on the same page), here's a primer.

Pattern matching is a way for us to match values against certain 'patterns'. Depending on the context, a pattern can be a wide range of things, including specific strings, numeric literals and list syntax; pattern matching can match on any and all data constructors. Pattern matching can even let us match on the inner structure of the thing we're matching on, such as a list or a tuple. Let's have some examples.

We can check whether a provided Integer is of a specific value, like in the following case:

isTheAnswer :: Integer -> Bool
isTheAnswer 42 = True
isTheAnswer _ = False

We can check whether a list contains zero, one, or many elements:

listState :: [a] -> String
listState [] = "The list is empty"
listState [_] = "The list has one element"
listState (_:_:_) = "The list has at least two elements"

We can also check which data constructor has been used to create a value and extract the constructor parameters from it or even check them for specific values:

data User = LoggedIn String | Anonymous

userInfo :: User -> String
userInfo Anonymous = "The user is anonymous"
userInfo (LoggedIn "admin") = "The user is an administrator"
userInfo (LoggedIn username) = "The user is " ++ username

This covers some of the patterns you might see used with pattern matching. Keep in mind that the order of the patterns matter: they are evaluated from top to bottom, and once a pattern matches, no more cases will get checked.

Covering all the cases

When pattern matching, you should always handle all cases to avoid partial functions. This might seem like a hassle, but you don't need to handle all the cases explicitly. As seen in the first example (isTheAnswer), the _ pattern works as a catch-all, meaning that if no other patterns have matched yet, this will match anything.

In addition to just being vigilant about matching all possible combinations, you can also turn up the compiler's crankiness by using the -Wall flag. This will give you warnings if your patterns are non-exhaustive.

Case expressions (or: more pattern matching)

Similarly to the pattern matching above, we also have case expressions in Haskell. They work the same way as basic pattern matching, but the syntax is a bit different. In short, it gives you all the power that you get from pattern matching, but with a bit more syntax. Let's rewrite the listState function above using a case expression to see the similarities:

listState :: [a] -> String
listState xs =
  case xs of
    [] -> "The list is empty"
    [_] -> "The list has one element"
    (_:_:_) -> "The list has at least two elements"

This looks very much like pattern matching (and serves the same purpose in this case), except that we use the case ... of structure to match on a named variable instead of directly on an input. Which is more appropriate depends on your use case and preferences.

And that's it for today, kids. A shorter, more digestible format that covers only the most essential. I'll cover the rest of the chapter in this way, and then we'll see what works best moving forward. Next time: higher-order functions! Until then: take care!

Async Rust: async main

Mon, 25 Nov 2019 21:17:59 GMT

In the previous post we had an introductory glance at async/.await in Rust, looking at what it does and how you'd use it. However, the little bit of code we were left with at the end still felt a bit rough. Coming from languages where you can have an async main function (or at least one that lets you act as if it is), it felt like an extra hurdle to have to extract all async functionality out into a separate function.

Luckily, Reddit user mbuesing pointed out that there is an attribute available in async-std that you can use to make your main function asynchronous: async_std::main! Let's have a look at what changes we'd have to make to incorporate that.

First off, let's update the Cargo.toml file. It's mostly the same as last time, but we're going to have to add the "attributes" feature from async-std:

[package]
name = "async-basics"
version = "0.1.0"
authors = ["Your Name <your.email@provider.tld>"]
edition = "2018"

[dependencies]
async-std = { version = "1", features = ["attributes"] }
surf = "1"

Next up, let's update the main.rs file. If we're doing everything within the main function, we can cut it down to about 8 lines of code, compared to the 14 lines we had last time, and because it's such a simple program, it's not any less readable:

use surf;

#[async_std::main]
async fn main() {
    match surf::get("https://pokeapi.co/api/v2/move/surf").recv_string().await {
        Ok(s) => println!("Fetched results: {:#?}", s),
        Err(e) => println!("Got an error: {:?}", e),
    };
}

So there you have it. With some extra attributes, we can make async code pretty ergonomic in Rust. Now, let's make something cool!

Async Rust

Mon, 18 Nov 2019 08:21:10 GMT

Hot off the heels of RustFest Barcelona and the stabilization of async/.await, I think it's safe to say that one of Rust's most anticipated language features has finally landed. And for that occasion (and because I've had some trouble understanding certain bits of it myself), I wanted to write a little introduction to asynchronous programming in Rust. We'll be creating a super simple application that fetches some data from the internet using the our newfound async abilities. The stabilization of async/.await also coincides nicely with another event I'm excited about: This week saw the release of the most recent entries in the mainline Pokémon games, Pokémon Sword and Shield, and because I'm a bit too busy to pick them up just yet, I'll make do by fetching data from the PokéAPI for now.

I'm assuming some base knowledge of Rust's syntax and ecosystem, but I hope that this is pretty accessible even to people very new to the community.

But before diving into the coding part, let's cover some basic concepts of asynchronous programming and how it might be a bit different in Rust than what you'd expect.

What does `async` mean?

In Rust, when we talk about async, we're talking about running code concurrently, or having multiple overlapping (in time) computations run on a single thread. Multithreading is a related, but distinct concept. Multithreading is ideal for when you've got computationally intensive tasks (so-called CPU-bound tasks) that can be spread across multiple, separated cores. Concurrent programming is better suited for when the task spends a lot of time waiting, such as for a response from a server. These tasks are called IO-bound.

So asynchronous programming lets us run multiple of these IO-bound computations at the same time on a single thread. They can run at the same time because when they're waiting for a response, they're just idle, so we can let the computer keep working on something that isn't waiting. When we reach a point where we need the result of an asynchronous computation, we must .await it. In Rust, values that are 'awaitable' are known as 'futures'.

Rusty weirdness

async in Rust may be a bit different from what you're used to in other languages. Having done asynchronous coding mostly in JavaScript and C#, it certainly was to me. Here's a few key things to understand:

An `async` function does not (necessarily) start executing immediately

To start an asynchronous function, you must either .await it or launch a task using an executor (we'll get to that in a moment). Until this happens, all you have is a Future that has not started. Let's look at an example to make it clearer:

use async_std::task;
// ^ we need this for task spawning

async fn negate_async(n: i32) -> i32 {
    println!("Negating {}", n);
    task::sleep(std::time::Duration::from_secs(5)).await;
    println!("Finished sleeping for {}!", n);
    n * -1
}

async fn f() -> i32 {
    let neg = negate_async(1);
    // ... nothing happens yet
    let neg_task = task::spawn(negate_async(2));
    // ^ this task /is/ started
    task::sleep(std::time::Duration::from_secs(1)).await;
    // we sleep for effect.

    neg.await + neg_task.await
    // ^ this starts the first task `neg`
    // and waits for both tasks to finish
}

So in the above little code snippet, here's what's going on.

The first line imports async_std::task. There's more on this below, but we need an external library to run futures as the standard library does not come with an executor.
The async function negate_async takes as input a signed integer, sleeps for 5 seconds, and returns the negated version of that integer.
The async function f is more interesting:

The first line (let neg ...) creates a Future of the negate_async function and assigns it to the neg variable. Importantly, it does /not/ start executing yet.
The next line of code (let neg_task ...) uses the task::spawn function to start executing the Future returned by negate_async. Like with neg, the Future returned by negate_async is assigned to the neg_task variable.
Next: we sleep for a second. This is so that it will be obvious from the output when a task starts running.
Finally, we await both futures, add them together, and return them. By awaiting neg, we start executing the Future and run it to completion. Since neg_task has already been started, we just wait for it to finish.

So what's the result of this, then?

Negating 2
# <- there's a 1 second pause here
Negating 1
Finished sleeping for 2!
Finished sleeping for 1!

As we can see, the second future, neg_task, started executing as soon as it was called---thanks to task::spawn---while neg did not start executing until it was awaited.

You need an external library to use `async/.await`

As was briefly alluded to above, you need to reach for an external library to do asynchronous programming in Rust. This took me a while to understand, as I'm used to it being part of the language experience. In Rust, however, *you need a dedicated executor/*[fn:1]. The executor is what takes care of /executing the futures, polling them and returning the results when they're done. The standard library does not come with an executor, so we need to reach out to an external crate for this. There are a few ones to choose from, but the two most prominent ones are ~async-std~ (which we're using here) and ~tokio~.

A minimal async example!

Alright, let's get practical. This is the reason that I'm writing this post. As mentioned at the start, we'll be creating a super simple application that fetches some Pokémon data and prints it to the console. For preparation, make sure you've got at least version 1.39 of Rust and cargo available.

Step 1: creating the application

Let's create a new application! Simply run this command in your preferred directory:

cargo new async-basics

Step 2: Dependencies

We're going to be using ~async-std~ for spawning tasks, and ~surf~ to fetch data from the API. Let's add them to the Cargo.toml file. Your whole file should look something like this:

[package]
name = "async-basics"
version = "0.1.0"
authors = ["Your Name <your.email@provider.tld>"]
edition = "2018"

[dependencies]
async-std = "1"
surf = "1"

Nice! This is going swimmingly!

Step 3: Fetch data

Okay, final step. Let's modify the main.rs file. We'll make it as simple as possible. Here's what we want to use:

use async_std::task;
use surf;

// fetch data from a url and return the results as a string.
// if an error occurs, return the error.
async fn fetch(url: &str) -> Result<String, surf::Exception> {
    surf::get(url).recv_string().await
}

// execute the fetch function and print the results
async fn execute() {
    match fetch("https://pokeapi.co/api/v2/move/surf").await {
        Ok(s) => println!("Fetched results: {:#?}", s),
        Err(e) => println!("Got an error: {:?}", e),
    };
}

fn main() {
    task::block_on(execute());
    // ^ start the future and wait for it to finish
}

That's all the code you need. In fact, it's more than what you need, as some parts have been broken up for legibility. Let's walk through it!

~use~ statements: Nothing exciting here. Just importing the crates we declared in the Cargo.toml file: surf and async_std.
~fetch~: This is simply a thin wrapper around the surf::get function which returns either the payload as a String or an Exception if something went wrong.
~execute~: This function calls fetch with the endpoint for the move Surf, waits for the result to return, and then matches on the result. If everything went well: print the output. Else: print the error.
~main~: main simply kicks off execute and waits for it to finish. task::block_on is a synchronous counterpart to task::spawn that starts an asynchronous operation, but blocks until it has finished. Because the main function can't itself be async (at least not at the time of writing), we can't use .await in it, but we can block on asynchronous operations.

Step 4: Extend it!

Hey, you made it this far; congrats! That's all I really have in store for you for this one, but if you want to play around a bit more, how about adding ~serde~ and try using surf's recv_json<T> instead? If you'd rather keep looking at async/.await, how about performing multiple requests simultaneously? Or how about making a PokéAPI CLI? (Ooh, that sounds like fun! Hit me up if you're doing this; I want in!)

Parting words and resources

So there you have it, dear reader. I hope you have found this useful. async/.await is finally stabilized and it feels like we've taken a major leap forward. I'm very much looking forward to seeing what happens in the coming months and what the community makes of this.

If you're looking for more resources on async Rust, be sure to check out the Async Book. I also recommend the async-std book for some extra insights.

Until next time: take care!

Footnotes

[1]

See this insightful Reddit comment thread for more on this.

Command Line Control: dirname

Mon, 11 Nov 2019 09:36:23 GMT

When you need to get the directory name of a file but you only have the full path to the file, what do you do? You could use some fancy parameter expansion, sure, but you might run into some weird edge cases. Personally, I'd suggest just using dirname.

If you're familiar with Python and the os library, the behavior is the same as os.path.dirname, but even if you're not, it's still pretty straightforward. Let's look at the man page for the command. It states that given a path, it will output it "with its last non-slash component and trailing slashes removed." If the path contains no path separators, it will output ., which is the current directory.

So in a nutshell, that means it will normalize path/to/file.extension to path/to, and f.txt to .. Because it removes trailing slashes and because it's not dependent on any file extensions, you can use it repeatedly to move further and further up the tree:

some/long/path/to/file.extension
-> some/long/path/to
-> some/long/path
-> some/long
-> some
-> .

So when is this useful? Well, if you've got something in your path and you want to find out what else is in that directory, you could do something like this:

ls $(dirname $(which <cmd>))

Maybe you want to navigate to that directory or open it in some other command? Yeah, you can do that too.

If you're still not convinced that you should use dirname, check out this amazing stack overflow reply about some of the differences between parameter expansion and dirname.

Rust 2020

Mon, 04 Nov 2019 07:11:56 GMT

The end of the year is fast approaching and that, my friends, means that it's time for a deluge of end-of-year Rust blogs. The Rust core team has put out a call for blogs to help decide where to take the language next, as they do every year. I've been quite invested in Rust for a few years now, so I thought it was about time I joined the fray.

Considering that I spend a disproportionately large amount of time thinking and reading about Rust compared to how much I actually use it, do take my thoughts and wishes here with a grain of salt; they may not be what the community needs the most, it's just what I'd want in my ideal, little world.

Stabilize slice patterns (`#![feature(slice_patterns)]`)

This is the one language feature I find myself wanting stabilized more than any other. Coming from languages like Haskell---and to a lesser extent Python and JavaScript---that have simple ways to destructure a list of values, doing this can feel quite clunky in Rust. Especially if all you want to do is get the head and the tail of a Vec or a slice. Yes, the ~split first~ function on slice does do that, but using it is tricky. In particular, how you handle destructured lists using case of statements in Haskell feels really ergonomic and intuitive to me.

Luckily, it seems we're headed in that direction. There is a tracking issue for the slice_patterns feature on GitHub, and save for some remaining compiler issues, it should be about ready to go in. For now, you can check it out on the nightly channel with the feature toggle #![feature(slice_patterns)].

What's more, because rust matches on slices and not linked lists like in Haskell, we get some extra goodies, such as being able to extract not only the first $n$ items, but also the last $m$ items. Here's a little playground link that I put together demonstrating some cool uses for slice patterns. (Note: you probably can do this in Haskell somehow---a little bit of searching tells me the ViewPatterns compiler extension might do the trick---but it's not baked into the base language like this.)

Easy async (documentation!)

I was holding a Rust workshop for some colleagues recently and one of them asked me to create an example of how you'd fetch web data using Rust. I thought this would be a great chance to have a little sneak peek at async/await before it stabilizes and get familiar with it: I figured I'd try and write a super simple application that fetches some data from an endpoint and dumps it to the terminal. However, I could not make it compile, and if it compiled, it didn't run properly (presumably because the future was never started).

What I'd like to see is more information on how to do basic async programming in Rust. The Async Book has a chapter on it, but I still find it confusing with all the different traits and unstable features. I've tried following the example with the Hyper server to make a request, but I keep running into trait bound errors. I'm sure if I sat down and spent some more time on it, it would work, but it feels like it's missing that 'get started in 2 minutes' section.

I'm sure this will improve very quickly over the next few weeks with the stabilization of async/await, but for now it's a hurdle.

Generic Associoted Types (GATs) and const generics

I've lumped these two together because I've not really followed either particularly closely, but reading up on them for the purpose of this post, I'm suddenly quite excited by both. In general I'm all for anything that makes the type system more expressive and allows us to create more compile-time constraints.

GATs ::

Const generics ::

In summary

I already had some ideas in mind, so I thought writing this post would be quick and easy, but I ended up diving down multiple rabbit holes, and in the end it took much longer than expected. That said, I learned a lot along the way, and I'm even more excited about the language and where we're headed now.

Apart from this short wish list, I also think it's important that we stay aware of the community and keep them in mind. As has been mentioned in a number of other blogs, let's try and make Rust sustainable for everyone: maintainers, core team, working groups, and community members alike. Let's also try and move forward with game development, GUI development, and web development. We're already great. Now let's be even better. And no matter where we go from here, I'm confident that as long as the community stays warm, welcoming, and inclusive, we'll have something to be proud of.

Much love ❤️

Let's read: Haskell Programming from First Principles, pt VI

Mon, 28 Oct 2019 07:24:29 GMT

It's been a while, but welcome back to yet another installment in our read-through of Haskell Programming from First Principles. This time we're looking at typeclasses.

What are typeclasses?

Typeclasses are like interfaces to your data. The description that I find the most intuitive is that a typeclass is a set of functions that must be implemented for your type to be an 'instance' of a typeclass. In classic OO-terms, this is very much like an interface; a contract that you must fulfill to have your type be able to stand in for this interface. In this way, typeclasses are a vehicle for ad-hoc or constrained polymorphism, and let you write code that operates on a more generic set of types than concrete implementations.

Typeclasses allow us to generalize over a set of types and to write functions that can abstract over concrete implementations. For instance: if all a function does with one of its input parameters is test it for equality, then the Eq typeclass contains all the functionality we need, and we can make that function take any type that implements the Eq typeclass. Similarly, all numeric literals and their various types implement the typeclass Num, which defines a set of standard numeric operations, allowing functions to work on any numeric value.

To get a better feel for how this works, let's look at some of the basic typeclasses and some implementations!

`Eq`

Eq, the typeclass for equality. A very simple typeclass that will let you decide whether two items are the same or not.

The definition is as follows:

class Eq a where
 (==) :: a -> a -> Bool
 (/=) :: a -> a -> Bool

Pretty simple, right? All it needs is a definition for (==) and one for (/=), and even better, the minimal complete definition --- that is, the least we need to implement to have a complete definition --- is just one of the two functions. The one we don't define will default to being the negation of the one we do define.

Most data types you'll work with implement this one already, so you don't need to worry much about it. Interestingly, because Haskell deals with structural equality, whether a tuple is an instance of Eq or not, depends on its components. If all the components can be compared for equality, then so can the tuple. This also applies to lists and a range of other data structures.

Writing an instance of the `Eq` typeclass

As an aside, here's how you'd write your own instance of the Eq typeclass. It's a simple instance to write, but that makes it easier to learn from. Let's look at a basic example using a simple type we define, Bool2:

data Bool2 = No | Yes

instance Eq Bool2 where
  Yes == Yes = True
  No  == No  = True
  _   == _   = False

So to create an instance, we put instance, the name of the typeclass, the name of the type to implement it for, and the where keyword. What follows is an implementation of the functions required for the typeclass. As mentioned above, the minimal complete definition of the Eq typeclass only requires one of the functions to be defined, so this is a valid implementation.

`Num`

The Num typeclass is one we've already looked at quite a bit previously, so we're not going to dwell on it for too long. It's a typeclass implemented by most of the numeric types, and the definition is as follows:

class Num a where
  (+) :: a -> a -> a
  (*) :: a -> a -> a
  (-) :: a -> a -> a
  negate :: a -> a
  abs :: a -> a
  signum :: a -> a
  fromInteger :: Integer -> a

`Fractional`

There are various other typeclasses related to Num, but let's look at Fractional, a subtype of Num:

class (Num a) => Fractional a where
  (/) :: a -> a -> a
  recip :: a -> a
  fromRational :: Rational -> a

This typeclass requires its type argument, a, to have an instance of Num. This means that a Fractional is a specialized Num and can be used anywhere a Num can, but also specifies an extra set of functionality that is not required of the regular Num typeclass.

`Ord`

This typeclass takes care of ordering things, and as such, has a definition concerned with orders and comparisons:

class Eq a => Ord a where
  compare :: a -> a -> Ordering
  (<) :: a -> a -> Bool
  (>=) :: a -> a -> Bool
  (>) :: a -> a -> Bool
  (<=) :: a -> a -> Bool
  max :: a -> a -> a
  min :: a -> a -> a

Note that it has a typeclass constraint of Eq. If you're going to be comparing things, you're going to have to be able to tell whether they are equal or not. These functions all work the way you'd expect them to.

Ord instances (when derived automatically) rely on the way the data type is defined. Looking back on our Bool2 data type from earlier (data Bool2 = No | Yes), Yes will be greater than No because it is defined later. This is similar to how enums work in the C-familiy of languages where they're usually just fancy names for successive integers starting at an arbitrary point and incrementing by one for each entry. Naturally, if you want to change the ordering, you can, but you're going to have to implement the typeclass yourself.

`Enum`

The Enum typeclass is quite different (yet weirdly similar) to most other things known as enums in other programming languages (at least in my experience). In Haskell, the Enum typeclass is for types that are enumerable, that have known predecessors and successors. What does this mean? It means you can use instances of this typeclass to generate ranges and lists.

Definition:

class Enum a where
  succ :: a -> a
  pred :: a -> a
  toEnum :: Int -> a
  fromEnum :: a -> Int
  enumFrom :: a -> [a]
  enumFromThen :: a -> a -> [a]
  enumFromTo :: a -> a -> [a]
  enumFromThenTo :: a -> a -> a -> [a]

The most obvious instance of this typeclass would be something like integers, where succ 4 is 5 and pred 3 is 2. Similarly, characters can be enumerated just as well, as can ordered data types like Bool.

Some of the function names listed above are a bit confusing (enumFromThenTo anyone?), but when you look at it in context it makes more sense. For instance,

enumFromThenTo 1 10 100
-- yields [1,10,19,28,37,46,55,64,73,82,91,100]

Where this becomes more interesting is when you see that you can use them in list comprehensions and figure out that there's alternative ways of using these methods. Here's the equivalent written as a list comprehension instead:

[1, 10 .. 100]
-- yields [1,10,19,28,37,46,55,64,73,82,91,100]

`Show`

Show is a typeclass that provides human-readable string-representations of structured data. GHCi uses this to create String values that it can print in the terminal. In fact, to have a type be printed in the terminal, it must implement Show.

class Show a where
  showsPrec :: Int -> a -> ShowS
  show :: a -> String
  showList :: [a] -> ShowS

Unless you're writing this for your own data structures, you don't need to worry much about this one. It's already implemented for all basic types. The minimal complete definition requires either show or showsPrec.

Definitions

We've covered a lot of ground here, so let's take a step back and look at some definitions for this chapter.

Typeclass inheritance: When a typeclass has a superclass. This means that only data types that implement the super class can have an instance of the subclass. Going back to the Num and Fractional typeclasses: Every Fractional is also a Num, but not every Num is a Fractional. This makes Num a superclass of Fractional.
Instance: An instance is the definition of how a typeclass should work for a given type. Instances must be unique for a given combination of typeclass and type, i.e. you cannot have two different instances of a typeclass T for a single type a.
Derived instances: Some typeclasses are so obvious or common that they can be automatically implemented for a data type. Automatically deriving typeclasses like this is done by specifying it after the data declaration: data MyType = A | B deriving (Eq, Show)

Command Line Control: vipe

Mon, 21 Oct 2019 21:11:58 GMT

Every now and then you come across a little tool that makes you go "/Oooh, that's neat. Wish I'd known about that before./" vipe is one of those. Granted, you might not need it every day, but sometimes it's just the right tool for the job.

The tl;dr: is that it's a way for you to pipe something into your $EDITOR and then pipe it on into another command when you're done editing it. That's right! There's no need to save it to an intermediate file and clutter your workspace, nor do you have to do everything via sed (though that can be a fun experience).

How did this come about? Well, we wanted to migrate a config map from one OpenShift project to another at work. Rather than saving the config to an intermediate file, we wanted to see if we could pipe it through an editor and back into oc apply with the required changes. And sure enough, we could:

# kubectl should work just the same as oc
oc get configmap <config-map> -o yaml \
| vipe | oc apply -f -

The first command gets the config map from the server in yaml format, the second (vipe!) pipes it into your $EDITOR, letting you make any necessary changes, and the last one takes your updated config map, applying it to the server for you.

vipe is available as part of the moreutils package, available via various package managers including Apt, Nix, and Homebrew. There's a couple more interesting packages in there too, so it's worth checking it out just for that!

On differentiation

Mon, 14 Oct 2019 06:54:56 GMT

Let's talk math. Specifically, calculating derivatives through differentiation. I've been taking a calculus course at my local university lately, and one of the things that's come up and bit me a couple times is how a lot of the reference material we're given for differentiation shows off differentiation in a way that isn't entirely clear. So in this post I want to record for the future me just how differentiation works.

Implicit derivatives

When the common derivatives are listed, they're listed for the variable $x$, and the fact that you'll need to apply the chain rule ($(f \circ g)' = (f ' \circ g)g'$ or $(f(g(x)))' = f'(g(x)g'(x)$) when working with expressions is not really made clear. For instance, let's look at the derivative for natural logarithms: $$ (\ln |x|)' = \frac{1}{x} $$

This works great for a single variable $x$, because the derivative of $x$ is $1$. However, in cases where $x$ is an expression ($2x$, $x^2$, $\sin x$, ...), you'll need to multiply that fraction with the derivative of the expression $x$. For this reason, I think it'd be much clearer if the formula sheet had it listed like this: $$ (ln |x|)' = \frac{x'}{x} $$

Similarly, this also goes for other derivatives, like sines, cosines, powers of $e$, and so on:

$$(\arcsin x)' = \frac{x'}{\sqrt{1 - x^2}}$$

Add updated files only

Mon, 07 Oct 2019 20:49:51 GMT

Sometimes you're just a hair too quick when working with git, run a little git add -A or git add ., and then let an unwanted file or two come along for the ride. Regardless of whether it's just random scribbles or actual secrets, it's pretty annoying. And if no-one notices before you push it to the remote (or open a merge request), it could be a good few levels worse. Today I learned a way to avoid this, and I want to share it with you, dear reader.

Now, just to be clear: I don't actually recommend you blindly add all files in a directory or project to git. It's usually better to be explicit about what you're adding. Sometimes, though, it's just easier to add everything. Claiming that I've never done it or that it's never caused me grief would be disingenuous at best.

So what's the solution you ask? It's a simple flag you can pass to git add: -u or --update. Much like the flag name would suggest, this makes it add only updated files.

The docs say that this option will "/update the index just where it already has an entry matching <pathspec>. This removes as well as modifies index entries to match the working tree, *but adds no new files*/". So you can point it to a directory and it will add all changed or deleted files in that directory, but no new files.

You can also use it without a path, which will just add all deleted and modified files: "/If no <pathspec> is given when -u option is used, *all tracked files in the entire working tree are updated*/".

So next time you want to blindly add all your changes (but no new files!) to git:

git add -u

Psst! Here's a little bonus resource for ya: For more information on the behavior of git add with different options, check the answers to this stack overflow question, which is what put me on the trail of git add -u in the first place.

A mentor's thoughts

Mon, 30 Sep 2019 06:52:57 GMT

I was a mentor at Oslo Legal Hackathon this past weekend, and while I've been to hackathons before, this was my first time participating as a mentor. I wasn't quite sure what to expect, but I felt ready.

And indeed, everything went pretty swimmingly. Well, mostly. There was a shortage of tech people, and I ended up co-mentoring a team consisting solely of people from the legal industry. Not ideal at an event that's supposed to bring legal and tech together, but they made it to the final round by focusing on the things that they could do rather than what they couldn't. But even though I think my team did a terrific job, that's not really what I want to focus on. Rather, I want to talk about the act of mentoring and how it feels different from participating.

Stepping down

Admittedly, I like being in charge. I like overseeing what's going on and giving my input on it. I often end up as team lead or in similar positions. Being a mentor is different.

Yes, I could still get an overview of where the team was headed, and yes, I could still give my input on it, but in the end, I knew that I had no real say in the matter.

I'm also very competitive. It's probably one of the main reasons that I work as hard as I do. Sure, I enjoy what I do, but I also want to be the very best. Like no one ever was.

By being slightly divorced from the process and by not actually being on the team, I could take a step back, breathe, and just enjoy seeing them work and the solutions they came up with.

In fact, I think that might be the most important thing I learned about myself: I enjoy enabling a team and seeing them succeed at least as much as I enjoy being on the team. They're two very different, yet very similar experiences.

For future reference

To close this all out, then: would I do it again? Yes. No doubt. Not only was it great fun, it was also a fantastic chance to grow my network, both within tech and within the legal industry, and to get to better know my coworkers and fellow mentors.

I've been trying to come up with some cons, but I haven't really been able to; the event was well organized, the participants and organizers were all nice people, there was food and drinks, and it was just a fun event all around.

A bit tongue-in-cheek, perhaps, but it felt great being able to come in, give feedback, express some opinions, and then go away and have someone else do the dirty work for you. I think I understand why people want to be managers now.

And finally, even if I've only been at this for a comparatively small amount of time, I felt that I really had something to contribute to the team, that I could make a difference, that I could add value. So remember: Even if you're not the most decorated or recognized individual, don't sell yourself short! As long as you invest your time and are genuinely passionate about what you do, you'll be a valuable asset to any team, no matter the role.

So don't quit, don't stop exploring new things, and don't stop making yourself uncomfortable. If you're curious, go for it!

Amending authors

Mon, 23 Sep 2019 07:02:37 GMT

If you juggle multiple git user names or emails in your job and/or spare time, it's not unlikely that you'll end up committing code with the wrong user. At a glance, there's no easy way to fix this; you can't just change the author of a commit---they're immutable, after all---but you can delete a commit and create a new one. This sounds tedious, and doing it manually is tedious, but fear not, for I am here to show you how we can do it automatically!

The trick is based on the answers to this Stack Overflow question. In short, it's just a matter of rebasing off a specific commit and then automating the process of accepting the commit as-is while changing the author.

The basics: `git rebase -i`

Interactively rebasing lets you choose a commit to rebase off and then choose what you want to do with all the commits since (edit, pick, delete, etc.). In this case, we want to amend the commits, so change them all to edit. If you want to leave some commits untouched, just mark them as pick.

When rebasing interactively, git will stop after each commit we've chosen to edit, saying you can amend the commit by running git commit --amend. We want to reuse the same commit message, but edit the author, and the trick for that is using the --reset-author and --no-edit options.

So far, our process looks like this:

Set the right user details in our config.

git config user.name <correct user name>
git config user.email <correct user name>

Initiate the rebase.

git rebase -i <commit hash>

Change the commit's author and continue the rebase.

# amend the commit, changing the author, but leaving the message intact.
git commit --amend --reset-author --no-edit

git rebase --continue

Repeat step 3 until you're done

Tedious, but at least it works. Most likely, this doesn't happen often enough that you'll need to automate it, so we could just leave it, but where's the fun in that?

Taking it further

Let's think about what we want to do: For each commit since a specific one, we want to change the author and continue the rebase. Sounds pretty easy, right? We'll just need to figure out how many iterations we need, and we have ourselves a one-liner!

So how do we go about finding out how many commits we need to change? We could count, but git does offer a command we can use for it: git rev-list. It doesn't do much without any arguments, but with the --count flag and a commit, we can start working some magic. The --count functionality, as you might expect, counts the number of commits up until the provided hash by default, but, conveniently, it can also operate on commit ranges! (As an aside: git rev-list has a lot of interesting uses, so go read the docs if you want to know more!)

To get the number of commits since the one we're using as the rebase root (thus not changing):

git rev-list <rebase root hash>...HEAD

This is starting to look pretty good. If we put something together, it might look something like this, with <root> being the hash of the commit we want to rebase off:

n=$(git rev-list --count <root>...HEAD);\
git rebase -i <root>;\
for i in $(seq $n);\
do\
  git commit --amend --reset-author --no-edit;\
  git rebase --continue;\
done

or in Fish for all you cool kids:

set n (git rev-list --count <root>...HEAD);\
git rebase -i <root>;\
for i in (seq $n);\
  git commit --amend --reset-author --no-edit;\
  git rebase --continue;\
end

Note that this is a pretty naive way to do it. As outlined in this Stack Overflow question, the number of commits you're looking at might not be what's returned by git rev-list directly. For instance, if you have multiple branches that have been merged in, you might need to add the --first-parent flag to get the desired behavior.

Special case: what about rebasing off the root of the repo?

Ah, yes; this is what I've actually had to do each time I've been amending authors. We've tackled rebasing off a repo root in a prior post, so we already know that we can pass the --root flag to git rebase. When we rebase off the root, we don't need to deal with commit hashes, so it actually becomes a bit simpler:

In bash:

n=$(git rev-list --count HEAD);\
git rebase -i --root;\
for i in $(seq $n);
do\
  git commit --amend --reset-author --no-edit;\
  git rebase --continue;\
done

And in fish:

set n (git rev-list --count HEAD);\
git rebase -i --root;\
for i in (seq $n);\
  git commit --amend --reset-author --no-edit;\
  git rebase --continue;\
end

Git config management

Mon, 16 Sep 2019 07:00:10 GMT

We've been looking a lot at how to tweak git to your liking, which is usually done through your git config file. Sometimes, though, you'll find that the configuration isn't working as you wanted, so I thought it'd be a good idea to look at how you can inspect, unset, and list configuration values from the command line.

Before we dive in, let's have a few words about how your git config files work. Your config files use a specific format which, according to the docs, "consists of sections and variables. A section begins with the name of the section in square brackets and continues until the next section begins." That's enough for what we need right now, but go read up if you're interested.

Furthermore, git config files 'cascade', and git, by default, searches four locations on your system, where later entries override earlier ones:

~/etc/gitconfig~: The system wide config
~$XDG_CONFIG_HOME/git/config~: Falls back to $HOME/.config/git/config if $XDG_CONFIG_HOME is not set or empty.
~~/.gitconfig~: Your user-specific, 'global' config file
~$GIT_DIR/config~: Repo-specific config file

Again, see the specific section of the documentation for more information.

One last thing: All setting and unsetting of values apply only to your user configurations, and they are local---repo-specific---by default. To make them apply to your global config, pass the --global flag.

Setting values

To set a value, simply run a command on the form

# default (local)
git config <section>.<key> <value>
# global
git config --global <section>.<key> <value>

For instance, to change the value of the commentChar entry of the core section to ;:

git config core.commentChar ';'

Unsetting values

Similarly, if you want to unset a value, use the --unset flag:

git config --unset <section>.<key>

So if you have overridden the core.commentChar value, this is the command to undo that and reset it to the default value, #:

git config --unset core.commentChar

Inspecting config values

Sometimes, you want to know what a value has been set to. For this, pass the --get flag and the section and key of the value you want to look up:

git config --get <section>.<key>

So if you're wondering what your user's name is, for instance:

git config --get user.name

A few things to note about getting the values

Git will show you the last value it finds in the chain: So if you set your username in your global config, but set a different one in a repo and invoke this command from the repo, it would show you only the repo-specific one.
The output will be empty if you have not explicitly set the value: In other words, if it's not in your config files, git won't show it to you. So if you're looking to inspect git's default values, that's not going to work.

Finding the config value sources

Sometimes you just want to inspect what values you're setting in your config. You can do this by passing the --list flag:

git config --list

By default, this will just throw all your configuration at you, which isn't all that helpful. However, it takes a pretty handy flag, --show-origin, which lists your configuration in two columns, with the first one being the file in which it was set and the second being the the key-value pairs.

If you ever find that some value isn't taking effect the way you expect, this is a great way to check what values are getting set in what locations:

git config --list --show-origin

For instance, if you've changed your global user name, but it's not taking effect in the current repo, you could pipe it into grep and look for duplicates:

git config --list --show-origin | grep "user.name="

Commit message templates

Mon, 09 Sep 2019 03:53:29 GMT

Here's a statement for ya: if you're working on a project with someone else, you should be using a commit message template. Not only will it help increase the consistency and usefulness of your messages, but also, and perhaps more importantly, it will reduce the cognitive load required to write a message and the time it takes to write it. I can only speak for myself, but I know that having to go look up information about how a commit message should be structured and what it references causes some much unneeded context switching, resulting in a less efficient workflow.

Setup

Setting up commit message templates requires only a few steps:

Create a template.

You can use any text file as a template; just put what you want to show up in the commit message buffer in there. Plain lines of text will show up as content, while lines starting with the comment character (# by default) show up as comments.

Add it to the git config as a commit template.

git config commit.template <path/to/file>

Done! Now try committing something.

When doing this, you're assigning the template locally (to the current repo). If you'd rather do it globally, pass the --global option.

If you want to share the template across multiple projects in a certain subdirectory, you can do that by combining the template with git's functionality for conditional includes. Be aware, however, that given a relative path to a file, git will look for the file relative to the including configuration's location. While this can potentially be exploited to use different templates for different projects, I've not found the need for it thus far, and have stuck with a single template with an absolute path.

If you want to unset the template, use the git config --unset functionality:

git config --unset commit.template

Commit template ideas

Different teams and projects have different needs when it comes to commit messages, so there's no 'one size fits all', but here's a few ideas based on message formats in teams I've been a part of.

Label listing

If you label your commits with the type of work they contain to and have a defined set of labels (e.g. 'feature', 'bug', 'chore', 'refactor', etc.), it might be a good idea to list all the different options in the template, so that you can look through them whenever you're committing some code. List them using commented lines and you don't even have to delete them; they'll just show up as extra info in the commit buffer.

While experienced members of the team are likely to know these by heart, it's very helpful for a new person to have the list right there in the commit buffer so that they don't have to go look them up.

Issue identification

If the standard says to have an issue/ticket ID listed in the commit message, you could put a little placeholder in the template that tells you to replace it with the relevant ID.

If you also work with feature branches and name them according to the feature/issue ID that they correspond to, you could even have git fill in this issue number automatically by using hooks, as explained in my previous post. I have found this to save me a surprisingly large amount of time and mental overhead.

Very specific formats

For when your message needs to match a very specific format and it's hard to remember exactly what goes where, templates are a perfect solution.

For instance, if you're working with a combination of the above two, where you want both the label and the ID, and you want them in a specific order, maybe even with a specific separator (seems like a hassle, but just roll with it), you could create a template that you could just fill in with the relevant parts, saving you having to look up what the format is every time.

Rebasing off a repo root

Mon, 02 Sep 2019 07:52:55 GMT

Ever had to change something about all the commits in a git repo before pushing it to a remote? As in from the very first commit in a repo? Maybe you've started a repo locally and when you're ready to push it, you realize that you've used the wrong author or messed up the format of the commit messages; or maybe you just want to squash those first few commits into a single concise package.

In these situations, my first response is to do to an interactive rebase (git rebase -i). Usually when I'm rebasing, though, I'm in a project that has an upstream and where I'm rebasing off a specific commit or branch. For situations where you don't have a commit to rebase off, but you want to rebase the entire history or at least the very first commit: What do you do?

The answer, my friend, is that you pass the --root option:

git rebase -i --root

That'll let you pick, reword, edit, squash, fixup, exec, drop, label, reset, or merge all of your commits from the 'dawn of time.'

Have fun!

Automate your commit messages

Mon, 26 Aug 2019 07:48:50 GMT

Git has a lot of inbuilt functionality that you might not use all the time, but which is oh-so-handy when you need it. One such thing is hooks. These can be used to do stuff like formatting your code before committing or running tests before pushing to the remote. Today, though, let's look at how we can use it to automatically fill in part of your commit messsage.

How did this come up?

The team I'm on at work uses a system for commit messages and branch names where they should both start with the issue id of whatever task we're working on. Using GitLab, they end up looking something like #2 Adds sorting functionality to UI (we can argue about commit message tenses later) and #42/new-sorting-algorithm, respectively.

If you're like me, your mind should be poking you right about now, saying that that's a lot of duplication; surely we don't need all of that? Well, you could argue that with feature branches scoped to issues and no fast-forward merges, the branch name should be enough, but it doesn't really show up in condensed commit logs etc., so it's nice to have the issue number available for when you need it.

But not to worry; we can make the computer fill it in for us!

Picking the right hook

Git has quite a few hooks available; if you check the .git/hooks directory of a tracked project, you should find a bunch of files named <hook name>.sample. Most of the hook names are fairly descriptive, but you could always look them up if you want to know more.

In our case we're interested in hooking into the system right before the commit message buffer appears on our screen, so we're going to go with the prepare-commit-msg hook.

Auto-filling the message

Now, my team has a template we use for commits, which starts with #[task number]. Using this knowledge, we can do some fancy shell magic and replace it with the task from the branch name.

The whole script looks like this:

#!/bin/sh

TASK_NO=$(git rev-parse --abbrev-ref HEAD | cut -d '/' -f 1)
# BSD sed (macOS)
sed -i'' -e "s/#\[task number\]/$TASK_NO/" "$1"

# GNU sed (Linux)
# sed -i -e "s/#\[task number\]/$TASK_NO/" "$1"

Depending on whether you use BSD sed  (macOS) or GNU sed (Linux), the ~sed~ command will behave slightly differently. For GNU sed, use the commented out line instead of the first one.

There is a slight difference in how you make the two versions of the programs write files in place.

The git rev-parse --abbrev-ref HEAD command gives you the name of the current branch, which we then split at the first / character using cut.

The next (and last) step is a simple sed replacement, replacing the #[task number] string with the branch's task number.

The $1 variable at the very end of the last line is the name of the commit message file and is something the script receives automatically. We use this to tell sed what file to replace the text in.

And for the curious, these are all the variables you get access to in the prepare-commit-msg hook (taken from the prepare-commit-msg.sample file):

COMMIT_MSG_FILE=$1
COMMIT_SOURCE=$2
SHA1=$3

Caveats

There is a pretty significant exception to when this script does what you want it to: when you're still on the master branch, where commit messages will start with the text master. I considered adding a special case for this, but then realized that it's actually quite helpful in that it helps me remember when I should branch off, so I decided to keep it for now.

Another drawback is that because the commit message appears different from what git expects to find on disk (presumably), it will assume you put an actual message in and complete the commit. At least that's what I found when using Vim. A simple workaround is deleting everything in the buffer (dae if you use the awesome text-obj-entire plugin) before exiting. Most of the time, though, I use Magit with Emacs, where canceling the commit works as expected.

Alternate approach

While I have opted for modifying the commit message before it's shown to the user, you could potentially also have the machine add the id automatically in the pre-commit hook.

You would make the system check the message for a pattern and amend the commit with the expected issue id if it's not present.

This solution doesn't give you any way to verify whether it's correct or not, though, so you'd probably want some sort of way to bypass it.

Change your git comment character

Mon, 19 Aug 2019 06:21:57 GMT

You prepare to commit your files, having made sure that all the right changes are in, that they belong to one logical change, and that you have the id of the task that they belong to. Your editor opens up and you see those familiar commit message instructions, but wait! You suddenly remember that you should start your commit with the id of the task prepended with a # character, but that would lead git to think that that line is a comment.

How do we solve this?

Easy! We simply tell git not to use the # character for comments, but pick a different character instead.

In short

There's two ways to do it, both producing the same result. You can either run a command from the command line or edit your git config directly. In this case, imagine we want to use a semicolon (;).

Command line:

git config --local core.commentChar ';'

Directly modifying your git config file:

[core]
  commentChar = ";"

Motivation

So why might you want to do this? As mentioned in the introduction, maybe your team has a standard format for commit messages, where each commit should lead with the id of the task that the commit relates to.

We use GitLab at work, where if you put a number after a # in your commit message, it will create a link to the issue with that id, which also adds an entry to the history of said issue. In addition to this, if all commits are prefaced with the task id, it becomes very easy to find out what task a commit relates to when looking at the history.

What might go wrong?

This is such a simple modification that there's not much you need to be aware of, but if you try and use a comment character consisting of more than one character, git will fail when you try and invoke a command:

error: core.commentChar should only be one character
fatal: bad config variable 'core.commentchar' in file '.git/config' at line 6

Finishing thoughts

It's a simple thing, but it's one of those things that improves your life just that little bit. You can even combine it with the custom config we mentioned last time to have this apply to all repos in your work directory, new and old.

Now as a final thought, what should you pick as a comment character? It's really up to what you need for your specific use case, but I find that ; is very rarely something that you'll want to start your sentences with, so that's my go-to.

Modularizing your git config with conditional includes

Mon, 12 Aug 2019 08:27:09 GMT

Do you track your git config across machines or keep it in sync with your dotfiles? Is it full of settings that you want to share between all your different contexts? Are there certain things that you want to change depending on what the context is? Well, I've got news for you!

tl;dr:

If you want to use an additional config file for all subdirectories below a certain path, use the includeIf functionality that was introduced in git 2.13.0.

[includeIf "gitdir:<path to top directory to use the config in>/"]
path = <path to extra config>

That's all you need to get going, but if you want to know more about how this works and some caveats you might to be aware of, keep reading.

`include` and `includeIf`

First off, I'd be remiss not to direct you to the official documentation for this feature, which can be found here. However, there's loads more info there than you might need (and want), so let's extract the important bits.

The include and includeIf sections work the same, except the includeIf has a condition that must be satisfied. They let you insert configuration from another file into your main one.

In my case, this meant that I could automatically change my email whenever the git directory was a subdirectory of ~/projects/work, allowing me a simple and efficient way to always use my work email for work, without changing my global git config or branching off from my main dotfiles repo.

Here's the relevant extract from my config:

[includeIf "gitdir:~/projects/work/"]
path = ~/projects/work/.gitconfig

I keep a separate config file in the ~/projects/work directory that overrides my email.

This would also be useful if your team has specific rules about whitespace, merge strategies, comment characters, etc. that you want to enforce.

Things that might trip you up

Syntax: Note the trailing slash after the directory path in the includeIf line. This makes it so that it will match all subdirectories of the specified directory. As explained by the docs:

If the pattern ends with ~/~, ~**~ will be automatically added. For example, the pattern ~foo/~ becomes ~foo/**~. In other words, it matches "foo" and everything inside, recursively.

This also means that if there is no trailing slash, it'll match only that specific directory.

File insertion: Here's another quote from the documentation:

The contents of the included file are *inserted immediately, as if they had been found at the location of the include directive*. If the value of the variable is a relative path, the path is considered to be relative to the configuration file in which the include directive was found.

So you'll probably want to put the includes at the bottom of your files to make sure the included config isn't overridden later on in the source file.

I hope that cleared some things up for you and that you found it useful; I know I did. If you want more in-depth information, see the documentation.

Let's read: Haskell Programming from First Principles, pt V

Mon, 05 Aug 2019 18:28:46 GMT

Oh, hey, it's you again! Always a pleasure to have you along for the ride. This time, let's have a deeper look at one of the basic underpinnings of the Haskell language: types. Being an implementation of a typed lambda calculus, types are a big part of what makes Haskell Haskell, and a good understanding of them is essential to master the language.

The chapter opens with a couple of pages asking what types are good for and describing how it can help us enforce invariants, avoid (certain types of) bugs, enable compiler optimizations, and reason about our systems.

For this summary, we'll explore how to constrain argument types to typeclasses, have another dive into currying and partial application, and have another look at polymorphism.

Constraining your types

Let's start off by looking at how we can constrain a function's domain by using typeclasses and how that impacts the function.

By default, when you create a function and you don't annotate it, the compiler will give it as wide a definition as it can, using the least specific types, constraining it to a typeclass if necessary, but leaving it completely open if possible. Of course, if one or more of the parameters can only be of a concrete type, it will infer that too.

This generalization is why we can define a function for numbers and then use that function for any type that has an instance of Num.

To add a typeclass constraint to a type signature, specify the typeclasses in before the arguments and separate them with a fat arrow (=>). If you have multiple constraints, put them in parentheses and separate them with commas. The below signatures illustrate this:

Num a => a -> a -> a

(Num a, Num b) => a -> b -> b

(Ord a, Num a) => a -> a -> Ordering

The type constraints only show up at the type level and are never visible at the term level.

As we'll see when we talk about polymorphism a bit later, constraining our types like this enables us to create more powerful and specialized functions at the cost of decreasing general applicability.

Currying

We've talked about currying before, but this time it's given an entire subchapter! If you can't quite remember what it is, let me give you a quick summary:

Because any function in a lambda calculus can only be applied to one argument, the way to construct functions that can be applied to multiple arguments is to nest multiple lambdas. Haskell abstracts this away for us and makes it seems like a function can be applied to an arbitrary number of arguments. The only place where this actually shines through is in function signatures, where the function operator ((->)) separates the parameters, hinting that it's actually just a bunch of nested lambdas

This is handy because it allows partial application, allowing us to pass around functions that haven't been fully applied and add the remaining arguments when it suits us.

Haskell is curried by default, but it's possible to uncurry functions. What does that mean? It means that if you have a function that you need to apply to two arguments, you can turn it into a function that only accepts a single argument: a tuple containing the two expected arguments. The opposite is also true: if you have a function that expects a tuple, you can curry it to turn it into a function expecting two separate arguments.

The type and possible implementation of uncurry is as follows:

uncurry :: (a -> b -> c) -> (a, b) -> c
uncurry f (x, y) = f x y

We can play around with it a bit:

uncurriedPlus :: Num a => (a, a) -> a
uncurriedPlus (x, y) = x + y

-- uncurriedPlus could also be implemented using the uncurry function:
uncurriedPlus' :: Num a => (a, a) -> a
uncurriedPlus' = uncurry (+)

-- and in the same way, we could redefine a plus function using curry:
plus :: Num a => a -> a -> a
plus = curry uncurriedPlus

Can you see how you can implement fst and const in terms of each other composed with either curry or uncurry? Look up the types in the REPL and play around with it a bit if not.

In short:

Curried functions: many nested functions, each taking one argument each
Uncurried functions: one function, many arguments (grouped in a tuple)

Sectioning makes a little return in this chapter too. Quick reminder: sectioning is partial applications of infix operators or functions, where we can leave out the term on either side and Haskell will cleverly know that we are missing the value to fill it up: (2^) 3 would give $8$, for instance, while (^2) 8 would give 64. Sectioning works for any binary, infix function.

Polymorphism

Polymorphism was also mentioned last time, but this time we dive a bit deeper into what it is. Our definitions of parametric and constrained polymorphism return, and we're introduced to something called ad-hoc polymorphism, so let's recap and explore a bit.

Parametric polymorphism is the broadest form of polymorphism, and is when the type variables are fully polymorphic, completely free of any and all constraints. This means that their actual concrete implementation could by anything and the function would still do what it should. The function will work on any type. An example of a parametrically polymorphic function is id:

id :: a -> a

Can you tell what it does? When a function is this polymorphic, there is only one thing it can do, and that is return whatever it receives as an argument. Hmm? Useless, you say? Well, it has its uses, but this is not the time or place to explore them.

Constrained polymorphism is when there are certain constraints placed upon the types by specifying that they must belong to a typeclass. This makes the function able to use functions defined for a given typeclass, increasing the number of things the function can do by simultaneously decreasing the number of values it can accept. Any numeric function is a good example of this:

(+) :: (Num a) => a -> a -> a

Constraining your types will give you more power within the function, but a smaller domain of input values. A good rule of thumb would be to always try and make your functions as general and unconstrained as possible to give the caller maximum freedom, but not to be afraid of putting constraints on the inputs when it is required by the task at hand.

Sometimes you'll come across something known as a polymorphic constant. In fact, we've seen a lot of these already: any time you only specify a literal numeric value without specifying the type, it will be polymorphic until it's given a more specific type such as by function application.

Haskell's type inference is built on extended version of the Damas-Hindley-Milner type system. It will infer the most generally applicable (polymorphic) type that is still correct for your code. But while declaring types might not be necessary, it's still considered good practice and is recommended.

Definitions

Ad-hoc polymorphism: Polymorphism that applies one or more typeclass constraints to a polymorphic type variable.
Higher-order functions: these are mentioned in the chapter, but not expanded upon. A higher-order function is a function that either accepts as argument or returns a function when evaluated. map is probably the poster child for having a function from a to b as its first parameter.
Parametricity: Parametricity states that the behavior of a function will be uniform across all concrete applications of the function.
Polymorphism: In Haskell: type variables that may refer to more than one concrete type. Usually manifested as ad-hoc or parametric polymorphism.
Type inference: The ability of some programming languages to infer types from terms (your code) without explicit type annotations.
Type variable: A way to refer to an unspecified type or set of types in type signatures.
Typeclass: A collection of functions that must be implemented for any type wishing to create an instance of said typeclass. Allows for ad-hoc polymorphism and abstractions.

Next time

Next up is the chapter on typeclasses, where we'll finally get an explanation for how all these numeric types work together and look at some basic, easily derivable typeclasses. Stay tuned!

Let's read: Haskell Programming from First Principles, pt IV

Mon, 29 Jul 2019 04:07:56 GMT

Welcome back to yet another foray into the wonderful world of Haskell, where this time, we're turning our gaze towards basic data types. The book presents this quote by Robin Milner at the start of the chapter:

There are many ways of trying to understand programs. People often rely too much on one way, which is called “debugging” and consists of running a partly-understood program to see if it does what you expected. Another way, which ML advocates, is to install some means of understanding in the very programs themselves.

Note that in this context, ML is not machine learning, but the language known as ML. If you don't know who Robin Milner was (I didn't), Wikipedia has this to say about him:

He developed LCF, one of the first tools for automated theorem proving. The language he developed for LCF, ML, was the first language with polymorphic type inference and type-safe exception handling. In a very different area, Milner also developed a theoretical framework for analyzing concurrent systems, the calculus of communicating systems (CCS), and its successor, the pi-calculus.

So I'd say he's reasonably influential.

But that's enough of a history lesson for now. Let's move on to some more theory, shall we?

Let's start right at the beginning: what are types? In short, types are sets of values, where a set is a collection with no duplicate entries. Every expression---when evaluated---yields a value, and that value has a type. Each type is defined by the members (the values) that inhabit (are part of) the type. Boolean values, for instance, have two inhabitants: True and False. Integers have an infinite number of inhabitants, every integral value from negative infinity to positive infinity.

Types help us group multiple values that have something in common. It could be a specific domain or model or something more abstract. Whatever it is, they let us talk about a set of values under a shared name.

Data declarations

Data Types in Haskell are defined by data declarations. Data declarations consist of a type constructor with belonging data constructors.

A type constructor is the name of the type, such as Bool or Int, and is used in type signatures, or the type level of your code.

Data constructors are the values that inhabit the type they are defined in, such as the aforementioned True and False for Bool. These are the values that appear in the logic of your code, at the so-called term level.

Let's look at the data declaration of the Bool data type to make it clearer:

data Bool = False | True
--   [1]    [2]  [3] [4]

The name of the type, aka the type constructor. This is what you see in function signatures.
Data constructor for the value False.
A pipe operator, indicating that this is a sum type (or discriminated union). This tells us that a Bool value is either False or True, but never both.
Data constructor for the value True.

This is a very simple data declaration. Others may take arguments or use product types instead of sum types, but what they all have in common is the data keyword followed by the name of the type. Most are followed by an equals operator and a set of data constructors.

Numeric types

We've worked a bit with numeric types previously, but never really looked too deeply at them. They work quite differently in Haskell than what they do in a number of other languages, so let's see what we can find.

The book lists two categories of numbers, which each have multiple types:

Integral numbers ::

Int: Fixed-precision integer. Has a minimum and a maximum value.
Integer: Integer that supports arbitrarily large and small numbers.

Fractional numbers ::

Float: Single precision floating point numbers. Like in most languages, limited in precision.
Double: Double-precision floating point number. Twice as many bits as a Float. Still limited in precision.
Rational: Fractional number that represents a ratio of two integers. A value such as x / y :: Rational will be a value consisting of two Integer values and represents the ratio of $x$ to $y$. Arbitrarily precise.
Scientific: Space efficient and almost arbitrarily precise. Represented using scientific notation, storing the coefficient as an Integer and the exponent as an Int. Because Int is bounded, there is technically a limit to the size of the number, but hitting it is very unlikely. More efficient, but less precise than Rational.

All these data types have instances of a typeclass called Num (more about typeclasses in the coming chapters), which lets us use any which one of them for most functions that only require standard operations.

Choosing an integral data type

As noted above, Integer can represent arbitrarily large integers, while Int (and the related Int8, Int16, and so forth) can only express as many as their predefined size allows them, falling back to overflow wrapping if they exceed their limits.

Because of this, the book advises us to always choose Integer over Int unless we really understand the limitations and the additional performance has an impact.

Fractional numbers

Of the common fractional types mentioned above, three of them come bundled with GHC, while the fourth, Scientific, comes from an external library.

As was the case with the integral numbers, we are advised to stick to the most precise or efficient type we can use. This would usually be Rational or Scientific, but for certain scenarios, such as graphics programming, you might want to use Float.

Certain mathematical operations require the inputs to be fractional rather than just numeric; division being a great example. The type of the division operator is (/) :: Fractional a => a -> a -> a. The Fractional a => part tells us that this is a typeclass constraint, and that the type a must have an instance of this typeclass.

The Num typeclass is a supertype of the Fractional typeclass, meaning that to create an instance of Fractional, the type must also define an instance of Num. In other words: all Fractional instances are also instances of Num, while the opposite is not true.

Comparison

Like most languages, Haskell defines a number of operators to perform comparisons. Most of these are the same as you see in most languages, but the inequality operator may be a bit different. Anyway, for your viewing pleasure, here they are:

~(<)~: Less than
~(>)~: Greater than
~(<=)~: Less than or equal to
~(>=)~: Greater than or equal to
~(==)~: Equal to
~(/=)~: Not equal to

The equality operators take two values that have instances of the Eq typeclass, and the comparison operators take instances of the Ord typeclass, which is a subclass of Eq. As such, an Ord instance can be used with any of the above operators, while an Eq instance can only be used with the equality operators.

`if ... then ... else ...`

Haskell operates with ~if~ expressions, rather than ~if~ statements. What this means is that the expression itself evaluates to a value; so you can assign a variable to an if expression, for example.

The syntax is:

if CONDITION
then EXPR_A
else EXPR_B

An assignment might look like this:

-- the value of a will evaluate to 2
let a = if True then 2 else 0

This is similar to how the ternary operator (?:) works in most C-like languages, and much in the same way, you must have an else clause. This is because it is an expression, and an expression must evaluate to a value.

Tuples

The Tuple type is a way to pack multiple values together and pass them around. Tuple syntax is the same at both type and term levels and is easily distinguished from other types by its distinct look: (x, y), a set of parentheses surrounding two or more values or types separated by commas.

At the type level, the tuple would contain types---e.g. (Int, Bool)---, while at the term level, it would contain expressions: (1, True), (x, y), etc.

A tuple has a fixed number of constituents (parts, members). A tuple with two element is called a pair or a tuple ((a, b)), while a three-element tuple is known as a three-tuple or a triple ((a, b, c)). The number of constituents is known as the tuple's arity.

The data declaration for a pair can be found by looking up the (,) operator in the GHCi:

data (,) a b = (,) a b

This is quite different from what we saw earlier with the Bool declaration: It takes two parameters, represented by type variables a and b, and it's a so-called product type, rather than a sum type, meaning that the number of possible combinations is the product of the number of possible values for each of the types it's applied to.

The two-element tuple comes with some default convenience functions in Haskell, fst and snd, which gets the first or second element of the tuple, respectively. Their type signatures are:

fst :: (a, b) -> a
snd :: (a, b) -> b

This way to describe the types of the tuples also extends to when we use them in the function implementation, where we can destructure it and name and use the elements directly. The two above functions can be implemented like this:

fst :: (a, b) -> a
fst (a, b) = a

snd :: (a, b) -> b
snd (a, b) = b

This destructuring and matching on the shape of the data is known as pattern matching, and is something we'll see a lot more of further down the road.

Lists

We had a little run-in with lists last time, and they're only barely mentioned in this chapter. The reason is that there's a full chapter devoted to lists coming up, so we'll save the deep-dive for then.

The one thing they do mention this time, though, is that, much like tuples, lists have special syntax too, where the type or the values are enclosed in square brackets, e.g. [Integer] for describing the type of a list of integral values, and [1, 2, 3, 4] to construct one of them.

As opposed to tuples, they can only take values of a single type, and there is no limit to how many or how few elements a list can have.

Names and variables

To close off the chapter, there is a little section on naming conventions in Haskell.

Functions, when used as parameters, are typically labeled with variables starting with f, continuing alphabetically (g, h, ....). May also have numbers (f1, f2) or an ending apostrophe (f'~)---pronounced /f-prime/---if they're closely related to a function labeled ~f.

They may also be given variable names that describe what they do, such as a function fetching text from some source being called txt.

Type variables, the ones in type signatures, are commonly given names starting from a and going along alphabetically.

Arguments to functions usually go from x, though they might also have some other letter assigned to serve a mnemonic function (such as n for a number or r for the radius of a circle).

Of course, variable names don't have to be single letters, but in smaller programs they usually are. In more domain-specific code, it might make more sense to use longer names.

If you have a list of things, it's common to use a name such as xs, especially if one element from the list might be called x (as if xs was the plural form of x).

This is demonstrated by the commonly seen construct (x:xs) which is a way to pattern match on a list using the cons operator we saw last time. This can be used to destructure the list if it has at least one element, assigning the variable x to the first element, and xs to the rest of the list:

sum :: Num a => [a] -> a
sum [] = 0
sum (x:xs) = x + sum xs

The above function will recursively sum a list by first checking whether it's empty. If it is, it returns $0$. Otherwise, it returns the sum of the current head and the value of the sum of the rest of the list.

Definitions

To tide us over until next time, this chapter provides a bunch of juicy definitions!

Tuple: An ordered grouping of values. You cannot have a tuple with only one element, but the type unit or () can be thought of as a zero-element tuple.
Typeclass: A set of operations defined for a polymorphic type. If a type has an instance of a typeclass, it can be used in any function requiring a value of that typeclass. In Haskell, you can only define one instance of any one typeclass for a given data type. In other words, type a can have at most one instance of Eq.
Data constructors: How we create values that inhabit a type in Haskell. Can be constant values (True), or take one or more arguments ((,) a b)
Type constructors: The name of a type. Can only be used in type signatures
Data declarations: Definition of a data type in Haskell. Consists of a type constructor and zero or more data constructors.
Type alias: A way to refer to a type constructor or type constant by a different name, usually to communicate something more specific. type Name = String would let you use Name in place of String in your code, allowing you to say something more about what kind of string you want on the type level.
Arity: The number of arguments a function accepts. In Haskell (lambda calculus), all functions are actually 1-arity (unary) due to currying, but we tend to let Haskell abstract that away and think of the total number of arguments needed to fully evaluate the function.
Polymorphism: The ability to write code using values that may be one of several, or any, type. There are two types of polymorphism: parametric and constrained.

Parametric polymorphism is when a function can take any type of value because it doesn't use any information about it. Parametric polymorphism is easily recognised by there being no constraints placed on the type variable.

Constrained (or bounded) polymorphism is when the valid set of types is constrained by a typeclass, such as for the equality operators that require their arguments to have an instance of Eq.

Summary

Whew; well done for making it to the end! There was a lot to get through this time around, but it's been very insightful. Next time we'll be looking more at types in Haskell, exploring constraints and polymorphism.

Let's read: Haskell Programming from First Principles, pt III

Mon, 22 Jul 2019 04:43:39 GMT

This will be a rather short and focused post. Chapter 3 of the book doesn't spring any exciting new concepts on us or cover large topics such as general syntax. Instead it chooses to zoom in on the String datatype. We look at how they are represented in Haskell and how we can treat them like lists because, spoiler alert: that's what they are.

Like in all compiled languages that I've used, there's a difference between the data types Char and String and how you represent the literals. As is the convention, a Char is represented using single quotes (~'a'~), while a string uses double quotes: ~"hey, tiger"~.

However, unlike in a most languages I'm aware of, a String is actually just syntactic sugar for a linked list of characters ([Char]). So ['h', 'e', 'l', 'l', 'o'] and ~"hello"~ are just two ways of writing the exact same thing. This has some interesting effects on how we handle strings, in that it means anything we can do with a list, we can do with a string, but it also means that certain operations will be very costly. There are alternatives for when you need performance, but let's not concern ourselves with that for now.

String/list operations

In the introduction to handling strings (secretly lists), we are introduced to the functions head, tail, take, drop, and the operator !!, so let's have a quick look at what they do.

The head and tail functions return the first element of the list and the list without the first element respectively; so that if you take the head of a list, the tail of a list, and then put them back together, you'd have the original list back:

splitAndPutBackTogether list =
  let h = head list
      t = tail list
   in h : t

Side note: this is the first time we're seeing that ~:~ operator. It's known as ~cons~ and it adds a value to the front of a list.

Be aware that these functions---~head~ and tail---are not total, and will throw exceptions at you if you give them an empty list.

A *total function* is one that is well defined and gives you a result for every input value---i.e. maps every value from the /domain/ into a value in the /image/ (the subset of the /codomain/ containing possible output values), to use the terminology from chapter one.

Functions that are defined only for a subset of their valid arguments are known as *partial functions*.

More on this later on in the book.

The take and drop functions, though, are total, and will not throw anything in your face. They operate on lists, take an Int as the first argument and then split the list after as many elements as specified by the Int, giving you either the first portion or the second. They're pretty much like head and tail, except you can choose how many elements go in each pile, and you'll always get a list back. If there aren't enough elements in the list to reach the specified number, take will give you back the whole list, and drop will give you an empty list.

!! is the index operator and returns the element at the specified index in the list. The equivalent of [n] in a lot of other languages. Zero-indexed and unsafe: ~"hey" !! 2~ would return ~'y'~, while ~"hey" !! 5~ would crash.

Definitions

There are only a few interesting definitions in this chapter:

String: a sequence of characters. In Haskell, the String type is just an alias for a linked list of characters, [Char].
Type (or datatype): "A classification of values or data", such as Int, Bool, String, etc. Notably does not have any behavior associated with it.
Concatenation: Joining sequences of values together. Often used with lists, it's the act of putting one before the other and creating a new list. The operator (++) takes care of it in Haskell.

That's it for this chapter. Next time, we'll be looking at basic datatypes. This is when things start to get interesting, so strap yourself in and get ready.

Let's read: Haskell Programming from First Principles, pt II

Mon, 15 Jul 2019 07:15:14 GMT

Welcome to the second part of reading Haskell Programming from First Principles. This time around we finally see some actual Haskell code. Sort of. It's mostly just a tour of Haskell's basic syntax, along with a brief introduction to the REPL and how that works. As such, let's do a quick recap of the most essential parts.

There are quite a few references to /the REPL/ in this article. /REPL/ is short for /read eval print loop/ and is a command line interpreter for a programming language. In our case, GHC's REPL is invoked by either the ~ghci~ or the ~stack ghci~ command, depending on whether you're using GHC directly or through Stack.

General notes

Let's start with some general notes about how Haskell code works, shall we?

First off, Haskell is a whitespace-sensitive language. Much like in Python, the indentation of your code matters. A lot. However, unlike in Python, indentation doesn't come in multiples of four. Or two. Or any number, really. Instead, it's dependent on the code (and as such, tabs are out). As the book states:

The basic rule is that code that is part of an expression should be indented under the beginning of that expression, even when the beginning of the expression is not at the leftmost margin. Furthermore, parts of the expression that are grouped should be indented to the same level.

To make that clearer, let's have an example:

calc x =
  let y = x ^ 2
      z = x - 1
   in x + y + z

Notice how all the variable assignments line up and how the in block (or line in this case) is indented one space more than the let block. Like many languages, Haskell has multiple valid ways to structure your code, so just find a style you like and stick to it. The key thing is: don't freak out if the indentation isn't what you expect at first.

Point two: capitalization matters. Haskell is quite strict about this and won't compile unless you follow the rules. Functions and variables start with lowercase letters (and conventionally use camelCase for longer identifiers), while types and type constructors (e.g. Bool and its variants True and False, Int, etc.) are captitalized (and use what's sometimes known as PascalCase).

Point three: Haskell is 'lazy', or 'lazily evaluated' or 'non-strict'. What this means is that Haskell won't evaluate anything until it really needs to. This is why we can work with infinite lists with no problems---unless you try and consume the whole thing, of course.

Remember when we talked about beta normal form and beta reduction in the last chapter? Haskell doesn't evaluate everything to normal form by default. Instead, it evaluates to what is known as weak head normal form. Because of referential transparency, it knows that it can compute everything on demand, so until a value is required from an expression, the expression will be left unevaluated. Imagine it's a pointer to an expression instead of a value, which can be lazily evaluated when it's first used.

Finally, on comment syntax: Haskell single-line comments start with a double dash (a literal --, not the Nintendo kart racing kind), while multiline comments are put between {- and -}.

Functions

Now, let's have a closer look at function definitions. A function definition is made up of the name of the function, the parameters (separated by whitespace), an equals sign, and the expression that is the body. Example:

add x y = x + y

This function takes two arguments---~x~ and y---and returns the sum. Notice that we don't say anything about the types here. The compiler is smart enough to figure that out. If we use the REPL and the :info (or :i) command to describe this function, it tells us that the type of the expression is: Num a => a -> a -> a. It's automatically generic for all numeric types (well, all instances of the typeclass Num, but we're getting ahead of ourselves). Neat!

We won't be looking any deeper into type annotations and function signatures for the time being. They're not covered in this chapter, but do show up later; so if all the arrows in the the type signature above confuse you: don't worry.

It's also worth mentioning that, much like lambda calculus, Haskell uses curried functions. If you forgot what that means, the authors describe it like this: "In Haskell, when it seems we are passing multiple arguments to a function, we are actually applying a series of nested functions, each to one argument." Because we return a new function for each argument we apply, we can assign the result of a partially applied function to a variable and save it for later, just like we did with the lambda expressions last time.

Operators and infix functions

In addition to regular functions, Haskell also has operators. These are a mix of what you'd expect from any programming language (+, -, *, /, etc.) and more exotic ones (>>=, <$>). In fact, Haskell lets you define your own operators too. This might sound unconventional to some, but when you realize that a binary operator is really just a binary function placed in between its arguments, it makes a lot of sense.

See, binary operators are what we call infix, which means that they go in between their arguments. But operators aren't the only things that can be infix: Any binary function in Haskell can be used as an infix function if you wrap it in backticks (e.g. div 2 2 becomes 2 `div` 2).

Using partially applied infix operators is called sectioning. It's briefly mentioned in the book and there's also a very easily understood article on the Haskell wiki on it, so let's use an extract from the latter to clarify:

Essentially, you only give one of the arguments to the infix operator, and it represents a function which intuitively takes an argument and puts it on the "missing" side of the infix operator. - ~(2^)~ (left section) is equivalent to ~(^) 2~, or more verbosely ~\x -> 2 ^ x~ - ~(^2)~ (right section) is equivalent to ~flip (^) 2~, or more verbosely ~\x -> x ^ 2~

When working with operators, it's important to keep in mind the precedence rules. Most of us probably know that multiplication has higher precedence than addition, for instance---which is why $1+3*5$ is $16$ and not $20$---but it's not always immediately obvious for all operators. In Haskell, precedence is defined as a number between 0 and 9, where higher numbers denote higher precedence. If you ever wonder about a specific operator, you can use the REPL's :info command to get information about precedence for any operator (e.g. :i (*)).

Most of the operators introduced in this chapter are pretty self-explanatory---don't worry, the weird ones I referenced before aren't mentioned---but there is one that is singled out into its own little section: $. This operator is a little special in that it has a precedence of 0 and that it is defined as f $ a = f a.

What?

Indeed, it doesn't actually do anything, but it's all about the precedence. This allows us to use this operator to write an expression (a) after it that will get fed into the expression before it (f).

Err ... okay? So what's the point?

Oh, it helps us avoid parentheses and a lot of nesting of functions. It's very useful for composition. A contrived example would be:

-- is the result of the calculation even?

-- without the $ symbol
even (2 + 2)

-- with the $ symbol
even $ 2 + 2

It probably still seems pretty pointless, but you'll grow to appreciate it as we progress further. Trust me.

Assigning variables in expressions

As we saw in the first section, you can assign variables with a let expression, but there is another way too: the where declaration.

let introduces an expression, and can thus be used wherever you can use expressions, while where is a syntactic construct, and is only valid in certain parts of the code. If you're in a function, though, both will serve you just fine. These two functions evaluate to the same result:

letFunction x =
  let y = x + 1
   in y + x

whereFunction x = y + x
  where
    y = x + 1

Apart from mentioning that a let expression is just that, an expression, and that where is a declaration, the book doesn't go into any more details of when to prefer one over the other, so neither will we. However, if you're interested, I encourage you to check out the Let vs. Where article on the Haskell wiki.

Definitions

Definition time! This is a selection of the definitions from the end of the chapter, leaving out the definitions not related to what we have touched on.

Expression: Anything that conforms to the Haskell syntax and that can be reduced to a result. In theory, even constants that can not be reduced (such as the number $1$) are expressions, but these are generally referred to as values in common parlance.
Value: A value is an expression that can not be reduced further.
Function: A mathematical object that maps a set of inputs (the domain) to a set of outputs (the codomain). A transformer of values. It might be interesting to note that the book also mentions that functions can be described as a list of ordered pairs of their inputs and corresponding outputs: Example: A function (^2) (a simple square function) defined for natural numbers would start with the entries (0, 0), (1, 1), (2, 4), (3, 9)
Infix notation: A style of notation where the operator is placed between the operands.
Operators: In Haskell: Functions that are infix by default. Must use symbols only.

Conclusion and next time

Not bad! We've seen some actual code this time and played around a bit at the REPL. We learned a bit about the basic syntax and about how operators and infix notation work, about sectioning and the $ operator.

Next time, we'll be looking at the String type and how it is implemented in Haskell.

See you then!

Building a custom NixOS installer

Mon, 08 Jul 2019 10:45:52 GMT

Just a few days ago, I finally brought my old MacBook Pro (mid-2012 retina) back from the dead and, naturally, wanted to install Linux on it. My main distro being NixOS, that was the obvious choice. However, when I booted into the installer, I found that I had no networking available. After some digging, I found that it was because it needs proprietary Broadcom wi-fi drivers to connect to anything.

The standard NixOS installer is very dependent on an internet connection to download packages and configure your system, so how do you deal with that? Without a wi-fi connection (and no ethernet ports), it's not like I can just download the drivers either. Luckily, NixOS lets you build your own custom installer ISOs with a selection of packages available.

I found this GitHub issue that discusses solutions and provides a custom image you can use. The problem is that the provided image is for NixOS 17.03, which uses an older version of Nix (pre-2.0). At first I figured I could just perform the update during the installation or after, but due to the Nix version bump, I was unable to upgrade. Obviously, I had to take matters into my own hands.

Now that I knew I could build my own custom ISOs, I went looking for documentation on it and found the section in the manual and the wiki article. They're both pretty short, but that's because they don't need to be any longer: It's surprisingly easy and straightforward.

Simply create a Nix file that imports their installers and specifiy what packages you want available and any changes you want to make to the system.

The base config file they provide is this:

# This module defines a small NixOS installation CD.  It does not
# contain any graphical stuff.
{config, pkgs, ...}:
{
  imports = [
    <nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix>

    # Provide an initial copy of the NixOS channel so that the user
    # doesn't need to run "nix-channel --update" first.
    <nixpkgs/nixos/modules/installer/cd-dvd/channel.nix>
  ];
}

We can extend this to add our proprietary drivers and even additional packages to have available in the installer. So if you prefer the Fish shell to Bash or want to have access to Git? Yeah, we can do that.

{config, pkgs, ...}:
{
  imports = [
    <nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix>
    <nixpkgs/nixos/modules/installer/cd-dvd/channel.nix>
  ];

  # configure proprietary drivers
  nixpkgs.config.allowUnfree = true;
  boot.initrd.kernelModules = [ "wl" ];
  boot.kernelModules = [ "kvm-intel" "wl" ];
  boot.extraModulePackages = [ config.boot.kernelPackages.broadcom_sta ];

  # programs that should be available in the installer
  environment.systemPackages = with pkgs; [
    fish
    git
  ];
}

Building the image is a simple command. Assuming your file is called iso.nix:

nix-build '<nixpkgs/nixos>' -A config.system.build.isoImage -I nixos-config=iso.nix

And that's it. Now, just flash it onto a drive or burn it onto a disc and you're off to the races.

Before rounding out this post, there's a few things I think it's worth to be aware of, even if we won't look too much into them:

The wiki states that you don't need an internet connection when using a custom ISO with packages, which makes me think that if you specify in this config all the packages you'll need (for your final install), you won't need to download them again when actually installing.

nix-build will build the image based on your system's nixos Nix channel. I use the unstable branch, so it built an image based on a NixOS 19.09 preview. You can of course change channels in the installer if you want to use a different base for the install (though this would definitely require internet access).

From what I understand, you need a NixOS system to build one of these images. So if you need one of these but don't run NixOS already, it seems you're locked into finding someone who could build the image for you. This isn't great for onboarding new people, and you'd think that it'd suffice with just the Nix package manager---honestly: it might. I don't know---but the documentation talks specifically about building using "an existing working NixOS installation". As was pointed out to me on Reddit, you need the Nix package manager to build one of these images, but not necessarily a NixOS system. However, it was also mentioned that it does probably have to be a Linux system, so Nix on macOS likely wouldn't work.

And with that: goodbye!

Emacs, VSCode, and me

Mon, 01 Jul 2019 07:51:32 GMT

It was about a year ago. My life was good. I had things in order (mostly). And most importantly: I had an editor I liked. VSCode and I had been going steady for about a year. With its excellent support for Python and JS---which was most of my work in those days---I rarely ever needed to venture into new territory. It felt good. I was happy.

I was happy!

Then it all came crashing down. A good friend and I decided to hang out and program one Saturday. The sky was a clear blue, the temperature high, and summer was just around the corner. We got into his office, sat down, and started setting up. That's when it happened. Completely out of left field:

--- So I found this cool thing the other day. It's called Spacemacs.

Spacemacs?

--- Yeah, it's Emacs, but with "evil mode". So you can get your Vim bindings!

It's true. I had been seduced into the deep, dark depths of Vim about a year prior, and had never found my way out. Never tried, really. I had let it consume me, and there was no way I'd let it go. Not for the world. Of course I had been using the VSCode Vim plugin, but while it had given me most of the functionality I wanted, I did sometimes find myself longing for something more. So I thought I ought to at least give this Spacemacs thing a go.

With one eyebrow slightly cocked, I went ahead and downloaded Spacemacs. I had tried to run Emacs previously, but had been met with that sheer, white light that comes as the default, and had been unable to figure out how to even close the program without resorting to my mouse. Naturally, I was skeptical.

So I launch it, resist the white light as I choose 'evil mode' and watch the window turn dark. The logo appears. It says it's loading. Fetching packages. Eventually, the words "Spacemacs is ready" appear at the bottom of the window. I get it up and running and do some basic programming. I don't know what it is, but something just feels ... good. I spend the rest of the day with Spacemacs, and when the time comes, I don't want to let it go. I go home, dreaming of what I'll do to it. VSCode is nowhere to be found.

That weekend, neither Emacs nor I left my apartment. We were consumed in the fiery throes of passion and configuration, and when the sun rose the following Monday, I knew my heart had been turned.

In truth, it would be another few weeks before I left VSCode completely and migrated to Emacs even at work, but it was coming. Oh, it was coming. To this day, I still stay faithful to Emacs for most of my endeavors, only venturing out when there is something I can't find proper support for (and even then, it's just a matter of time before I take matters into my own hands!).

What I love about it

The extensibility: The fact that you can make it do pretty much whatever you want to is nothing short of amazing. I have never seen an editor this powerful and versatile.
It's the 'self-documenting editor': Everything in Emacs is documented. You ever wonder what a key does? C-h k <key (or combination)> has got you covered. You wonder what the value of a variable is? C-h v <variable name>. Want help with Emacs? C-h C-h. Boom.
EXWM: The Emacs X Window Manager. Just recently, I finally got it set up to the point where Emacs now manages all of my windows (yes, for all applications) for me. How many other editors can do that? Your move, /'Code/.
Emacs Lisp: I think Lisp is a pretty cool guy. He lets you configure emacs down to the bone. Much like the monarch of beef patties, he let's you 'have it your way', and honestly, that's the way I like it.

What I'm not crazy about

Its single threaded nature: This is usually not much of an issue, unless you somehow end up freezing the main thread, which can happen. In that case, best restart your editor.
Lacklustre support for certain languages: While most languages I use are very well supported in Emacs, I struggle with others. In particular, Microsoft's .NET languages have been hard to get set up properly.
Emacs Lisp: Yeah, it's back. Mostly because I don't really know Lisp yet (relax; it's on my todo list). Until you're comfortable with it, it makes changing your config just that little bit trickier.

What I miss from the VSCode days

So it's not all roses. There are some things I miss from VSCode.

The ease of use: I can't really that the learning curve is steeper; getting started with VSCode is just so easy. Also, installing extensions is a piece of cake.
The massive amount of plugins: Not that Emacs doesn't have packages, but most communities I see seem to have settled on VSCode as their darling, so that's just where you're going to see the most activity and up-to-date plugins in a lot of languages.
The amount of documentation: I mentioned that Emacs is self-documenting. This is great. But it's also ... over 40 years old (1976?! Gee!)! So finding documentation online is tricky in comparison.
The community: Everybody loves ~~Raymond~~ VSCode. That's great and makes it super easy to find articles on how to set it up and configure it and so on. Need help? Ask anyone. Chances are they're using VSCode. The Emacs community is great too, but much, much smaller.

In the end, do I regret it? Not for a second. I think it's a worthwhile tradeoff if you can take the initial learning curve and are willing to put some time into your editor. So if you feel inspired, should you do it? All I can say is that if you pay the upfront cost, Emacs will return it tenfold.

QWERTY to Dvorak

Mon, 24 Jun 2019 05:55:24 GMT

Ever thought about changing up your keyboard layout? Maybe something as small as going from an ISO layout to an ANSI one (hint: tall and thin to short and fat return key), or from Norwegian (or whichever your language of choice is) to American QWERTY? Maybe something more comprehensive, like learning a whole new layout, such as Dvorak, Colemak, or Workman? Well, I've done all of the above, and while the smaller changes can offer some pretty significant benefits, they're pretty easy to get through. The layout-changes are more interesting. I'd like to talk briefly about how I went from QWERTY to Dvorak and what I learned along the way.

First off: you might wonder why I'd do that. I'd love to tell you that I had some grand epiphany and suddenly realized that the QWERTY life wasn't worth living or that I discovered that Mr Dvorak himself was actually my great-great-grandfather; but nah, it was actually just an offhand comment someone made.

See, I had just recently started learning Vim and was talking about it with some fellow students when one of them jokingly suggested I do it in Dvorak instead. As if it wasn't challenging enough already, right? I laughed it off at first, but it sparked something in me. Thirty minutes and some googling later and I had made my mind up. I didn't know whether I was gonna make the switch, but I knew I had to check it out to see what all the fuss was about.

Before starting my degree in computer science (about two years prior to that fateful comment), I had never been much into computers and never properly learned to type, so my keyboard skills weren't exactly award-winning: I wasn't very fast (though not remarkably slow either), and I quite often had to look at the keyboard to find the key I was looking for.

Taking the above into consideration, I realized this was a chance to learn proper form and improve my typing skills; so after some investigation, I decided I was gonna go for it. And seeing this as a chance to pick a layout designed specifically for my purposes, I decided to go for the Programmer Dvorak layout.

Pros and cons

Now, about two years and thousands of key strokes later, I can take stock of what I have gained from switching and what issues arose because of it.

Benefits

Proper touch typing form: This might not seem like a big one, and from what I understand, it's not necessarily the holy grail that some people make it out to be, but it has been a big boon to me. I can't comment on it being better or more efficient than any other technique, but it sure beats my original, awkward, four-fingered approach.

Speed: My typing speed has gone up dramatically. I don't know how many words per minute I could get out of my old QWERTY skills, but it wasn't breaking any necks. I reckon that after about 3--4 months of Dvorak I had already caught up.

Accuracy: A nice feature of learning Dvorak on keyboards where you can't change the physical layout is that you need to memorize the layout because---with the exception of 'A' and 'M'---no letters are placed in the same spot. A side effect of this is that you can't look down on the keyboard to find a key. In fact, that'll probably just confuse you further.

Comfort: There's something about the alternating hand-motion of the Dvorak layout that I really like. In fact, while learning it, I'd often come across sequences of letters that just felt good to write.

It might be more ergonomic: You often hear that the Dvorak layout is more ergonomic and better for avoiding RSI and the like. I can't comment on the veracity of these claims, but it sounds plausible.

Drawbacks

The ramp-up time: It took a while until I got up to a respectable speed. This period where things just don't move as fast as you want can be annoying and mentally taxing.

Extra set-up time: Dvorak is by no means the default layout for any system (as far as I'm aware), and so---if you're using a software-defined layout---you'll always need to go through some extra steps to set it up properly. Even more so if you go for some esoteric layout like Programmer Dvorak (though Linux comes with this available out of the box). This can of course be offset by going for a programmable keyboard (or getting a Dvorak keyboard).

Pair programming: If you're getting help from a coworker, keep a QWERTY layout handy. It's okay to have a little laugh when they stand there, though, perplexed at why nothing seems to work.

Losing touch with QWERTY: When I switched, I switched completely. As a result of this, every time I use a QWERTY keyboard now, I need to go looking for keys and I'm super slow. Though, for some reason, this does not apply to phone keyboards, where Dvorak feels awkward but QWERTY is perfectly fine.

Is it worth it?

Absolutely.

Seeing as I spend most of my day using a keyboard, it makes sense to invest in building skills. This might be 'survivor bias' talking, but I got so much out of it that I'd recommend it to anyone who's interested. At least have a little sniff. If you've got the time and energy, it can be very fun and offer a real feeling of accomplishment. Plus, it'll keep your brain active!

If that was all you wanted to know, you can stop reading now, but if you're looking for some tips on how to learn a new layout, let's finish it off with what I found worked for me.

How to learn

After picking my target layout (Programmer Dvorak), I had to find out how to learn. While it might be tempting to just sit down with a key map next to you (or shuffle the physical keys around if you can do that) and go for it, this is quite likely to be counterproductive for a number of different reasons; the biggest ones being that you're more likely to slip out of proper technique and that it's less efficient (based on my empirical studies with a sample size of one) when it comes to memorizing layout.

So what do I recommend? Glad you asked! I've got some tips for ya:

Use typing training software: These will typically introduce you to one new letter for each hand at a time and slowly work their way through the full layout, often focusing on common patterns. If you want tips on which one to go for, I'd recommend Typing Club; it's free, effective, and even includes some pretty interesting bits of writing to copy as you progress, so you can learn while you learn. Yo, dawg.

Take it everywhere (work, studies, whatever): Once you're comfortable enough with the position of the keys to not feel like you need to look up where a key is before pressing it (though thinking is okay), start using it all the time. Once you've got the basics down, the most important thing is to get exposure and experience with it. Depending on your situation, this can be more or less acceptable, so keep important deadlines and the like in mind, but don't let that stop you. Keep your other layout around so you can swap back in case of emergency (or pair programming!).

Don't give up!: Even when it seems hard and you're tired of making the same typo for the two hundred and twenty-seventh time this hour, stick with it. Take a deep breath, have a cup of tea, and clear your mind. Go back to the keyboard and start again. Consistency is key. This goes doubly for memorizing fingerings and modified keys: Make sure you always hit the key with the correct finger. Some keys might be awkward at first, but you'll get used to it eventually.

Accurate is the new fast: It doesn't matter how fast you are if you hit the wrong key half the time. It's important to focus on accuracy, especially at the start. Speed will come.

If you decide to embark on this exciting journey, be aware: it won't be easy and the learning curve is pretty steep; but it's not as difficult as you might think. Time and tenacity is all you need: Once you start, you simply need to stick with it.

Good luck!

Let's read: Haskell Programming from First Principles, pt I

Sun, 16 Jun 2019 16:36:21 GMT

So you wanna learn Haskell, huh? Well, you've come to the right place. Maybe.

This is the first in a series of posts where I try to firm up my understanding of the Haskell programming language by going through the book Haskell Programming from First Principles by Christopher Allen and Julie Moronuki (ISBN: 9781945388033). I'll work my way through the book and write a summary of the most important bits from each chapter as I go along.

A little bit of background

For a while now, I've been very interested in Haskell and using it whenever I can for my side projects, but I've also been very busy, and haven't had as much time to play with it as I'd wish. I read Learn You a Haskell for Great Good and far too many monad tutorials when I first started out, but since then, it's mostly been a just-in-time sort of process when looking for answers to problems I've come across. So, while I feel I have a basic grasp of the language, some of the more advanced features still escape me.

This book has been on my reading list for a while, but I've not made a serious effort to get through it before. Now, though, I have the time and I have the motivation. It's time to learn Haskell. For real.

Alright, here we go. Deep breaths.

Deep breaths.

Chapter 1: All you need is Lambda

.... aaaand there's not a single line of Haskell in this whole chapter.

I'm sorry to disappoint you, dear reader, but this chapter is entirely about the theoretical underpinnings of the Haskell language: the lambda calculus. But that doesn't mean that we can't make anything of it! Quite the opposite, in fact.

So put on your math hat and do some light stretches.

...

Ready?

Lambda calculus

While a thorough introduction to the lambda calculus is outside the scope of this post (not to mention, not something I'm qualified to do), there are certain concepts that would aid our learning, so let's try a basic introduction.

In the summary of the first chapter, the lambda calculus is defined as "a formal system for expressing programs in terms of abstraction and application." Wow. Such clarity.

Let's see if we can find something more digestible.

Earlier on, the book defines a calculus as a method of calculation or reasoning. In other words, it's a system we can use to solve problems. The lambda calculus is but one process for formalizing one of these methods or systems. In other words, it's a way of thinking about and calculating certain problems and it gives you a set of tools that you can use to solve these.

So why are we looking at the lambda calculus before getting into the code? Well, because functional programming is made up of expressions (values, variables, functions) that model typical mathematical behavior, and understanding how they interact will help us understand how the language works later. Or something.

That's about as much as you'll need to get started. We'll cover anything else as and when it comes up.

Lambda terms

Let's talk about lambda terms, the three basic components of the lambda calculus. They are:

Variables: A variable is just something you can assign a value to. It has no meaning or value, but only exists as potential input to a function. Simple.
Abstractions: An abstraction is a function. It is a lambda term that has a head (a lambda (𝜆) followed by a variable name) and a body (another expression) and is applied to an argument. An argument is an input value.
Expressions: An expression can be a variable name, an abstraction, or any combination of those two.

A function in the lambda calculus is nothing special. It's just an expression that can be applied to another expression and that returns yet another expression. If this sounds confusing, keep in mind that expressions can be both values and other functions.

A lambda abstraction (that is, a function) looks like this:

$𝜆x.x$

where everything from the $𝜆$ to the $.$ ($𝜆x.$) is known as the head, and the body is what comes after the $.$ ($x$ in this case). The head binds variables to be used in the body (i.e. function parameters).

Now, an interesting thing to note is that each lambda can only take one argument. Does that mean that in lambda calculus you can only have functions that take a single argument?

Not quite. See, functions that take multiple arguments are actually just nested heads:

$𝜆x.𝜆y.xy$

But this is quite verbose, so we'll simplify it by simply writing

$𝜆xy.xy$

It might seem trivial but it is a very important property of lambdas, known as currying (after Haskell Curry (no, really)).

This is also what allows for partial application. Because an expression can be partially evaluated, we can also apply it to one value and get a new function back.

For a concrete example, let's say we have a function $mul$ that takes two arguments and returns their product:

$mul=𝜆xy.x*y$

If we want to create a function that doubles a value, we can do that by applying $mul$ to $2$:

$double=mul(2)$

or, written out as the result of the above expression:

$double=𝜆y.2*y$

The $double$ function expects one argument and will return the double of whatever it receives and is defined as a partially applied $mul$.

Glossary

How lambda terms interact is probably the most important part of this chapter, but let's have a look at some other terms mentioned in the chapter---mostly because you can use them to sounds smart on internet message boards, but also because they help us understand what we're talking about when we're talking about the lambda calculus.

Alpha equivalence

Two expressions are alpha equivalent if they are the same function, even if they are written with different symbols. The following functions are alpha equivalent:

$𝜆a.a$
$𝜆b.b$
$𝜆c.c$

Application

How we evaluate/reduce lambdas.

Beta normal form

When an expression cannot be beta reduced any further, either because there are no more application to do (no more lambdas), or because there are no free variables to apply the function to.

Beta reduction

Evaluating expressions. This is what happens when you go from the expression

$𝜆a.a (2)$ and apply the lambda to the number, thereby ending up with $2$

Bound variables: Variables that are declared in the head of a function (y'know, the parameters)
Codomain: The set of possible outputs of a function.
Combinators: A lambda term with no free variables. In other words, all variables are in the head.
Divergence: Any function that never terminates is divergent (hint: recursive functions that never exit, loops of the while(true) type).
Domain: The set of possible inputs to a function.
Free variables: Variables that exist in a function body that are not defined in the head, such as the $y$ in this expression:

$𝜆x.xy$

Lambda: The Greek letter 𝜆. Used to indicate abstractions (functions) and bind variables.
Lambda abstraction: An anonymous function or lambda term. You can think of the head as an abstraction for the body, i.e. something we can put in place of the computation.
Referential transparency (/aka/ purity): This means that given the same input, a function will always produce the same output.

Conclusion

While this may seem a strange place to start, I can see the logic in doing it, and getting a better understanding of the lambda calculus is helpful anyway, so I'm fine with starting it this way.

Anyway, there should be more code in the later chapters. Until next time!

Command Line Control: awk

Sat, 08 Jun 2019 15:19:18 GMT

Unless you spend a whole lot of time working primarily on the command line, awk is one of those commands you've likely come across a few times, but never really learned how to use properly. Similar to my recent adventures with xargs, though, I recently came across a little use case where I could benefit from using it, so I decided to sit down and look into it.

Even more so than with xargs, this is a really powerful tool, but let's focus on the basics and see what we can make of it.

What is `awk`?

awk is a command that processes text using programs written in the AWK language.

To understand what it can do, understanding the language it uses is probably a good place to start. Let's use the words of Alfred Aho---one of the creators of the AWK language---from a 2008 /Computerworld/ interview to get a rough grasp of what it is:

AWK is a language for processing files of text. A file is treated as a sequence of records, and by default each line is a record. Each line is broken up into a sequence of fields, so we can think of the first word in a line as the first field, the second word as the second field, and so on. An AWK program is of a sequence of pattern-action statements. AWK reads the input a line at a time. A line is scanned for each pattern in the program, and for each pattern that matches, the associated action is executed.

That may be a bit dense, so let's take it apart:

"AWK is a language for processing [..] text": AWK is a domain-specific language (DSL) focused on text processing. The awk command expects that its first argument is a script or a string in this language.
"AWK reads the input a line at a time": AWK is line-oriented and works through the input line by line. (It's actually record-oriented, but the default separator is a newline character, so this is the same thing by default.)
"Each line is broken into a sequence of fields": Each word in a line maps to a field. These fields are accessed with the $ operator, e.g. $1 for the first word, $2 for the second, and so on. $0 is the whole line. By default, fields are delimited by whitespace (which is why I've called them words) but this can be customized.
"An AWK program is of a sequence of pattern-action statements": This means it's a sequence of predicates with actions. If a predicate evaluates to true, perform the specified action. If no predicate is specified, it will always evaluate to true, and if no action is specified, it will default to printing the whole line.

Huh? Yeah, this is still a bit confusing, but maybe some examples will make it clearer:

Patterns: This is a predicate to check each line against. It usually takes the form of a regex enclosed in forward slashes. /foo/, /b[ar]?/, /^baz/, /(fizz|buzz)$/ are all examples. Most of the regex skills you have will be applicable here (character sets, character classes, alternations, etc.).

You can also match specific fields against a regex. Only want to match lines where the second field contains 'cheese'? $2 ~ /cheese/

The pattern can also consist of functions and comparisons; so if you wish to act only on lines that aren't empty: length > 0

If no pattern is given, every line will match.

Actions: These are commands telling awk what to execute. They are enclosed in curly braces ({}). This is where you might instruct awk to print a certain field of the string---~print $3~, for instance---or increment a counter if you're counting words: word_count += NF (yeah, I'll get to what NF means in a bit).

If no action is given, awk will print the matching line.

Basic auth `awk`

That's a quick overview of how the language is structured. Before we start playing with it, let's explore some of the features.

Built-in variables

awk has a number built-in variables, and while I won't cover all of them, these are the ones that I've found the most useful:

~NR~: Gives you the line number of the current match. Could be used for adding line numbers to an input file: ~awk '{print NR, $0}'~. Or maybe looking for lines that contain 'ice cream' is more your speed: ~awk 'ice cream {print NR}'~.

~NF~: This is the number of fields in a line. Useful if you're looking for the last field, either for finding out how many fields are in a line or for seeing if it contains a pattern: awk '$NF ~ /out/ {print NR}

~FS~: The field separator value. This is what awk will use to split each line into fields. By default this is whitespace. If you have a file full of comma-separated values and want to split each line on commas instead of whitespace: BEGIN {FS=","}

~RS~: This is the line ('record') equivalent of the field separator. The default is \n. Say you want to print your PATH over multiple lines: ~echo $PATH | awk 'BEGIN {RS=":"} {print}'~

`BEGIN` and `END`

awk lets you supply commands to be executed at the start and end of a script by using BEGIN and END. While they may seem like it, they're not really special at all. Instead, think of them as being patterns that evaluate to true before anything is evaluated and after everything is evaluated, respectively.

BEGIN could be used to set the field separator or initialize a variable.

END is useful for printing out results accrued through the life of the program, such as a word count. If we bring back our word counting example: ~awk '{word_count += NF} END {print word_count}'~

Functions and conditionals

Like most languages, AWK has a number of built-in functions (such as the print and length functions we saw earlier) and also lets you define your own functions if you so please. This is probably overkill for most trivial operations but could come in handy in certain cases.

And AWK has conditionals too! While you can use the if else construct within actions, I'd like to highlight that you can do conditional statements based on the supplied pattern. I.e. awk '/foo/ {print "FOO"} {print "bar"}'~ will print 'FOO' for lines that match ~/foo/ and 'bar' for lines that don't.

Applications

Now that we've got some idea how it works, what can we use it for? Let's look at some sample applications for it:

Sorting lines in a file by length

Here's a fun little application: Let's take a file, count the number of characters in each line, and then sort it based on the number of characters:

awk '{print length, $0}' <file> | sort -n

And if you want to exclude empty lines, try this:

awk 'length > 0 {print length, $0}' <file> | sort -n

Friendly `PATH`

This is the same one that was listed above, and is useful if you're looking for certain entries that can easily get lost when the path is on one line:

# bash, zsh
echo $PATH | awk 'BEGIN {RS=":"} {print}'

# fish
echo $PATH | awk 'BEGIN {RS=" "} {print}'

Counting words

Another example that was used previously. Count the words in a file or any input stream:

awk '{word_count += NF} END {print word_count}'

Parsing git logs

Maybe your team has agreed that all commit messages should start with #<task_no> if it relates to a task and #-- if it doesn't. To find all commits that relate to a specific task---say #42---we could do this:

git log --oneline | awk '/#42/ {print $1}'

Or how about finding the ratio of commits that belong to a task versus those that don't?

git log --oneline --no-decorate | \
awk '$2 ~ /#[0-9]+/ {task += 1} {notask +=1} \
  END {printf "Tasks: %d\nNo-tasks: %d\nTask to no-task ratio: %f", \
  task, notask, task / notask }'

(Yeah, the printf function works pretty much the same as in C---so much so that I haven't looked it up yet!)

Killing processes

This is actually what triggered me to look into awk and what eventually led to this post being brought into existence.

There are a number of ways to kill applications from the command line, and for a while, my default fallback has been the classic ps aux | grep <app> combo.

While this lists out all the relevant processes and their process IDs (PIDs), you then have to manually copy the PID and put it into the kill command to shut it down. This is annoying at best, and gets even worse if the process has spawned children that we want to take down as well.

How do we deal with this? Well:

ps aux | awk '/<app>/ {print $2}' | xargs kill

ps lists all processes on the system

awk then goes over every line that contains the application name, extracts the second whitespace-delimited word---which in this case is the PID---and prints that.

For the last stretch, we use xargs to feed the PIDs into the kill command, thus killing all the processes.

This works just fine, though it comes with the caveat that it'll also try and kill the awk process (because the application name is part of the command, so it gets listed by ps), but that's only a minor annoyance, and I'll leave fixing that as an exercise for the reader.

Now, I'm sure you can make killall do something quite similar to this, but I've found this way to be more effective (by which I mean: closer to what I expect).

Closing up

Phew. We've learned a lot today, and this post grew much longer than what I had originally imagined---about four or five times longer---but it's been quite the journey and I'm glad you were there with me, partner. I hope you've gleaned some new insight too and that you'll find some application for this in your day-to-day.

See you next time!

Hakyll CI builds in one fifth of the time

Sun, 02 Jun 2019 10:50:37 GMT

As I've mentioned before, one of the big pain points relating to this blogging business is deployment time: Haskell is slow to compile and Hakyll has multiple large dependencies, so the builds would initially take up to an hour. Yeah, you read that right. 60 minutes 😱. Something goes wrong towards the end of the build? Sucks to be you.

Thanks to Saksham Sharma and their post on speeding up Haskell CI builds, however, I have been able to bring it down to 7-8 minutes in GitLab's CI/CD systems (excluding time spent waiting for runners to spin up etc.). That said, it wasn't quite as easy as I'd hoped it would be (when is it ever?): Due to how Stack and Nix interact, building of the site would crash when it ran into UTF-8-encoded characters. Not cool.

Let's fix it.

Step 1: using an image with Hakyll pre-built

In Sharma's post, they mention that they've created an image that you can use for your build systems. The simplest version would look a little something like this (freely updated from their minimal configuration example):

image: sakshamsharma/docker-hakyll:v3

pages:
  script:
    - stack build
    - stack exec site build
  # ... rest of stage omitted

An important thing to note is that your stack config's resolver must match the one used in the Docker image, otherwise the build system would have to recompile Hakyll and its dependencies for your version, taking us back to the hour-long builds.

For v3, the resolver is lts-12.21, so make sure your project's stack.yaml contains the following line:

resolver: lts-12.21

If this works for you and is all you need: great! If it doesn't and you get errors talking about invalid byte sequences like the one below: don't panic. I'll sort you out.

Compiling
  [ERROR] ./about.rst: hGetContents: invalid argument (invalid byte sequence)

Step 2: This one weird trick

As described in [[https://github.com/commercialhaskell/stack/issues/2358#issuecomment-466818624][this GitHub issue]], a fix for the above error is available in Stack's master branch and as of Stack v2.1---the release candidate for which was released while I was writing this post---will be included with the tool.

From the [[https://github.com/commercialhaskell/stack/releases/tag/v2.1.0.1][release notes for the release candidate]]: "Use en_US.UTF-8 locale by default in pure Nix mode so programs won't crash because of Unicode in their output".

So if you're using Stack v2.1 or later, the steps outlined in this section should not be necessary.

As evidenced by a fair few GitHub issues^[1], this is something that a number of users run into and it might be difficult to troubleshoot, but what it boils down to is this: When running Stack in Nix mode it defaults to building in pure mode. This isolates the build environment by removing environment variables and other things on your system that could influence the build and lead to a lack of reproducibility. This is usually a good thing, but it also unsets the LANG variable, which Stack relies on to know how it should handle encodings.

Ok. So all we gotta do is re-set that variable, then? Yes. But how to do that might not be immediately apparent. You might be used to running shell commands like this:

MY_VAR="my-value" ls -lah

But this won't work with Stack, because it'll still isolate the environment. What you can do, however, is to use the --no-nix-pure option. This tells Stack not to isolate the build environment, so you'll still be able to access external variables. Here's an extract from my current build file that does just that:

image: sakshamsharma/docker-hakyll:v3
script:
  - stack build
  - stack exec --no-nix-pure site build

This works perfectly on GitLab's CI runners, but if this still doesn't solve your issue, you might want to check what the locale is actually set to by using the locale shell command. The output should look something like this:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

If the output doesn't show a UTF-8 format, that seems like a good place to start (I'd try EXPORT LANG=en_US.UTF-8 before running the Stack commands), but now we're wading out past the scope of this post, so you're gonna have to go it on your own, I'm afraid. Sorry, kiddo.

Wrapping up

And that's it! Simple, but not immediately obvious. It's likely that a similar approach---the prepared Nix container---would work for other Haskell projects as well, though I can't say for certain one way or the other.

Footnotes

^[1] A selection of GitHub issues relating to the unicode problem:

xargs and the unruly tags

Sat, 25 May 2019 14:01:57 GMT

I thought I was really clever when I configured my CI/CD pipeline to tag commits that got deployed and push the tags back into the repo, but I'm rarely as clever as I like to think: I had forgotten to put the proper checks in place to avoid these tag pushes triggering subsequent runs of the pipeline, and things got a little ... out of hand.

I'd gone to bed just after pushing an update, and when I arose to check on it, I found that the deploy tagging stage had been running over and over and over and over and ... you get the point. Thankfully, it had failed after about 130 rounds, so it could have been a lot worse, but I was left with a large amount of useless and unwanted tags in the remote repo.

So how do you fix something like this? Yup, xargs to the rescue!

Where there's a will ...

At first, I didn't really know how I'd go about it. I was hoping git would have some nice, built-in functionality for mass-deleting remote tags, but while I have found in retrospect that it does (see the postmortem), I couldn't find it at the time.

However, because all the tags were for a specific commit, I did know that I could list all the relevant tags separated by newlines, using git tag --contains <SHA>.

So, with some helpful advice from Stack Overflow and this guy, I constructed this little command which sorted me out just fine:

git tag --contains 9216e97ce7e66090f79eba4d1abe6548d72dd638 \
| xargs -I % git push origin :refs/tags/%

Now, I'd come across xargs before, even done the ol' copying and pasting from Stack Overflow trick, but it had always looked really complicated and no-one had ever told me why I'd need it or what it does; so I just carried on in blissful ignorance. Not this time, though. It was time to figure out what was going on.

Groking `xargs`

The way xargs was sold to me was: "execute a command for each item in a list". It's actually more powerful than that, but that's a great place to start.

Let's use the man page to find out what that -I % bit means :

~-I~: replace-str: "Replace occurrences in the initial-arguments with names read from standard input"

The string to use to indicate where to place arguments in the command to run. In the command above, we chose to use %, but you're not limited to this.

Similar to printf and format strings in general, this places your arguments at your desired place in the command. In our case it both limits us to using one argument (tag) at a time, and it lets us append it to :refs/tags/ without being separated by a space.

That means that in the above snippet, xargs would, for each tag listed, run the command git push origin :refs/tags/<tag_name>, which pushes that tag with an empty reference, thereby deleting it.

If all you want is to put the argument at the end of the command, you can even do without the -I. Say you want to recursively delete all the .swp files in a directory:

find -name "*.swp" | xargs rm

Be aware, though, that without either using a -I or -n (to limit the number of arguments to use for each command), xargs will split the list you give it into sizeable chunks and apply as many arguments to the command as it can each time. That means that in this case, it'd likely end up looking something like this:

rm a.swp b.swp c.swp ...

which is usually fine and what you want, but keep this in mind for when it isn't.

This is only scratching the surface of what xargs can do, but it's enough to make it do some pretty heavy lifting. It might not be something to reach for very often, but for when you do need it, it's a great tool to have in your belt.

Postmortem ⚰️

<<postmortem>> Now, you might have noticed that I did a git push for each tag that I was deleting, and you might be thinking that for over a hundred tags, it must have taken quite some time. You would be right. Luckily, I was working on something else, so I could happily let it run in the background. But we can do better!

xargs has an option -P or --max-procs, which you can use to decide how many processes to run in parallel. The default is 1, but if you set it to 0, it will run as many as it can. This could have saved us quite some time, assuming git would let us run multiple push operations from the same repo at the same time. But there is an even better way:

As outlined in this Stack Overflow response, you can use a whitespace-separated list of tag names (<tags>) with git push~; so we could have run ~git push --delete origin <tags> to achieve the same outcome as deleting them one by one.

If we rewrite the command from earlier, we can both simplify it and do it all in a single push:

git tag --contains 9216e97ce7e66090f79eba4d1abe6548d72dd638 \
| xargs git push --delete origin

... yeah, that would have been a lot more efficient 😅

Docking pains

Sat, 18 May 2019 15:53:10 GMT

You know how some things are a lot more difficult than they seem? In an attempt to speed up deployments for this blog, I wanted to look into building a Docker image with Hakyll and all the required build dependencies available. To be able to do this effectively, I figured I'd need to have the ability to work with Docker locally. Turns out this was one of those things.

The goal: Enable Docker virtualisation and development on a NixOS system
Challenges: The root partition---which is where Docker stores data---keeps running out of space

In theory, it's simple:

enable Docker
configure it to store data somewhere that is not /var/lib/docker

In practice, it turns out to be a bit more difficult than expected, but don't worry: We'll figure it out together!

Enabling

According to the NixOS manual and the the Wiki article on Docker there's really not much to it: To enable Docker, all you need to do is update your configuration.nix to include

virtualisation.docker.enable = true;

As pure and simple as Nix should be.

According to the manual: "/This option enables docker, a daemon that manages linux containers. Users in the "docker" group can interact with the daemon (e.g. to start or stop containers) using the docker command line tool./"

Read that last line carefully: "/Users in the "docker" group can interact with the daemon [...]/". Yup. That means we need to make sure our user is in the correct group:

# replace thomas with the name of your user
users.extraUsers.thomas = {
  extraGroups = [
    "docker"
    # ... other groups
  ];
# ... remaining configuration
};

What isn't immediately obvious is this: You must log out and back in before this setting change takes effect. Let's repeat that to make sure we understand:

You *must* log out and back in before this setting change takes effect.

From what I can tell, this goes for any change to a user's groups, but it isn't particularly well documented anywhere. (Psst: I am not the only one to have run into this.)

But wait; there's more! This is only vaguely referenced in the manual ("/using the docker command line tool/"), but to have access to the Docker CLI, you're going to have to install Docker (pkgs.docker) for your user, either by putting it in configuration.nix's systemPackages or by using a solution such as this.

And that's it. If all you wanted to do was set up docker to run with the default configuration, you're done now. Congrats! Have a donut. You've earned it.

The space race

Ah, yes, disk space ... We've got Docker in place now, and, assuming the group change setting has taken effect, we can start playing with it. That's what I did. For a day or so. As per the usual NixOS song and dance, I wanted to change some configuration settings, so I tried to rebuild my system and got this fateful message:

No space left on device

Now, this isn't anything new. I've realized since setting up the OS, that I should have probably allocated more space for the root partition (someone once told me that NixOS "trades disk space for sanity"). "Oh, well," I thought. "Guess I have to delete some old generations again." So I ran the garbage collector. This usually frees up about 7--10GB of space, but now it was hardly removing two! I tried all the tricks that I knew of, but nothing seemed to make a difference. And then the thought struck me: "Docker is installed as a system service. That means it probably stores images system-wide too!". And indeed, after looking through the 'docks' (har har), I found that the default place Docker stores data is in /var/lib/docker.

So I killed all of my containers, deleted all of my images, and lo and behold: My root partition had suddenly lost nearly 10GB! Superb!

The next step, then, would be to figure out how to store the data somewhere else. Luckily, the docs (NixOS and Docker) are quite clear on this point: For your Docker configuration, you can specify an option, --data-root, and have the data stored there instead. In general, I prefer not to mess around with where things are stored too much, but in some cases it makes life easier (until I get around to repartitioning my drive, anyway), so I decided I'd put it under /home/docker for now. This is easily done like this:

virtualisation.docker.extraOptions = "--data-root /home/docker";

This setting means that my /home partition carries some extra data, but it's got more than enough space to deal with it.

Putting it into practice

Now, having experienced first-hand how space-hungry Docker can be, and having read through the documentation, I found that there are other options that might come in handy. For now, I decided to have the system aggressively auto-prune on a weekly basis. This should keep me from running into space issues any time soon, and if it gets annoying I can always change the settings.

At the end of this little adventure, the resulting configuration.nix should look something like this:

# Docker CLI (either put this here or in your user config)
environment.systemPackages = with pkgs; [ docker ];

# Put your user in the correct group
users.extraUsers.thomas.extraGroups = [ "docker" ];

# Set up the Docker daemon
virtualisation.docker = {
  enable = true;
  autoPrune = {
    enable = true;
    flags = ["--all"];
  };
  extraOptions = "--data-root /home/docker";
};

In summary, these are the steps needed:

Enable virtualisation.docker
Make sure your user is in the ~"docker"~ group. (Log out and back in!)
Install Docker for your user
(Optional) If, like me, you have issues with space, change the data-root to somewhere else, such as a different partition or an external drive.

So there you have it, folks! It really is quite simple ... once you figure out all the tricky parts.

Hello, World!

Tue, 07 May 2019 04:14:30 GMT

As a first little post and an introduction to the blog, I thought it might be cute to have a little overview of how it's made. This has been my first time setting up most of the surrounding architecture and I have learned more than a few things in the process (though there is, of course, lots left to learn). This post won't go into great detail about any particular points, but will serve as more of an overview of what 'the system' looks like at the time of writing.

That is to say: don't expect any brilliant insights from this post, but read on if you're interested in how I've organized things.

Hakyll

Let's start with the most important part of this whole thing, shall we? The truth is that without [[https://jaspervdj.be/hakyll/][Hakyll]], this blog would not be up and running now. I have been wanting to get into writing for a bit, but I have been lacking a platform that met my criteria:

I wanted to be able to write my posts using Emacs' [[https://orgmode.org/][Org mode]]
I wanted to be able to put them in a version controlled repo and have the blog auto-update whenever I pushed a new update.
I wanted it to be low-ish effort---at the very least I didn't want to mess with servers and so on---but I also wanted to be able to host it myself, so that I would not be dependent on some other platform.

Maybe I didn't look hard enough, but I couldn't really find any alternatives that would let me tick all these boxes. The closest was GitHub pages, which would have been fine, except that I would have had to restrict myself to markdown. ... but then a friend of mine told me about Hakyll and I found this blog post on using it with Org mode. That was all I needed to set out.

Notable changes and additions

While Hakyll comes with a lot of great features out of the box, I found that I wanted just a little bit more out of it, and these things had to be configured:

Feeds (RSS and atom): I've never used an RSS reader before, but in setting up this project I wanted to find out how the format works and ended up getting addicted myself. If you've not tried it out: you might never wanna go back. Overall, this was quite easy, though it did require some tricksy loading of posts and applying different url treatment.
Sitemap: Another thing I've never looked twice at. This project provided me a reason to understand what it is, why you might want it, and how you could set it up. Oh, and how you'd link it in your robots.txt. Pretty simple.
Pagination: For some reason, it was quite important for me to be able to have a paginated stream of all blog posts (yup, that's the blog page). Luckily, it was fairly easy to set up, though it did require me to change how I store post drafts.

Where I want to go

I think I've got it mostly where I want it now, but I'm still not happy with how I store drafts. At the moment, they're in a drafts directory under the posts directory. In an ideal world I'd be able to store them along with all other posts with the only way to tell whether they've been published or not is whether they have a published tag in their metadata. However, this causes some issues with Hakyll's build process when it tries to process unpublished posts for different parts of the site. It should be as simple as loading the correct snapshot based on a filtered group of posts, but I haven't quite gotten around to that yet.

Netlify

Overall I've been very happy with what Netlify has offered me thus far. Their documentation is great, their CLI tool is pretty nice, and they seem to have their stuff together.

External domain providers

Stupidly, I bought the domain from a different domain provider and then went to Netlify later. If I'd had the foresight to check with Netlify first, I wouldn't have had to deal with setting up name servers and so on. It hasn't been that big a deal, and at least now I know you can, but it would just have been easier if I'd gone all in on Netlify, I think. Maybe next time.

It might be worth pointing out that by changing DNS records to have it all managed by Netlify, I got https support for 'free' (when the option would have been to pay for it, had I stayed with my domain provider).

Haskell support (or lack thereof)

However, even though Netlify does offer a pretty solid service, their CI/CD system does not, at the time of writing, offer Haskell support^[1]. This means that for now, I'm stuck using an external service to build and then publish to their systems. While this isn't too bad, it does mean that I miss out on certain benefits that using their integrated system gives you, including minification and dynamic image serving.

GitLab

As Netlify doesn't support Haskell static site generators out of the box, I have to work around them and use an external CI/CD system instead. This is actually a big part of the reason why I chose GitLab for hosting this site: I've used their systems a fair bit lately and find them rather nice to work with.

For my CI/CD setup, there where a couple of rules that I wanted to have in place:

If I push an update to anything that would impact the site (styles, posts, templates), the system should deploy.
If I push changes that include new posts, the system should add the appropriate timestamps to the relevant posts before deploying. The timestamp additions should be commited and pushed back to the main repo before deploying.
I should be able to manually deploy, either by going through the gitlab UI, or by pushing a commit that includes a specific tag ([deploy])
I should be able to say that a commit should skip CI, even if it would normally trigger it, using a tag ([noci])

Most of this sounds pretty basic, but it was a surprising amount of effort to get things set up right. The first big snag was that GitLab doesn't have built-in support for pushing changes back to the repo from the CI pipeline. There are ways to do it, but it was surprisingly convoluted for something I would expect to be reasonably common.

The second thing, and this is probably the most pressing matter relating to this site at the moment, is that building the site for deployment takes almost an hour. From what I can tell, this is because stack (the Haskell build tool) has to download and compile Hakyll and its dependencies before compiling the site. However, there are solutions to this: The most promising one I found is the one documented in this blog post by Saksham Sharma. I did try using the provided docker image, but due to an issue with locale info being unset in pure nix shells^[2] (I suspect), the site won't build properly. I expect I'll be looking into this pretty soon.

Python helper scripts

As any good software developer, I want to automate as much of the process as possible, and the first obstacle I ran into was related to timestamping posts, both for first publish and for subsequent updates. Beyond that, I also wanted an easy way to move files from the drafts directory to the published posts, adding the required data while doing so.

~add_timestamps.py~

A pretty simple script meant to be run in CI. It goes through all the posts with 'published' tags that have changed since last push and adds the current timestamp. If a file has no value for the 'published' tag, the current time is added, else, it adds or updates the 'modified' tag with the current time.

~publish.py~

To automate moving files between directories and adding the 'published' tag to drafts, I wrote a simple script that does just that. Super easy. This led me to my first real revelation using nix-shell (still just scratching the surface here), where I could package this script and use it as a command anywhere! I was super excited until I failed at packaging it correctly due to the dependencies on hakyll.py ... This hasn't been resolved yet, but I'll figure it out at some point!

~hakyll.py~

This is just some shared functions that deal specifically with Hakyll and my specific system. Most notably: lets you get metadata as a dict, update metadata values, sort tags, decide whether a post is new or updated and whether it is published at all. Also my first time using type annotations in Python (I don't actually know whether I did it right, but the system isn't complaining so far?), so that's a win.

In closing

In short, this has been a very enlightening and rewarding experience, and while I'm certainly glad that to have gone through it, I am looking forward to not having to worry about it for a while (hah, as if).

Now, let's get writing!

Footnotes

^[1] GitHub issue: Add support for Haskell ^[2] GitHub issue: "hakyll can't handle unicode?"

Whisper of the Heartman

Bevy: getting started on NixOS

Initial setup

Problem 1: fast compilation

Problem 2: Apps

Problem 3: Plugins

Summary and full files

shell.nix

.cargo/config.toml

2020 wrap-up

2020: Review

Programming language theory

Community engagement

Emacs

Nix

OpenShift

Containers/infrastructure/Haskell/Nix

2021: New goals?

Rust 2021

Nested OR-patterns

Trait aliases

That's it for now!

Understanding lifetimes

Introducing lifetimes

Lifetime annotations

Why we need lifetimes

The explanatory example

Lifetime elision

Examples and common problems

Returning references from functions

Structs with references

Summary and further reading

Footnotes

Meet the borrow checker

The ... borrow checker?

Garbage collection vs manual memory allocation vs the borrow checker

The stack and the heap

Garbage collection

Other borrow checker superpowers: paralyzed or parallelized?

Drawbacks

In practice

Attack of the clones

References

Summary

New domain: blog.thomasheartman.com

Notmuch: (un)tag outgoing mail

The problem

The solution

org-babel-load-file doesn't tangle correctly

70,000 words

The ups and downs

The Good

The Bad

Tangible outcomes

Stats

Traffic generators

Now what?

Was it worth it?

The results of splitting empty strings

Splitting on a substring that doesn't appear in the string

The algorithmic approach

The user experience approach

Splitting an empty string

The empty string as separator

Summary

Footnotes

Save on typing and improve legibility with Rust's macros

Post purpose

My Little Macro 🦄

Replacement as a form of abstraction

Further reading

Footnotes

Configuring Emacs with Org mode and literate programming

Who is this for?

What is literate programming?

Org mode

Configuring Emacs

Benefits of using Org mode for configuration

Summary

Further reading and references

`shell.nix`

`.cargo/config.toml`

The `main` function

Serializing the `HeaderMap`

`Maybe`

`Either`

Authentication: using `.authinfo`