z - Advanced topic: Rolling out your own binary cache

Introduction

At its core, the Nix package manager installs software from source, which can take a very long time. For example, installing R or RStudio from source could take quite some time, depending on your hardware. Some R packages also must be compiled, so depending on the environment you need, this could take several hours to build. In practice however, this is rarely the case as most, if not all packages get pre-built and made available through NixOS’s public cache. The Nix package manager first checks if what you need is in the public cache, and if yes, it downloads a binary from there instead of building it on your computer. This means that building an environment is just a matter of waiting for packages to download.

However, in some cases, you might need to use packages that have not been pre-built and cached yet, for example if you use rix(r_ver = "bleeding_edge") or rix(r_ver = “frozen_edge”)` to generate environments with bleeding edge packages. Building these environments could potentially take quite long, because these packages need to be built from source on your computer.

This vignette will explain how you can cut down on building times by building the environment once, and then cache it, so you don’t need to rebuild it on new machines. But before rolling out your own cache, try using ours as explained in vignette("z-bleeding_edge"), and if our cache didn’t cover your needs, set up your own. This vignette explains how.

Using Github Actions to build the environment

Before creating a cache to hold the binaries for your development environment, you first need to build the environment once. You can build the environment on your computer, or on Github Actions. The advantage of using Github Actions is that you can automate the process of building and pushing the binaries each time you change the definition of the environment.

For example, imagine you have the following generate_env.R file for a project, which you version on Github:

library(rix)

rix(r_ver = "bleeding_edge",
    r_pkgs = c("dplyr", "ggplot2"),
    system_pkgs = NULL,
    git_pkgs = NULL,
    tex_pkgs = NULL,
    ide = "rstudio",
    project_path = ".")

if you add packages to it, or re-run it, you’ll end up with a new default.nix file, and so you will need to rebuild the environment. Again, depending on the packages you include, this could take quite some time to build. To generate a Github Actions workflow file that will build the environment on Github Actions, run rix::ga_cachix(cache_name = "", path = "") where cache_name is the name of the cache you made on Cachix, and path is the path to the default.nix file generated by generate_env.R. By default, the environment gets rebuilt every time you push changes to the master/main branch of your repository, but you can also re-build the environment periodically, by changing these lines:

on:
  push:
    branches: [ master, main ]

to these:

on:
  push:
    branches: [ master, main ]
  schedule:
    - cron: '30 0 * * *'

By using cron syntax you can specify how often you want the environment to be re-built. This can be useful if you need to develop against the current state of CRAN every day (for instance, for package development).

Using your cache

Whether you decide to build the packages on Github or locally, to then use your cache, you need to open an account on Cachix. The free tier includes 5GB of space, which is more than enough for several development environments. Once your account is done, create a personal auth token so that the Github Actions workflow (or your computer, if building locally) can authenticate to your Cachix account. Then, copy this token, and go to your Github repository’s settings, then Secrets and variables > Actions and add a new repository secret. Copy the token in the Secret field and name the secret CACHIX_AUTH. Now the action can authenticate to Cachix and push the binaries it builds! If building locally, simply run cachix authtoken <TOKEN> in your terminal before building (follow the instructions on Cachix website to learn how to push the binaries afterwards).

To use your personalized cache on your computers, run the following commands on your computer. First, install the Cachix client:

nix-env -iA cachix -f https://cachix.org/api/v1/install

then use the cache:

cachix use your-cache-name

Anyone can pull binaries from your cache, so if you work in a team, you can ensure everyone can benefit from it. You can also use several caches at once, NixOS’s public cache, our rstats-on-nix cache, and your own, so your cache will only end up holding the binaries not found in the other two caches!

Take a look at this package’s repository for an example of how this is done in practice.