Introduction

On the 9th of March in the year 2023 or our era, Topiary was born or, at least, its version 0.1.0 was released. Thus began the quest to package it, to make it available in various ecosystems. Topiary is a universal formatter engine written in Rust and packaged with Cargo. Packaging it in nixpkgs was a breeze because of nixpkgs’ support for Rust, but mostly because someone else did it for us, for which we have to thank our contributor figsoda. It is therefore already possible to do:

$ echo '{"foo"  : "bar"   }' | nix run nixpkgs#topiary -- -l json
{ "foo": "bar" }

Great.

Now, Topiary comes with support for OCaml and we want to be able to reach this community and make it easy for them to use Topiary in their projects. Waiting for Topiary to be available in distribution package managers, such as APT for Debian and Ubuntu, or RPM for Fedora and CentOS, is not an option. For one thing, the release schedules of distribution means that Topiary wouldn’t be available for months. More critically, different distributions may package different versions of Topiary, and it wouldn’t do at all if a contributor on Debian would format the code differently than a contributor on Fedora. No, the best way was to provide it directly in the same way as other OCaml development tools are provided, and that is via OPAM.

OPAM is a package manager for OCaml. In fact, OPAM stands for “OCaml Package Manager”. It is the de facto standard way of distributing packages in the OCaml ecosystem, and installing the OCaml LSP, for instance, is only a matter of running one of the following, depending on whether we just want any latest version or an exact one, for instance to match that of a colleague:

$ opam install lsp
$ opam install lsp.1.11.6

Now that sounds amazing; let us allow people to simply run one of:

$ opam install topiary
$ opam install topiary.0.2.1

Except that Topiary is a Rust project, so how is it supposed to get into OPAM? This question has in fact two aspects: is it reasonable to add Rust packages to an OCaml package repository, and how do we get OPAM to install a Rust package correctly? Basically: can we, and should we?

The pièce de resistance

For the “should we” part, this was not really our call, so we turned to the community. This showed that there was a real interest in having Topiary available in the OCaml ecosystem and that it made sense, in this specific case, to have it distributed via OPAM. Done. Let us now focus on the technical part.

OPAM is described on its own web page as “a source-based package manager for OCaml”. The fact that it is for OCaml can be misleading, though: yes, OPAM is written in OCaml by OCaml enthusiasts in order to provide a package manager for the OCaml ecosystem. However, the tool itself is completely language-agnostic. So while 67%¹ of OPAM packages contain a single call to Dune, a build system for OCaml:

build: [
  ["dune" "build" "-p" name "-j" jobs]
]

nothing prevents 18% of them to call make:

build: [
  ["./configure" "--prefix=%{prefix}%"]
  [make]
  [make "opt"]
]
install: [
  [make "install"]
]

About 6% of them even call OCaml itself on a custom file. So, in fact, nothing prevents us from writing:

build: [
  ["cargo" "build" "--release" "--package" "topiary"]
]

Also, OPAM contains special packages that check for the presence of system dependencies. For instance, the package conf-rust-2021 ensures that Rust is available on the machine and that it is at least the 2021 edition. We can add it as a dependency of our OPAM package, and tadaaaa…

$ opam install topiary
The following actions will be performed:
  ∗ install conf-rust-2021 1     [required by topiary]
  ∗ install topiary        0.1.0
===== ∗ 2 =====
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved conf-rust-2021.1
⬇ retrieved topiary.0.1.0
∗ installed conf-rust-2021.1
[ERROR] The compilation of topiary.0.1.0 failed at "cargo build --release --package topiary".

#=== ERROR while compiling topiary.dev ========================================#
### output ###
# error: failed to get `clap` as a dependency of package `topiary-cli v0.1.0`

…aa? Oh, right, sandboxing. Packages are built in isolation and therefore cargo build cannot download the dependencies.

There are different directions we could take from here. A first solution, when facing this situation, is for package managers to resort to publishing binary packages. This would even be pretty easy with Rust since most dependencies are statically linked anyways. However, OPAM is source-based so that is not going to fly.

The other typical solution is to replicate the packages of the foreign ecosystem in ours; for instance, for OCaml, nixpkgs contains a lot of ocamlPackages.*, while Debian has a lot of packages in *-ocaml and *-ocaml-dev. This however implies a much higher maintenance burden and only makes sense if these packages will be reused, which is not the goal here.

The last solution would be to go towards vendoring all the dependencies of Topiary in one OPAM package; and building this package would involve calling Cargo to build all the dependencies and then Topiary. This introduces potentially a lot of duplication and compilation time compared to the previous solutions, which is only an issue if there are many Rust-based utilities packaged in this way in OPAM. It does solve all our problems though! Except… vendoring is usually quite annoying, right?

NAME
       cargo-vendor — Vendor all dependencies locally

SYNOPSIS
       cargo vendor [options] [path]

Ah. Turns out cargo vendor is a thing, and it works wonderfully well. All that remains is therefore to:

• create a standalone² Git repository and copy Topiary in it,
• call cargo vendor to add all the dependencies to it,
• add an OPAM package file containing a call to cargo build,
• send that as a package in opam-repository,
• profit.

We therefore proceeded to do that in our repository tweag/topiary-opam, opened a pull request on opam-repository (that later got merged), and voilà:

$ opam install topiary
The following actions will be performed:
  ∗ install conf-rust-2021 1     [required by topiary]
  ∗ install topiary        0.1.0
===== ∗ 2 =====
Do you want to continue? [Y/n] y

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved conf-rust-2021.1
⬇ retrieved topiary.0.1.0
∗ installed conf-rust-2021.1
∗ installed topiary.0.1.0
Done.

$ type topiary
topiary is ~/.opam/ocaml-system/bin/topiary

$ echo 'let  x= f ( ) in {x; y =278   }' | topiary -l ocaml
let x = f () in { x; y = 278 }

Conclusion

Topiary is a universal formatting engine written in Rust that, among other languages, supports OCaml. OPAM is a package manager for OCaml, but it is in fact language-agnostic, and therefore it is possible to package Rust packages in it, as long as we find a way to please OPAM’s sandboxing mechanism. Luckily for us, there exists a cargo vendor command that can help us package a Rust project in a standalone way, and this works great. You can now go run:

$ opam install topiary

and start hacking!

Credit is due to the wonderful participants of this discussion around OPAM packaging and to the maintainers of the OPAM package tezos-rust-libs, which already contained all the necessary ideas.