Ravenports

[WarFox]

Wanted to share a little of the work I a currently doing as it is pretty interesting. Recently I have been investigating means of using one system of package management across different UNIX systems as a part of a project. Off the top of my head, there are 3 real good options. The first and oldest is Pkgsrc from the NetBSD community (will proabbly do a write up later), Nix (from the NixOS project) and Ravenports. I wanted to highlight Ravenports as it is pretty interesting.

Ravenports was started by a former FreeBSD developer. Prior to creating Ravenports, he had developed a package management system written in ada called Synth. As ada is not very popular and there were some early design decisions and new approaches that came out, he rewrote Syn and incorporated many improvements with creating Ravenports (writing it in C).

As of the writing of this post, Ravenports supports FreeBSD, Linux, MacOS, Solaris, and DragonFlyBSD right now with ports to others in the pipeline. However, I personally have only played with it on FreeBSD so far and looking into jumping in on the porting effort on it to NetBSD.

Ravenports is interesting in that it incorporates lessons learned from the FreeBSD ports system and Pkgsrc. Here is a guide that was written for me that I am sharing, it covers the install process and instructions for working with Ravenports the package manager and Ravenports the ports tree.

 

Quote

So for you to get familiar with RP, I suggest that you first follow the quickstart
guide on the ravenports homepage
(https://github.com/jrmarino/Ravenports/wiki/quickstart-freebsd). Feel free to use
the jail for it. In contrast to Pkgsrc where you'd do a bootstrap on each host with
a foreign system via the script, Raven is only bootstrapped once for each platform.
When that's done, a bootstrap package containing a static version of the package
manager and the repository configuration is prepared. To start using Raven on a
fresh system, you just fetch that bootstrap tarball and extract it to the root
directory. After optionally including the binary paths below the standard prefix
(Raven can use something else than the standard "/raven"; done that out of
curiosity, but it's a bit more involved), the user is good to just install packages.

Raven's package manager, ravensw(8), is basically identical to FreeBSD's pkg(8). In
fact it's a fork, created to address issues that arose on non-native platforms like
Linux and Solaris. It was later renamed to ravensw both to avoid possible confusion
and to solve a name clash with modern Solaris that also has a "pkg" package manager
for its IPS ("Image Packaging System") which is a completely different thing,
though.

The main difference between vanilla FreeBSD's and Ravenports' packages is the
package names. Raven uses a naming scheme for packages that consists of the
software's name, the subpackage and variant. If the program "foo" is a simple one
that doesn't need any subpackages, the package will be named "foo-single-standard".
If there are subpackages, the main program will be in "foo-primary-standard".
Optional components like i18n or documentation will be in subpackages
"foo-nls-standard" and "foo-doc-standard" respectively. These are common subpackage
types, but others exist as well or can be introduced where it makes sense.

Variants are packages built with different compile-time options selected. With
variant names like "lite", "loaded", "x11" and so on, one can easily guess what they
are about. Different versions of packages for interpreded languages like Python are
also expressed by variants like e.g. "py38" and "py39". Unlike FreeBSD ports,
Ravenports are meant to be the building plans for binary packages only. Thus is is
generally preferred to offer variants for valid use cases than trying to be as
flexible as possible and adding as many user tunable options as one can come up
with.

That's basically all that a average user has to know. Next thing to try out is
building packages. If you install the turnkey package
("ravenports-single-freebsd_amd64") as described in the guide, you get the
toolchain, the administration tool and a couple of dependencies for those. Execute
"ravenadm configure", make any changes you wish (or leave it at the default which is
generally fine) and confirm. Then run "ravenadm update-ports". Now you can start
building packages.

The latter command fetched the compiled buildsheets from Github. It will allways get
the latest release. Now have a look at the package catalog online (link here:
http://www.ravenports.com/catalog/) play around with it a little to see what
information you can get from it, then search for git. You'll see that git has two
variants: The default "standard" as well as a "lite" one.

The advanced user can run "ravenadm build PORTNAME", but for anyone interested in
development, I recommend to always use "ravenadm test PORTNAME" instead if you've
got a somewhat fast system. This will build packages, too, but runs some additional
tests after each successful build. It's meant for developing new ports, though, and
I know that John doesn't run it all the time. With "ravenadm test git:lite" you can
build the lite variant of the git port and all dependencies. Unless you turned it
off, you'll see a nice curses-based UI that informs you of the progress. There's a
subtle bug with it, though, so that very long port names can make the UI hang. So
whenever I build a larger set of ports and leave the machine, I first turn the
curses UI off (might be a better idea to fix this eventually, but there are more
important issues right now, so yeah).

Try pressing CTRL-Q while something's building. It will initiate a "graceful
shutdown", meaning that it'll let the currently building package(s) finish but not
start any new one. Usually this is preferable over CTRL-C'ing out hard.

Ravenadm supports different profiles to quickly switch between configuration
settings (it's good to know, but ignore this for now). The default profile is called
"primary". Logs are in /var/ravenports/PROFILE/logs/logs unless you changed the
setting. Yes, that's two levels deep; the upper directory is for serving a useful
web application that is similar to the curses UI and allows quick access t the logs.
Should we decide to go with Ravenports, we should definitely invest the little
effort to install a webserver and basic auth for it (or perhaps even allow open
access?).

Maybe you want to take a look at the logs while the git package is building. Once
it's done, I suggest that you build the standard variant, too. If you omit the
variant, standard is implicit. So "ravenadm test git:standard" and "ravenadm test
git" are equivalent. The packages will end up in /var/ravenports/PROFILE/packages by
default. You could install git using "ravensw add
/var/ravenports/primary/packages/All/git-primary-lite-2.31.1.tzst". The better way
however would be to do "ravenadm generate-repository" first and then edit
/raven/etc/ravensw/repos/01_raven.conf, changing the URL to the local repo (i.e.
file:///var/ravenports/primary/packages/All). Then run "ravensw update" and when
it's done you can do "pkg search" and "pkg install" from the local repo.

So much for that. You should have a pretty good impression of what RP feels like and
whether you like the tooling or not. Next step is looking at ports and messing with
existing ones as well as adding new ports. Before you can do that, clone the
ravensource repository (link here: https://github.com/jrmarino/ravensource.git) as
your unprivileged user. The repo that you just cloned is the source for the
buildsheets used by Raven.

It's organized in a bucket structure (I think performance considerations regarding
parallelism lead to what we have). Each port lives in one of 255 buckets named
"bucket" + underscore + two-digit hex value. IIRC the hex value is derived from the
first two digits of a md5 checksum of the port name or something like that. Note
that there's also a directory called "Scripts". Feel free to take a look at it.
There's one particularly useful script in there that I'd like to point you to: The
"regenerate_conspiracy.sh" one.

"Conspiracy" is as in "a flock of ravens", BTW, not as in "conspiracy theory". Since
English is your native language, you probably figured that it'd be, but I wondered
about it for quite a while until I finally asked and John explained it! ;)
Conspiracy is the regular buildsheet collection. With the script mentioned before,
all of the ports source files will be compiled to buildsheets. This is useful if no
official release has been made for a while (so you cannot "ravenadm update-ports" to
build more recent versions) but changes have been pushed to the ravensource
repository.

In addition to Conspiracy there's also Unkindness. This refers to a custom ports
directory which is very useful for developing ports. You can think of it as a
batteries-included overlay over Conspiracy. If you create a port in your Unkindness
directory, it will take precedence over a port of the same name in Conspiracy (if it
exists). Also there's no need to compile buildsheets after adding a port or making
changes - Unkindness handles this automatically (which is veeery convenient).

A great port to take a look at is vim. It frequently has new releases, so that
there's always the chance that a new one is released when you take a look, even if
John or I have recently updated it. Which leads me to an important characteristic of
RP: There are no ports maintainers in the classical sense of an "owner"! The person
who or persons listed with any port are considered "contacts" who are responsible
for the port in the sense of "they make the decisions on how a port works". Anybody
can choose to update any port at any time (a few of exceptions exist for ports that
John reserved for himself e.g. due to regularly break other ports when updated or
things like that - in such cases it's noted in a comment at the top of the port).

For major changes you should contact the maintainer (e.g. turning on a feature that
draws in a huge additional dependency). If you just update to a newer version, no
communication is needed, you can just go ahead (if the new version requires an
additional deendency or for minor adjustments you don't need to ask). This approach
is really genius if you ask me. It has allowed RP to gain and maintain an
unparalleled "freshness" of versions over a long time (several years).

To figure out where vim is, use "ravenadm locate PORTNAME". As I said, newer vim
versions are released extremely often, so in this case I know by heart that it's E1.
For most ports however, I also use "ravenadm locate" all the time.

 

[cwade12c]

Really high quality post, man. This is very impressive. You and @ryoh have hooked my interest with universal package systems lately. Between this and Nix, I have a lot of reading to do.

[WarFox]

Just a nice little update on the Ravenports project. It mainly only worked on FreeBSD and Linux. I recently toyed with the code and got it compiling under NetBSD. John Marino, the creator of RavenPorts, snagged my code changes and now RavenPorts has an official build of NetBSD! John also made some fixed to build sheets and the whole RavenPorts ports tree builds on NetBSD.

However, support for OpenBSD does not look it is coming since RP relies on nullmounts, which OpenBSD does not have as an OS feature.