Hi!

I would like to share a part of the plan for future Laniakea improvements which might likely bind a significant amount of my time, but may also provide a huge amount of improvements for the PureOS tooling.

What is Laniakea? -------------------------

In order to build a Debian derivative, many tools are needed - besides managing the archive, you also want to do QA, migrate packages between suites, sync packages from Debian, show some web UI about packages, etc. If all tools are separate and don't know of each other, you have to manage them separately, which is a lot of work, but also coordinate them somehow and manage their configuration (e.g. if a suite stabilizes, you need to adjust the new development target 5 times for 5 different tools). Laniakea is a suite of tools which either provides features for archive management on its own, or wraps existing tools and manages their configuration. All tools share one database for configuration which is the single source of truth in the system, and individual Laniakea modules can also communicate directly via a ZeroMQ-based messaging protocol.

The Problem -----------------

Laniakea was originally created as part of the Tanglu Debian derivative, and then adapted for PureOS. One of the original ideas of Laniakea was to work with any archive management software, bei it dak (Debian Archive Kit), reprepro, aptly, ... and just wrap it like it wraps any other module. Because that's what it's design was based on: Be very modular and make each module easily replaceable in case something better comes along. This also enabled Laniakea to for a while host two ways to get packages built, while nowadays this has pretty much been collapsed into one singular job management system, which handles package builds as well as image builds.

The problem with making the archive manager pluggable is that it goes against Laniakea's principle of having a single source of truth in its database. Currently, only dak is properly supported as archive manager for Laniakea, which keeps its own package database, handles all incoming packages and generates data which Laniakea then imports into its own copy of the archive database. This is terrible for multiple reasons:

* Laniakea and dak can - even if just briefly - desynchronize. In reality this makes people wonder which version of a package currently is in what suite, as the web UI does not reflect the actual archive status.

* Due to potentially desynchronized states, Laniakea can not trigger certain actions, or can only trigger them in a static sequence of operation when it *knows* that the archive data is currently the same as the data in its database. This results in a lot of archive operations being executed in strict sequences and some changes will not take effect until those were run (like package removals, migrations, syhfornizations, ...)

* Laniakea does not know of certain actions taken by dak: For example, if dak rejects a package from upload, Laniakea can only guess that this has happened as there is no direct communication channel that would be race-free. This results in heuristics for the package build system to figure out whether an upload actually reached its destination suite.

* The constant interactions between Laniakea and dak are a huge performance hit - Laniakea needs to constantly synchronize its database with the dak data, which can occasionally take more than 20min. That combined with the sequences that Laniakea has to run anyway destroys performance and the "immediateness" that archive actions could have, as everything runs at timed intervals by cron jobs.

The Solution -----------------

The solution seems simple: Just implement archive management in Laniakea itself! I did shy away from this for years though. Reasons are that creating the apt archive actually is not an easy task, and a lot of work went into dak for a lot of nice features that would all have to be implemented again. Also, the Debian archive is constantly evolving, and by detaching from dak we would not be able to benefit from them unless we put in work from our side to implement them again. Another reason why this is a bad idea is that it would bind my time, so other features which I consider important would get realized much later. There's also personal reasons for not attempting that rewrite: Dak has some insane SQL to generate indices performantly directly from database tables, and me being no experienced database developer would not help with achieving a better result.

However, despite all of these concerns I think I should attempt to write the Laniakea archive module, with the aim to replace dak mid-term. Reasons being the following:

* The issues due to interfacing with dak are touching every area of our archive, and it is honestly a pain to work with. Also, 90% of feature requests brought forth by users directly tie in with this issue and could be easily addressed if Laniakea could control the archive directly, and would be the only entity doing so (instead of having some external thing to synchronize with).

* Ubuntu's Launchpad is a good example of what can be built if the archive is integrated a bit more tightly with other distribution management features.

* Launchpad also uses apt-ftparchive for some metadata read actions. Performance and flexibility is a huge worry for me in an endeavour like this, so I looked into using apt-ftparchive for a "Laniakea archive" prototype. It's not a tool that makes sense to run directly on the archive, but it is great to obtain exactly the metadata we need to build archive index files. Those can be transformed into JSON and stored in Postgres JSON table elements, so we can very easily extend the extra metadata elements in future, if we need to. Also, reading a large quantity of data elements from Postgres tables with SQLAlchemy's ORM works very quickly, and converting and writing them with C++ code is almost instant. Building on top of apt-ftparchive and APT's libapt-pkg also ensures we benefit from improvements made on these tools in Debian.

* We don't actually need to support all features initially. Some features we would certainly lose initially are pdiffs, by-hash etc, but a functioning APT archive, without any legacy support but also without any of the very fancy new features, could actually be built in a reasonable amount of time. Of course that would be annoying for some users, but the audience for PureOS would unlikely mind these changes much. And we could gradually add them back later.

So, due to that I think it makes sense to spend some serious time to look into implementing this module. I would build on top of apt-pkg and apt-ftparchive, using C++ code where necessary for performance, and Python code everywhere else. I would also make heavy use of json/jsonb entries in Postgres for many of the metadata elements, as it's a convenient, performant but most of all easily extensible way to store most of the auxiliary metadata. For actions such as dependency-resolution I would also use the existing APT library. There are a few open questions on design details that I haven't nailed down, but unlike a year ago, I think a project like this is actually doable and will not take years to complete. But as always with these things, what may look easily achievable now can turn out to be much harder in future ;-)

So, what do you think? Am I crazy, or is this a good thing to finally attempt?

Cheers, Matthias

P.S: To address the obvious question "Why not adjust dak?" - I looked into this, but the amount of changes we would need is quite big, and since there is no chance to merge these upstream, we would essentially be forking dak and still having a separate database and separate-ish tooling. Also, dak serves its own purpose and serves it well for Debian - trying to make it something that it isn't just to work for Laniakea is not a great plan.