Hello! Since last week, all of PureOS infrastructure is now switched to the latest, mostly Python-based version of Laniakea[1].
## What is Laniakea? Laniakea is a suite of tools and adapters which originated in the Tanglu Debian derivative. Its purpose is to bring together data and configuration from the multitude of separate tools that are required to run a Debian derivative, as well as adding missing functionality in an integrated way. Tools which are wrapped by Laniakea include Britney, dose3, germinator, dak and more. Due to Laniakea's database being the authoritative source of information for a derivative, there is always only one place to change configuration, individual configuration for the wrapped tools is automatically generated from database information. Also, the centralization of data in a database and well-defined directory structure allows Laniakea modules to cross-reference information and make decisions based on data from other tools, e.g. the debcheck information is used to determine whether a package is actually buildable in the distribution. Using the database, all information Laniakea knows about is accessible (currently read-only) via modern web interfaces. Laniakea also contains a global worker pool to execute arbitrary long-running tasks. These tasks currently are image builds and package builds. All tasks are run on separate machines in systemd-nspawn containers via debspawn[2], for build tasks static-analysis reports are generated in the Firehose XML format[3]. Laniakea also contains a message-bus based on ZeroMQ and JSON to let modules running on other machines know about changes and react to events when running in a more isolated environment. This also allows notifications via email or Matrix.
## What changed now? Laniakea previously was written entirely in the D programming language. Since the project was a fun experiment in Tanglu, using the language also wasn't much of an issue. With our new requirements in PureOS and limited manpower though, it became apparent that the choice of D was hindering progress in this case, simply due of D's much smaller ecosystem of existing tools and libraries. Time spent at improving the database connection code in D could be better spent at implementing a new feature we need.
So, Laniakea was mostly ported to Python 3.6+ with some code still remaining in D. Since D has great bindings to Python, writing a module to run existing D code from Python was very easy. This also allows us to reuse a lot of the more complex logic in Laniakea without changes (and the chance to introduce new bugs). Porting progress can be followed via this document[4].
Since Monday last week, PureOS' infrastructure is now running on the Python-ified version of Laniakea, removing the extra maintenance burden of developing two versions of the same software in parallel.
## How does the change affect me? Ideally, a user or developer of PureOS should not see any changes. The new implementation should work just as the previous one. The web UI on master.pureos.net looks slightly different, but that is it. Since the new Laniakea database has no indices yet, certain queries could take a bit longer.
One notable change is the absence of software.pureos.net - this module could not be easily ported and needs to be rewritten entirely. I am working on that, and we will definitely get it back, better than before.
The Laniakea message piblishing code also has not been ported yet, so modules can not communicate with each other via ZeroMQ. This should not ceuase any issues, but when it is finally implemented, we will get a new Matrix bot to notify about archive changes immediately.
## What's next? ### Create an integration test framework for Laniakea We need better testing, or rather, any testing for some Laniakea modules and larger pieces of the codebase. Since a large chunk of Laniakea is interfacing with existing code or with its own modules, we need some way to do intergration testing properly.
### More unit testing! The Python code doesn't have many tests written yet, so this needs to be added as well.
### Bring back software.po.n (webswview) The web application just needs to be rewritten. This is a work in progress at time.
### Reimplement Laniakea messaging The Lighthouse module of Laniakea needs to get its message publication support back. I intend to make use of signedjson[5] for message signing this time. When implementing this feature, the very crude current Lighthouse implementation also needs a complete overhaul. Additionally, we can write a Matrix bot for Laniakea again at this stage.
### Better dak integration The Debian Archive Kit (dak) and Laniakea basically exist alongside each other, not interacting all that much. That should be changed to make dak and Laniakea talk to each other better to reduce the reliance on cronjobs.
### Better performance There is a lot of potential for performance improvements in Laniakea, most notably in the module that imports archive data into the database. Making these improvements will allow for many nice things, like faster package processing.
### Package for Debian Laniakea should be available from withing Debian. This is a long-term goal as it requires at least some stability of the database schemas and more and better testing. Also, the JavaScript dependencies of the web modules need to be sorted out properly. Ultimately though, at some point you can just apt install Laniakea to create your own Debian derivative.
## What can I do? Contribute! Either by writing code for Laniakea (documentation is currently updated to reflect the Python changes) or filing bugs. If you see anything weird happening in PureOS infrastructure that hasn't happened before, please file a bug in the PureOS bugtracker[6] or against Laniakea directly if you know it is to blame already.
Happy hacking! Matthias
[1]: https://github.com/lkorigin/laniakea [2]: https://github.com/lkorigin/debspawn [3]: https://github.com/fedora-static-analysis/firehose [4]: https://github.com/lkorigin/laniakea/blob/master/docs/porting-status.md [5]: https://github.com/matrix-org/python-signedjson [6]: https://tracker.pureos.net/
On Wed, 2019-05-01 at 23:53 +0200, Matthias Klumpp wrote:
Hello! Since last week, all of PureOS infrastructure is now switched to the latest, mostly Python-based version of Laniakea[1].
This is great news. Congratulations Matthias!
Laniakea also contains a message-bus based on ZeroMQ and JSON to let modules running on other machines know about changes and react to events when running in a more isolated environment. This also allows notifications via email or Matrix.
Do we intend to send messages to email and/or Matrix? If so, perhaps we use a separate Laniakea channel for Matrix so as not to flood the PureOS channel. I suppose we can use the pureos-changes mailing list for messages too? pureos-project seems like it might be better left to higher level topics.
<snip>
The Laniakea message piblishing code also has not been ported yet, so modules can not communicate with each other via ZeroMQ. This should not ceuase any issues, but when it is finally implemented, we will get a new Matrix bot to notify about archive changes immediately.
+1
## What's next? ### Create an integration test framework for Laniakea We need better testing, or rather, any testing for some Laniakea modules and larger pieces of the codebase. Since a large chunk of Laniakea is interfacing with existing code or with its own modules, we need some way to do intergration testing properly.
When you say integration testing, what do you mean? Do you mean how the various components of Laniakea fit together?
### More unit testing! The Python code doesn't have many tests written yet, so this needs to be added as well.
I think a description of the test framework you use (TAP? PyTest), and then a TEMPLATE python script might be a great way to get folks to contribute. I know I would.
### Bring back software.po.n (webswview) The web application just needs to be rewritten. This is a work in progress at time.
What is the framework for the web app? Is it Angular, Vue, or perhaps something from Python like flask?
### Package for Debian Laniakea should be available from withing Debian. This is a long-term goal as it requires at least some stability of the database schemas and more and better testing. Also, the JavaScript dependencies of the web modules need to be sorted out properly. Ultimately though, at some point you can just apt install Laniakea to create your own Debian derivative.
I think a lot of derivative would use this if you could just apt-get install laniakea. It ought to make their lives much, much easier. I know this would have helped Maemo when Nokia essentially abandoned it.
## What can I do? Contribute! Either by writing code for Laniakea (documentation is currently updated to reflect the Python changes)
URL? Are you using Readthedocs?
or filing bugs.
I filed a bug. You fixed it too quickly. :P
If you see anything weird happening in PureOS infrastructure that hasn't happened before, please file a bug in the PureOS bugtracker[6] or against Laniakea directly if you know it is to blame already.
Happy hacking! Matthias
Cheers,
Jeremiah
Am Do., 2. Mai 2019 um 21:23 Uhr schrieb Jeremiah C. Foster jeremiah.foster@puri.sm:
On Wed, 2019-05-01 at 23:53 +0200, Matthias Klumpp wrote:
Hello! Since last week, all of PureOS infrastructure is now switched to the latest, mostly Python-based version of Laniakea[1].
This is great news. Congratulations Matthias!
It's a major milestone, and I am incredibly happy about the reduced maintenance burden :-)
Laniakea also contains a message-bus based on ZeroMQ and JSON to let modules running on other machines know about changes and react to events when running in a more isolated environment. This also allows notifications via email or Matrix.
Do we intend to send messages to email and/or Matrix?
Mails aren't currently on my todo list, but if/when that feature is implemented I assume it will be subscribe-only or dependent on the user. E.g. we may send the user who caused a build failure an email, but not every single project member.
If so, perhaps we use a separate Laniakea channel for Matrix so as not to flood the PureOS channel. I suppose we can use the pureos-changes mailing list for messages too? pureos-project seems like it might be better left to higher level topics.
<snip>
When the feature existed for Tanglu back in the days, we had a #tanglu-changes IRC channel where messages were just dumped as-is. I imagine we could have that on Matrix as well. Sending build failures and results of QA runs to the mailinglist may be a lot of spam, I am not sure whether we should do that. The mailinglist currently holds direct changes to the archive (package uploads).
[...]
## What's next? ### Create an integration test framework for Laniakea We need better testing, or rather, any testing for some Laniakea modules and larger pieces of the codebase. Since a large chunk of Laniakea is interfacing with existing code or with its own modules, we need some way to do intergration testing properly.
When you say integration testing, what do you mean? Do you mean how the various components of Laniakea fit together?
Exactly that, as well as how those modules interact with other components such as dak, Britney, etc. I do not believe much in mocking components, so I am currently looking into how we could possibly test the whole thing and components working together easily. One such a test could be to automatically set up Britney, perform a run on a sample package repository, save the result to a temporary Postgres database, sort the data and assert the result.
### More unit testing! The Python code doesn't have many tests written yet, so this needs to be added as well.
I think a description of the test framework you use (TAP? PyTest), and then a TEMPLATE python script might be a great way to get folks to contribute. I know I would.
Having people write tests would be amazing, but probably also not the most fun task to start with ^^ At the moment I am thinking of using PyTest and Hypothesis[1] together, the latter of which I find very appealing from its approach.
[1]: https://hypothesis.readthedocs.io/en/latest/
### Bring back software.po.n (webswview) The web application just needs to be rewritten. This is a work in progress at time.
What is the framework for the web app? Is it Angular, Vue, or perhaps something from Python like flask?
It's plain boring Flask. I am not a web developer, and Flask is a thing that makes a lot of sense to me. Also, it can just directly use the ORM of Laniakea ;-)
### Package for Debian Laniakea should be available from withing Debian. This is a long-term goal as it requires at least some stability of the database schemas and more and better testing. Also, the JavaScript dependencies of the web modules need to be sorted out properly. Ultimately though, at some point you can just apt install Laniakea to create your own Debian derivative.
I think a lot of derivative would use this if you could just apt-get install laniakea. It ought to make their lives much, much easier. I know this would have helped Maemo when Nokia essentially abandoned it.
It's purpose is like a pluggable "Launchpad lite" - in theory you could have all features it provides with other tools, but that requires a lot more human work. In the long run I fear that I have to make Laniakea absorb dak to be as efficient as I want it to be, but I will try to avoid that as much as possible for now.
From long discussions last Debconf, Laniakea is exactly what many
projects and derivatives need, but unfortunately it's not ready yet to be used as easily as I want it to.
## What can I do? Contribute! Either by writing code for Laniakea (documentation is currently updated to reflect the Python changes)
URL? Are you using Readthedocs?
It's using MkDocs: https://lkorigin.github.io/ - the documentation hasn't been updated for about a year though, and really needs a complete overhaul. The documentation also requires some practical examples to do basic actions (e.g. scheduling builds with the CLI).
or filing bugs.
I filed a bug. You fixed it too quickly. :P
Neat :-D
Cheers, Matthias
Matthias Klumpp wrote:
Since last week, all of PureOS infrastructure is now switched to the latest, mostly Python-based version of Laniakea[1].
This is great news. Congratulations Matthias!
Indeed, congratulations and thank you for posting about it here too. My apologies for this late reply getting lost between various cracks in my inbox. :(
If so, perhaps we use a separate Laniakea channel for Matrix so as not to flood the PureOS channel.
I completely +1 the idea of a #pureos-changes channel.
I use this pattern in many projects; having that "soft" visibility is extremely helpful for all sorts of reasons (both technical and non- technical) in terms of encouraging more of a team feel, avoiding any feeling of being isolated from others. Indeed, I even wrote the #debian-devel- changes bot back in 2008 for these very reasons and it has been remarkably successful in these regards.
Can I thus resubmit my request for this to be re-prioritised? :)
Oh, one adjunct question; how can we contribute/learn more about this codebase? Whilst justifiable for many reasons, as it is on Github it definitely feels a little separate from the rest of the PureOS code. Can we ameliorate this somehow? I don't have any concrete suggestions at this time, alas.
Best wishes,
-
Chris Lamb -
Jeremiah C. Foster -
Matthias Klumpp