Andaman: Difference between revisions
m (→Terra: fix wrong repo url in note) |
(→Installation: add instructions on how to install anda) |
||
(2 intermediate revisions by the same user not shown) | |||
Line 4: | Line 4: | ||
simplify building different package formats for different projects in the same repository. | simplify building different package formats for different projects in the same repository. | ||
It mainly supports monorepo setups, especially ones with CI support like GitHub. It currently supports building RPMs, OCI images and Flatpak packages. | It mainly supports monorepo setups, especially ones with CI support like GitHub. It currently supports building RPMs, OCI images and Flatpak packages. | ||
It is written by [[Fyra Labs]] with the Rust programming language. | It is written by [[Ultramarine Linux]] originally<ref group="note">The original GitHub repository was located at https://github.com/Ultramarine-Linux/anda, but now it redirects to [https://github.com/FyraLabs/anda the new URL].</ref> and [[Fyra Labs]] with the Rust programming language. | ||
Anda's simplicity makes the packaging process seamless and it is most notably used in [[Terra]]. | Anda's simplicity makes the packaging process seamless and it is most notably used in [[Terra]]. | ||
== Installation == | |||
On systems with [[Terra]], you can install it with <code>sudo dnf5 in anda</code> (or dnf). | |||
On systems with [[w:Rust (Programming langauge)#Cargo|Cargo]], you can install it with <code>cargo install anda</code>. | |||
Alternatively you can also manually download the sources from GitHub: | |||
<syntaxhighlight lang="bash"> | |||
git clone https://github.com/FyraLabs/anda | |||
cd anda | |||
git checkout 0.1.18 | |||
cargo install --path . | |||
</syntaxhighlight> | |||
== Mechanisms == | == Mechanisms == | ||
Currently, Anda is simply a meta-build system that calls upon other build systems to build and distribute packages. It reads the project manifest in <code>anda.hcl</code> | Currently, Anda is simply a meta-build system that calls upon other build systems to build and distribute packages. It reads the project manifest in <code>anda.hcl</code> and calls the corresponding build system to build the package. Multiple projects can be included in a single repository (known as a ''monorepo'') recursively in different directories. The root directory contains a main configuration file (also <code>anda.hcl</code>) and the configurations will be inherited. All the other <code>anda.hcl</code> files in its subdirectory can independently represent multiple ''projects'', and multiple ways to build a package can be specified in each project. Once properly configured, a project can be built by <code>anda build ''subdirectory_path''/''project_name''</code>. | ||
and calls the corresponding build system to build the package. Multiple projects can be included in a single repository (known as a ''monorepo'') recursively in different directories. The root directory contains a main configuration file (also <code>anda.hcl</code>) and the configurations will be inherited. All the other <code>anda.hcl</code> files in its subdirectory can independently represent multiple ''projects'', and multiple ways to build a package can be specified in each project. Once properly configured, a project can be built by <code>anda build ''subdirectory_path''/''project_name''</code>. | |||
All configurations should be written in the HashiCorp Configuration Language with the filename <code>anda.hcl</code> and will be parsed in <code>anda-config</code>. | |||
=== Building process === | |||
When Anda needs to build RPMs, it first loads all the <code>anda.hcl</code> configuration files, then run all executable <code>pre_script</code>s. | |||
Then, Anda will build RPMs, Flatpak, Podman and Docker artifacts, then run the build scripts and finally the <code>post_script</code>s. | |||
All the scripts in Anda can either be a command that can be executed with <code>sh -c</code>, or a path to an AndaX script with the extension <code>.rhai</code>. | |||
Other package types are processed as follows: | |||
* ''RPM'': Anda calls either <code>mock</code> or <code>rpmbuild</code> depending on the <code>--rpm_builder</code> option which defaults to <code>mock</code>. | |||
* ''Flatpak'': The manifest is read using the <code>flatpak</code> crate and the application is built using <code>flatpak-builder</code>. As of 0.1.17, the output is always located at <code>.flatpak-builder/build/</code>, as opposed to <code>anda-build/</code> used by other package types. | |||
* ''Docker''/''Podman'': Anda calls <code>docker build</code> or <code>podman build</code>. | |||
=== AndaX === | === AndaX === | ||
'''AndaX''' is an embedded scripting system powered by [https://rhai.rs Rhai]. It is mainly for supporting automatic updates, but it is also the scripting system used behind | '''AndaX''' is an embedded scripting system powered by [https://rhai.rs Rhai]. It is mainly for supporting automatic updates, but it is also the scripting system used behind | ||
<code>pre_script</code> and <code>post_script</code>. | <code>pre_script</code> and <code>post_script</code>. | ||
AndaX loads all the Rhai module script files from the following paths: | |||
* <code>/usr/lib/anda/</code> | |||
* <code>/usr/local/lib/anda/</code> | |||
* <code>/usr/lib64/anda/</code> | |||
* <code>/usr/local/lib64/anda/</code> | |||
* <code>~/.local/lib/anda/</code> | |||
The following static modules (built into AndaX directly) are also available: | |||
* <code>io</code> | |||
* <code>tsunagu</code> | |||
* <code>kokoro</code> | |||
* <code>tenshi</code> | |||
* <code>anda::rpmbuild</code> | |||
* <code>anda::cfg</code> | |||
Most if not all of the functions in the above static modules are well-documented in [https://developer.fyralabs.com/terra/autoupdate DevDocs]. | |||
You can pass labels into the AndaX executor during with the <code>--labels</code> command-line argument. A label is a key-value pair that can be used to tag packages (is it a nightly package?), adding special variables, etc. Different labels should be separated by commas, like so: <code>--labels commit_sha=abc1234,environment=gh_ci,key_value_pair=k=v</code>, which will be parsed to: | |||
<syntaxhighlight lang="json"> | |||
{ | |||
"commit_sha": "abc1234", | |||
"environment": "gh_ci", | |||
"key_value_pair": "k=v" | |||
} | |||
</syntaxhighlight> | |||
=== Automatic Updates === | === Automatic Updates === | ||
Line 27: | Line 76: | ||
} | } | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Then, AndaX executes the script. Projects can also be filtered by adding <code>--filters key=value,key2=value2</code> to the command. | Then, AndaX executes the scripts in parallel, where each update script gets its own thread. This greatly reduces the time required to fetch package versions. Note that Rhai does not support asynchronous programming. Around 200 update scripts in Terra can be processed under a second. | ||
For example, the update script for the above project will be executed for <code>--filters a=1</code> or even without filters related to <code>a</code>, but | |||
it won't be executed for <code>--filters a=2</code>. | Projects can also be filtered by adding <code>--filters key=value,key2=value2</code> to the command. For example, the update script for the above project will be executed for <code>--filters a=1</code> or even without filters related to <code>a</code>, but it won't be executed for <code>--filters a=2</code>. | ||
== Configuration == | == Configuration == |
Latest revision as of 07:45, 17 September 2023
Andaman, often abbreviated as anda, is a modern package build toolchain designed to simplify building different package formats for different projects in the same repository. It mainly supports monorepo setups, especially ones with CI support like GitHub. It currently supports building RPMs, OCI images and Flatpak packages. It is written by Ultramarine Linux originally[note 1] and Fyra Labs with the Rust programming language.
Anda's simplicity makes the packaging process seamless and it is most notably used in Terra.
Installation
On systems with Terra, you can install it with sudo dnf5 in anda
(or dnf).
On systems with Cargo, you can install it with cargo install anda
.
Alternatively you can also manually download the sources from GitHub:
git clone https://github.com/FyraLabs/anda
cd anda
git checkout 0.1.18
cargo install --path .
Mechanisms
Currently, Anda is simply a meta-build system that calls upon other build systems to build and distribute packages. It reads the project manifest in anda.hcl
and calls the corresponding build system to build the package. Multiple projects can be included in a single repository (known as a monorepo) recursively in different directories. The root directory contains a main configuration file (also anda.hcl
) and the configurations will be inherited. All the other anda.hcl
files in its subdirectory can independently represent multiple projects, and multiple ways to build a package can be specified in each project. Once properly configured, a project can be built by anda build subdirectory_path/project_name
.
All configurations should be written in the HashiCorp Configuration Language with the filename anda.hcl
and will be parsed in anda-config
.
Building process
When Anda needs to build RPMs, it first loads all the anda.hcl
configuration files, then run all executable pre_script
s.
Then, Anda will build RPMs, Flatpak, Podman and Docker artifacts, then run the build scripts and finally the post_script
s.
All the scripts in Anda can either be a command that can be executed with sh -c
, or a path to an AndaX script with the extension .rhai
.
Other package types are processed as follows:
- RPM: Anda calls either
mock
orrpmbuild
depending on the--rpm_builder
option which defaults tomock
. - Flatpak: The manifest is read using the
flatpak
crate and the application is built usingflatpak-builder
. As of 0.1.17, the output is always located at.flatpak-builder/build/
, as opposed toanda-build/
used by other package types. - Docker/Podman: Anda calls
docker build
orpodman build
.
AndaX
AndaX is an embedded scripting system powered by Rhai. It is mainly for supporting automatic updates, but it is also the scripting system used behind
pre_script
and post_script
.
AndaX loads all the Rhai module script files from the following paths:
/usr/lib/anda/
/usr/local/lib/anda/
/usr/lib64/anda/
/usr/local/lib64/anda/
~/.local/lib/anda/
The following static modules (built into AndaX directly) are also available:
io
tsunagu
kokoro
tenshi
anda::rpmbuild
anda::cfg
Most if not all of the functions in the above static modules are well-documented in DevDocs.
You can pass labels into the AndaX executor during with the --labels
command-line argument. A label is a key-value pair that can be used to tag packages (is it a nightly package?), adding special variables, etc. Different labels should be separated by commas, like so: --labels commit_sha=abc1234,environment=gh_ci,key_value_pair=k=v
, which will be parsed to:
{
"commit_sha": "abc1234",
"environment": "gh_ci",
"key_value_pair": "k=v"
}
Automatic Updates
- See also: Fyra Developer: Terra autoupdate for the supported functions in AndaX
Anda has refined support over automatically updating packages via the anda update
command. The command finds all the projects with the file update.rhai
in the same directory or if the configuration file has the update
property set to a value:
project project_name {
update = "path/to/update_script.rhai"
labels {
a = "1"
}
}
Then, AndaX executes the scripts in parallel, where each update script gets its own thread. This greatly reduces the time required to fetch package versions. Note that Rhai does not support asynchronous programming. Around 200 update scripts in Terra can be processed under a second.
Projects can also be filtered by adding --filters key=value,key2=value2
to the command. For example, the update script for the above project will be executed for --filters a=1
or even without filters related to a
, but it won't be executed for --filters a=2
.
Configuration
Andaman uses the HashiCorp Configuration Language (HCL) for its configuration format, known for being used in software by HashiCorp such as Terraform, Consul, Vault and Packer. Its syntax is similar to UCL and NGINX's configuration format.
Consider the following. for { }
and ""
, they represent, respectively, an dictionary (empty by default) with keys and string values, and a string with no default value. Unless otherwise stated, all the attributes are optional.
config {
mock_config = "" // default mock config for `anda build -c custom_mock_config`
strip_prefix = "" // strip prefix to form new project alias
strip_suffix = "" // strip suffix to form new project alias
project_regex = ""
}
project "project_name" { // the string quote is optional
rpm {
spec = "path/to/file.spec" // REQUIRED
sources = "."
package = ""
pre_script = "rpm_pre.rhai"
post_script = "rpm_post.rhai"
mock_config = ""
enable_scm = false
scm_opts { }
config { }
plugin_opts { }
macros { }
opts { }
}
podman tagname {
dockerfile = ""
import = ""
tag_latest = false
context = "" // REQUIRED
version = ""
}
docker tagname {
dockerfile = ""
import = ""
tag_latest = false
context = "" // REQUIRED
version = ""
}
flatpak {
manifest = "path/to/manifest.yml" // REQUIRED
pre_script = ""
post_script = ""
}
pre_script = "pre.rhai"
post_script = "post.rhai"
env { }
alias = [] // array of strings
scripts = [] // array of paths to AndaX scripts
labels { }
update = "update.rhai"
arches = ["x86_64", "aarch64"]
}
History
umpkg
In the very beginning, the Ultramarine Project had been using Koji for building RPM packages. Koji is the packaging system used by the Fedora Project. However, Koji's sophistication makes it difficult to create and maintain RPM packages — it was said that the only person in the Ultramarine team who actually knew how to use Koji was the lead developer, Cappy Ishihara. Cappy was annoyed by the sheer complexity of building and uploading packages to Koji and made umpkg, a packaging toolchain that runs on Mock and Koji. The creation of umpkg made it easier to create and maintain packages. However, umpkg still tightly integrates with Koji and uses the Koji API to trigger builds. Thus, an entirely new CI server was proposed.
RPM Frontend
The Ultramarine Project started work on Andaman on 9 Jun, 2022. The name Andaman refers to the Andaman Sea next to Thailand, and follows the naming patterns for other projects related to Ultramarine. Initially, Anda was written as a custom RPM frontend as an alternative to DNF that is similar to that of an AUR helper. However, there were issues with rust and RPM support.
CI Server
Anda was then rewritten as a CI Server that handles builds via BuildKit, stores artifacts and build infos, and later included a frontend to the server. However, it slowly turned into a massive feature creep of projects.
Final rewrite
Thus, Anda was rewritten again and it finally became a build system for various package formats.
Terra
- Main Article: Terra
After the creation of Anda, the Ultramarine Project has migrated their repositories into GitHub monorepos. However, a repository containing packages for not only Ultramarine Linux, but also all other Fedora distributions was wanted so that packages can be shared between both Ultramarine Linux and TauOS, which led to the creation of the Andaman Common Repositories, also known as the Andaman Subatomic Repositories.[note 2] It was later renamed to Terra, and now contains many common packages not found in the Fedora repositories.
Notes
- ↑ The original GitHub repository was located at https://github.com/Ultramarine-Linux/anda, but now it redirects to the new URL.
- ↑ The original GitHub organisation was called "andaman-common-pkgs", and the original repository url was
https://subatomic.fyralabs.com/ad37/
.