test(snap): add spread integration tests#1465
Conversation
zyga
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
The integration test suite runs on typical Linux distribution images and ensures that openshell can create a sandbox and execute a hello-world program inside. Tests have two components: 1) The `image-garden` program uses built-in rules, as well as `.image-garden.mk` to prepare virtual machine images for testing. The virtual machines are vanilla cloud images booted once with a cloud-init profile that prepares them for testing. At runtime garden downloads images to ~/.cache/garden/dl or $SNAP_USER_COMMON/cache/dl (when using the snap) and then saves customized differential images in `.image-garden/` in the project directory. In practice the pre-created environment has snapd, the docker snap and the "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" docker image pre-pulled for faster test iteration. 2) The `spread` program uses `spread.yaml` and a collection of `task.yaml` files to run tests. The top-level file defines the set of test systems, contains project wide preparation logic where we install the snap and defines a single test suite which corresponds to the `tests/` directory. An initial smoke test that creates a sandbox and ensures it can run a shell hello world is provided. This ensures that the locally built snap really works on the set of test environments: - centos-cloud-10 - debian-cloud-13 - fedora-cloud-44 - ubuntu-cloud-24.04 - ubuntu-cloud-26.04 Those tests are typically used with the `image-garden` snap, which also includes spread: https://snapcraft.io/image-garden/ Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
|
/ok to test 6e77fc0 |
We need to give docker a moment to finish setting up. We an also add a docker system group while we are at it. With this setting it is worth to keep prepared images around in CI but the time advantage is worth it. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
CentOS 10 cloud image has a truncated checksum file causing each download to fail validation. Let's remove it from spread.yaml while keeping the .image-garden.mk entry for another day. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
|
/ok to test 551d47f |
drew
left a comment
drew
left a comment
There was a problem hiding this comment.
Can you explain a little more how this is intended to be used?
There was a problem hiding this comment.
can we instead add .image-garden/* to the root .gitignore?
There was a problem hiding this comment.
When image-garden make or image-garden allocate is used, it loads this file to compute the cloud-init profile to give to the VM. In practice this defines what is in the image that is perhaps cached by the CI stack. Locally it plays a larger role as the cached image is simply in the workspace (no separate caching step). This effectively allows iteration with one-off preparation costs paid once, regardless of how many test cycles you have.
There was a problem hiding this comment.
how is this used? do you see this getting expandded for non snap use cases? i'd prefer to keep snap specific infra in deploy/snap for now. we have an existing install canary that we run in .github/workflows/release-canary.yml.
There was a problem hiding this comment.
This is one of the files that spread loads to understand test cases and execute them.
I would love to expand this to cover the non snap case (note that in prepare we can simply install the pre-build binaries from target/release). Spread is used heavily across many of our products as it allows to perform realistic full system testing on many releases of ubuntu (with the real kernel, with the real bugs people would face) as well as to extend this testing to other distributions (this is what garden provides: distribution images).
Here we only have one test case but in our products we aim to have full coverage of all interactions expressed as spread tests. They provide invaluable support in ensuring quality and in preventing regressions in complex products.
|
Crossing linking this discussion here, #1494 (comment). |
3 participants
Summary
Add
spreadsmoke test suite for the snap package.Related Issue
N/A
Changes
Testing
mise run pre-commitpassesChecklist
Please let me know if I should update architecture docs. The next step after this is to wire this all to GitHub to gate snap publishing on the smoke test passing.