# Proposed CI/CD architecture Automated hardware validation requires GitHub self-hosted runners labelled `e810`, `e830`, and `e835`. The build runner produces reusable, content-addressed outputs; bare-metal runners restore and validate those outputs before testing. The proposed ICE and JPEG XS changes are tracked in [GitHub Actions issue: prebuilt host dependencies](../.github/github_actions_issue.md). The pipeline is organized as follows: ```mermaid flowchart TD PR_TRIGGER([PR TRIGGER]) LINTER[LINTER] BUILD[BUILD] SMOKE_FILTER{SMOKE\nPATH\nFILTER} GTEST_SMOKE[GTEST / SMOKE] CHECK_HASH_1([COMPUTES\nSOURCE SHA-256]) UPLOAD_STASH([UPLOADS TO\nGITHUB STASH]) GITHUB_STASH[(GITHUB STASH\n\nMTL ARTIFACT\nTEST ARTIFACT\nFFMPEG ARTIFACT\nGSTREAMER ARTIFACT\nDPDK ARTIFACT)] NIGHTLY[NIGHTLY / GTEST NIGHTLY] %% PR Pipeline - Left side BM_HOST_1[[BARE METAL HOST]] BM_HOST_2[[BARE METAL HOST]] CHECK_HASH_2([COMPUTES\nSOURCE SHA-256]) CHECK_HASH_3([COMPUTES\nSOURCE SHA-256]) DL_ART_1([DOWNLOADS\nARTIFACTS]) DL_ART_2([DOWNLOADS\nARTIFACTS]) VALIDATE_1([VALIDATE HOST\n\nRESTORE AND VERIFY\nARTIFACTS\nACTIVATE ICE\nONLY IF CHANGED]) VALIDATE_2([VALIDATE HOST\n\nRESTORE AND VERIFY\nARTIFACTS\nACTIVATE ICE\nONLY IF CHANGED]) SHADOW_HOST_1[[SHADOW HOST]] TESTS_1([TESTS]) TESTS_2([TESTS]) %% Nightly Pipeline - Right side BM_HOST_3[[BARE METAL HOST]] BM_HOST_4[[BARE METAL HOST]] CHECK_HASH_4([COMPUTES\nSOURCE SHA-256]) CHECK_HASH_5([COMPUTES\nSOURCE SHA-256]) DL_ART_3([DOWNLOADS\nARTIFACTS]) DL_ART_4([DOWNLOADS\nARTIFACTS]) VALIDATE_3([VALIDATE HOST\n\nRESTORE AND VERIFY\nARTIFACTS\nACTIVATE ICE\nONLY IF CHANGED]) VALIDATE_4([VALIDATE HOST\n\nRESTORE AND VERIFY\nARTIFACTS\nACTIVATE ICE\nONLY IF CHANGED]) SHADOW_HOST_2[[SHADOW HOST]] TESTS_3([TESTS]) TESTS_4([TESTS]) RAPORT[RAPORT] CREATE_REL([CREATE REL\nCANDIDATE]) %% PR Trigger connections PR_TRIGGER --> LINTER PR_TRIGGER --> BUILD PR_TRIGGER --> GTEST_SMOKE %% Main pipeline flow LINTER -->|WAITS| BUILD SMOKE_FILTER -->|WAITS| BUILD BUILD --> CHECK_HASH_1 BUILD --> SMOKE_FILTER SMOKE_FILTER --> GTEST_SMOKE %% Build artifact upload CHECK_HASH_1 --> UPLOAD_STASH UPLOAD_STASH -->|UPLOAD| GITHUB_STASH %% GTEST/SMOKE to Bare Metal Hosts GTEST_SMOKE --> BM_HOST_1 GTEST_SMOKE --> BM_HOST_2 %% Left branch - BM_HOST_1 BM_HOST_1 --> CHECK_HASH_2 CHECK_HASH_2 --> DL_ART_1 GITHUB_STASH -.->|UPLOAD| DL_ART_1 DL_ART_1 --> VALIDATE_1 VALIDATE_1 -->|IF NEEDS AND EXISTS| SHADOW_HOST_1 VALIDATE_1 --> TESTS_1 SHADOW_HOST_1 <--> TESTS_1 %% Right branch - BM_HOST_2 BM_HOST_2 --> CHECK_HASH_3 CHECK_HASH_3 --> DL_ART_2 GITHUB_STASH -.->|UPLOAD| DL_ART_2 DL_ART_2 --> VALIDATE_2 VALIDATE_2 --> TESTS_2 %% Nightly pipeline NIGHTLY --> BM_HOST_3 NIGHTLY --> BM_HOST_4 %% Nightly Left branch - BM_HOST_3 BM_HOST_3 --> CHECK_HASH_4 CHECK_HASH_4 --> DL_ART_3 GITHUB_STASH -.->|UPLOAD| DL_ART_3 DL_ART_3 --> VALIDATE_3 VALIDATE_3 -->|SETS| SHADOW_HOST_2 VALIDATE_3 --> TESTS_3 %% Nightly Right branch - BM_HOST_4 BM_HOST_4 --> CHECK_HASH_5 CHECK_HASH_5 --> DL_ART_4 GITHUB_STASH -.->|UPLOAD| DL_ART_4 DL_ART_4 --> VALIDATE_4 VALIDATE_4 --> TESTS_4 %% Report and Release TESTS_3 -->|WAITS| RAPORT TESTS_4 -->|WAITS| RAPORT RAPORT --> CREATE_REL ``` ## The stash, as implemented today `BUILD` and the `GITHUB STASH` above are `.github/workflows/build.yml`. The artifact store is `actions/cache`, keyed on a checksum of the sources that produce each artifact, so a run only pays for the components a pull request actually touched. | stage in the diagram | implementation | | ------------------------- | ---------------------------------------------------- | | `LINTER` / `WAITS` | `wait-for-linter` job | | `COMPUTES SOURCE SHA-256` | `checksums` job, `script/hash_sources.sh` | | `GITHUB STASH` | `actions/cache`, keys `stash--` | | `BUILD` | `build` job on the self-hosted `e835` fleet runner | The checksums waterfall, so that rebuilding a component invalidates everything downstream of it: ```mermaid flowchart LR DPDK[dpdk sources] --> H_DPDK([hash: dpdk]) MTL[mtl sources] --> H_MTL([hash: mtl]) H_DPDK --> H_MTL H_MTL --> H_FFMPEG([hash: ffmpeg]) H_MTL --> H_GST([hash: gstreamer]) H_MTL --> H_PLUGINS([hash: plugins]) ``` The path lists that feed each hash live in `script/hash_sources_*.env`. ## Testing other MTL sources with the current tests On a manual run, `nightly-pytest` and `custom-pytest` take an optional `branch` input. The run builds and tests the MTL sources of that ref, while the workflow, the CI scripts and the acceptance tests stay at the workflow commit, the commit of the branch picked in "Use workflow from". It is the overlay `perf-pytest` already uses (`task ci:workflow -- overlay-tests`), and its typical use is running today's tests against older code to tell a regression from a test change. Left empty, nothing is overlaid. - `branch` takes a branch, a tag or a **full 40-character commit SHA**. A shorter SHA is read as a branch or tag name and fails the `checksums` job. - The ref must contain 0aff97f0 (#1700). Older trees lack the `Taskfile.yml` and `overlay-tests` that perform the overlay, and `script/` interfaces the current CI calls. They fail in the `checksums` job, before any hardware runner is taken. - The overlay adds and replaces files but deletes none: a file removed under the overlaid paths after the `branch` commit is still there. - `Build` takes the same input on a manual run, so a ref can be built on its own, without a test job. Use it to fill the stash before `perf-pytest`, which restores caches with `fail-on-cache-miss` but builds nothing itself. It still queues on the shared `dpdk` build runner. | Part | Comes from | | -------------------------------------------------------------------- | --------------- | | Workflow YAML | workflow commit | | `.github/` (actions, CI scripts, `setup_environment.sh`) | workflow commit | | `Taskfile.yml` | workflow commit | | `tests/acceptance/` | workflow commit | | `script/hash_sources.sh`, `script/hash_sources_*.env` | workflow commit | | `lib/`, `include/`, `app/`, `manager/`, `ld_preload/` | `branch` | | `tests/tools/RxTxApp/`, `tests/tools/gstreamer_tools/` | `branch` | | `ecosystem/`, `plugins/`, `patches/`, `versions.env` | `branch` | | The rest of `script/` (`nicctl.sh`, `build_*.sh`, `common.sh`) | `branch` | | The first `overlay-tests` pass | `branch` | | The second `overlay-tests` pass | workflow commit | `build.yml` resolves `branch` to a commit once, in the `checksums` job, and exports it as `mtl_commit`. The `build` job and the test jobs check out that commit and apply the same overlay. They have to: the source hashes include `.github/` files, so a job that hashed any other tree would miss the stash the build saved, or restore the stash of another tree. The hash script and its lists come from the workflow commit too, so every `branch` keys its stash on the CI files the current build uses; older lists leave some of them out. `overlay-tests` runs twice for this: the first pass is the `branch` copy, which may predate that, and checks out the workflow commit's; the second pass runs it. The test jobs check this, with or without `branch`. They pass `build_outputs: ${{ toJSON(needs.build.outputs) }}` to `validate-host`, which logs the built commit, the checked-out commit and each checksum. If any checksum differs from the build's, it fails before restoring a cache. ## Proposed host dependency outputs ICE and JPEG XS should join the existing cache waterfall, but they have different compatibility boundaries. ```mermaid flowchart LR ICE_SRC[ICE version and patches] --> H_ICE([hash: ICE source]) KERNEL[target kernel ABI and arch] --> H_ICE_ART([key: ICE module]) H_ICE --> H_ICE_ART SVT[SVT-JPEG-XS revision] --> H_JPEG([hash: JPEG XS bundle]) H_MTL([hash: MTL]) --> H_JPEG H_JPEG --> H_FFMPEG([hash: FFmpeg with JPEG XS]) ``` The JPEG XS bundle is installed under `.local_install/jpegxs` and contains the SVT runtime, `pkg-config` metadata, and MTL bridge plugin. Its hash includes the SVT revision, MTL hash, architecture, toolchain/build options, and bridge sources. Consumers update `LD_LIBRARY_PATH`, `PKG_CONFIG_PATH`, and the Kahawai plugin registry to use that tree; they do not install it into `/usr/local`. The ICE output contains the patched `ice.ko` plus metadata describing its source hash, target kernel release, architecture, compiler, and vermagic. Kernel release and architecture are part of the cache key. `validate-host` rejects incompatible modules before touching the NIC and uses SHA-256, not MD5, for identity. There is one ICE producer in `build.yml`, running on the `e835` fleet runner so the artifact is built against the validation kernel ABI. The dedicated `e810`, `e830`, and `e835` hardware validation hosts share that ABI, so all validation jobs restore the same ICE cache entry. NIC type is not part of the key and does not require separate producer jobs. A kernel mismatch indicates runner drift and fails validation instead of triggering a host-side build. Driver activation is separate from driver compilation. Activation asks one question — is the running driver already the cached module? — and does nothing if it is. The kernel exposes no hash of a loaded module, so the answer comes from the artifact being the file at the `updates/` path modprobe prefers, the loaded `srcversion` matching that file, and the reported Kahawai version. On a mismatch it stops NIC users, zeroes the VFs of the ice PFs, unloads dependent modules and ICE, installs the validated module, and confirms the running driver is that artifact. VFs are not recreated: the test jobs build the VF state they need. Gtest and pytest jobs never compile, install, or reload a driver. Preparing the NIC is likewise the job's step and not the suite's work. A gtest job runs `sudo task ci:bind-test-ports`, which creates the trusted VFs on one PF, two more on the other port of its card for NoCtx, and binds two DMA channels on that PF's NUMA node; `.github/scripts/gtest.sh` then only reads what that left, runs the cases and reports. A retry re-runs the case on the same ports: a card rebuilt under a running suite is how a bare-metal runner ends up wedged, and a case that only passes after its NIC was rebuilt is not a pass worth reporting. The detailed artifact contracts and acceptance criteria are maintained in [the GitHub Actions issue](../.github/github_actions_issue.md). ### The failure mode this design has `actions/cache` saves in a post step that runs whether the job passed or not. A run that dies part-way through installing a component still stores that half-written tree under the key derived from its sources. Every later run with the same sources restores it, reads `cache-hit == true`, skips the build, and fails on whatever the tree is missing. The source checksum is unchanged, so the poisoned entry cannot expire on its own — it has to be evicted by hand or outlived. Observed as: `MTL=HIT` in the cache notice, followed by `ERROR: mtl >= 22.12.0 not found using pkg-config`, on a branch that had not touched a single file feeding the `mtl` hash. Two rules close it: 1. a restored tree counts as a hit only if it is structurally usable — the artifact consumers resolve is present (`libdpdk.pc`, `mtl.pc`, `libavcodec.pc`, a plugin `.so`) — otherwise it is treated as a miss; 2. the cache is written only when the job succeeded, which for `actions/cache` means splitting it into `actions/cache/restore` plus an explicit save step gated on `if: success()`. ## Workflow authoring practices Workflow YAML describes ordering, permissions, conditions, and runner selection. It must not become the implementation layer. Do not add multi-line `run: |` programs to workflows or composite actions. Add a named task to the repository [Taskfile](../Taskfile.yml), and put substantial shell logic in a focused script called by that task. A developer and GitHub Actions must invoke the same task. Two narrow one-line exceptions remain. A composite action that runs before Task is installed may invoke a focused checked-in script through `$GITHUB_ACTION_PATH`. An `actions/github-script` step may use a one-line `require(...)` loader for a checked-in module because `github`, `context`, and `core` only exist inside that action. The policy checker still rejects multi-line `run` and `script` values recursively on active workflow and action YAML; `.github/legacy` is intentionally excluded. This gives every operation a short, searchable log label and a command that can be reproduced outside GitHub. Tasks should be idempotent where practical, validate their inputs, fail on the first unsafe condition, and leave enough diagnostic output to identify the failed operation. Apply these rules to new and modified workflows: - pin third-party actions to immutable commit SHAs; - use minimal job-level `permissions`; - set explicit timeouts and per-host concurrency controls; - keep build, host validation, test execution, and report collection separate; - use caches for reusable dependencies and artifacts for run-specific outputs; - validate restored cache contents and save caches only after successful builds; - make cache-key inputs and job dependencies explicit; - keep secrets out of logs, command lines, caches, and artifacts; - use environment files or action outputs for structured hand-off between steps; - keep cleanup steps explicit and guarded with `if: always()` where required; - run the matching `task ci:*` command through the local CI harness before merging workflow changes. These practices are informed by DevOps Directive's [Complete GitHub Actions Course - From BEGINNER to PRO](https://www.youtube.com/watch?v=Xwpi0ITkL3U&t=6767s) and adapted for MTL's persistent self-hosted runners and hardware lifecycle.