MTL Acceptance Tests — Setup and Run

MTL Acceptance Tests — Setup and Run#

Everything needed to get tests/acceptance/ pytest running, and to fix it when it breaks. For how the framework is built see acceptance-design.md; for MTL itself see build.md and run.md.

The .local_install rule — read this first#

Pytest resolves every binary it launches under .local_install/, a tree separate from the build/ + /usr/local install that gtest/KahawaiTest uses. This is hardcoded in tests/acceptance/mtl_engine/const.py (PREFIX = ".local_install") and is the single most common reason a host “with MTL built” still cannot run a test.

Pytest launches

Resolved path

RxTxApp

.local_install/mtl/bin/RxTxApp

MtlManager

.local_install/mtl/bin/MtlManager

FFmpeg / ffprobe

.local_install/ffmpeg/bin/

GStreamer plugin

.local_install/gstreamer/gstreamer-1.0/

A working build/manager/MtlManager or tests/tools/RxTxApp/build/RxTxApp does not satisfy pytest. Symptom when the tree is missing: the mtl_manager fixture errors with Failed to start MtlManager on host while gtest runs fine.

Prerequisites#

  • Python 3.9+ and root access — the framework connects over SSH to root@ even on a single host, because NIC binding and hugepage setup need it. There is no unprivileged mode.

  • An Intel E810/E830/E835 NIC. VFs are created automatically; do not create them by hand.

  • Test media, on NFS or generated locally (see below).

  • FFmpeg/GStreamer only if you run the integration suites — they are built into .local_install, not taken from the system packages.

Manual setup#

Only needed to debug or customise one stage; the script above automates all of it.

Python environment#

cd tests/acceptance
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -r common/integrity/requirements.txt

Configuration#

configs/gen_config.py generates both files; hand-edit only to adjust.

configs/topology_config.yaml — hosts, connections, and NICs:

hosts:
  - name: host
    instantiate: true
    role: sut                    # "sut" or "client" (dual-host only)
    network_interfaces:
      - pci_device: 8086:1592    # vendor:device, NOT a BDF
        interface_index: 0
    connections:
      - ip_address: 127.0.0.1
        connection_type: SSHConnection
        connection_options:
          port: 22
          username: root         # must be root
          password: ''           # required key even when using a key file
          key_path: /home/<you>/.ssh/id_rsa

pci_device takes the vendor:device ID (lspci -nn | grep Ethernet), not the 0000:18:00.0 bus address. Supplying a BDF raises Incorrect format ... PCIDevice.

configs/test_config.yaml — paths and per-run behaviour:

build: /home/<you>/Media-Transport-Library/
mtl_path: /home/<you>/Media-Transport-Library/
media_path: /mnt/media
interface_type: VF               # VF | PF | VFxPF | 2VFxPF | 3VFxPF
capture_cfg:
  enable: false                  # true enables EBU LIST compliance capture
  pcap_dir: /mnt/ramdisk/pcap
  capture_time: 5
  sniff_pci_device: 8086:1592    # vendor:device of the capture NIC
  phc_sync: true                 # discipline the local PHC to system TAI
ebu_server:                      # omit to leave compliance disabled
  host: 10.0.0.9
  user: <user>
  password: <password>
ramdisk:
  media: {mountpoint: /mnt/ramdisk/media, size_gib: 32}
  pcap:  {mountpoint: /mnt/ramdisk/pcap,  size_gib: 768}

Capture needs a second NIC port in a different IOMMU group from the one carrying traffic. Without ebu_server, tests still run — the compliance verdict is simply skipped.

Test media#

Production hosts mount an NFS share at media_path. Tests skip rather than fail when an asset is missing. To generate a local set instead:

cd tests/acceptance/common
./gen_frames.sh

This needs an FFmpeg build with text filters; No such filter: 'drawtext' means the system FFmpeg is too minimal (sudo apt install ffmpeg).

The ST2110-40 GStreamer suites stage inputs and outputs on ramdisk.media.mountpoint — mount and size that ramdisk or they spill into /tmp.

Running tests#

Run from tests/acceptance/, as root, using the venv interpreter:

cd tests/acceptance
sudo -E ./venv/bin/python3 -m pytest \
    --topology_config=configs/topology_config.yaml \
    --test_config=configs/test_config.yaml \
    -m smoke -v

sudo python3 uses the system interpreter and fails with No module named pytest. The config arguments are registered by the local conftest.py, so running from another directory fails with unrecognized arguments.

Selecting tests#

$PY -m smoke -v                                   # by marker
$PY tests/single/st20p -v                         # by folder
$PY tests/single/st40p -k multicast -v            # by substring
$PY "tests/single/st20p/test_fps.py::test_st20p_fps[|fps = p60|-Penguin_1080p-|application = rxtxapp|]"
$PY <selector> --collect-only -q                  # dry run

Quote selectors containing [ ] or |.

Markers#

Suite markers:

Marker

Use

smoke

Smallest set that proves the host works

nightly

Bulk single-host coverage

performance

Dual-host capacity sweeps, long and hardware-bound

base_performance

Added at collection to a test ID that holds 1080p and 59fps

dual

Requires two hosts

ptp

Uses MTL’s internal PTP (phc2sys suppressed)

Neither performance marker selects a tests/single/ test. Select the tests/single/performance/ modules by path.

Descriptive markers: verified (see below), refactored (uses the Application-based harness), tx_side/rx_side/tx_and_rx (which side a test validates, and therefore whether a compliance verdict is meaningful), and allow_wide_compliance (accept ST 2110-21 wide instead of failing).

tests/ contains single/ and dual/ only.

Useful options#

Option

Effect

--time N

Seconds each test runs (default 15)

--keep all|failed|none

Retain result media files

--dmesg clear|keep

Whether to clear the kernel ring before each test

--media, --build, --nic, --dma

Override the config values

--num_sessions N, --sch_quota N

Performance sweeps only

Output#

Logs land in tests/acceptance/logs/<UTC-timestamp>/ (root-owned) with a logs/latest symlink — per-test logs plus MtlManager and application output and the rendered JSON configs.

sudo grep -E "EAL|hugepage|VF|RxTxApp|Traceback|err:" logs/latest/*.log | head -40

An HTML run report:

$PY -m smoke --template=html/index.html --report=report.html

A performance report across timestamped log directories:

./venv/bin/python3 common/generate_report.py logs/

Troubleshooting#

Start with ./.github/scripts/acceptance_setup.sh status — it checks kernel, hugepages, governor, ICE driver, NIC PFs, .local_install build state, NFS, venv, and configs in one read-only pass, and usually names the broken stage directly. Re-run setup --base-only or setup --pytest-only to repair it.

Symptom

Cause and fix

No module named pytest

Used sudo python3. Use sudo -E ./venv/bin/python3.

unrecognized arguments: --topology_config

Not in tests/acceptance/.

Failed to start MtlManager on host

.local_install tree missing — re-run setup. A system-wide build/ install does not count.

RxTxApp: command not found

Same cause: build into .local_install, not tests/tools/RxTxApp/build/.

error while loading shared libraries: librte_*.so

Stale linker cache. sudo ldconfig, then check ldd .local_install/mtl/bin/RxTxApp.

Media file not present ... (SKIPPED)

NFS unmounted or empty. Re-run setup with the NFS source.

Incorrect format ... PCIDevice / sniff_pci_device ... not found

A BDF was used where vendor:device is required.

SSHConnection.__init__() missing ... 'password'

Add password: '' to the topology connection.

Permission denied (publickey) for root@127.0.0.1

Public key not in /root/.ssh/authorized_keys.

ValueError: q must be exactly 160, 224, or 256 bits long

DSA key. ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa && ssh-copy-id root@localhost.

netsniff-ng: command not found

sudo apt install -y netsniff-ng.

EBU server configuration not found, test still PASSED

Data path passed, compliance skipped. Configure ebu_server + capture_cfg.

Test hangs past --time + ~30 s

Stale process: sudo pkill -9 RxTxApp MtlManager ffmpeg gst-launch-1.0.

EAL hugepage or VF binding errors

Hugepages exhausted, or VFs not bound to vfio-pci. Re-run setup.

RxTxApp segfault in iavf_tm_node_add

Stock kernel ice loaded instead of the patched out-of-tree build. Re-run setup.

A segfault anywhere other than iavf_tm_node_add is a real MTL/DPDK bug — capture a backtrace (coredumpctl gdb RxTxApp) and report it rather than working around it.

The exhaustive symptom table, kept in sync with the engine source, is in mtl-acceptance-tests.instructions.md.

Verified test criteria#

@pytest.mark.verified is a claim about evidence, not just a passing run. A test may carry it only when all three hold:

  1. Telemetry matches inputs. If the input is 60 fps, the logs and collected metrics report 60 fps within a documented tolerance. Configuration flags (e.g. rss_enabled) in the logs agree with the test inputs.

  2. The stated goal is actually checked. The test carries a Doxygen-style preamble giving objective, steps, expected outcome, and metrics, and enables every check that is feasible for it.

  3. It passes in CI on a framework-provisioned host — no manual setup — and the CI artifacts include the logs and metrics proving point 1.

See also#