Media-Transport-Library

MTL Acceptance Tests — Setup and Run

Everything needed to get tests/acceptance/ pytest running, and to fix it when it breaks. For how the framework is built see acceptance-design.md; for MTL itself see build.md and run.md.

The .local_install rule — read this first

Pytest resolves every binary it launches under .local_install/, a tree separate from the build/ + /usr/local install that gtest/KahawaiTest uses. This is hardcoded in tests/acceptance/mtl_engine/const.py (PREFIX = ".local_install") and is the single most common reason a host “with MTL built” still cannot run a test.

Pytest launches Resolved path
RxTxApp .local_install/mtl/bin/RxTxApp
MtlManager .local_install/mtl/bin/MtlManager
FFmpeg / ffprobe .local_install/ffmpeg/bin/
GStreamer plugin .local_install/gstreamer/gstreamer-1.0/

A working build/manager/MtlManager or tests/tools/RxTxApp/build/RxTxApp does not satisfy pytest. Symptom when the tree is missing: the mtl_manager fixture errors with Failed to start MtlManager on host while gtest runs fine.

Prerequisites

.github/scripts/acceptance_setup.sh is the single entry point. It composes acceptance_setup_base.sh (apt, ICE driver, DPDK, MTL + plugins into .local_install, hugepages, CPU governor) and setup_acceptance.sh (NFS media, localhost root SSH, venv, generated configs). No agent or MCP client is required.

./.github/scripts/acceptance_setup.sh status   # read-only probe, changes nothing
./.github/scripts/acceptance_setup.sh setup    # interactive: asks PF, NFS source, EBU

Non-interactive, e.g. for CI:

./.github/scripts/acceptance_setup.sh setup --auto \
    --pf-bdf=0000:c9:00.0 \
    --nfs-source=10.0.0.5:/mnt/NFS/mtl_assets/media

It prints a ready-to-run pytest command when it finishes. -h lists all flags (--base-only, --pytest-only, --check-only, --ffmpeg/ --no-ffmpeg, --gstreamer/--no-gstreamer, --ebu-ip/--ebu-user for compliance capture, --nr-hugepages, --build-mode). Both phases are idempotent — re-running on a prepared host is safe and takes seconds.

Cold setup takes roughly 7–10 minutes. Do not interrupt and retry during the first run.

Manual setup

Only needed to debug or customise one stage; the script above automates all of it.

Python environment

cd tests/acceptance
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -r common/integrity/requirements.txt

Configuration

configs/gen_config.py generates both files; hand-edit only to adjust.

configs/topology_config.yaml — hosts, connections, and NICs:

hosts:
  - name: host
    instantiate: true
    role: sut                    # "sut" or "client" (dual-host only)
    network_interfaces:
      - pci_device: 8086:1592    # vendor:device, NOT a BDF
        interface_index: 0
    connections:
      - ip_address: 127.0.0.1
        connection_type: SSHConnection
        connection_options:
          port: 22
          username: root         # must be root
          password: ''           # required key even when using a key file
          key_path: /home/<you>/.ssh/id_rsa

pci_device takes the vendor:device ID (lspci -nn | grep Ethernet), not the 0000:18:00.0 bus address. Supplying a BDF raises Incorrect format ... PCIDevice.

configs/test_config.yaml — paths and per-run behaviour:

build: /home/<you>/Media-Transport-Library/
mtl_path: /home/<you>/Media-Transport-Library/
media_path: /mnt/media
interface_type: VF               # VF | PF | VFxPF | 2VFxPF | 3VFxPF
capture_cfg:
  enable: false                  # true enables EBU LIST compliance capture
  pcap_dir: /mnt/ramdisk/pcap
  capture_time: 5
  sniff_pci_device: 8086:1592    # vendor:device of the capture NIC
  phc_sync: true                 # discipline the local PHC to system TAI
ebu_server:                      # omit to leave compliance disabled
  host: 10.0.0.9
  user: <user>
  password: <password>
ramdisk:
  media: {mountpoint: /mnt/ramdisk/media, size_gib: 32}
  pcap:  {mountpoint: /mnt/ramdisk/pcap,  size_gib: 768}

Capture needs a second NIC port in a different IOMMU group from the one carrying traffic. Without ebu_server, tests still run — the compliance verdict is simply skipped.

Test media

Production hosts mount an NFS share at media_path. Tests skip rather than fail when an asset is missing. To generate a local set instead:

cd tests/acceptance/common
./gen_frames.sh

This needs an FFmpeg build with text filters; No such filter: 'drawtext' means the system FFmpeg is too minimal (sudo apt install ffmpeg).

The ST2110-40 GStreamer suites stage inputs and outputs on ramdisk.media.mountpoint — mount and size that ramdisk or they spill into /tmp.

Running tests

Run from tests/acceptance/, as root, using the venv interpreter:

cd tests/acceptance
sudo -E ./venv/bin/python3 -m pytest \
    --topology_config=configs/topology_config.yaml \
    --test_config=configs/test_config.yaml \
    -m smoke -v

sudo python3 uses the system interpreter and fails with No module named pytest. The config arguments are registered by the local conftest.py, so running from another directory fails with unrecognized arguments.

Selecting tests

$PY -m smoke -v                                   # by marker
$PY tests/single/st20p -v                         # by folder
$PY tests/single/st40p -k multicast -v            # by substring
$PY "tests/single/st20p/test_fps.py::test_st20p_fps[|fps = p60|-Penguin_1080p-|application = rxtxapp|]"
$PY <selector> --collect-only -q                  # dry run

Quote selectors containing [ ] or |.

Markers

Suite markers:

Marker Use
smoke Smallest set that proves the host works
nightly Bulk single-host coverage
performance Dual-host capacity sweeps, long and hardware-bound
base_performance Added at collection to a test ID that holds 1080p and 59fps
dual Requires two hosts
ptp Uses MTL’s internal PTP (phc2sys suppressed)

Neither performance marker selects a tests/single/ test. Select the tests/single/performance/ modules by path.

Descriptive markers: verified (see below), refactored (uses the Application-based harness), tx_side/rx_side/tx_and_rx (which side a test validates, and therefore whether a compliance verdict is meaningful), and allow_wide_compliance (accept ST 2110-21 wide instead of failing).

tests/ contains single/ and dual/ only.

Useful options

Option Effect
--time N Seconds each test runs (default 15)
--keep all\|failed\|none Retain result media files
--dmesg clear\|keep Whether to clear the kernel ring before each test
--media, --build, --nic, --dma Override the config values
--num_sessions N, --sch_quota N Performance sweeps only
--sut_app_args='ARGS' Performance sweeps only: extra RxTxApp arguments for the measured app
--shard K/N Run only shard K of N of the selected tests; the N shards together run each test once

Output

Logs land in tests/acceptance/logs/<UTC-timestamp>/ (root-owned) with a logs/latest symlink — per-test logs plus MtlManager and application output and the rendered JSON configs.

sudo grep -E "EAL|hugepage|VF|RxTxApp|Traceback|err:" logs/latest/*.log | head -40

An HTML run report:

$PY -m smoke --template=html/index.html --report=report.html

A performance report across timestamped log directories:

./venv/bin/python3 common/generate_report.py logs/

Troubleshooting

Start with ./.github/scripts/acceptance_setup.sh status — it checks kernel, hugepages, governor, ICE driver, NIC PFs, .local_install build state, NFS, venv, and configs in one read-only pass, and usually names the broken stage directly. Re-run setup --base-only or setup --pytest-only to repair it.

Symptom Cause and fix
No module named pytest Used sudo python3. Use sudo -E ./venv/bin/python3.
unrecognized arguments: --topology_config Not in tests/acceptance/.
Failed to start MtlManager on host .local_install tree missing — re-run setup. A system-wide build/ install does not count.
RxTxApp: command not found Same cause: build into .local_install, not tests/tools/RxTxApp/build/.
error while loading shared libraries: librte_*.so Stale linker cache. sudo ldconfig, then check ldd .local_install/mtl/bin/RxTxApp.
Media file not present ... (ERROR) NFS unmounted or empty. Re-run setup with the NFS source.
Incorrect format ... PCIDevice / sniff_pci_device ... not found A BDF was used where vendor:device is required.
SSHConnection.__init__() missing ... 'password' Add password: '' to the topology connection.
Permission denied (publickey) for root@127.0.0.1 Public key not in /root/.ssh/authorized_keys.
ValueError: q must be exactly 160, 224, or 256 bits long DSA key. ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa && ssh-copy-id root@localhost.
netsniff-ng: command not found sudo apt install -y netsniff-ng.
EBU server configuration not found, test still PASSED Data path passed, compliance skipped. Configure ebu_server + capture_cfg.
Test hangs past --time + ~30 s Stale process: sudo pkill -9 RxTxApp MtlManager ffmpeg gst-launch-1.0.
EAL hugepage or VF binding errors Hugepages exhausted, or VFs not bound to vfio-pci. Re-run setup.
RxTxApp segfault in iavf_tm_node_add In-tree ice loaded instead of the out-of-tree build. Re-run setup.

A segfault anywhere other than iavf_tm_node_add is a real MTL/DPDK bug — capture a backtrace (coredumpctl gdb RxTxApp) and report it rather than working around it.

The exhaustive symptom table, kept in sync with the engine source, is in mtl-acceptance-tests.instructions.md.

Verified test criteria

@pytest.mark.verified is a claim about evidence, not just a passing run. A test may carry it only when all three hold:

  1. Telemetry matches inputs. If the input is 60 fps, the logs and collected metrics report 60 fps within a documented tolerance. Configuration flags (e.g. rss_enabled) in the logs agree with the test inputs.
  2. The stated goal is actually checked. The test carries a Doxygen-style preamble giving objective, steps, expected outcome, and metrics, and enables every check that is feasible for it.
  3. It passes in CI on a framework-provisioned host — no manual setup — and the CI artifacts include the logs and metrics proving point 1.

See also