This document defines how tests/acceptance/ selects cases. Framework
architecture and execution are documented in
doc/acceptance-design.md and
doc/acceptance_quickstart.md.
A test verifies an MTL behavior such as packet pacing, format conversion, stream integrity, or backend selection. Application adapters are transport mechanisms for expressing that behavior. They are not a normal Cartesian test dimension, but their capability coverage must remain visible.
Use RxTxApp for library parameter sweeps. Use FFmpeg and GStreamer in focused adapter-conformance and interoperability cases. Execute each behavior only through applications that can express it. Also retain an explicitly skipped application leg when an adapter cannot express that behavior: the collected skip is the coverage record and its reason names the missing capability.
Keep capability skips sparse: use the smallest representative input, avoid duplicating the same gap across unrelated media or parameter axes, and remove the skip when support lands. Do not use a skip to hide a reachable behavior that is unfinished or failing; that case needs an executable test or a bug investigation.
Choose sweep depth from the implementation selected by the parameter.
| Class | Meaning | Coverage |
|---|---|---|
| Path selector | Selects another tasklet, backend, queue model, codec, or hardware engine | Representative load ladder and coupled arithmetic axes |
| Arithmetic modifier | Changes packetization, pacing, timestamp, or buffer calculations | Boundary values and the axis used by the calculation |
| Value parameter | Copied or compared without changing the implementation | Two or three representative values |
| Deployment parameter | Controls placement, logging, or capacity | Performance or dedicated deployment tests |
Do not build full Cartesian products. Add a second axis only when source code shows that the two values share control flow or arithmetic. For larger coupled sets, use pairwise cases.
A test should vary one named behavior and pin unrelated inputs. Media filenames
must not hide changes in resolution, frame rate, format, or interlace state.
Select explicit entries from mtl_engine/media_files.py as load anchors:
| Rung | Media | Purpose |
|---|---|---|
| LS | i576i50 |
Interlaced SD and pacing fallback |
| L0 | i720p59 |
Low-cost progressive stream |
| L1 | i1080p59 |
Baseline functional stream |
| L2 | i2160p59 |
High bandwidth |
| L3 | i2160p119 |
High packet cadence |
| L4 | i4320p29 |
Large frames at throughput comparable to L3 |
Use the smallest subset that reaches the mechanism under test. Expensive rungs require a mechanism-based justification. Tests that require a specific pixel or transport format, codec, interlace state, audio layout, ancillary payload, or metadata item use their specialized registry instead; the generic video ladder does not replace those behavior-specific inputs.
Every test must declare what makes it pass:
Path-selecting tests must assert the path MTL actually resolved. Requesting a pacing way, RSS mode, or transport backend is insufficient because MTL can fall back while continuing to transmit.
An unavailable oracle is explicit. Do not silently omit integrity or compliance to make a failing case green. Lossy codecs may use application and compliance results when byte-exact integrity is not a valid property.
Cross-application coverage is one baseline case for each meaningful TX/RX application pair. Resolution, frame rate, and codec matrices remain in their own functional suites unless negotiation specifically depends on them.
Each application is paired with RxTxApp in both directions, and its adapter runs the pair. GStreamer puts RxTxApp in place of its own element on one end.
ST 2022-7 is tested as one session with two independent member streams. Each member uses a distinct interface path and each captured destination must be included in compliance analysis. A redundant result is valid only when both member streams satisfy their required oracle.
Acceptance assets must be deterministic and registered in
mtl_engine/media_files.py. Generation metadata must agree with the declared
pixel and transport formats. New tests must not depend on an undocumented
host-local media corpus.
Before merging a matrix change, verify that: