Session Statistics Guide#
How to read MTL per-session counters. Field-level docs live in the headers
(st_api.h, st20_api.h, st30_api.h, st40_api.h, st41_api.h).
API#
st<NN>_<rx|tx>_get_session_stats(handle, &stats);
st<NN>_<rx|tx>_reset_session_stats(handle);
<NN> ∈ 20 video, 30 audio, 40 ancillary, 41 fast metadata.
All counters are uint64_t, monotonic, thread-safe (per-session spinlock).
RX packet processing pipeline#
┌──────────────────────────────────────────────────────────────────────────────┐
│ NIC queue │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 1. VALIDATE (payload type, SSRC, packet length, interlace F-bits) │
│ │
│ FAIL ──► port[i].err_packets++ │
│ stat_pkts_wrong_{pt,ssrc,len,interlace}_dropped++ (video) │
│ stat_pkts_wrong_{pt,ssrc,interlace}_dropped++ (anc/fmd) │
│ stat_pkts_{wrong_pt,wrong_ssrc,len_mismatch}_dropped++ (audio) │
│ packet DISCARDED │
└──────┬───────────────────────────────────────────────────────────────────────┘
│ pass
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 2. PER-PORT SEQUENCE CHECK │
│ │
│ forward gap ──► port[i].lost_packets += gap_size │
│ stat_lost_packets += gap_size │
│ backward seq ──► port[i].reordered_packets++ │
│ same seq ──► port[i].duplicates_same_port++ (audio/anc/fmd only) │
│ (video charges port[i].lost_packets at frame recycle — see step 8) │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 3. PRE-REDUNDANCY TOTALS │
│ │
│ port[i].packets++ ◄── "what arrived on this wire" │
│ port[i].bytes += pkt_len │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 4. REDUNDANCY FILTER (timestamp/seq already seen on the other port?) │
│ │
│ YES ──► stat_pkts_redundant++ (expected on 2-port sessions) │
│ packet DISCARDED │
└──────┬───────────────────────────────────────────────────────────────────────┘
│ unique
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 5. POST-REDUNDANCY LOSS DETECTION │
│ │
│ Audio/Anc/FMD: session_seq_id gap ──► stat_pkts_unrecovered += gap │
│ Video: frame completion ──► stat_pkts_unrecovered += missing │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 6. POST-REDUNDANCY TOTAL │
│ │
│ stat_pkts_received++ ◄── "what the app actually got" (per pkt) │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 7. ENQUEUE to slot / RTP ring (transport) │
│ │
│ no free slot ──► stat_pkts_no_slot++ (video) │
│ ring full ──► stat_pkts_rtp_ring_full++ (video RTP) │
│ anc enqueue fail ──► stat_pkts_enqueue_fail++ (anc/fmd) │
│ audio offset overflow ──► stat_pkts_dropped++ (audio) │
│ no free framebuff ──► stat_slot_get_frame_fail++ (video/audio) │
│ │
│ first pkt of a new frame on this port │
│ ──► port[i].frames++ (audio/anc/fmd) │
└──────┬───────────────────────────────────────────────────────────────────────┘
│ frame complete (last pkt arrived, or timeout closed it)
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 8. FRAME COMPLETION (transport notify_frame_ready → pipeline) │
│ │
│ Video: per-port frame accounting │
│ port P delivered enough pkts ──► port[P].frames++ │
│ port P was short ──► st20_rx_user_stats │
│ .frames_partial[P]++ │
│ (same for port R) │
│ gap → estimated missing pkts ──► stat_pkts_unrecovered += est │
│ per-port deficit (deferred) ──► port[i].lost_packets += deficit │
│ deficit = max(0, span − recv_on_port[i]) │
│ span = pkts_received + this-frame all-port holes │
│ charged at frame recycle, not at completion, so late │
│ redundant twins are counted first (stats appear one frame later) │
│ │
│ ST20 / ST22 / ST30 (transport): intra-frame loss detected │
│ ──► stat_frames_incomplete++ │
│ │
│ ST20p / ST30p / ST40p: frame has unrecovered loss │
│ ──► stat_frames_corrupted++ │
│ (ST_FRAME_STATUS_CORRUPTED; see │
│ Frame-level counters) │
│ │
│ Pipeline (ST20p / ST30p / ST40p): │
│ no free user framebuff ──► stat_frames_dropped++ │
│ frame NOT delivered │
│ handed off to user ring ──► stat_frames_received++ │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 9. APP get_frame() │
└──────────────────────────────────────────────────────────────────────────────┘
port[i].packets = what arrived on wire i. stat_pkts_received = packets the
session accepted (post-redundancy). stat_frames_received = frames the app
actually got.
The five counters you monitor#
Counter |
Meaning |
|---|---|
|
Packets missing on this port (pre-redundancy). Video: per-frame deficit |
|
Backward arrival on the same port |
|
Same seq twice on the same port (audio/anc/fmd; always 0 for video) |
|
Cross-port duplicate filtered — expected on 2-port sessions |
|
Missing on all ports — real loss |
stat_lost_packets == Σ port[i].lost_packets
stat_pkts_unrecovered <= stat_lost_packets
save_rate (%) == 100 * lost / (lost + unrecovered)
save_rate answers: “is redundancy covering the gaps?” 100% = yes.
Note. A same-port duplicate or retransmit inflates that port’s received count, so a genuine
port[i].lost_packetsdeficit can be masked (read as 0). Aggregate cross-port recovery accounting is unaffected.
Reading table#
|
|
|
|
Verdict |
|---|---|---|---|---|
0 |
0 |
0 |
≈ received |
Healthy |
50 |
0 |
0 |
≈ received |
P degraded, R covers ( |
50 |
30 |
5 |
< received |
5 pkts lost on both → real loss |
10 |
n/a |
10 |
0 |
Single-port, no redundancy |
Frame-level counters#
Session-wide, type-agnostic counters living in st_rx_user_stats /
st_tx_user_stats. Use these (not port[i].frames) to answer “how many
frames did the app receive / drop / send”.
Counter |
Side |
Meaning |
|---|---|---|
|
RX |
Frames delivered to the app via the get-frame / notify path |
|
RX |
Frames the pipeline could not deliver (no free user slot) |
|
RX |
Frames delivered with |
|
TX |
Frames whose final packet was committed to the wire ( |
|
TX |
Frames the pipeline dropped because the app handed them too late ( |
Populated by pipeline session types (ST20p, ST30p, ST40p for both
RX and TX). For transport-only paths and types with no per-frame
integrity concept (stat_frames_corrupted on ST41 RX),
the relevant counters stay 0.
ST20 / ST22 / ST30 only: the transport-layer field
stat_frames_incomplete(inst20_rx_user_stats/st30_rx_user_stats) is not the same asstat_frames_corrupted.stat_frames_incompletefires whenever the transport detects intra-frame loss, including when the frame is then silently discarded becauseRECEIVE_INCOMPLETE_FRAMEis not set;stat_frames_corruptedonly counts corrupted frames the app actually consumed viaget_frame().stat_frames_incomplete - stat_frames_corrupted= corrupted frames dropped before reaching the app.
port[i].frames — two flavors (per-port, not a session total)#
Sessions |
|
Partial counter |
|---|---|---|
Video (ST20/ST22) |
This port delivered enough pkts to complete the frame |
|
Audio/Anc/FMD |
New frame’s first packet arrived on this port (race winner) |
n/a |
port[i].frames is per-port and the two flavors above do not compose: summing
across ports does not yield total frames delivered to the app. For end-to-end
frame accounting, use the session-wide common counters:
stat_frames_received, stat_frames_dropped, stat_frames_corrupted (RX)
and stat_frames_sent, stat_frames_dropped (TX) — see
Frame-level counters above. Use port[i].frames /
st20_rx_user_stats::frames_partial[i] only for per-port redundancy debugging.
Video frame-mode invariant (ST20 RX with frame callbacks, per port):
port[i].frames + frames_partial[i] == stat_frames_received.
Each delivered frame (status COMPLETE or RECONSTRUCTED) bumps exactly one
of the two counters per port. Frames that never complete on either port bump
stat_frames_incomplete instead and leave both per-port counters untouched.
Does NOT hold for: ST22 single-port (port R is left at zero, since there is no port R to account for) or any RTP-mode session (only the winning port’s
framesis bumped;frames_partialis unused).Example. A 2-port redundant RX session reports:
stat_frames_received = 1000 (TX sent 1000, all delivered to app) frames_dropped = 0 unrecovered = 0 port[P].frames = 950 frames_partial[P] = 50 port[R].frames = 50 frames_partial[R] = 950The invariant holds on both ports: 950 + 50 = 1000.
Reading it: P completes 95% of frames on its own; R only 5%. Every frame still reaches the app because whenever R is short, P fills the gap (and vice versa) — that’s redundancy doing its job. But R is missing packets on 950 of 1000 frames (95%). R is one switch hiccup away from real loss.
Investigate R:
mtl_get_port_stats(R).rx_hw_dropped_packets, switch port stats, MTU, SFP/cable.
err_packets — not data loss#
port[i].err_packets = packets that arrived but the session refused (wrong PT, wrong
SSRC, length mismatch, etc.). Does not reduce stat_pkts_received or cause
stat_pkts_unrecovered. Most common benign cause: another stream on the same multicast
group leaks in — confirm via stat_pkts_wrong_pt_dropped / stat_pkts_wrong_ssrc_dropped.
TX packet processing pipeline#
┌──────────────────────────────────────────────────────────────────────────────┐
│ APP put_frame() / put_frame_abort() │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 1. PIPELINE INTAKE (ST20p / ST30p / ST40p) │
│ │
│ put_frame_abort() ──► stat_frames_dropped++ │
│ notify_frame_done(DROPPED) │
│ put_frame() ──► hand frame to transport ring │
└──────┬───────────────────────────────────────────────────────────────────────┘
│ frame ready for transport
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 2. USER-TIMESTAMP VALIDATION (only if the app supplied an RTP timestamp) │
│ │
│ timestamp invalid ──► stat_error_user_timestamp++ │
│ (the frame is still scheduled) │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 3. PACING DECISION │
│ │
│ app handed the frame too late, slots had to be skipped │
│ ──► stat_epoch_drop += skipped_slots │
│ notify_frame_late() (if app registered) │
│ │
│ app handed the frame too early, scheduled far in the future │
│ ──► stat_epoch_onward += onward_slots │
│ │
│ pacing snapped to a different epoch than requested │
│ ──► stat_epoch_mismatch++ (audio/anc/fmd) │
└──────┬───────────────────────────────────────────────────────────────────────┘
│ frame slot assigned
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 4. PER-PORT BUILD (transport tasklet, P and R independently) │
│ │
│ packet built and queued ──► port[i].build++ │
│ packet enqueued to NIC ──► port[i].packets++ │
│ port[i].bytes += pkt_len │
│ transient build error ──► stat_recoverable_error++ │
│ fatal build/send error ──► stat_unrecoverable_error++ │
│ (session needs restart) │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 5. FRAME COMPLETION (transport — last packet of frame committed to wire) │
│ │
│ all packets sent on this port ──► port[i].frames++ │
│ │
│ pipeline late-drop watchdog (post-send): │
│ cur_tai > frame_tai + frame_period │
│ ──► stat_frames_dropped++ │
│ notify_frame_done(DROPPED) │
│ otherwise ──► stat_frames_sent++ │
│ notify_frame_done(COMPLETE) │
└──────┬───────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ 6. APP notify_frame_done(status) │
└──────────────────────────────────────────────────────────────────────────────┘
port[i].packets = what was put on wire i. stat_frames_sent = frames the
app successfully delivered end-to-end. stat_frames_dropped is the only
counter that reflects ST_FRAME_STATUS_DROPPED — stat_error_user_timestamp
only counts user-supplied RTP timestamp validation failures and is unrelated
to drops.
Per-session quirks#
Video (ST20/ST22). Loss uses frame-internal pkt_idx.
stat_pkts_unrecovered is estimated as (frame_size − recv_size) / avg_pkt_size.
Cross-frame reorders are not tracked. Watch
stat_frames_dropped, stat_frames_incomplete, stat_pkts_rtp_ring_full,
stat_pkts_no_slot, stat_slot_get_frame_fail.
Audio/Anc/FMD (ST30/40/41). Loss uses post-redundancy session_seq_id →
stat_pkts_unrecovered is exact. ST30 adds stat_pkts_dropped,
stat_pkts_len_mismatch_dropped, stat_slot_get_frame_fail,
stat_frames_incomplete. ST40/41 add
stat_pkts_wrong_interlace_dropped, stat_pkts_enqueue_fail. ST20p,
ST30p and ST40p RX mark frames whose constituent packets had unrecovered
(post-redundancy) gaps as ST_FRAME_STATUS_CORRUPTED and count them in
stat_frames_corrupted (the app should consult frame->status; for whether
such a frame is delivered at all see Frame-level
counters). For count-based assembly (ST30 audio /
ST40 anc) a fully-lost frame is absorbed — the delivered frame count drops
rather than emitting an extra incomplete frame — so under heavy contiguous
loss stat_frames_corrupted is a lower bound on frame-integrity impact.
ST40p RX seq stats are bitmap-derived (transport-side). The
seq_lost/seq_discontfields onst40_frame_info(and the underlyingst40_rx_frame_meta) are computed in the RX session from a per-frame 64-bit session-merged sequence bitmap. Each accepted packet (after the per-port redundancy filter) flips its bit; at marker timeseq_lost = (max_offset + 1) - popcount(bitmap). Packets that arrive on the redundant port and fill a gap left by the primary port flip the same bit, so cross-port redundancy is reflected directly in the count — there is no false positiveseq_discontfrom cross-port reorder. Theport_seq_lost[]/port_seq_discont[]arrays still report the per-port intra-frame view for observability.
TX (any type). stat_epoch_drop = app handed frame too late, slots were
skipped (counter advanced by the number of skipped slots).
stat_epoch_onward = system clock ran past max_onward_epochs ahead of the
next free slot — the frame was still scheduled, but pacing skipped onward
slots (advanced by the onward delta). stat_epoch_mismatch = pacing snapped
to a different epoch than requested. stat_recoverable_error /
stat_unrecoverable_error = TX faults during build/send.
stat_error_user_timestamp counts user-supplied RTP timestamp validation
failures during pacing setup; it does not track frames dropped because
the pipeline handed them to TX too late — those are
stat_frames_dropped (and surface to the app as
notify_frame_done(status=ST_FRAME_STATUS_DROPPED)).
Periodic stat log lines#
Emitted every stat_dump_period_s (default 10s); zero-delta lines are suppressed.
RX_<TYPE>_SESSION(idx): fps F.f frames N pkts M [(redundant R)]
RX_<TYPE>_SESSION(idx): per-port arrivals P=<n> pkts (<f> frames <verb>), R=<n> pkts (<f> frames <verb>)
RX_<TYPE>_SESSION(idx): per-port loss covered by redundancy: <L> of <T> pkts
(P:<dp>=<%>, R:<dr>=<%>), unrecovered (lost on both) <U>, save_rate=<z>%
RX_<TYPE>_SESSION(idx): unrecovered pkts <U> (single-port form, err)
<verb> = complete (video) or first (audio/anc/fmd).
Line |
Level |
|---|---|
|
|
|
|
|
|
|
|
|
|
Troubleshooting with Stats#
Symptom |
Counter to check |
Likely cause |
|---|---|---|
No packets at all |
|
Wrong IP/port, multicast not joined, NIC link down |
Packets arrive but |
|
PT or SSRC mismatch between sender and receiver |
|
|
Network congestion, packet loss, or RSS misconfiguration |
|
Compare with |
Real data loss — redundancy could not cover; check path diversity |
|
Upstream switch / path |
Switch loop, cable fault, LAG misconfig, or tcpreplay loop |
|
Upstream switch / path |
ECMP/QoS reorder; for ST20 limited to intra-frame reorders |
|
|
Redundant port not receiving data |
|
— |
Normal — each packet arrives on both ports, one copy filtered |
Video |
|
Incomplete frames due to packet loss or frame buffers exhausted |
TX |
|
Application providing frames too late; callback blocking |
|
— |
Receiver too slow: return frame buffers faster or increase ring size |
RX log: |
video: |
Back-pressure summary. Raise |