NMSF — Neural Video Streaming over MoQ

Architecture

End-to-End Neural Video Pipeline

🎬

Source

MP4 decoded to YCbCr 4:4:4 via FFmpeg

➜

🧠

NVC Encode

DCVC-RT neural compression via OpenVINO + rANS entropy

➜

🌐

MoQ Relay

Any standard relay. No codec awareness needed.

➜

🧠

NVC Decode

Entropy decode + DCVC-RT synthesis via OpenVINO

➜

📺

Output

Reconstructed video via FFmpeg

The Innovation

A New Packaging Format for Neural Video

Why NMSF?

Neural Video Codecs produce compressed bitstreams that are fundamentally different from H.264, HEVC, or AV1 — entropy-coded latent tensors, not NAL units or fMP4 boxes. Existing packaging formats can't represent them without unnecessary overhead and semantic mismatch.

NMSF extends MSF (MOQT Streaming Format) with a new "nvc" packaging type, following the same pattern as CMSF for CMAF. NVC tracks sit alongside LOC and CMAF tracks in the same MoQ Broadcast.

Wire Format

Neural keyframes (Intra) are the first Object in each MoQ Group. Delta frames (Inter) follow. Any MoQ relay forwards them without modification.

frame_type 1 byte 0x00=Intra 0x01=Inter
frame_number 4 bytes absolute sequence (u32 BE)
width 4 bytes frame width (u32 BE)
height 4 bytes frame height (u32 BE)
payload_len 4 bytes payload byte count (u32 BE)
payload variable NVC compressed bitstream

Packaging Comparison

NMSF vs CMAF

Aspect	CMAF (CMSF)	NVC (NMSF)
Container	fMP4 (moof + mdat)	None (17-byte header)
Init segment	moov box	Catalog metadata only
Codec signaling	`avc1.42c01e`	`dcvc-rt`
Keyframe	IDR NAL unit	Neural Intra (context init)
Per-frame overhead	~100-200 bytes	17 bytes
Relay awareness	None	None

Design Goals

Built for the MoQ Ecosystem

Codec Agnostic

Works with any NVC: DCVC-RT, SSF, FVC, RLVC. The packaging doesn't depend on the codec.

Relay Transparent

Standard MoQ relays forward NMSF Objects without modification. No plugins or codec awareness.

Late-Join Ready

Every MoQ Group starts with an Intra frame. Subscribers begin decoding from any Group boundary.

MSF Compatible

NVC video tracks coexist with LOC and CMAF audio tracks in the same Broadcast catalog.

Minimal Overhead

17-byte fixed header per Object. No container boxes, no padding, no alignment constraints.

Open Specification

IETF Internet-Draft. draft-herz-moq-nmsf-00

Codec Registry

Supported Neural Video Codecs

Codec	Full Name	Status
`dcvc-rt`	DCVC Real-Time (Microsoft)	Tested
`dcvc-fm`	DCVC Feature Modulation	Compatible
`ssf`	Scale-Space Flow	Compatible
`fvc`	Feature-space Video Coding	Compatible
`elfvc`	Efficient Learned Flexible VC	Compatible

Get Started

Download NMSF Publisher & Subscriber

Coming Soon

Pre-built Linux binaries for dcvc-pub and dcvc-sub will be available with a push key from moqcdn.net.

Requirements: Ubuntu 24.04, Intel Xeon (AMX recommended), OpenVINO 2025.1+, DCVC-RT model weights from Microsoft.