MSEdgeExplainers

RTCRtpReceiver Decoder State Changed and Error Events

Authors

Much of this explainer synthesizes and consolidates prior discussions and contributions from members of the WebRTC working group.

Participate

Introduction

Game streaming platforms like Xbox Cloud Gaming and Nvidia GeForce Now rely on hardware decoding in browsers to deliver low-latency, power-efficient experiences. During a stream, the decoder's state can change. The codec can be renegotiated, the receiver can fall back from hardware to software decoding, or the decoder can fail outright. Applications have no event-driven way to observe these changes and failures as they occur. The existing statistics must be polled and terminal decoder errors are not surfaced at all.

This proposal has two parts. The first part adds two events on the receiver. The decoderstatechange event fires when the decoder's state changes. Codec changes always fire it, while decoder implementation changes, such as hardware-to-software fallback, fire only while hardware exposure is allowed. The decodererror event fires when the decoder hits a terminal error. Together the events replace inefficient polling.

The second part expands the existing hardware-exposure check for actively used interactive media receivers. The current check has no input because context capturing state applies to the entire context. To add a receiver-specific condition without broadening access to protected outbound statistics, the check would accept an optional RTCRtpReceiver. This would allow qualifying non-capturing applications to receive implementation-change events and read the protected decoderImplementation and powerEfficientDecoder statistics.

User-Facing Problem

When the decoder fails terminally, playback freezes. The failure is not surfaced to the application, so there is no direct signal that decoding has stopped. The decoder's state can also change during a stream, for example when the codec is renegotiated. There is no event for these changes either. The only way to observe decoder state today is to poll getStats() repeatedly, which is inefficient.

A related concern is decoder fallback. When the receiver falls back from a hardware to a software decoder, end users may experience increased latency, degraded quality, and battery drain. Developers would like to detect this in real time. They previously relied on the decoderImplementation statistic. As of Chromium M110+, it is available only while the application is capturing camera or microphone input. The first part of this proposal preserves the existing hardware-exposure check while replacing polling with an event for applications that already qualify. Today, the check allows exposure only when the context capturing state is true. That condition fits real-time communication but not interactive streaming, where media capture is unrelated to the application's need to react to decoder fallback.

The second part of this proposal would broaden decoder hardware exposure for a specific RTCRtpReceiver without requiring capture. The receiver's document would need to be visible and focused, the receiver would need to be actively receiving and decoding live WebRTC video, and the user would need to be demonstrably interacting with the experience. Qualifying interaction conditions could include pointer lock, keyboard lock, recent meaningful gamepad activity, or fullscreen combined with recent user input. These conditions could allow applications to react to fallback during an active interactive session without exposing protected decoder information to passive or background contexts.

Goals

Non-goals

User Research

Feedback from Xbox Cloud Gaming, Nvidia GeForce Now and similar partners shows:

Proposed Approach

Part 1: Decoder state and error events

Introduce two events on RTCRtpReceiver:

Codec changes and decoder errors are surfaced without requiring getUserMedia() permission and are not subject to the decoder hardware-exposure gate. The decodererror event is coarse, carrying no decoder- or device-specific detail. Changes that reveal hardware-versus-software decoding are surfaced only when exposing hardware is allowed. This condition already gates the existing decoderImplementation and powerEfficientDecoder stats, so the event reveals nothing the page cannot already read (see Privacy Considerations). This enables applications to alert users, re-negotiate codecs, and debug issues at runtime.

Event triggers

Two changes trigger the decoderstatechange event:

The decodererror event fires when the decoder hits a terminal, unrecoverable failure, for example when hardware decoding fails and no software decoder is available for the negotiated codec (such as H.265). The failure is surfaced as an EncodingError DOMException. When a fallback succeeds (a software decoder is available), the receiver keeps decoding and may fire decoderstatechange instead, subject to the gating above.

Proposed IDL

partial interface RTCRtpReceiver {
    attribute EventHandler ondecoderstatechange;
    attribute EventHandler ondecodererror;
};

interface RTCDecoderStateChangeEvent : Event {
    constructor(DOMString type, RTCDecoderStateChangeEventInit eventInitDict);

    // The RTP timestamp of the media frame associated with this event.
    readonly attribute unsigned long rtpTimestamp;
};

interface RTCDecoderErrorEvent : RTCDecoderStateChangeEvent {
    constructor(DOMString type, RTCDecoderErrorEventInit eventInitDict);

    // The inherited rtpTimestamp reports when the error occurred.
    readonly attribute DOMException error;
};

Example

const pc = new RTCPeerConnection();

pc.addEventListener('track', (event) => {
  const receiver = event.receiver;

  // Codec changes are always reported. Decoder implementation changes are
  // reported only while hardware exposure is allowed for this receiver.
  receiver.addEventListener('decoderstatechange', async (ev) => {
    // Query getStats() for the codec currently in use on this receiver.
    const stats = await receiver.getStats();
    let codec = 'unknown';
    for (const report of stats.values()) {
      if (report.type === 'inbound-rtp' && report.codecId) {
        const codecStats = stats.get(report.codecId);
        if (codecStats) {
          codec = `${codecStats.mimeType}|${codecStats.sdpFmtpLine}`;
        }

        // decoderImplementation and powerEfficientDecoder are protected. They
        // are present only while hardware exposure is allowed.
        if (report.decoderImplementation) {
          logMetric(`Decoder implementation: ${report.decoderImplementation}`);
        }
        if (report.powerEfficientDecoder !== undefined) {
          logMetric(`Power efficient decoder: ${report.powerEfficientDecoder}`);
        }
        break;
      }
    }
    logMetric(`Decoder state change: codec=${codec}, time=${ev.rtpTimestamp}`);
  });

  // The error event reports a decoder failure.
  receiver.addEventListener('decodererror', (ev) => {
    // ev.error is a DOMException describing the failure. The inherited
    // rtpTimestamp marks when it occurred.
    showToast('Video playback error');
    logMetric(`Decoder error: ${ev.error.name} - ${ev.error.message}, time=${ev.rtpTimestamp}`);
  });
});

Part 2: Expand the hardware-exposure check

Part 2 proposes extending the existing hardware-exposure check with an optional receiver input. When no receiver is supplied, the check would behave as it does today and allow exposure only when the context capturing state is true. When a receiver is supplied, the receiver's active interactive media state would provide another way for the check to return true.

To avoid requiring applications to repeatedly reestablish user intent after temporary focus or visibility changes, this proposal distinguishes recognition of an interactive media session from its current eligibility for hardware exposure.

Interactive media session recognition

An RTCRtpReceiver would be recognized as belonging to an interactive media session when all of the following entry conditions are true at the same time:

Recognition would be specific to one receiver. Activity on one receiver would not recognize another receiver as belonging to an interactive media session.

Once established, recognition would persist until the receiver's video track ends, its associated transceiver is stopped, its peer connection is closed, its document navigates or is discarded, or it stops receiving and decoding video for a sustained session-termination period. The document becoming hidden or losing focus, an interaction lock ending, or a recent-input window expiring would not by itself end recognition.

Active interactive media state

A receiver would be in the active interactive media state while all of the following are true:

Hardware exposure through the receiver-specific condition would be suspended when any condition becomes false. If the user temporarily switches tabs or applications, the receiver would remain recognized as part of the same interactive media session, but protected decoder information would not be exposed while its document is hidden or unfocused. Exposure could resume automatically when the user returns and the receiver is again actively decoding, without requiring another pointer lock, keyboard lock, fullscreen interaction, or gamepad input.

The time windows used for recent frame decoding, recent input, and session termination remain to be defined. They should tolerate ordinary network jitter, temporary interruptions, and pauses in user input without allowing recognition or exposure to persist after the interactive session has ended.

Conceptually, the WebRTC Stats algorithm would be updated as follows:

To check if hardware exposure is allowed, given an optional RTCRtpReceiver receiver, run the following steps:

  1. If the context capturing state is true, return true.
  2. If receiver was given and receiver is in the active interactive media state, return true.
  3. Otherwise return false.

The relevant receiver would be supplied when checking whether to expose decoderImplementation or powerEfficientDecoder for that receiver, or whether to dispatch a decoder-implementation-change event. No receiver would be supplied when checking outbound statistics, including encoderImplementation, powerEfficientEncoder, psnrSum, and psnrMeasurements. Those fields would therefore remain available only when the context capturing state is true. Future callers would likewise receive the existing behavior unless they explicitly supply a receiver.

Eligibility transitions

If the receiver leaves the active interactive media state, protected decoder statistics would no longer be exposed and decoder-implementation-change events would be suppressed. This suspension would not clear the receiver's interactive media session recognition.

If the receiver later reenters the active interactive media state and its current decoder implementation differs from the last implementation exposed to the application, the receiver would fire one coalesced decoderstatechange reflecting the current observable state. Intermediate changes while exposure was suspended would not be replayed. The event would use the rtpTimestamp of a frame decoded after exposure resumes rather than reveal when a hidden transition occurred.

Alternatives Considered

  1. Use decoderImplementation info via WebRTC Stats API
    • Rejected because it now requires getUserMedia() permissions, which are invasive and have a high failure rate.
  2. Use MediaCapabilitiesInfo.powerEfficient
    • Rejected because this is a static hint that does not update when the browser silently switches from hardware to software decoding.
  3. Guess based on decode times
    • Unreliable and has masked bugs in production.
  4. Add decoderFallback field to RTCInboundRtpStreamStats
    • Rejected because relying on stats to trigger a change felt like an anti-pattern and the recommendation was to explore an event driven solution. Additionally, there were concerns around fingerprinting.
    • WebRTC March 2023 meeting – 21 March 2023

Privacy Considerations

Event exposure

The events carry only the media frame's rtpTimestamp and expose no hardware vendor, device identity, or decoder implementation detail.

Hardware-exposure safeguards

The proposed decoderstatechange event for decoder implementation changes and the protected decoder statistics would invoke the hardware-exposure check with the relevant receiver. Outbound statistics would invoke the same check without a receiver. This keeps one hardware-exposure algorithm while ensuring that active interactive media state can allow only receiver-related exposure.

Interactive media session recognition is scoped to a specific receiver and requires active video decoding in a visible, focused document together with a qualifying indication of user interaction. Recognition alone does not expose protected information. Receiver-specific exposure remains limited to periods when the recognized receiver is actively decoding and its document is visible and focused. This allows temporary focus or visibility changes without requiring the user to reestablish the session, while preventing background observation of decoder state. The additional condition does not unlock protected outbound statistics, whose exposure remains tied to context capturing state.

Relationship to MediaCapabilities

MediaCapabilitiesInfo.powerEfficient can expose whether a hypothetical configuration is expected to be power efficient without requiring active capture. The protected WebRTC statistics differ because they describe the actual decoder and can change during a session, potentially revealing contention for shared hardware resources across tabs or applications. Requiring an active receiver, a foreground document, and ongoing user interaction limits this additional exposure to contexts that need the live operational signal.

Security Considerations

This proposal does not introduce a new network transport, media source, script execution mechanism, or ability to select, configure, reserve, or control a decoder. It reports state changes and terminal failures associated with an existing RTCRtpReceiver.

Decoder events and protected statistics must be scoped to the affected receiver and its associated document. A qualifying interaction associated with one receiver, document, or origin must not enable access for unrelated receivers or origins. Cross-origin iframe support is out of scope. Only browser-observed interaction state may satisfy the gate; synthetic events must not qualify.

Documents that are not fully active, including documents in BFCache or disconnected documents, must not receive decoder events or access protected decoder statistics through the expanded gate. Changes that occur while access is blocked must not be queued or replayed later.

The decodererror event exposes only a generic EncodingError. It must not include decoder, hardware, driver, platform error-code, or resource-availability details. Applications should treat these events as notifications that the decoder changed or failed, not as proof of the underlying cause.

Open Questions

Stakeholder Feedback

Last discussed in the 2025-11-13 Media WG Meeting (TPAC): Slides 110-117 & minutes

References & Acknowledgements

Many thanks for valuable feedback and advice from:

Links to past working group meetings where this has been discussed:

This site is open source. Improve this page.