WingNews logo WingNews
top | item 38071823

(no title)

DaleCurtis | 2 years ago

Thanks for the nice write up! I work on the WebCodecs team at Chrome. I'm glad to hear it's mostly working for you. If you (or anyone else) has specific requests for new knobs regarding "We may need more encoding options, like non-reference frames or SVC", please file issues at https://github.com/w3c/webcodecs/issues

discuss

order

hokkos|2 years ago

I use the WebCodecs API with VideoDecoder with a very specific use case, to get data arrays using the great compression of video codec and the data having temporal coherency. Demo here : https://energygraph.info/d/f487b4fd-45ad-4f94-8e7e-ea32fc280...

And I have some issues with the copyTo method of VideoFrame, on mobile (Pixel 7 Pro) it is unreliable and output all 0 Uint8Array beyond 20 frames, to the point I am forced to render each frame to an OffscreenCanvas. Also the many formats of frame output around RGBA/R8 with reduced range 16-235 or full range 0-255 makes it hard to use in my convoluted way.

DaleCurtis|2 years ago

Please file an issue at https://crbug.com/new with the details and we can take a look. Are you rendering frames in order?

Android may have some quirks due to legacy MediaCodec restrictions around how we more commonly need frames for video elements, frames only work in sequential order since they must be released to an output texture to access them (and releasing invalidates prior frames to speed up very old MediaCodecs).

vlovich123|2 years ago

There’s a few that would be neat:

* maybe possible already, but it’s not immediately clear how to change the bitrate of the encoder dynamically when doing VBR/CBR (seems like you can only do it with per-frame quantization params which isn’t very friendly)

* being able to specify the reference frame to use for encoding p frames

* being able to generate slices efficiently / display them easily. For example, Oculus Link encodes 1/n of the video in parallel encoders and decodes similarly. This way your encoding time only contributes 1/n frame encode/decode worth of latency because the rest is amortized with tx+decode of other slices. I suspect the biggest requirement here is to be able to cheaply and easily get N VideoFrames OR be able to cheaply split a VideoFrame into horizontal or vertical slices.

DaleCurtis|2 years ago

* Hmm, what kind of scheme are you thinking beyond per frame QP? Does an abstraction on top of QP work for the case you have in mind?

* Reference frame control seems to be https://github.com/w3c/webcodecs/issues/285, there's some interest in this for 2024, so I'd expect progress here.

* Does splitting frames in WebGPU/WebGL work for the use case here? I'm not sure we could do anything internally (we're at the mercy of hardware decode implementations) without implementing such a shader.

jampekka|2 years ago

I'm currently working with WebCodecs to get (the long awaited) frame-by-frame seeking and reverse playback working in the browser. And it even seems to work, albeit the VideoDecoder queuing logic seems to give some grief for this. Any tips on figuring out how many chunks have to be queued for a specific VideoFrame to pop out?

An aside: to work with video/container files, be sure to check the libav.js project that can be used to demux streams (WebCodecs don't do this) and even used as a polyfill decoder for browsers without WebCodec support!

https://github.com/Yahweasel/libav.js/

Rodeoclash|2 years ago

Wow, great to see some work in this space. I've been wanting to do reverse playback, frame accurate seek and step by step forward and back rendering in the browser for esports game analysis. The regular video tag gets you somewhat of the way there but navigating frame by frame will sometimes jump an extra frame. Likewise trying to stop at an exact point will often be 1 or 2 frames off where you should be. Firefox is much worse, when pausing at a time you could +-12 frames where you should be.

I must find some time to dig into this, thanks for sharing it.

kixelated|2 years ago

Thanks for WebCodecs!

I'm still just trying to get A/V sync working properly because WebAudio makes things annoying. WebCodecs itself is great; I love the simplicity.