Super User is a question and answer site for computer enthusiasts and power users. It only takes a minute to sign up.
Sign up to join this community
is it possible to do partial edits to each frame of video (e.g. adding a logo) without transcoding? And save video with the same format?
I mean, pixel positions of each frame are calculable; so is it possible to modify file bytes in such a way that they result in the desired change on a frame (e.g. adding a logo)?
I know in formats like XviD this is not possible (easily at least) but it is completely impossible or is possible for some kind of formats (e.g. MPG or raw AVI)?
This is only (practically) possible for lossless video encoding.
It's very simple to do what you want for losslessly encoded video codecs. Many of those store the video data pixel-by-pixel, mostly in the YUV colorspace. Also, every frame stands on its own.
It would be pretty easy to edit such a video because you know where every frame is, and where every pixel position is in the byte stream. You can simply overwrite parts of the byte stream without consequences to the remaining stream. Even if the lossless video was arithmetically compressed, you could uncompress it first, edit, and then compress and save again—no problem with that.
So, for example, if you have a raw YUV video in an AVI container, you could edit it on a frame-by-frame basis and save it to YUV in AVI again.
But if you want to do lossy video encoding—or keep the original lossy encoding—after the editing step, this is impossible. There are two problems with that: the encoding process itself, and the fact that often, frames depend on each other.
Lossy compression tries to remove redundancy as much as possible, by removing details the human eye doesn't see. This is done in several steps, but the most important one involves transforming the pixel domain into the frequency domain, often with variants of the Discrete Cosine Transform.
What this step does is that it takes a block of, say, 8×8 pixels and transforms it into a block of frequency coefficients. From this block, certain coefficients are dropped, which reduces the amount of information (and thus compresses the size), but also throws away visual information, which reduces the quality of the video. Which coefficients are dropped depends on the quality setting of the encoder. The video is then not stored as pixels, but as frequency coefficients.
When you want to edit a lossy video, you first have to reconstruct the frequency coefficients into a pixel-by-pixel representation again (like mentioned above). At this point you could edit the video, and insert a logo, but once you'd want to store it again, you'd have to perform the transformation step again—and throw away information. This is, in essence, the main cause for generation loss.
Another problem is the fact that for most lossy video, some frames depend on the information contained in other frames. More specifically, B- and P-frames only contain offsets of earlier (or later) B/P or I-frames. If you changed the contents of an I-frame, all the other dependent frames would change when they are decoded, which is typically not what you want. Peter Cordes' answer below hightlights this point. And indeed he's right that in principle, you could edit a lossy I-frame only video in place, but it would be practically very hard to accomplish.
So, very simply speaking, unless you can store a video losslessly, you can't edit a video in-place without sacrificing quality.
slhck is correct if we're talking about complex video codecs with motion vectors (edit: actually, any kind of intra or inter prediction). You could modify a h.264 stream by re-encoding only the macroblocks in the area you want to replace with your logo. Logo in I frames, skip blocks in P and B frames. But any time the camera panned, there would be nearby blocks that get their picture data by copying from the part of the picture that's now logoed. So the edges of your logo would get copied around, corrupting other parts of the picture until the next I frame.
So he's right, but it's because of references, not lossy-ness. (edit: His edits clear up most of the confusion.)
For simpler codecs where every block is independent (not even any intra prediction), e.g. mjpeg, you could replace the coded blocks only in the area you wanted to logo, without decoding / re-encoding the rest of the picture. I think there's support for doing this in a single jpeg image file, but IDK if anything supports it for mjpeg, without breaking out the frames to separate jpeg files.
http://jpegclub.org/jpegjoin/jpegjoin.txt has info on how to use jpegtran to replace some blocks of jpeg files, without decoding/encoding the rest of the image.
Here's an example of mucking around with a video stream without decoding/re-encoding. Hiding data in a h.264 stream, in the low coefficients of iPCM macroblocks. (literal, not DCTed, coding.) You have to decode / re-encode the CABAC entropy-coding layer. (It's the final zip-style compression for the bitstream.)
If the source is in a lossless format, you're not incurring a huge filesize cost by decompressing, editting, and recompressing the whole thing, just CPU. Many lossless formats essentially zip each whole frame separately, rather than operating on blocks, so you'd have to do a full decompress-edit-recompress. Lossless h.264 has all the inter and intra prediction of normal lossy h.264, so even though it does have macroblocks, a full decode-edit-encode is needed there, too.
None of this changes anything for the actual practical purpose of logoing a video, UNLESS your lossy source was mjpeg. I'm just writing this for those curious about why.
I would say it should be possible.
During the US changeover to OTA HD (ATSC), almost all the networks provided video from their home studios via satellite, using a very high bit rate (much higher than the OTA bit rate) video stream. They then provided each local television station with a decoder and lower bitrate encoder, so they could completely decode the high-quality stream, add their station logo, picture inserts, etc., and then encode to the broadcast standard bitrate for transmission.
However, the Fox network opted to encode the stream in their home studios, presumably using a higher quality (and expense) encoder, but to the broadcast bit rate. Then, they supplied a "splicer" to each local affiliate which could partially decode the stream, insert logos, crawls, etc., and then re-encode just that portion before broadcast.
I do realize that the MPEG-2 stream used in OTA broadcast differs from h.264/MPEG-4, but from what I know of the technical details of the formats, this is most likely still feasible with h.264/MPEG-4. (In fact, MPEG-4 actually adds features that support alpha compositing, although I suspect few decoders may actually handle this feature.)
That said, finding such a piece of software has eluded me. I have streams to which I'd like to add a graphics bar (mostly a sports score), without re-encoding the already mediocre quality from a cheap camera. But I haven't found one yet.
Edit: I did find this paper on "Replacing picture regions in H.264/AVC bitstream by utilizing independent slices," that appears to provide the technical details to achieve this. But I'm still looking for some proof-of-concept software.
