I am starting this topic to discuss ways to achieve constant quality encoding with variable bit rates. This idea can be applied to GOPs of different lengths.
Cake v2.3, for the GH2 v1.1 firmware image and PTool 3.64d or later
Designed for reliability and spanning in 720p, HBR, 24p, and Variable Movie Mode (NTSC and PAL)
2 - 2.5 times stock bit rates
Support for all lenses, high ISOs, and ETC mode
SanDisk Extreme cards recommended, Class 10
For higher quality in HBR 30p and 25p modes, see http://personal-view.com/talks/discussion/2123/gh2-cake-topic/p23#Item_568 .
Cake was designed with stability and spanning in 720p, HBR, 24p, and VMM as its highest priorities. It is intended for event coverage, where long recording times and reliability are required. For each mode I've set the bit rate limit with a wide margin of safety below the highest bit rate that is stable and spanning, to allow for variability in the rate control. This gives high confidence that the settings will span each time when you use a recommended memory card. Cake has spanned 100% reliably in my experience, and I don't recall anyone reporting a spanning failure.
The choice of memory card will affect reliability. I designed the settings using SanDisk Extreme cards. Some people have reported success with other Class 10 cards. I recommend the SanDisk Extreme Pro 32-GB 95 MB/s card. (not 64 GB, because of the "file number limit exceeded" problem) I strongly recommend that you periodically perform several full-overwrite formattings in your PC using the SD Card Association Formatter utility, followed by a formatting in the camera, to put your card in the best possible condition.
There are no restrictions on which lenses or camera settings you can use: Lumix lenses, high ISOs, and ETC mode are fine.
With these settings I strive for identical frame-to-frame quality, and the highest possible image quality within the reliability and spanning constraints, especially for content of moderate-to-high detail and motion. The encoder uses adaptive quantization rate control for a nearly constant bit rate at the bit rate limit, and constant quantization / variable bit rate encoding when not constrained by the bit rate limit.
Cake is optimized for 24H, HBR, and SH modes. 24L and H modes are set approximately 15-20 Mbps lower.
Cake release history:
1.0: initial release; for v1.0e firmware; rate control using frame size limits
1.1: rate control using fallback mode
1.2: for v1.1 firmware; adds support for all AVCHD modes
2.0: rate control using adaptive quantization; improved quality in HBR
2.1: adds spanning in 720p
2.2: improved rate control in 24L
2.3: improved rate control in 24H
Quality is subjective. A proper constant quality encoder will have models of human perception of quality, and try multiple quantization parameters on each macroblock until it finds the one that yields the quality it is targeting.
The GH2 has nothing so complicated in its encoder, but you can get very close to constant quality by using a constant quantization parameter or auto quantizer setting, while having no frame limit or bit rate limit to constrain the frame sizes and raise the QP. In my testing, I've found that this approach yields nearly identical quality in I-frames and P-frames, with no need to change quantizer table assignments. With stock quantizer table assignments, B-frames are way too small and don't match the quality of the other frames, even when they have the same QP. My observation of the B-frames in Flow Motion is that they don't match the quality of the I-frames as closely as P-frames do even with stock quantizer table assignments. Perhaps there is additional work that can be done to get B-frames to match more closely, but personally I don't see a lot of value in trying to include B-frames, since P-frames are not that much less efficient.
Generally you don't want to have truly unconstrained bit rates and constant quality: we have hardware limitations, after all. In Cake, I keep the GOP size small enough to span by limiting the frame sizes to 3.33 megabits, which if all the frames are that size, is 80 Mbps in 24-fps mode. Without that limit, the bit rate for QP22 can go as high as 100 Mbps when you shoot at ISO 12800. Using a frame limit is not ideal, since you will in some cases be limiting the sizes of I-frames even when you are not at 80 Mbps, which will decrease their quality relative to the P-frames. For a frame limit of 3.33 megabits and QP of 22, the limit is really only apparent on the I-frames for high detail, low noise, low motion scenes (ISOs 250 and below on a tripod). Once you add a bit of motion or noise, the I-frames fall below 3.33 megabits. In high noise, the frame limit makes the I-frames have lower quality than the P-frames, but you'll hardly notice such a small difference among all that noise.
Ideally we could use the GH2 encoder's bit rate limiter to keep the GOP sizes small enough to span, instead of using frame limits. That would open up the possibility of constant quality encoder settings at longer GOPs, like 6 or 9. GOP6 would be an awesome sweet spot for quality and efficiency. I'm working now on how to get the bit rate limiter to work correctly at GOPs shorter than the defaults.
Shorter GOPs span at higher bit rates. GOP12 spans at 32 Mbps; GOP6 at 60 Mbps; GOP3 at 80 Mbps; GOP1 at 88 Mbps or perhaps higher. My theory is that the camera is writing a GOP at a time to the memory card, and that the write operation needs to complete in less than some length of time for spanning to succeed.
And of course we know that with the faster SanDisk Extreme Pro 95 MB/s 64 GB SDXC card, GOP1 spans at 146 Mbps. But I'm not keen on using that card, because, besides being very expensive, SDXC cards suffer from the occasional "file limit exceeded" error. If you need long recording times, imagine what happens when you encounter that error: to continue recording to the same card, you need to reformat it. That's a frightening scenario, avoidable by having multiple cards on hand (expensive), or by offloading the data before reformatting (slow). And yes, there is the technique of recording a short clip after spanning and before turning the camera off to avoid the error, but I don't trust myself to always adhere to that protocol to avoid a disaster.
Ah, I've tasted this Cake before! :) And since it's a whole different approach (VBR) I also think it deserves it's own thread.
Thanks again!
@balazer when you say "cake beats it for everything else" you mean FlowMotion L50 and not 24H setting of FlowMotion, yes?
I mean for scenes that are not on a tripod or at ISOs higher than 250, Cake beats Flow Motion's 24L (50 Mbps) setting for quality.
Flow Motion 24H (100 Mbps) of course always wins for quality. The bit rate is always higher (more than double, on average) and it doesn't span.
Thank you, @duartix. I want to quote what you said in the other thread about Cake:
Cake test 24p @ISO3200 & Panasonic 14-42mm. Very reactive bitrate. :) Amazing quality and detail. Great grainy shadows. No blocks. Absolutely no noticeable difference between I/P frames in my eyes.
I'm looking forward to having more people try these settings out and let me know how they're working.
@balazer interesting approach , would this be possible even for fw 1.1 when hacked ?!
I would think so, but we'll have to wait and see.
Thanks for your efforts balazer.
Just tested it. (Cake) Quality looks good. With Quantum 50, although also generally good, I sometimes got weird looking 'low resolution parts' in darker segments of an image. This looks more stable in the dark area's. I only tested it at ISO 160, since I like to minimize noise.
Test, after test after test... listen up motherfuckers. Youre never gonna be happy.
listen up motherfuckers. Youre never gonna be happy.
Amen
Thanks for testing, @erw456.
Just to be clear, there's no question that 100 Mbps and 146 Mbps intra patch settings will beat Cake for quality every time. If you need the maximum quality, Cake isn't for you. Cake is about increasing efficiency while maintaining consistently high quality and spanning with a variety of cards. Cake is an improvement over existing 44-66 Mbps patch settings.
@balazer :
1 - About B-Frames and after some footage observations, I couldn't agree with you more, neither can the WIKI: http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Controversies
2 - How's the progress on the bit rate limiter?
3 - The pitfall.
For someone that does his encodes in VBR for more than 10 years, this is now the approach that makes the most sense to me, however there is something that I wonder if it could be pursued to produce more satisfactory results: the L mode! In a perfect world we would have a bigger and different quantizer for 24L. But this is a Constant Quantizer ~ Constant Quality patch and there is only one quantizer parameter. Even though it works wonders on 24H because it's practically not limited, what is to be expected of 24L when all that is there to work with are bitrates and frame limits? It's a hard if not impossible task to make it Constant Quantizer/Quality. Could there be a solution in the quantizer/scaling tables? Or should we assume right away 24L will always be a fallback and shouldn't be used unless we're prepared to give up true VBR?
And isn't it strange that the GH2 works originally in 24H & 24L modes with the same quantizer?
Thanks to LPowell's work on Flow Motion, I have a new version of Cake that does away with the frame limit and instead uses LPowell's method of limiting the bit rate. This approach combines the best features of both patch settings:
GOP3 for better efficiency than GOP1
Consistently high quality at a variable 20-80 Mbps; under 50 Mbps, on average
Spanning. (tested with SanDisk Extreme HD Video cards)
Now you can have quality and spanning at the same time - no need to choose one or the other as you do in Flow Motion. The camera automatically varies the bit rate so as to achieve consistently high quality, using high bit rates only when necessary.
P-frame quality matches I-frame quality extremely well (better than Flow Motion's B-frames)
Frame sizes are unconstrained, for better quality in high detail scenes
I've done a fair bit of testing, and the results look extremely good. In the old version of Cake, I-frame quality could be limited for detailed, low motion, low noise video. But now for these scenes Cake does just as well as Flow Motion, and potentially better than 100-Mbps intra settings. Playing back footage shot on a tripod at ISO 160, the frames are so similar to each other that I thought my player was stuck, until I saw a truck drive by in the video.
At this point just 1080/24p and Variable Movie Mode are working. Other settings are copied as-is from Flow Motion, and might not work correctly.
In my testing, the patch settings are stable. But we'll need to have other people test before I can call these settings non-experimental. The new version is in the second post of this topic.
Big thanks to LPowell for making these new settings possible.
I've been trying to follow these conversations about P-frames, b-frames, GOP, etc, and I have to admit it's over my head. I'm still not sure which patch I need. The real issue I find with the GH2, more than anything else, is noise. I just want decent bitrate, good motion, and extremely low noise (or as low as possible) in low light/low detail areas. Is cake what I'm looking for, or would I be better off with an older patch with a higher AQ level? I only need the camera to be stable in 1080 24p. Someone please help. My head is spinning over here.
The best thing you can do to keep the noise low is shoot at a low ISO. Do some tests and see how the noise increases at higher ISOs. Of course to shoot at low ISOs you'll want to do what you can with lighting, choose a fast lens, and keep the shutter speed as slow as you can tolerate.
If you shoot at low ISOs, your choice of patch settings doesn't matter too much. Cake, Flow Motion, Aquamotion, or Quantum 100 would all be fine. At higher ISOs, you're going to want to record that noise as accurately as you can, so it can be removed using Neat Video. Cake has no significant loss up to about ISO 3200. Above 3200, you'll be better off with 146-Mbps intra setting like Driftwood's Reaquanted or Seaquake, or Gop3zilla.
In the FAQs you'll find some generic patch setting recommendations and tips for low-light shooting. There's a link to the FAQs at the top of this page.
I appreciate the response balazer. I know to keep my iso levels down. The problem I have is often my subject is lit but the background isn't. In that case the background is always noisy. Never heard of Neat Video, I'll have to check it out. What about Driftwood's Quantum v7? I saw some pretty impressive footage shot with that patch. Having trouble finding it though.
Quantum is still beta. There are other 146-Mbps intra settings that are stable and perform similarly.
And definitely check out Neatvideo, it can work wonders!
@balazer Do you know if the 720P/60 mode works well in Cake 1.1? I need that sometimes? If not, what do I do to merge a good 720p/60 patch with Cake 1.1? Thanks Izhar
what duartix said.
(bit late but) @yrowan, no patch will remove noise for you. In fact the high quality patches will preserve as much noise grain as possible, as the camera counts it as detail. That's actually a good thing, because the raw noise can be removed better later (with something like NeatVideo) than the splotches you get from noise-reduction or heavy compression.
@balazer, just tried Cake 1.1 with 1.1 firmware/3.64d - quick indoor test shows it working fine, with expected 24H bitrates (~50 average, 70+ top). Very exciting from reading your detailed description, I've been using Chris' 66mb as my go-to as it was totally reliable for me (except for spanning) & the right quality/filesize tradeoff.
I'll need to test outdoors in anger (will report back), but apart from the expected quality & motion increase and spanning @ similar file sizes, the lower GOP should also be faster to edit: win/win/win/win : ).
It looks like you're new here. If you want to get involved, click one of these buttons!