"The h setting is 110Mbps, GOP3, AQ2, X1.5 on FBs and FL. The j setting is 110Mbps, GOP3, AQ3, X1.39 on FBs and FL. Both worked. The attached diagram is from the j setting. My gut feeling is that... the B-frame seems little too big. Or is that a typical behavior from AQ3 setting?"
I think it is typical of AQ3 and 4... and to my mind it's a good thing, those big b-frames, especially with non moving shots. Going to test your latest right now.
@driftwood Unfortunately stream parser several times showed the same thing. Maybe this card? Maybe this is my model of camera? When I changed FBs and Fl to 1.3 .... Hmmm, I think I'm doing something wrong. Anyone can help me ?
When you lower frame limit it will end up raising the effective QP value to accommodate the lower bandwidth allocated to I frames and do other strange things. It's sort of like clipping - not really a good thing. Better to leave frame limit a little higher and lower AQ until it works. According to my calculations (based in part on your feedback), the ideal AQ value for GOP3 at 100Mb would be between 1 and 2. It seems that 2 works pretty well, 3 is clearly too high according to my latest math. With GOP1, you probably want to use AQ1.
For GOP 6 I would estimate that you could go to AQ3 at any bitrate. With GOP1 I think it's AQ 1-2 no matter the bitrate. This is because AQ is automatically scaled according to bitrate.