[Bug]: Using hevc_amf encoder in ffmpeg>=7.1, -maxrate is ignored resulting in huge files with very high bitrates

[TanMan1217]

Describe the bug
In previous versions of ffmpeg<7.1, using hevc_amf with "-quality 5" was sufficient to encode with a bitrate ~ 2mbps. With current versions of ffmpeg, this isn't working anymore. I tried setting "-maxrate 2000k", but this was also ignored. The encoded video is now huge with a very high bitrate. The problem only exists using the hevc_amf encoder with ffmpeg versions >= 7.1.

To Reproduce
Steps to reproduce the behavior:

1. Use ffmpeg (full) for Windows (from Gyan) with version >= 7.1
2. ffmpeg -i input -c:v hevc_amf -quality 5 output
3. Alternatively, ffmpeg -i input -c:v hevc_amf -maxrate 2000k output

Setup (please complete the following information):

• OS: Windows 23H2 22631.4391
• ffmpeg >= 7.1
• I'm using the integrated GPU on the Ryzen 5700G
• Encoder

I originally posted the problem as an ffmpeg bug, but they suggested I also post this here.

[rhutsAMD]

1. The correct command line is expect to be:

   ffmpeg.exe -i input.mp4 -c:v h264_amf -b:v 1M -max_rate 1.6M out.mp4

2. If the command line is without -max_rate, then there is a warning message “rate control mode is PEAK_CONSTRAINED_VBR, but rc_max_rate is not set”. The encoding will still work normally, and the default max_rate of 1.5x (1.5 * bitrate) will be applied.

   ffmpeg.exe -i input.mp4 -c:v h264_amf -b:v 1M out.mp4

3. If the command line is without -b:v 1M, then a default bitrate of 20Mbps will be applied.

   ffmpeg.exe -i input.mp4 -c:v h264_amf out.mp4

4. Considering the reported case plus -maxrate,

   ffmpeg -i input -c:v hevc_amf -maxrate 2000k output

Since the command line does not specify the bitrate -b:v, then the default bitrate of 20M will be applied by AMF.
Since the -maxrate 2000k is smaller than the default bitrate of 20M, then the -maxrate 2000k will be ignored and won’t take effect.
As a result, a large file will be produced where the actual bitrate is 20M.

A patch will be submitted to remove the warning message “rate control mode is PEAK_CONSTRAINED_VBR, but rc_max_rate is not set”. It brought a little confusion.

For the bitrate parameter -b:v, we previously set it to 2M in FFmpeg, but it has now been changed to -1. In AMF, the default value is 20M. This explains the difference seen between past and present behavior.