While testing new updates to the VES testing suite, I discovered some weird behavior in NVENC. Here's a list of them, maybe NVIDIA can shed some light on it:

G-Sync affects NVENC Encoding Speed

For unknown reasons G-Sync affects the rate of encoding provided by NVENC, no matter how you submit frames to it. So in case you're hitting "Encoding Overloaded" in OBS Studio for no actual reason, try disabling G-Sync globally.

Constant Quality is a lie?

The "Constant Quality" encoding method often used for archival and also often mistaken for an alternative to x264 CRF has a default bitrate limit. For H.264/AVC the maximum bitrate it will pick is 135mbit/s, while for H.265/HEVC it will pick is 25mbit/s. You can affect these limits by explicitly setting the maxBitRate ("-maxrate" in FFmpeg) and vbvBufferSize ("-bufsize" in FFmpeg) to a higher value.

• H.264/AVC
  • 6.2: -maxrate 800000k -bufsize 800000k
  • 6.1: -maxrate 480000k -bufsize 480000k
  • 6.0/5.2: -maxrate 240000k -bufsize 240000k
  • 5.0: -maxrate 135000k -bufsize 135000k (Driver Default)
• H.265/HEVC
  • 6.2:
    • High: -tier high -maxrate 800000k -bufsize 800000k
    • Main: -tier main -maxrate 240000k -bufsize 240000k
  • 6.1:
    • High: -tier high -maxrate 480000k -bufsize 480000k
    • Main: -tier main -maxrate 120000k -bufsize 120000k
  • 6.0/5.2:
    • High: -tier high -maxrate 240000k -bufsize 240000k
    • Main: -tier main -maxrate 60000k -bufsize 60000k
  • 5.1:
    • High: -tier high -maxrate 160000k -bufsize 160000k
    • Main: -tier main -maxrate 40000k -bufsize 40000k
  • 5.0:
    • High: -tier high -maxrate 100000k -bufsize 100000k
    • Main: -tier main -maxrate 25000k -bufsize 25000k (Driver Default)

Please note that doing this means that you will need a more modern decoder to view or edit the footage. I still recommend to go with the highest possible maximum bitrate in order to get the best out of your footage.

H.265/HEVC struggles representing Foliage

Even with the above maximum bitrate fix applied, the HEVC NVENC encoder has a weird affinity to just murder foliage for no actual reason. It seems that there is a noise pattern detection method that freaks out once the foliage gets too large, and fails to adapt to the fact that the foliage now has even more detail than before.