High Quality Recordings with NVIDIA NVENC (in OBS® Studio)

With StreamFX's NVENC you can achieve the quality targets in one of two ways: Constant Quantization Parameters (CQP) or Variable Bitrate with Constant Quality (VBR-CQ). Both have their own advantages and downsides, and it depends entirely on your requirements which one you should use. Modern NVIDIA GPUs (Pascal and newer) also all support 4:4:4 and 4:2:0 encoding at most of the quality levels shown here.

Constant Quantization Parameter (CQP)

Of the two methods available, CQP is by far the faster method, with hardly any noticable impact on other parts of the GPU. It also supports vastly higher framerates compared to the alternative method, thanks to NVIDIA optimizing their presets further. With the "Fastest (P1)" preset it is now possible to record 1920x1080 NV12 at nearly 460 FPS on an RTX 3080 or higher. Unfortunately none of my PCI-E SSDs, not even the PCI-E 4.0 from Samsung, managed to withstand the amount of data requested at once, so the exact upper limit is unknown.

Option Lossless Near Lossless Indistinguishable High Quality Acceptable Quality
Preset Fastest (P1)
Tune Lossless High Quality
H264
Profile High 4:4:4 Predictive High 4:4:4 Predictive (when using I444)
High (when using NV12)
Level Automatic
H265
Profile Main
Tier High
Level Automatic
Rate Control Options
Mode Constant Quantization Parameter
Multi-Pass Single Pass
Look Ahead 0 frames
Adaptive I-Frames Disabled
Adaptive B-Frames Disabled
Quantization Parameters
I-Frame QP
P-Frame QP
B-Frame QP
0 0 to 4 5 to 10 11 to 16 17 to 22
Adaptive Quantization
Spatial Adaptive Quantization Disabled
Temporal Adaptive Quantization Disabled
Other Options
Maximum B-Frames Any
B-Frame Reference Mode Any
Zero Latency Default
Weighted Prediction Default
Non-reference P-Frames Default
Reference Frames -1 frames
Low Delay Key-Frame Scale -1
Key Frames
Interval Type Frames Frames
Seconds
Interval 1 frame 1 frame
0.5 seconds
OBS Studio Advanced Video Settings
Color Format I444 I444 or NV12
Color Space Identical to Input (sRGB on PC, Bt.709/sRGB for Console)
Color Range Identical to Input (Full on PC, Partial on Console)

Variable Bitrate with Constant Quality (VBR-CQ)

The second method for recording is VBR-CQ, which is similar to x264's CRF but not a perfect match. While x264's CRF will improve with higher presets and lower CRF values, NVENC's VBR-CQ has a flat ceiling that simply can't be bypassed, no matter what settings are applied. Still, it has its uses for Indistinguishable, High quality and Acceptable Quality recordings. Performance wise it is far slower and has much more impact on the machine, so it should not be used for Single-PC setups.

Option Indistinguishable High Quality Acceptable Quality
Preset Medium (P4) or slower
Tune High Quality
H264
Profile High 4:4:4 Predictive (when using I444)
High (when using NV12)
Level Automatic
H265
Profile Main
Tier High
Level Automatic
Rate Control Options
Mode Variable Bitrate
Multi-Pass Two Pass at Quarter Resolution
or: Two Pass at Full Resolution
Look Ahead 32 frames
Adaptive I-Frames Enabled
Adaptive B-Frames Enabled
Limits
Target Quality 1.00 to 10.99 11.00 to 17.99 17.00 to 22.99
Target Bitrate 0 kbit/s
Maximum Bitrate 0 kbit/s
Buffer Size 0 kbit
Quantization Parameters
Minimum QP
Maximum QP
-1
I-Frame QP
P-Frame QP
B-Frame QP
-1
Adaptive Quantization
Spatial Adaptive Quantization Enabled
Spatial Adaptive Quantization Strength 1 (Lowest) to 15 (Highest)
Temporal Adaptive Quantization Enabled
Other Options
Maximum B-Frames As many as your GPU supports
B-Frame Reference Mode Each if supported
Half if supported
None in any other case
Zero Latency Default
Weighted Prediction Default
Non-reference P-Frames Enabled
Reference Frames As many as your GPU supports
Low Delay Key-Frame Scale -1
Key Frames
Interval Type Seconds
Interval 1.00 seconds or more
OBS Studio Advanced Video Settings
Color Format I444 or NV12
Color Space Identical to Input (sRGB on PC, Bt.709/sRGB for Console)
Color Range Identical to Input (Full on PC, Partial on Console)

Additional Information

  • Multi-Pass at Full Resolution is a new option, and has measurably increased quality in VBR-CQ. However, it is very expensive and should be the first option changed if performance is a problem.
  • Look Ahead in VBR-CQ can be freely adjusted as needed, as long as it is not lower than the number of B-Frames and Reference Frames.
  • Adaptive B-Frames is required for VBR-CQ, as B-Frames can both increase and decrease quality if used in the wrong place.
  • Stronger Spatial Adaptive Quantization will focus more bitrate towards visually complex areas, but may starve smoother areas of the bitrate it needs.
  • The Key Frames Interval can be freely adjusted for anything but Lossless, as I-only encoding is required to guarantee the least data loss.
  • The old presets were deprecated by NVIDIA and now map to the new P1-P7 presets, which come with massive performance or quality boosts. High Quality is now Medium (P4), while High Performance is closer to Faster (P2).
Comments for: High Quality Recordings with NVIDIA NVENC (in OBS® Studio)