The Art of encoding with NVIDIA NVENC

Streaming with more than one PC has been the leader in H.264 encoding for years, but NVIDIAs Turing and Ampere generation has put a significant dent into that lead. The new generation of GPUs with the brand new encoder brought comparable quality x264 medium - if you can find a GPU that is. Let's take a look at what's needed to set up your stream for massively improved quality.

The guide has been updated for:
StreamFX v0.11 and OBS Studio 27.0

Setting up NVENC (for Streaming)

Modern OBS Studio has two ways to achieve the expected quality: the built-in NVENC H.264 (new) and the addition from StreamFX called NVIDIA NVENC H.264/AVC (via FFmpeg). Both Options can achieve similar quality to x264 medium, but the latter is able to exceed that and rival x264 medium/slow in various situations. Whichever you pick, both of them support zero-copy encoding, and they're both valid options for streaming.

Built-In: OBS Studio NVENC H.264 (new)

Image reference for Turing/Ampere
The built-in NVENC option in OBS Studio is by far the simplest option and will give you almost identical quality on Maxwell, Pascal, Turing and Ampere, though Turing and Ampere will make use of the new improvements of the NVENC chip. Maxwell and Pascal users can expect to reach x264 veryfast/faster-like quality, while Turing and Ampere users can expect to hit fast/medium-like quality. Below are the settings you need to set:

Option Maxwell/Pascal Turing/Ampere
Preset Quality Max Quality or Quality
Profile high high
Look-ahead Enabled/Checked Enabled/Checked
Psycho Visual Tuning Enabled/Checked Enabled/Checked
Max B-Frames 1 or 2 3 or 4
Best Built-in NVENC settings

StreamFX: NVIDIA NVENC H.264/AVC (via FFmpeg)

Image reference for Turing/Ampere
If you're new to StreamFX's NVENC integration, it will most likely overwhelm you with the settings it offers. But thanks to all those settings, you can actually go above the default quality by quite a significant amount. Note that I will only cover critical settings, as other settings like Bitrate, Buffer Size and Key Frame Interval are explained elsewhere.

Option Kepler, Maxwell & Pascal Volta, Turing & Ampere
Preset High Quality High Quality
► Profile High High
► Level Automatic Automatic
Rate Control Options
► Mode High Quality
Constant Bitrate
High Quality
Constant Bitrate
► Two Pass Disabled Enabled
► Look Ahead 8 - 12 16 - 32
► Adaptive I-Frames Enabled Enabled
► Adaptive B-Frames Enabled Enabled
Adaptive Quantization
► Spatial (AQ) Enabled Enabled
► Spatial (AQ) Strength <Any> <Any>
► Temporal (AQ) Enabled Enabled
Other Options
► Maximum B-Frames 1 or 2 3 or 4
► B-Frame References "middle" "middle"
► Zero Latency Disabled Disabled
► Weighted Prediction Disabled Disabled
► Non-reference P-Frames Enabled Enabled
► Reference Frames -1 (Automatic) -1 (Automatic)
FFmpeg Options (Optional)
► Custom Settings
Ideal NVENC Settings for StreamFX's NVENC

For certain platforms it may be necessary to turn off Adaptive I-Frames due to how their internal processing works, such as Twitch. This has a drastic quality impact and should only be done if you rely on that platform alone to reach your audience.

Setting up Resolution and Framerate to match the Bitrate

It is no secret than H.264/AVC is an outdated codec and that platforms should have paved the way for better codecs a long time ago, but it is the solution that we are stuck with until AV1 is adopted by the masses. So in order to get the best quality out of our stream, we should aim to also set up our stream according to the bitrate and codec we use. Below is the average result of a few hundred thousand tests at various resolutions and bitrates, according to PSNR and VMAF (weighted PSNR 30:70 VMAF):

Resolution 3.5mbit
30 FPS
60 FPS
30 FPS
60 FPS
30 FPS
60 FPS
640x360 6 6 8 7 9 8
960x540 5 4 6 5 7 7
1280x720 4 3 4 3 5 4
1536x864 3 2 4 3 4 4
1600x900 3 2 3 2 4 3
1920x1080 2 1 2 2 3 3
Rating from 1 to 10 based on VMAF and PSNR, weighted towards producing useful ranges. Tests performed with x264 slow.
A 10 is perfect, 9 is near lossless, 8 is indistinguishable, 7 is high quality, 5 is acceptable quality and 3 is watchable.

Please note that watchable in video encoding means that you can decode information within it with reasonable accuracy, instead of it having turned to full garbage. Higher resolutions than 1920x1080 were omitted from the table as the rows would be filled with values between 0 and 1, which just are not very useful to us.

This means that at 3.5mbit, the highest resolution and framerate for a variety streamer is 1280x720 at 30 FPS, or 960x540 at 60 FPS. The equation shifts slightly for 6.0mbit, where you can either go for 1536x864 at 30 FPS or 1280x720 at 60 FPS. Finally at 8.5mbit you are looking at a maximum resolution and framerate of 1920x1080 at 30 FPS or 1536x864 at 60 FPS.

Final Words

In the past few years NVIDIA has made massive improvements to their encoder, which has evened the playing field far beyond what was expected. With no need to transfer frames from the GPU to the CPU, and quality comparable to x264 medium (or better), NVIDIAs Turing NVENC is pushing the boundaries of what is possible in a single consumer PC.

Whether you use it or not is entirely up to you however. If you already have a working Dual-PC setup that can achieve x264 medium (or better) quality, then you don't gain much from moving to Turing NVENC. But if you're currently stuck on anything below x264 medium, or have a Turing GPU ready to test it out - why not give it a shot?

Video Examples

ARMA 3 "Walk through the Jungle" #002

Black Mesa #001

Call of Duty Modern Warframe TDM Broadcast (by EposVox)

Forza Horizon 4 #002

GRIP Combat Racing #007

Comments for: The Art of encoding with NVIDIA NVENC