OBS video output is appearing to a bit less sharp/blurred compared to Nvidia App/Nvidia Overlay output. Details in the post body.

[clatzeo]

I've been going nuts over this, so bear with me please. I'll tell you the exact problem I encountered. My objective. My system, hardware etc. My relevant OBS settings, Some info I found in the output file, and of course the log's attached. Also the things I have already tried and failed. Here's a video that somewhat visualize what I see.

Video 1:

My Problem: I haven't really noticed this, till this day, but the videos I've been recording through OBS seems to be a little bit blurry or "less" sharp on edges of objects, as my youtube video shows. However, when I record it from Nvidia App overlay, the recording comes out sharp(As sharp as the rendered game itself!), with the same bitrate (12-18Mbps). What the magic possibly be?

Yeah, I can slap a sharpness filter on any video editor as a workaround, but I really want to clear out any mess that I might have tweaked on the software side, or am I just not aware of somethings?
The game is Kingdom Two Crowns, which is made on 2D pixel art, rendered on 1920x1080 res. The pixel art accentuates the defects even more, and made it more visible to me.

My Objective: If possible, somehow clear this blurriness on the output from OBS side(or some other setting; though no filters right now.). Find the options that can fix it.

- My System: CPU = i7-7700k, 8GB DDR4, Graphics card = GTX 1650 Super 4GB, Monitor is 1366x768, 60hz, non-HDR.
HDR is obviously off from Windows10 settings. Also, from Settings>Display>Graphics Settings, Hardware-accelerated GPU scheduling is ON. OBS is set on High-performance mode.
- Nvidia control panel settings (like: Image scaling, Ambient Occlusion, Anisotropic filtering, Antialiasing and all) are also OFF (Both for game and also for OBS).

- My OBS settings:
Base canvas Resolution = 1920x1080, Output Resolution=1920x1080 (No downscale filter as both are same. )
The game's also rendered on 1920x1080 resolution, JTLYK

- Advanced tab: Renderer = Direct3D 11, Color format = I444 (8-bit: 4:4:4, 3 planes) (Also tried NV12 and I420), Color Space = Rec. 709, Color range =Full(better than Limited), SDR White level = 300nits, HDR Nominal Peak Level = 1000nits.

- Recording settings:
format = Hybrid MP4(mp4),
Rescale output = Disabled
Rate control = Variable Bitrate with Target Quality,
Target quality = 12, Max bitrate = 50,000 Kbps,
Keyframe Interval=1s,
Preset = P6:Slower,
Tuning = High Quality,
Multipass mode = Two passes (Quarter resolution),
Profile=main,
Look-ahead=Off, Adaptive quantization=On,
B-Frames = 4, B-frame as reference = Each, (also custom parameters level=5.2)

- Scenes setup at OBS UI:
Using Game capture: Selected game(Kingdomtwocrowns.exe), Only "Use anti-cheat compatibility hook" is checked, Hook rate=normal, RGB10A2 Colorspace=Rec. 2100 (PQ).
The only other option to this, which I'd already tried is Display Capture. Game Capture allows to record the exact resolution that the game is being rendered. Display capture would have my Monitor's resolution.
Also, there are no filters in game capture or anything.

- Nvidia App Overlay settings:
Encoder: Doesn't allows me to chose, but uses H.264 (NVEnc).
Recording resolution: Does 1366x768(My monitor), even though I have chosen 1080p
Bitrate: 12Mbps/18Mbps, and 60Fps

Some more information out of output files via MediaInfo:
- Nvidia App video capture output file:
Encoder: H.264 (AVC), Profile=High@L4.2, 3 Ref frames, GOP: M=1, N=30, codec ID: avc1,
Bitrate: ~12Mbps, Resolution: 1366x768, Frame rate mode: variable,
Color space: YUV, Chroma Subsampling: 4:2:0, Bit depth =8bits, Scantype: Progressive,
Color range: Limited, Color Primaries & Transfer Characteristics & Matrix Coeff = BT.709

- OBS file:
Encoder: NVENC HEVC, Profile=Main 4:4:4@L5.2@Main, 4 Ref frames, codec ID: hvc1,
Bitrate: ~36Mbps, Resolution: 1920x1080, Frame rate mode: Constant,
Color space: YUV, Chroma Subsampling: 4:4:4, Bit depth =8bits, Scantype: Progressive,
Color range: Full, Color Primaries & Transfer Characteristics & Matrix Coeff. = BT.709

What I had already tried in OBS (All the switches):
- Base canvas to 1366x766: Still the blurriness exists.
- Color formats = NV12 (8-bit: 4:2:0, 2 planes), I420 (8-bit: 4:2:0, 3 planes): I don't think changes here did anything at all really, but I am recording so I can go 4:4:4. I have read something about 4:2:0 & 4:4:4 difference, but IDRK. Blurriness exists.
- Color Space = sRGB: Doesn't seem to show any visible effects.
- Color Range = Full->Limited: Colors more washed/pale, but the blurriness exists.
- At recording output -- Preset: P7 (still blurry), Multipass mode = Two passes (Full resolution): Still blurry
- Tuning = Ultra high quality: blurry (P7, Multipass, Tuning - all simultaneously max)
- In Game capture -- Allow transparency, and Premultiplied Alpha checked: I though this would do, but the results aren't sharp even now!

My hypothesis:
- The obligatory - me doing something wrong here.
- Either there's something on the OBS pipeline that's not getting what it should. Something... something..
- Or, Nvidia could be applying a secret sharpening filter, who knows. But I understand that Nvidia can get all the lines and all that better as it's a close job.

Can someone help to uncover this? Can someone perform this experiment on their end? Like using OBS on pixel art game, and then also the Nvidia Overlay recorder -- then compare.
- I wanna know what even is happening. Ultimately, how to get OBS to give me as clear as the Nvidia's Output. (I like the flexibility and control with OBS and also love the colors it provides. The blurriness is hurting me)
More comparison attached:
01

02

03 Cropped from above image from Nvidia Screenshot at lamp.

04 Cropped from above image from BS Screenshot at lamp. You can see the blurriness more clearly here. This is what I'm talking about.

 

[clatzeo]

I have created 1 more comparison, which still ended up highlighting the same problem:

- This time I have used the level 6.2 of HEVC encoder(the highest), and set the maximum possible bitrate to be 180,000 kbps in OBS.
- Chroma subsampling is 4:4:4 to not have any misalignment issues.
- Base Canvas = 1920x1080, Output resolution is the same, and the game is also rendered on 1080p.
- I have further manually gone to change my display resolution to 1920x1080 before opening the game(In case the game doesn't really changes resolution.)

Nvidia recorder:

OBS recorder:

You can clearly see that slight shift. I'd thought it could be happening because the Game caputre scene is misaligned with the OBS canvas, but I have made sure to Ctrl+F (Fit) before pressing record.

 

[clatzeo]

The last option here is to re-install OBS and re-setup everything from scratch. Tomorrow, I'll be doing that and we will be back with the comparisons. Let's see.

 

[clatzeo]

(This message is for reference only, as I mistakenly uploaded the same images twice.)
I'd uploaded the wrong screenshot for Nvidia. My bad. The correct screenshots for both the videos are these:

Nvidia recorder output:

OBS recorder output:

If you zoom, you can see the shift of pixels at the edge of objects are visible (Can only be noticed relative to the Nvidia screenshot), which is making the image seems blurry.

I think I have figured out what could be happening. I will get back with the re-install, just to have clear starting point.

 

[Erick Smith]

My advice:

Set the .mp4 container to Microsoft. MERIT is set by default, so....

• Merit is about priority ranking of codecs in Windows’ codec list.
• Microsoft refers to the official codec packages that provide hardware-accelerated MP4 decoding.

Basically, by default, MERIT will either use its own muxer or the GPU (or whatever it wants based on priority), which produced inconsistent recordings from my research. Not saying that it's a problem with OBS, but it is in Windows 11. So I set mine to Microsoft. There is no third party muxing, only the GPU and its extremely consistent. You need Win7DSFilterTweaker to change the mode, which is in the K-Lite Codec Pack.

One thing I did differently....so the Recording tab is also associated to OBS's preview. I set my settings on the Recording tab to:

Type: Custom Output. This removes the safeguards. Why? Our AMD and RTX GPU's are extremely fast and this is not 2010 anymore.

Container format: .mp4. This will set the Video encoder to libx264. Don't change this. Else, OBS will not record.
Set your Video Bitrate and Keyframe intervals. I set my Keyframes to 5000 (because the safeguards are down). This is decoupled from the Stream tab, so you can set those keyframes to 1 or 2 (for live streaming).

You can also use x264's switches in this mode as well.

 

[clatzeo]

My advice:

Set the .mp4 container to Microsoft. MERIT is set by default, so....

• Merit is about priority ranking of codecs in Windows’ codec list.
• Microsoft refers to the official codec packages that provide hardware-accelerated MP4 decoding.

Basically, by default, MERIT will either use its own muxer or the GPU (or whatever it wants based on priority), which produced inconsistent recordings from my research. Not saying that it's a problem with OBS, but it is in Windows 11. So I set mine to Microsoft. There is no third party muxing, only the GPU and its extremely consistent. You need Win7DSFilterTweaker to change the mode, which is in the K-Lite Codec Pack.

One thing I did differently....so the Recording tab is also associated to OBS's preview. I set my settings on the Recording tab to:

Type: Custom Output. This removes the safeguards. Why? Our AMD and RTX GPU's are extremely fast and this is not 2010 anymore.

Container format: .mp4. This will set the Video encoder to libx264. Don't change this. Else, OBS will not record.
Set your Video Bitrate and Keyframe intervals. I set my Keyframes to 5000 (because the safeguards are down). This is decoupled from the Stream tab, so you can set those keyframes to 1 or 2 (for live streaming).

You can also use x264's switches in this mode as well.

Thanks for a reply Eric. I really appreciate every info.
I am currently on Windows 10(not that something's different here). The Win7DSFilterTweaker (6.5), shows only the option to decoders. I don't get how it's supposed to do anything to OBS? Like, isn't the recording and output about Encoding?

Also, if I had set the Advanced Graphic Settings from Windows setting to "Performance mode" (Which is basically selecting the GPU for any application for hardware scheduling), wouldn't it already select GPU for a lot of things?
(Those are two different questions).

I want to use the NVEnc HEVC encoder, which will use GPU and it's also the most efficient encoder available to me. Unfortunately I don't have NVEnc AV-1 (Which is even more efficient). I can't use CPU for encoding right now, and libx264 is CPU based.