You're approaching this the wrong way - you don't necessarily need to jump straight to a top-tier card if your current one can't handle the load. The problem often isn't the GPU's raw power, but how you're distributing tasks. An RTX 3080 is perfectly capable of running a game and encoding through NVENC simultaneously if you configure your bitrate and stream resolution correctly. Maybe you're just trying to max out everything at once - 4K in-game, 4K stream, maximum encoding quality?
My friend had a similar situation once: she bought a more powerful card, but it barely helped until she reconfigured OBS. Turns out she was running the game on high settings AND the stream at maximum bitrate at the same time, then trying to process video in After Effects on the same system. Once she split things up - game on one drive, stream recording on another, video processing after streaming - everything fell into place. The RTX 3080 handles it fine if you're not doing everything at once.
You're right about NVENC being more convenient than AMD solutions. If you do decide on a new card, look at the RTX 4090 or 5090, not the 5080 - those are the ones that give you the real boost for parallel tasks. But first try optimizing your current setup: lower your stream resolution to 1440p or even Full HD, reduce the bitrate, move your files to a separate SSD. It usually helps!