The frustrating part is that YouTube treats Shorts caching like a core feature, not an optional thing. You can't flip a switch to disable it entirely, but you've got more granular control than just torching your whole app cache. In the YouTube app itself, go to Settings > General and look for "Download" or "Offline" settings - some versions let you restrict auto-downloads. More importantly, check Settings > Apps > YouTube > Permissions and see if you can revoke or limit "Modify system settings" or "Write to external storage," though this might break some features.
The real solution most people miss is managing it at the Android level rather than fighting YouTube directly. Go to Settings > Storage > Manage Storage (varies by phone maker), and you can often see exactly how much YouTube is taking up. You can also set app-specific storage limits if your phone supports it - on some devices this is under Developer Options. Some folks set up a recurring task using an automation app to clear YouTube's cache on a schedule, say weekly, without wiping everything else. It's not elegant, but it actually works better than manually clearing it every time you notice the problem.
One last thing: if you're really fed up, there's always YouTube Vanced or similar third-party clients that give you actual control, though they come with their own trade-offs around updates and stability.