Your wedding budget is finite, and that's the only justification you actually need. Period. The tricky part isn't whether you *can* exclude people - you absolutely can - it's managing the fallout if someone finds out they weren't invited and feels slighted. That's the pitfall nobody mentions: distant relatives especially have a way of hearing through the grapevine that they were left out, and then you're dealing with hurt feelings for years.
The safest move is to be consistent with your criteria and keep it genuinely quiet. If you're inviting only close family and your actual friend group, that's defensible and easy to explain if it comes up: you wanted an intimate celebration. If you're inviting some coworkers but not others, that gets messier because people talk, and it looks less like budget management and more like preference ranking. So either draw a clear line (close friends and immediate family only) or don't invite coworkers at all unless they're actually your friends.
When someone inevitably asks why they didn't make the cut, a simple "we had to keep it really small due to budget and venue" is honest and leaves no room for debate. Don't elaborate or offer exceptions - that just makes people feel worse. And definitely don't post a ton of photos on social media right after if you're worried about hurt feelings; people are less likely to get angry if they don't see exactly how many guests you had versus how many they think they deserved.