A golden middle ground is to invite only those colleagues you actually hang out with outside of work, or whoever your wife wants to invite. If the relationship is really just office-based, you can be more honest about it - don't invite them, but give them a heads-up beforehand in a casual setting, explaining that you've decided to keep the celebration really tight and small. People usually take that approach way better than feeling like "I got invited because they couldn't figure out how to say no."
The main thing is don't waffle between options afterward and don't create double standards (inviting one person but not another). That'll get around the office fast and it'll genuinely hurt people's feelings.