You're not missing out on anything critical if your current drill is working well for shelves and light stuff. I've tackled plenty of deck projects and occasional furniture assembly at home, and honestly a regular drill handled all of it just fine - the main thing an impact driver gives you is less wrist strain when driving longer fasteners into denser wood, and it's noticeably faster if you're doing repetitive work, but that's more of a quality-of-life upgrade than a game-changer for occasional weekend projects.
If you find yourself regularly stripping out screws or feeling worn out after a session, then yeah, grabbing one makes sense. Otherwise save your money and keep the drill you've got.