i've seen this exact situation play out a few times when folks are doing extensions, and honestly the speed gain is real but comes with caveats. the panels themselves aren't the problem - a good manufacturer makes solid ones. the issue is usually in the details: how they're being customized for your space, whether the finishes actually blend with what you've got, and if your contractor has experience fitting them properly. i knew someone who went this route and saved maybe 2-3 weeks on framing, but then spent extra time on site adjustments because the panels didn't account for slightly uneven existing walls.
the bigger question is your contractor's track record with this method. have they done prefab work before, or are they just parroting a sales pitch? there's a difference between using panels as a genuine efficiency tool and using them because it's easier for them. ask to see examples of previous extensions they've done with prefab - specifically how the seams and transitions look once finished. bad installation makes quality panels look terrible.
one thing to push back on: just because panels are faster doesn't mean the whole job gets done faster. you still need electrical, plumbing, finishes, and everything else taking the same time. so that "half the timeline" claim deserves scrutiny. if your timeline is genuinely squeezed and your contractor can demonstrate they know this method well, prefab makes sense. if it feels like they're suggesting it mostly for their own convenience, that's when you should worry.