What's actually holding you back - the age itself, or the fear of looking out of place? Because those are two different problems, and one of them isn't real.
The earlier answer's right about recovery being slower, but I'd push back on the "patience" framing a bit. Yeah, you'll progress differently than a 22-year-old, but "differently" doesn't mean slowly in a way that matters. Most people starting BJJ are terrible at first regardless of age. The real advantage younger folks have isn't intelligence or potential - it's that they're often still figuring out what they want to do with their time, so they train five days a week. You'll probably train two or three times, and that's actually fine. Consistency matters way more than frequency once you're past the complete beginner stage.
The biggest thing people miss: starting at 35 means you often have better body awareness than teenagers, you're more likely to listen to what your coach actually says instead of just muscling through, and you'll tap before you get injured. I've seen way more 20-year-olds quit because they got hurt or got frustrated with their ego than I've seen people in their 30s or 40s just... keep showing up. Some gyms absolutely skew young, but if you find one with a decent older crew training, you'll realize there's a whole world of people exactly where you are. Brown belts who started at 40. Black belts who came to it in their 50s.
Go do a trial class and watch who's actually training, not who looks the strongest. That'll tell you everything.