First of all, measure the distance from where you sit to where the TV will go, because that's what determines everything. If you're more than two meters away, 40" is still your best bet; if less, then yeah, consider going smaller. What nobody's told you is that in small rooms light reflection is a real pain, so if you go with OLED make sure the room can get dark enough, because otherwise OLED panels won't get bright enough and you'll lose the whole point.
For the specific size, honestly 40" is pretty manageable in a small room. 32" is more for when space is really tight or if you want something that barely takes up any room. What I'd change is the technology: if you've got the budget, OLED is killer for contrast and colors, but Mini LED also works well and tends to be cheaper. That said, pay attention to the refresh rate in hertz (newer models usually offer 120Hz) and make sure it has good gaming features in case you want to hook up a console later.
One thing: don't let yourself be swayed just by specs on some online store. If you can, go check out actual models in person, because seeing how it actually looks on a smaller screen is different from reading numbers. And check that the stand or mount isn't garbage, because that matters too in small rooms where every centimeter counts.