The thing is, what will damage your relationship here isn't your refusal - it's his requests. You've already said "yes" several times, and nothing has changed. That means trying to be polite and considerate isn't working anymore. He's gotten used to you giving him money, and he'll keep asking because it works for him.
I had a similar situation - an acquaintance would borrow from me right in the work cafeteria between lunch breaks. I'd give her money, she wouldn't pay it back, then she'd ask again. I was embarrassed for a long time to just say "no" outright, afraid of damaging our relationship. In the end, when I finally refused, it turned out she wasn't even offended - she just moved on to the next person. Later I realized she never actually expected to pay the money back at all; she was just testing who would give it to her.
Next time he asks, just say: "Unfortunately, I haven't gotten back the previous amounts I lent you, so I'm not able to give you more." Don't apologize, don't explain how you budget your money. It's direct, calm, and fair. If the person is actually reasonable, they'll understand. If they get offended - well, that means the relationship was built on the idea that they could count on you for money, and that's not your fault.