I don't think it's rude to always arrive late, but it does affect the other person. When you make plans with someone, that person is setting aside their time for you, so showing up 15 or 20 minutes late is basically telling them their time is worth less than yours. It's not catastrophic if it happens occasionally, but when it's a pattern, people notice and get upset, even if they don't say so directly.
The fact that your friends don't say anything doesn't mean they don't care. A lot of times people avoid bringing it up so they don't sound bitter or create conflict, but internally they're building up resentment. If you do it all the time, eventually someone's going to be less available for you or hangouts are going to become less frequent, without them explaining exactly why.
What you could do is pretty straightforward: leave 15 minutes earlier than you think you need to. If you're always late, it's because your time calculation is off, so adjust your buffer. And if you can't make it one time, give advance notice. Your friends would probably appreciate a message saying you're going to be late way more than waiting around without knowing if you're even going to show up. It's a small gesture that shows respect.