Don't spray perfume directly on delicate fabrics like silk, linen or anything vintage - the alcohol in fragrance can leave water marks or discoloration, especially on light colors. You're not imagining the longevity thing. Perfume lasts longer on fabric because your skin has natural oils and heat that actually break down fragrance molecules faster. Fabric doesn't have that same biological activity, so the scent molecules just kind of hang around. Plus your clothes aren't constantly shedding dead skin cells the way your skin is, so there's less friction wearing away the fragrance.
The staining risk is real but depends on the perfume concentration and fabric. A light cologne spray on cotton or a sturdy blend is usually fine. Heavy perfumes or multiple sprays on pale fabrics? That's where you run into trouble. My approach is to spray on clothes that can handle it - jeans, cotton tops, sweaters - and keep it light. If you're worried, test it on an inconspicuous spot first or spray the inside of a garment where it won't show anyway.
The tradeoff is that while perfume lasts longer on fabric, it can actually smell slightly different than on skin. Your skin chemistry affects how a fragrance develops, and that warmth creates a more complex scent evolution. On fabric it's more static - what you spray is what you get. So maybe use clothing application for events where you want staying power, and stick with skin when you want the full olfactory experience.