How do I properly clean a fabric headphone headband without damaging it?

A soft microfibre cloth slightly dampened with warm water is your best friend for the fabric - gently wipe it down without soaking it and let it air dry. If it's really dirty, you can add a drop of mild soap to the water, but avoid alcohol or harsh products that'll discolour or weaken the fibres.