Watering is indeed a key factor, but I'd clarify: it's not just about how much water you give, but how exactly you deliver it. In summer, peaches need deep but infrequent watering - roughly once a week, but abundantly, so the water reaches the root depth. If you water often and just a little bit, the soil will be wet on top while staying dry below, and the tree will still be under stress. I'd recommend mulching the tree base with grass or straw - it stabilizes moisture and prevents sudden fluctuations.
Beyond that, pay attention to nutrition. In summer, when fruit is actively growing, peaches can lack potassium and magnesium - these are the elements needed to hold onto the fruit set. If you're not fertilizing it in summer, try applying a balanced fertilizer with an emphasis on potassium and phosphorus around June-July. Nitrogen during this time is better kept to a minimum, otherwise you'll get excessive leaf growth at the expense of the fruit.
And one more thing - check for pests or fungi. Sometimes fruit drops because of infection, and it looks just like stress from drought. If you see spots, deformations, or any coating on the fruit before it falls - that's a reason to treat it with fungicide.