Don't just reset them to factory settings without trying this first - there's a weird quirk where AirPods sometimes get stuck trying to reconnect to a device that's no longer in range. Go into Settings > Bluetooth on both your iPhone and iPad, forget the AirPods from both completely, then power off both devices for like 30 seconds. When you turn them back on, only pair to one device first and let it stabilize for a day or two before adding the iPad back.
The multidevice switching is definitely part of your problem, but I found that AirPods sometimes hold onto a "preferred" device even when you're trying to use another one, and a full cold restart of the ecosystem clears that ghost connection better than just unpairing.