Your phone's metronome is absolutely reliable enough for drum practice. The timing accuracy on modern smartphones is solid - the drift you're worried about basically doesn't exist in any noticeable way during a practice session. Built-in metronome apps are synced to your phone's clock, which is pretty precise. Unless you're doing something extreme like a multi-hour session where you'd fatigue anyway, you won't hear it falling out of time. Plenty of people practice drums with phone metronomes and get great results.
The volume is where it gets tricky though. Most phone metronomes are quieter than dedicated drum metronomes, especially the mechanical click kind. When you've got drums going, that little phone speaker might get drowned out depending on how loud you're playing and where you're sitting relative to the phone. You can crank the phone volume to max, but that only helps so much - a tiny speaker is still a tiny speaker. Some folks solve this by placing the phone closer to their ear, routing the audio through a Bluetooth speaker, or using headphones if that works with your setup.
If you're just starting out or doing casual practice at moderate volumes, your phone works fine. But if you're really whacking on the drums hard, you might find yourself straining to hear the click, which defeats the purpose. Try it for a few sessions and see if it bothers you - you can always grab an actual metronome later if the sound issue becomes a real problem.