Actually the device still turns on and can do more than it seems at first glance. It's true it's stuck on iOS 5.1.1 and most modern apps don't work, but Wi-Fi continues to work just fine. The kid can use Safari to open basic websites (some older ones work better), and if you have any app compatible with that version saved somewhere, it works. Syncing with Apple's servers is really a mess, you're right about that.
The trick everyone forgets is using the iPhone 4 as just a Wi-Fi device, kind of like an older iPod touch. If you install apps through compatible IPA files (it's possible to find old versions of some apps) or if you manage to jailbreak it with tools specific to iOS 5, you can extend quite a bit what's possible to do. There are online communities dedicated to reviving old iPhones and they have really detailed guides. It's not complicated at all if you have patience, and the phone won't stop working because you tried - worst case, you go back to how it was.
The messaging situation is more complicated though. Basic SMS works if you have a SIM card, but WhatsApp and any modern messaging app are out of reach. It's worth trying if it's just to have a toy device or to experience what life was like over a decade ago, but if you actually expected to communicate with people, better leave it in the drawer.