That's not necessarily a general problem with integrated USB outlets - many actually charge just fine. However, it really depends on the quality and model you bought. Cheaper variants often have limited total power output for all USB ports combined. So if you plug in multiple devices at once or the outlet only delivers 30-40W total, the power has to be distributed across all the ports - then of course it'll be slow.
The wiring inside the outlet also plays a role. Some cheaper models have thinner cables that lead to voltage drops. Your phone then notices that less voltage is arriving and automatically throttles the charging speed for safety reasons. Check the manual to see how many watts the USB ports provide in total and whether individual ports have a maximum. If you're only charging one device and it's still slow, your outlet is probably just undersized.
The safe solution: invest in a higher-quality outlet with at least 45-65W total output and good reviews. Or don't get rid of your old 20W power adapter - traditional power adapters are often more reliable when it comes to actual charging speed.