It's not really that math validates the transactions - it's more that cryptography makes it work. Basically, each block of transactions has a hash, which is like a fingerprint generated by a mathematical formula; if someone tries to tamper with an old transaction, the hash changes completely, and everyone can see there's been fraud. Bitcoin uses this to create a chain of blocks (hence "blockchain") where each one points to the previous one, making it practically impossible to falsify the history. Math also comes into play in the proof-of-work part, where computers have to solve complex puzzles to add new blocks - the more computers verifying, the more secure the network becomes.