I'm developing an API in Node.js with Express and when I test it with large CSV files (like 500MB) the whole process freezes and I have to restart it. I've already tried increasing the heap but it keeps freezing. Does anyone know if it's actually a memory problem or if there's something wrong with my code?
I have the same problems with huge databases, and it's almost always the same culprit: you're loading the entire file into memory before processing it! A 500MB CSV becomes absolutely monstrous when parsed into arrays, and increasing heap is just putting a band-aid on a wound. The real solution is to use streams - process line by line while you're writing the data to a database or file, and Node.js won't even need to keep more than a few KB at a time! Try the `csv-parser` library with streams or even write a custom parser with `fs.createReadStream()` and events, that'll completely change your game.
I've already dealt with something similar when I was trying to process large files all at once, and I realized the problem wasn't just memory - it was the fact that I was loading everything into RAM at the same time. With a 500MB CSV, if you try to read the entire file and then process it, you'll run out of memory pretty quick. The solution is to use streams instead of loading everything into memory. In Node.js you have the `fs.createReadStream()` module that lets you read the file in chunks. If you're using libraries like `csv-parser`, you can chain streams to process line by line without overloading the RAM.
Another thing that made a difference for me was realizing that even with streams, if your processing logic is slow (like synchronous operations in the loop, database queries without batching), you'll create a bottleneck. When the stream delivers data faster than you can process it, the memory still piles up. Use `pause()` and `resume()` to control the flow, or implement back-pressure properly. If you're doing database queries for each row, batch them - like, group 1000 lines and do one operation instead of 1000 separate ones.
What kind of operation are you roughly doing with the CSV data? And are you saving the results somewhere (file, database) or just processing and returning it through the API? The answer helps because the problem might also be in how you're sending the response back to the client.
Log into answer.
