2 answers

★ Best answer

Pandas is what you need here. With 500k rows, processing everything in memory with native Python loops is a speed disaster. Pandas is optimized in C under the hood and handles tabular data much more efficiently. Load the CSV with `pd.read_csv()` and then apply vectorized operations (filters, transformations, calculations) directly on the columns instead of iterating row by row.

If it's still slow, first check exactly what you're doing: if you're reading the entire file but only need some columns, use the `usecols` parameter in `read_csv()` to load only those. If you're processing in chunks, read the file in blocks with `chunksize` instead of all at once. Avoid applying custom functions with `apply()` when you can - those are super slow. If you need to process complex conditionals, keep the operations in pandas (with boolean indexing) instead of jumping to pure Python.

Another option if Pandas doesn't cut it is Polars, which is a modern replacement and even faster, but for 500k rows Pandas should bring the problem down to seconds instead of minutes. Try these things before switching approaches completely.

With 500k lines, Python's native loops are brutally slow, so the other comment about Pandas is on the right track. But heads up: if you're using Pandas and it's still taking forever, you're probably doing row-by-row operations inside an `.apply()` or something like that, which is almost as slow as loops. Try to vectorize everything you can (operations on entire columns, not on each row) and if Pandas is still the bottleneck, consider libraries like Polars that are faster with large datasets, or if the processing is simple, specialized tools like DuckDB for direct SQL on CSVs without loading everything into memory.

Your answer

Log into answer.