8 Tips & Tricks for Pandas

1. Use pandas >= 1.0.0

  • Performance boosts above this version, so upgrade your package asap

2. Specify the types of the columns when loading csv

  • Save a lot of memory in large files

3. Specify the necessary columns when loading

  • Do not load columns you don't need

4. Delete Dataframes you not longer need

  • Free memory when possible

5. Read a file and process per chunks

  • Useful for large files

6. Explicitly convert certain columns to datetime objects when loading

  • This helps you to do easy transformations on datetimes columns

7. Use multi-index when possible

  • Groups values and compress data, saving a lot of memory

8. Avoid Apply method, use built in functions

  • Save CPU processing time avoiding Apply()

10