Labtech: Concurrency and caching in Python for busy scientists
2024-08-24 , Breakout Room

Python has made it easier than ever for scientists and analysts to design and run computation-heavy experiments. However, as a project grows and experiment variants multiply, your experimentation loop can grind to a halt: your experiments take longer and longer to run and you spend most of your time managing their results.

Faced with these problems, I developed labtech (https://pypi.org/project/labtech/), an open-source library that makes it easy for you to manage your experiment variants, run them as concurrently as possible on your hardware, and cache intermediate results to be reused across experiments. All you need to do is pip install labtech and reformat your existing experiment code as labtech tasks!

Labtech simplifies the experimentation loop for many fields, including:

  • Data scientists evaluating alternative machine learning models
  • Researchers running simulations and data processing pipelines
  • Business analysts scraping many websites or documents over time while only re-scraping as needed
  • Any use case where you want to concurrently run interdependent tasks with result caching and re-use

You'll walk away from this talk with:

  • A picture of how labtech fits into the landscape of tools for parallelising Python code and managing experiments.
  • An understanding of how labtech works and the features it provides for managing experiments.
  • The practical knowledge to get your experiment loop running at light speed with labtech!

What is the anticipated audience for your presentation?:

Anyone

Ben loves using Python every day in his work as a data scientist to help organisations get more from their data, and he has a passion for teaching others how to get stuff done with Python.

He recently co-delivered the first-ever PyNoon lunchtime Python training course, and has presented at KiwiPycon and other conferences on a range of technologies.

Ben has over 12 years experience in software development and he previously worked as the original software architect for NZ security software startup DataMasque.

For his recently completed PhD thesis, Ben developed machine learning algorithms that can be applied despite common data deficiencies in collaboration with Fisher & Paykel Appliances.

This speaker also appears in: