I use python as my predominant language of choice for research. I don’t need to retread the obvious justification here, but the machine learning library ecosystem revolves around Python. There’s a growing argument to be had for Julia and perhaps Swift, but I’d rather move research forward with the tools I have availbale now than devote my time working around corner cases.
This list serves to document some of the major quality-of-life practices I’ve adopted with python, my thoughts on various useful libraries, and things I’m excited about.
Python packaging can be kind of a mess. Recently, several libraries (pip-tools, poetry, pipenv) that have emerged attempting to solve reproducible python environments. The community mindshare seems to have shifted to poetry. I’ve been using it happily for about a year.
Poetry uses a combination of a pyproject.toml and poetry.lock file to define compatible library/interpreter version ranges as well as a lockfiles which takes a snapshot of the exact library versions. This is terrific for ensuring reproducibility, as both files can be checked into version control.
This is a project I am very excited about. I often here people praise conda for its binary bundling, which allows you easy access to pre-compiled binaries for libraries such as cuda. This makes running things like opencv and tensorflow very easy on any major linux distribution. The combination of poetry and nix offers substantially more power (in my opinion) while sacrificing a bit of usability. You can declaratively define exact versions for system and python libraries, and reproduce this setup on any linux system.
poetry and docker
I’ve been using k8s to deploy models with kubeflow (or trying, at least), and one usecase that comes up often is that I would like to bundle my current python development environment into aa container and push it to my private registry. This is slightly annoying with Docker due to incrmental builds.
I really hate text lag, and my laptop is a 7 year old macbook pro which operates as a glorified terminal. I used to use pycharm, but realized for my purposes I use only a fraction of the featureset.