python

Posted on Experts, Research

Deep Learning in Python

So maybe after reading some of my past posts, you are fired up to start programming a deep neural network in Python. How should you get started?

If you want to be able to run anything but the simplest neural networks on easy problems, you will find that since pure Python is an interpreted language, it is too slow. Does that mean we have to give up and write our own C++ code? Luckily GPUs and other programmers come to your rescue by offering between 5-100X speedup (I would estimate my average speedup at 10X, but it varies for specific tasks).

There are two main Python packages, Theano and TensorFlow, that are designed to let you write Python code that can either run on a CPU or a GPU. In essence, they are each their own mini-language with the following changes from standard Python:

  • Tensors (generalizations of matrices) are the primary variable type and treated as abstract mathematical objects (don’t need to specify actual values immediately).
  • Computational graphs are utilized to organize operations on the tensors.
  • When one wants to actually evaluate the graph on some data, it is stored in a shared variable that when possible gets sent to the GPU. This data is then processed by the graph (in place of the original tensor placeholders).
  • Automatic differentiation (ie it understands derivatives symbolically).
  • Built in numerical optimizations.

So to get started you will want to install either Theano (pip install theano), TensorFlow (details here), or both. I personally have only used Theano, but if Google keeps up the developmental progress of TensorFlow, I may end up switching to it.

At the end of the day, that means that if one wants to actually implement neural networks in Theano or TensorFlow, you essentially will learn another language. However, people have built various libraries that are abstractions on top of these mini-languages. Lasagne is one example that basically organizes Theano code so that you have to interact less with Theano, but you will still need to understand Theano. I initially started with Theano and Lasagne, but I am now a convert to Keras.

Instead, I advocate for Keras (pip install keras) for two major reasons:

  1.  High level abstraction. You can write standard Python code and get a deep neural network up and running very quickly.
  2. Back-end agnostic. Keras can run on either Theano or TensorFlow.

So it seems like a slam dunk right? Unfortunately life is never that simple, instead there are two catches:

  1. Mediocre documentation (using Numpy as a gold standard, or even comparing to Lasagne). You can get the standard things up and running based on theirs docs. But if you want to do anything advanced, you will find yourself looking into their source code on GitHub, which has some hidden, but useful, comments.
  2. Back-end agnostic. This means if you do want to introduce a modification to the back-end, and you want it to always work in Keras, you need to implement it in both Theano and TensorFlow. In practice this isn’t too bad since Keras has done a good job of implementing low-end operations.

Fortunately, the pros definitely outweigh the cons for Keras and I highly endorse it. Here are a few tips I have learned from my experience with Keras:

  • Become familiar with the Keras documentation.
  • I recommend only using the functional API which allows you to implement more complicated networks. The sequential API allows you to write simple models in fewer lines of code, but you lose flexibility (for example, you can’t access intermediate layers) and the code won’t generalize to complex models. So just embrace the functional API.
  • Explore the examples (here and here).
  • Check out the Keras GitHub.
  • Names for layers are optional keywords, but definitely use them! It will significantly help you when you are debugging.

Now start coding your own deep neural networks!

Advertisement
Posted on General science, Research

General Programming Tips

I thought I would put together some useful programming tips that I have learned over the years. Most of these are general tips, but they are tailored towards Python.

  1. Zen of Python. Even if you don’t use Python, these are good ideas to internalize.
  2. The language documentation (Python’s standard library), StackOverflow, and Google searches are your best friends.
  3. Utilize modern IDEs (like Spyder for Python) and tab-completion to reduce the number of basic errors.
  4. Comments are not optional. The general logic of functions and objects should be understandable from the comments. Every block of code logic should have a short comment to aid future changes. If you find a chunk of code confusing now, it will be just as confusing if not worse in the future!
  5. Use sensible variable names. This cuts down on the number/length of comments.
  6. Try to adhere to the language standards (Python’s), but don’t obsess over it.
  7. Set your own consistent standards (Do variable names end in s or not? Do boolean variables have similar style names? Etc).
  8. When starting a project, do you best to get quickly get up to a basic working prototype. Working but incomplete code is always better than non-working code. Quick coding is aided by the next point…
  9. Outline your code before starting. My tips for outlining in Python are detailed after this list.
  10. Write modular code. Common tasks should be made into functions or objects.
  11. Avoid magic numbers and hard coded values. Better to include a set of named parameters in one section of your code where the basic logic of these variables is explained.
  12. Avoid multiple inheritance (check out this fun explanation of why this is bad).
  13. A program should have a standard interface, I like to call it main, and a way to run the standard interface with some default values. In Python, utilize if __name__ == ‘__main__’: to define standard parameters and then call main(parameters). This aids the goal of always having working code, as well as making it easier to interact with different programs.
  14. Check out these Python tricks (1-23 are the best, rest are more advanced).

 

Here are details on how I outline code in Python. I try my best to have running code at all times, even if it does absolutely nothing. If it isn’t real code, I leave it as a comment. Therefore, my programming tends to proceed as follows.

  1. Outline the general logic of the code in comments. Define needed functions, but at first have it take no actual variables (utilize pass to keep it as functioning Python code). In the comments inside a function, list the data type you think it should take in, what it should do, and what it should return.
  2. If you start to code a function or series of logic, you can safely leave it incomplete by having it raise NotImplementedError.
  3. Use assert to check any of your assumptions. A custom assert statement will save you lots of time later.
  4. While the Pythonic way is to utilize duck typing, I still prefer to do some type checking if there is potential for confusion. So I like to utilize things like isinstance or implement checks on attributes.
  5. Take advantage of your IDE’s additional formatting options. For example, Spyder specially highlights comments that start with TODO: with a little checkmark. Additionally, it supports code blocks and defines them by #%%. This lets you quickly run small chunks of a larger code.

What is the major advantages of coding like this?

  1. If your code always runs, it allows you to quickly find syntax errors and typos.
  2. You avoid implementing unused code. It sucks to really work on a code section to only realize later that you didn’t actually need it.
  3. You spend your time on the standard case and can add certain options or take care of edge cases when the appropriate time arises. Because sometimes that time will never arise…

Anyways, enough advice, start coding!

 

 

 

 

Posted on General science, Research

Learning Python for Science

Here I outline how to learn Python on your own with emphasis on solving science problems. The first section applies to anyone, but the end is specialized towards computational problems that arise in science.

Python Basics

I recommend the following two tutorials:

Some additional resources that may be helpful include:

My suggested workflow:

  1. Do Codecademy and Python the Hard Way at the same time.
  2. If Codecademy/Python the Hard Way is too difficult, also read a Byte of Python.
  3. If Codecademy/Python the Hard Way is easy, use Think Python as an additional resource.
  4. If you are confused about a specific chunk of code, put it into Python Tutor which will walk you step by step through the program.
  5. Additionally, Google and Stack Overflow are extremely useful for coding questions or go to the original Python documentation.

The essential things one needs to learn about Python include:

  • data types: int, float, string
  • data structures: lists, dictionaries, tuples, sets
  • control statements: for, while, if else
  • print function
  • open / write to a text file
  • custom functions and objects
  • list comprehensions – comes up less often in numerical code, but still good to know

 

Numpy and Scipy

Numpy is the essential mathematics module in Python and is part of the larger Scipy project. All standard numerical needs are covered in Numpy, while more advanced functions are in Scipy.

I recommend the following tutorials:

  • Numpy’s tutorial
  • This fun tutorial that programs the Game of Life (GoL). I only recommend the section that implements GoL in Numpy, the rest of it is not essential. It also has a useful quick reference guide.
  • This Numpy tutorial from Scipy Lecture Notes

 

Matplotlib

Visualizing data is essential to understanding and communicating science ideas. Matplotlib is the standard plotting module. While it has its limitations, I still personally use it for my everyday plots. For more advanced plot types, check out Plotly, Seaborn, Mayavi, ggplot, and Bokeh.

And assuming you are new to making scientific figures, there are some good habits you should get into. First, read these tips from Plos. Second, never ever use rainbow colors aka jet. Color challenged people like myself will hate you. Please stick with a color map that uses shading sensible. Besides making me happy, it also is easier to print to gray scale.

 

Posted on Everybody, Research

Python on a Mac

I personally do most of my coding on my laptop, which is a Mac. Eventually that code gets run on a Linux server, but all initial coding, exploratory data analysis, etc is done on my laptop. And since I advocate for Python, I thought I would lay out all the steps I needed to do to setup my Mac in the easiest manner. (Note: probably similar steps on Windows, but I haven’t used a Windows computer in so long that I don’t know the potential differences).

8rYYWPNYNN-2

Unfortunately, the Python 2.x vs 3.x divide exists and so far, I have yet to be able to completely commit to 3.x due to a few packages with legacy issues. Luckily, there is a pretty easy solution below. Note, your Mac has Python preinstalled (go to terminal and type python to start coding…). However, if you want to update any packages, you can quickly run into issues. So it is easiest to install your own version of Python.

  1. Install Anaconda (I advocate version 2.7, Anaconda will call this environment root)
  2. I recommend using Anaconda Navigator and using Spyder for an IDE
  3. Install version 3.5 and make an environment (in Anaconda Navigator or terminal commands below): 
    $ conda create -n python3.5 python=3.5 anaconda
  4. You can switch between python environments  {root, python3.5} 
    $ source activate {insert environment name here}
  5. To add new python packages use conda or pip (anaconda has made its own pip the default)
  6. WARNING: always close Spyder before using conda update or pip. I got stuck in some weird place where Spyder would no longer launch. Apparently it can happen if Spyder is open and underlying packages get changed.

To get around the 2.x vs 3.x issue, go to your terminal and use pip install for the following packages: future, importlib, unittest2, and argparse. See the package’s website for details of any differences. Then, start your Python code with the following two lines:

from __future__ import (absolute_import, division, print_function, 
unicode_literals)

from builtins import *

For nearly all scientific computing applications, you are essentially writing Python 3 code. So make sure to read the correct documentation!

Personally, I found Anaconda to be a lifesaver. Otherwise, I got stuck in some weird infinite update loop to install all required packages for machine learning (specifically Theano).

Now you are ready to code! If you aren’t familiar with Python, my recommended tutorials will be in a future post.

 

Posted on General science, Research

The one ring to rule them all: Python

Ringfrodo

This lays out why I think all scientists should learn Python first and use it as their primary programming language. I think many of the reasons why scientists should learn Python first are equally applicable to everyone, but computer scientists and others probably have different demands for their primary language.

First, a quick history of languages I have programmed in. My first programming project was as a freshman in college and I used Fortran for simulations of water molecules. Then as a sophomore I used MATLAB for a summer research project that testing components for a high energy experiment. The following summer I used C for a summer research project on stochastic simulations. This was followed by me taking my first programming course where I used Java.

At this point I had used a slew of programming languages, but MATLAB was my primary language. I rarely had to deal with strings or statistics, so MATLAB had everything I needed. This continued into graduate school. Eventually, I ended up doing some bioinformatics, so I had to learn R. And finally, just for the hell of it, I decided to write some code in Python. Additionally, I have used Mathematica, but I wouldn’t count it as a true language.

So this is a really long-winded explanation of why I have some credibility when I say Python is the best (compared to Fortran, C, MATLAB, Java, and R which with I have personal experience). When I started my postdoc, it seemed like the perfect time to make the complete switch to only use Python.

So to start with, why would I recommend it as a first language?

  1. Correct level of difficulty
  2. Pythonic – simple expressions usually work as you would guess
  3. Versatile – can do everything one needs
  4. Open sourced
  5. Community – lots of great packages

And why should scientists use it?

  1. Versatile – handles all data types easily (unlike MATLAB)
  2. Fast enough – Cython, Theano, etc can be used when speed matters
  3. Plenty of scientific packages – Numpy, Scipy, Matplotlib, Scikit-Learn, etc
  4. Large science community – new packages all the time

Additionally, its a language that is popular enough (see here and here) to lead to a job in industry and will safely be around for years to come. So please drink the Kool-Aid and join the Python cult!