This semester Terry Sejnowski is teaching a graduate seminar course that is focused on Deep Learning. The course meets weekly for two hours to discuss papers. Here I’ll just outline the course and in later posts I’ll add some thoughts on each specific week.
Week 1: Perceptrons
- Rosenblatt, F. A comparison of several perceptron models, Yovits, Jacobi, Goldstein (Eds.), Self-Organizing Systems, Spartan Books, New York (1962)
- Gray, M. S. Lawrence, D. T. Golomb, B. A. Sejnowski, T. J. A Perceptron Reveals the Face of Sex, Neural Computation, 7, 1160-1164, 1995
- Pollack, JB, Book Review, Perceptrons.
Week 2: Hopfield Nets and Boltzmann Machines
Week 3: Backprop
Week 4: Independent Component Analysis (ICA)
Week 5: Convolutional Neural Networks (CNN)
- Clement Farabet, Camille Couprie, Laurent Najman, Yann LeCun, Learning Hierarchical Features for Scene Labeling, IEEE 35: 1915-29 (2013)
Week 6: Recurrent Neural Networks (RNN)
Week 7: Reinforcement Learning
Week 8: Information and Control Theory
So maybe after reading some of my past posts, you are fired up to start programming a deep neural network in Python. How should you get started?
If you want to be able to run anything but the simplest neural networks on easy problems, you will find that since pure Python is an interpreted language, it is too slow. Does that mean we have to give up and write our own C++ code? Luckily GPUs and other programmers come to your rescue by offering between 5-100X speedup (I would estimate my average speedup at 10X, but it varies for specific tasks).
There are two main Python packages, Theano and TensorFlow, that are designed to let you write Python code that can either run on a CPU or a GPU. In essence, they are each their own mini-language with the following changes from standard Python:
- Tensors (generalizations of matrices) are the primary variable type and treated as abstract mathematical objects (don’t need to specify actual values immediately).
- Computational graphs are utilized to organize operations on the tensors.
- When one wants to actually evaluate the graph on some data, it is stored in a shared variable that when possible gets sent to the GPU. This data is then processed by the graph (in place of the original tensor placeholders).
- Automatic differentiation (ie it understands derivatives symbolically).
- Built in numerical optimizations.
So to get started you will want to install either Theano (pip install theano), TensorFlow (details here), or both. I personally have only used Theano, but if Google keeps up the developmental progress of TensorFlow, I may end up switching to it.
At the end of the day, that means that if one wants to actually implement neural networks in Theano or TensorFlow, you essentially will learn another language. However, people have built various libraries that are abstractions on top of these mini-languages. Lasagne is one example that basically organizes Theano code so that you have to interact less with Theano, but you will still need to understand Theano. I initially started with Theano and Lasagne, but I am now a convert to Keras.
Instead, I advocate for Keras (pip install keras) for two major reasons:
- High level abstraction. You can write standard Python code and get a deep neural network up and running very quickly.
- Back-end agnostic. Keras can run on either Theano or TensorFlow.
So it seems like a slam dunk right? Unfortunately life is never that simple, instead there are two catches:
- Mediocre documentation (using Numpy as a gold standard, or even comparing to Lasagne). You can get the standard things up and running based on theirs docs. But if you want to do anything advanced, you will find yourself looking into their source code on GitHub, which has some hidden, but useful, comments.
- Back-end agnostic. This means if you do want to introduce a modification to the back-end, and you want it to always work in Keras, you need to implement it in both Theano and TensorFlow. In practice this isn’t too bad since Keras has done a good job of implementing low-end operations.
Fortunately, the pros definitely outweigh the cons for Keras and I highly endorse it. Here are a few tips I have learned from my experience with Keras:
- Become familiar with the Keras documentation.
- I recommend only using the functional API which allows you to implement more complicated networks. The sequential API allows you to write simple models in fewer lines of code, but you lose flexibility (for example, you can’t access intermediate layers) and the code won’t generalize to complex models. So just embrace the functional API.
- Explore the examples (here and here).
- Check out the Keras GitHub.
- Names for layers are optional keywords, but definitely use them! It will significantly help you when you are debugging.
Now start coding your own deep neural networks!