This semester Terry Sejnowski is teaching a graduate seminar course that is focused on Deep Learning. The course meets weekly for two hours to discuss papers. Here I’ll just outline the course and in later posts I’ll add some thoughts on each specific week.

Week 1: Perceptrons

• Rosenblatt, F. A comparison of several perceptron models, Yovits, Jacobi, Goldstein (Eds.), Self-Organizing Systems, Spartan Books, New York (1962)
• Gray, M. S. Lawrence, D. T. Golomb, B. A. Sejnowski, T. J. A Perceptron Reveals the Face of Sex, Neural Computation, 7, 1160-1164, 1995
• Pollack, JB, Book Review, Perceptrons.

• Minsky, M. and Papert, S., Perceptrons, expanded edition, Prologue and Epilogue, 1987

• Ng, Andrew, Support Vector Machines, Stanford Lecture notes

Week 2: Hopfield Nets and Boltzmann Machines

• J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities”, Proceedings of the National Academy of Sciences of the USA, vol. 79 no. 8 pp. 2554–2558, April 1982

• Hopfield JJ, Tank DW, (1985) ”Neural” Computation of Decisions in Optimization Problems. Biol. Cybern 52, 141-152

• Hinton, G. E., Sejnowski, T. J. (1986). D. E. Rumelhart, J. L. McClelland, and the PDP Research Group, eds. “Learning and Relearning in Boltzmann Machines” (PDF). Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations. Cambridge: MIT Press: 282–317.

• Sejnowski, T. J. Kienker, P. K. Hinton, G. E. Learning Symmetry Groups with Hidden Units: Beyond the Perceptron, Physica, 22D, 260-275, 1986

Week 3: Backprop

• Rumelhart, DE, Hinton, GE, Williams, RJ , Learning representations by back-propagating errors Nature 323, 533-536 (1986)

• Sejnowski, T. J. Rosenberg, C. R. Parallel Networks That Learn to Pronounce English Text, Complex Systems, 1, 145-168, 1987

• Hinton, GE, How to do backpropagation in a brain, lecture slides, 2014

• Bengio, Y. Towards Biologically Plausible Deep Learning, arXiv, 2015

Week 4: Independent Component Analysis (ICA)

• Bell, A. J. Sejnowski, T. J. An Information-Maximization Approach to Blind Separation and Blind Deconvolution, Neural Computation, 7, 1129-1159, 1995

• Jung, T.-P. Makeig, S. McKeown, M. J. Bell, A. J. Lee, T.-W. Sejnowski, T. J. Imaging Brain Dynamics Using Independent Component Analysis, Proceedings of the IEEE, 89(7), 1107-1122, 2001

• Laird, A.R. et al. Behavioral interpretations of intrinsic connectivity networks. J. Cogn. Neurosci. 23, 4022–4037 (2011).

• Hill, E.S Moore-Kochlacs, C. Vasireddi, S. K. Sejnowski, T. J. Frost, W.N. Validation of Independent Component Analysis for rapid spike sorting of optical recording data, Journal of Neurophysiology, 104, 3721-3731, 2010

Week 5: Convolutional Neural Networks (CNN)

• LeCun, Y., Bengio, Y., and Hinton, G. E.. Deep learning. Nature, 521(7553):436-444, 2015.

• Krizhevsky, A., Sutskever, I. and Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks, Advances in NIPS (2012)

• Hubel DH, Wiesel TN Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol. Jan;160:106-54 (1962)

• Zeiler, M. D. Fergus R. Visualizing and Understanding Convolutional Networks, arXiv: (2013)

• Clement Farabet, Camille Couprie, Laurent Najman, Yann LeCun, Learning Hierarchical Features for Scene Labeling, IEEE 35: 1915-29 (2013)

• Dauphin et al. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in NIPS 27, (2014)

Week 6: Recurrent Neural Networks (RNN)

• Hochreiter, S., Schmidhuber, J., Long Short-Term Memory. Neural Computation, 9(8):1735- 1780, (1997)

• Wang, C., et al. Image captioning with deep bidirectional LSTMs. ACM. arXiv (2016).

• Durstewitz D, Seamans J.K, Sejnowski T.J. Neurocomputational models of working memory. Nat. Neurosci. 3:1184–1191 (2000)

• Hinton, G., et al. (2012). Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6), 82-97

• Alex Graves, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538: 471–476, 2016

Week 7: Reinforcement Learning

• Sutton, R. S. Learning to predict by the method of temporal differences. Machine Learning, 3:9-44 (1988).

• Gerald Tesauro, Temporal difference learning and TD-Gammon.
  Originally published in Communications of the ACM, volume 38 number 3, pages 58-68. (1995)

• Silver, Huang et al., Mastering the Game of Go with Deep Neural Networks and Tree Search Nature 529 (2016)

• Montague, P. R. Dayan, P. Sejnowski, T. J. A Framework for Mesencephalic Dopamine Systems Based on Predictive Hebbian Learning, Journal of Neuroscience, 16(5), 1936-1947 (1996)

Week 8: Information and Control Theory

• Shannon, Claude E. A Mathematical Theory of Communication. Bell System Technical Journal. 27 (3): 379–423 (1948)

• Doyle, J. Teaching Control Theory in High School (2016)

• W. Brian Arthur, The Nature of Technology, Preface, Penguin, (2009)