# [ Most Cited Deep Learning Papers By Terry T. Um ]

A curated list of the most cited deep learning papers (since 2010)

I believe that there exist *classic* deep learning papers which are worth reading regardless of their applications. Rather than providing overwhelming amount of papers, I would like to provide a *curated list* of the classic deep learning papers which can be considered as *must-reads* in some area.

## Awesome list criteria

**2016**: Based on discussions**2015**: +100 citations (✨ +200)**2014**: +200 citations (✨ +400)**2013**: +300 citations (✨ +600)**2012**: +400 citations (✨ +800)**2011**: +500 citations (✨ +1000)**2010**: +600 citations (✨ +1200)

*I need your contributions!*

## Table of Contents

- Survey / Review
- Theory / Future
- Optimization / Regularization
- Network Models
- Image
- Caption
- Video / Human Activity
- Word Embedding
- Machine Translation / QnA
- Speech / Etc.
- RL / Robotics
- Unsupervised
- Hardware / Software (Total 80 papers)

### Survey / Review

- Deep learning (2016), Goodfellow et al.
*(Bengio)*[html] - Deep learning (2015), Y. LeCun, Y. Bengio and G. Hinton [pdf] ✨
- Deep learning in neural networks: An overview (2015), J. Schmidhuber [pdf] ✨
- Representation learning: A review and new perspectives (2013), Y. Bengio et al. [pdf] ✨

### Theory / Future

- Distilling the knowledge in a neural network (2015), G. Hinton et al. [pdf]
- Deep neural networks are easily fooled: High confidence predictions for unrecognizable images (2015), A. Nguyen et al.[pdf]
- How transferable are features in deep neural networks? (2014), J. Yosinski et al.
*(Bengio)*[pdf] - Why does unsupervised pre-training help deep learning (2010), E. Erhan et al.
*(Bengio)*[pdf] - Understanding the difficulty of training deep feedforward neural networks (2010), X. Glorot and Y. Bengio [pdf]

### Optimization / Regularization

- Taking the human out of the loop: A review of bayesian optimization (2016), B. Shahriari et al. [pdf]
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015), S. Loffe and C. Szegedy [pdf] ✨
- Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (2015), K. He et al. [pdf] ✨
- Dropout: A simple way to prevent neural networks from overfitting (2014), N. Srivastava et al.
*(Hinton)*[pdf] ✨ - Adam: A method for stochastic optimization (2014), D. Kingma and J. Ba [pdf]
- Regularization of neural networks using dropconnect (2013), L. Wan et al.
*(LeCun)*[pdf] - Improving neural networks by preventing co-adaptation of feature detectors (2012), G. Hinton et al. [pdf] ✨
- Spatial pyramid pooling in deep convolutional networks for visual recognition (2014), K. He et al. [pdf]
- Random search for hyper-parameter optimization (2012) J. Bergstra and Y. Bengio [pdf]

### Network Models

- Deep residual learning for image recognition (2016), K. He et al.
*(Microsoft)*[pdf] ✨ - Going deeper with convolutions (2015), C. Szegedy et al.
*(Google)*[pdf] ✨ - Fast R-CNN (2015), R. Girshick [pdf] ✨
- Very deep convolutional networks for large-scale image recognition (2014), K. Simonyan and A. Zisserman [pdf] ✨
- Fully convolutional networks for semantic segmentation (2015), J. Long et al. [pdf] ✨
- OverFeat: Integrated recognition, localization and detection using convolutional networks (2014), P. Sermanet et al.
*(LeCun)*[pdf] - Visualizing and understanding convolutional networks (2014), M. Zeiler and R. Fergus [pdf] ✨
- Maxout networks (2013), I. Goodfellow et al.
*(Bengio)*[pdf] - ImageNet classification with deep convolutional neural networks (2012), A. Krizhevsky et al.
*(Hinton)*[pdf] ✨ - Large scale distributed deep networks (2012), J. Dean et al. [pdf] ✨
- Deep sparse rectifier neural networks (2011), X. Glorot et al.
*(Bengio)*[pdf]

### Image

- Imagenet large scale visual recognition challenge (2015), O. Russakovsky et al. [pdf] ✨
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015), S. Ren et al. [pdf]
- DRAW: A recurrent neural network for image generation (2015), K. Gregor et al. [pdf]
- Rich feature hierarchies for accurate object detection and semantic segmentation (2014), R. Girshick et al. [pdf]
- Learning and transferring mid-Level image representations using convolutional neural networks (2014), M. Oquab et al.[pdf]
- DeepFace: Closing the Gap to Human-Level Performance in Face Verification (2014), Y. Taigman et al.
*(Facebook)*[pdf] - Decaf: A deep convolutional activation feature for generic visual recognition (2013), J. Donahue et al. [pdf]
- Learning Hierarchical Features for Scene Labeling (2013), C. Farabet et al.
*(LeCun)*[pdf] - Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis (2011), Q. Le et al. [pdf]
- Learning mid-level features for recognition (2010), Y. Boureau
*(LeCun)*[pdf]

### Caption

- Show, attend and tell: Neural image caption generation with visual attention (2015), K. Xu et al.
*(Bengio)*[pdf] ✨ - Show and tell: A neural image caption generator (2015), O. Vinyals et al. [pdf] ✨
- Long-term recurrent convolutional networks for visual recognition and description (2015), J. Donahue et al. [pdf] ✨
- Deep visual-semantic alignments for generating image descriptions (2015), A. Karpathy and L. Fei-Fei [pdf] ✨

### Video / Human Activity

- Large-scale video classification with convolutional neural networks (2014), A. Karpathy et al.
*(FeiFei)*[pdf] - A survey on human activity recognition using wearable sensors (2013), O. Lara and M. Labrador [pdf]
- 3D convolutional neural networks for human action recognition (2013), S. Ji et al. [pdf]
- Deeppose: Human pose estimation via deep neural networks (2014), A. Toshev and C. Szegedy [pdf]
- Action recognition with improved trajectories (2013), H. Wang and C. Schmid [pdf]

### Word Embedding

- Glove: Global vectors for word representation (2014), J. Pennington et al. [pdf] ✨
- Sequence to sequence learning with neural networks (2014), I. Sutskever et al. [pdf]
- Distributed representations of sentences and documents (2014), Q. Le and T. Mikolov [pdf]
*(Google)*✨ - Distributed representations of words and phrases and their compositionality (2013), T. Mikolov et al.
*(Google)*[pdf] ✨ - Efficient estimation of word representations in vector space (2013), T. Mikolov et al.
*(Google)*[pdf] ✨ - Word representations: a simple and general method for semi-supervised learning (2010), J. Turian
*(Bengio)*[pdf]

### Machine Translation / QnA

- Towards ai-complete question answering: A set of prerequisite toy tasks (2015), J. Weston et al. [pdf]
- Neural machine translation by jointly learning to align and translate (2014), D. Bahdanau et al.
*(Bengio)*[pdf] ✨ - Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014), K. Cho et al.
*(Bengio)*[pdf] - A convolutional neural network for modelling sentences (2014), N. kalchbrenner et al. [pdf]
- Convolutional neural networks for sentence classification (2014), Y. Kim [pdf]
- The stanford coreNLP natural language processing toolkit (2014), C. Manning et al. [pdf]
- Recursive deep models for semantic compositionality over a sentiment treebank (2013), R. Socher et al. [pdf] ✨
- Natural language processing (almost) from scratch (2011), R. Collobert et al. [pdf]
- Recurrent neural network based language model (2010), T. Mikolov et al. [pdf]

### Speech / Etc.

- Speech recognition with deep recurrent neural networks (2013), A. Graves
*(Hinton)*[pdf] - Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups (2012), G. Hinton et al. [pdf] ✨
- Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition (2012) G. Dahl et al. [pdf]✨

### RL / Robotics

- Mastering the game of Go with deep neural networks and tree search, D. Silver et al.
*(DeepMind)*[pdf] - Human-level control through deep reinforcement learning (2015), V. Mnih et al.
*(DeepMind)*[pdf] ✨ - Deep learning for detecting robotic grasps (2015), I. Lenz et al. [pdf]
- Playing atari with deep reinforcement learning (2013), V. Mnih et al.
*(DeepMind)*[pdf])

### Unsupervised

- Building high-level features using large scale unsupervised learning (2013), Q. Le et al. [pdf] ✨
- Contractive auto-encoders: Explicit invariance during feature extraction (2011), S. Rifai et al.
*(Bengio)*[pdf] - An analysis of single-layer networks in unsupervised feature learning (2011), A. Coates et al. [pdf]
- Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion (2010), P. Vincent et al.
*(Bengio)*[pdf] - A practical guide to training restricted boltzmann machines (2010), G. Hinton [pdf]
- Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion (2010), P. Vincent et al.
*(Bengio)*[pdf]

### Hardware / Software

- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016), M. Abadi et al.
*(Google)*[pdf] - MatConvNet: Convolutional neural networks for matlab (2015), A. Vedaldi and K. Lenc [pdf]
- Caffe: Convolutional architecture for fast feature embedding (2014), Y. Jia et al. [pdf] ✨
- Theano: new features and speed improvements (2012), F. Bastien et al.
*(Bengio)*[pdf]

## License

To the extent possible under law, Terry T. Um has waived all copyright and related or neighboring rights to this work.

Advertisements