Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Abstract
Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets), we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66%). To our knowledge, our result is the first to surpass human-level performance (5.1%, Russakovsky et al.) on this visual recognition challenge.
References 22
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- Improving neural networks by preventing co-adaptation of feature detectors
- Return of the Devil in the Details: Delving Deep into Convolutional Nets
- Deeply-Supervised Nets
- Deep Learning Face Representation by Joint Identification-Verification
- Maxout Networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- One weird trick for parallelizing convolutional neural networks