Improving Deep Neural Networks: Optimization, Regularization, and Generative Modeling
Date
2019-12
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
In the past decade, deep learning has revolutionized the fields of computer vision, speech recognition, natural language processing, and continues spreading to many other fields. Therefore, it is important to better understand and improve deep neural networks (DNNs), which serve as the backbone of deep learning. In this thesis, we approach this topic from three different perspectives: optimization, regularization, and generative modeling. Firstly, we address the generalization gap recently observed between adaptive optimization methods, such as Adam, and simple stochastic gradient descent (SGD). We develop a tailored version of Adam for training DNNs, which is shown to close the gap on image classification tasks. Secondly, we identify a side effect of a widely used regularization technique, dropout, and multiplicative noise in general. That is, multiplicative noise tends to increase the correlation between features. We then exploit batch normalization to efficiently remove the correlation effect. Finally, we focus on generative modeling, a fundamental application of DNNs. We propose a framework for training autoencoder-based generative models, with non-adversarial losses and unrestricted neural network architectures.
Description
Keywords
Citation
Zhang, Z. (2019). Improving Deep Neural Networks: Optimization, Regularization, and Generative Modeling (Doctoral thesis, University of Calgary, Calgary, Canada). Retrieved from https://prism.ucalgary.ca.