Li, ZongpengSamavati, FaramarzZhang, Zijun2019-12-132019-12-132019-12Zhang, Z. (2019). Improving Deep Neural Networks: Optimization, Regularization, and Generative Modeling (Doctoral thesis, University of Calgary, Calgary, Canada). Retrieved from https://prism.ucalgary.ca.http://hdl.handle.net/1880/111343In the past decade, deep learning has revolutionized the fields of computer vision, speech recognition, natural language processing, and continues spreading to many other fields. Therefore, it is important to better understand and improve deep neural networks (DNNs), which serve as the backbone of deep learning. In this thesis, we approach this topic from three different perspectives: optimization, regularization, and generative modeling. Firstly, we address the generalization gap recently observed between adaptive optimization methods, such as Adam, and simple stochastic gradient descent (SGD). We develop a tailored version of Adam for training DNNs, which is shown to close the gap on image classification tasks. Secondly, we identify a side effect of a widely used regularization technique, dropout, and multiplicative noise in general. That is, multiplicative noise tends to increase the correlation between features. We then exploit batch normalization to efficiently remove the correlation effect. Finally, we focus on generative modeling, a fundamental application of DNNs. We propose a framework for training autoencoder-based generative models, with non-adversarial losses and unrestricted neural network architectures.engUniversity of Calgary graduate students retain copyright ownership and moral rights for their thesis. You may use this material in any way that is permitted by the Copyright Act or through licensing that has been assigned to the document. For uses that are not allowable under copyright legislation or licensing, you are required to seek permission.Computer Sciencedoctoral thesis10.11575/PRISM/37338