Deep Learning and Neural Networks syllabus

Fundamental Neuron Models: Biological neuron vs. Computational units, McCulloch–Pitts unit and thresholding logic, Linear perceptron and perceptron learning algorithm (Architecture, weight update rule, and learning process). Multilayer Perceptrons and Convergence: Linear separability and convergence theorem for perceptron learning algorithm (Geometric interpretation and XOR problem), Introduction to multilayer perceptrons (MLPs) (Need for hidden layers and forward pass), Representation power of MLPs (Universal approximation theorem), Sigmoid neurons (Non-linearity, Mathematical formulation, and Comparison with threshold units).

Feedforward and Backpropagation: Feedforward neural networks representation (input, hidden, and output layers), Backpropagation algorithm (mathematical derivation using chain rule, error calculation, and step-by-step working mechanism). Gradient Descent Variants and Challenges: Gradient descent (batch GD), Stochastic gradient descent (SGD), Momentum based GD, Nesterov accelerated GD, AdaGrad, RMSProp, Adam (brief working and advantages of each), Saddle point problem in neural networks (Difference from local minima and mitigation strategies).

Bias-Variance and Basic Regularization: Bias-variance tradeoff (underfitting vs. Overfitting), L2 regularization (weight decay), Early stopping, Dataset augmentation, Parameter sharing and tying, Injecting noise at input, Ensemble methods (bagging and boosting). Advanced Regularization and Training Enhancements: Dropout and drop connect, Batch normalization, Better activation functions (ReLU, Leaky ReLU, and ELU), Better weight initialization methods (Xavier and He initialization), Greedy layer wise pre-training.

Basic Autoencoders and Regularization: Autoencoders (definition and basic architecture: Encoder, Bottleneck, Decoder), Regularization in autoencoders, Denoising autoencoders. Specialized Autoencoders and Training Overview: Sparse autoencoders, Contractive autoencoders, Overview of autoencoder training: Problem definition of reconstruction, Data encoding and decoding, Interpretation of latent representations, Decision-making for feature learning.

Convolutional Neural Networks: Introduction to CNN and building blocks of CNN (convolution, pooling, and fully connected layers), Overview of CNN architectures (LeNet, AlexNet, ZF-Net, VGGNet, GoogLeNet, ResNet), Visualizing CNNs and guided backpropagation, Fooling convolutional neural networks, Transfer learning. Recurrent Neural Networks and Attention: Introduction to RNN and backpropagation through time (BPTT), Vanishing and exploding gradients and truncated BPTT, Long short term memory (LSTM) and gated recurrent units (GRU), Bidirectional RNNs and bidirectional LSTMs, Encoder-decoder models and attention mechanism.