Prep 2: Deep Learning Fundamentals

📋 Main Topics¶

• Introduction to Neural Networks
• Components of an Artificial Neural Network (ANN)
• Universal Approximation Theorems
• Regularization for Neural Networks
• Vanishing/Exploding Gradient Problem
• Transfer Learning
• Backpropagation

🧠 Class Activity - Labs¶

• Google Neural Network Playground
• Lab: Mastering Deep Learning with Fashion MNIST and CIFAR10

📚 Required Readings¶

• A visual proof of the universal approximation theorem for Neural Networks
  • Interactive Live Neural Network Demo
• Brief Tutorials on Neural Networks:
  • Google’s Neural Network Crash Course
  • Neural Designer Tutorials
  • Adventures in Machine Learning Tutorial (PDF)

📚 Optional (Advanced) Reading¶

• Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville:
  • Section 6: Introduction to Deep Neural Networks
    • 6.1 (Pages 164-174)
    • 6.3 Hidden Units (Pages 187-193)
  • Section 7: Regularization for Deep Neural Networks
    • 7.4 Dataset Augmentation (Pages 255-258)
    • 7.12 Dropout (Pages 236-237)
    • 7.13 Adversarial Training (Pages 265-266)
  • Skip the remainder of chapters 6 and 7 unless explicitly mentioned above.

🎥 Recommended Videos¶

Neural Networks & Backpropagation¶

• Deep Neural Networks by Lex Fridman (Watch only the first 45 minutes)
• What is a Neural Network? (Part 1)
• How Neural Networks Work (Part 2)
• What is Backpropagation? (Part 3)
• Neural Network - Inside the Black Box (Part 1)
• Neural Network - Backpropagation (Part 2)

Vanishing/Exploding Gradient Problem¶

• Vanishing & Exploding Gradient Explained – DeepLizard
• Vanishing and Exploding Gradients – Andrew Ng (DeepLearning.AI)

Transfer Learning¶

• Transfer Learning – Andrew Ng (DeepLearning.AI)
• What is Transfer Learning? – Levity AI (includes video)

Universal Approximation Theorem¶

• CMU Lecture: The Universal Approximation Theorem

📝 Recommended Blog Posts¶

Vanishing/Exploding Gradient Problem¶

• The Vanishing/Exploding Gradient Problem in Deep Neural Networks – Towards Data Science
• Vanishing and Exploding Gradients Problems – GeeksforGeeks

Transfer Learning¶

• A Gentle Introduction to Transfer Learning – Machine Learning Mastery
• CS231n Transfer Learning Notes – Stanford
• Transfer Learning in PyTorch – learnpytorch.io

Universal Approximation Theorem¶

• A Visual Proof That Neural Nets Can Compute Any Function – Michael Nielsen