Skip to content

Latest commit

 

History

History
198 lines (113 loc) · 6.13 KB

README.md

File metadata and controls

198 lines (113 loc) · 6.13 KB
Raw

P2.13_seed

Deep Learning

This is the first part of the Deep Learning Course for the Master in High-Performance Computing (SISSA/ICTP): Introduction to Neural Networks.

You can find here the second part of the course, Natural Language Processing, by Cristiano De Nobili.

Main Topics

First Part

  • Artificial neural networks
  • Train, validate and test a deep learning model
  • Convolutional neural networks
  • Brief remarks on unsupervised models

Second Part

  • Natural Language Processing
  • Transformers and contextual word embeddings
  • PyTorch, SpaCy and Hugging Face libraries

Teachers

  • Alessio Ansuini (Research and Technology Institute, AREA Science Park)
  • Cristiano De Nobili (HARMAN International, a Samsung Company)

Follow-up course P2.14: Deep generative models with TensorFlow 2

  • Piero Coronica (RSE @ Research Computing Services - University of Cambridge)

Detailed Syllabus of the First Part

Day 1

  • The artificial neuron, activation functions, capacity of a single neuron
  • The limits of a single neuron, transformations of the input: the concept of representation
  • Fully connected architectures, exact count and scaling of the number of parameters
  • Universal approximation theorem (sketched)
  • Softmax, probabilistic interpretation of the output, discriminative models
  • Cross-entropy loss, probabilistic interpretation as maximum-likelihood inference, information-theoretical interpretation as KL distance between the probability of the data and the model
  • Optimization and learning rules for gradient descent and its stochastic counterparts
  • Regularization L2, L1 and their influence on training dynamics

Sources (see below): Hugo Larochelle's Neural networks class, Michael Nielsen's free book

Day 2

  • Representations as a scientific object of investigation
  • More on regularization: dropout
  • Convolutional networks: convolutional layers, pooling layers
  • Transfer learning

Sources: Michael Nielsen's free book, image kernels, PyTorch Tutorials, Intrinsic dimension, SVCCA, PWCCA, CKA, T-SNE

Day 3

Exam

The exam will consist in an exploration of a recent finding on network pruning called "The Lottery Ticket Hypothesis"

Videos

In order to see a video I suggest to download it (otherwise only shortened previews are available)

Resources

There are excellent free resources to deepen your knowledge on topics such as Deep Learning, Reinforcement Learning and more in general Artificial Intelligence.

Here is a selection of very good ones.

Books for free

Courses for free

Websites and Blogs

YouTube channels