day3/notebooks/Representation_learning_and_Self_supervised_learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "collapsed_sections": [
        "TCzcNAQUC5xz"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Outline\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "xW-1nTrJMCZ2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This notebook introduces self-supervised representation learning for vision and vision-language tasks, with a probabilistic view on it.\n",
        "\n",
        "*** General introduction: why representation learning***\n",
        "\n",
        "**Introduction on InfoNCE**\n",
        "1. InfoNCE (Non-contrastive loss estimation) as in [Oord et al](https://arxiv.org/pdf/1807.03748v2.pdf)\n",
        "  - *Math assignment*: show that InfoNCE correctly retrieves the desired conditional density ratio as an alternative to generative models.\n",
        "  - Link with InfoMax principle (see Oord et al): minimizing InfoNCE amounts to maximizing mutual information (MI).\n",
        "\n",
        "**InfoNCE applications in image classification and a new view on its role as a MI lower bound**\n",
        "2. InfoNCE applications in classification\n",
        "  - Application of InfoNCE to image classification large-scale:\n",
        "  - [CPCv2](https://arxiv.org/pdf/1905.09272.pdf)\n",
        "  - [SimCLR](https://arxiv.org/abs/2002.05709)\n",
        "  - Causal view on SimCLR-like training and its link to disentanglement\n",
        "3. New view away from mutual information maximization: [Wang and Isola](https://arxiv.org/abs/2005.10242) explanation of uncertainty and alignment.\n",
        "<!-- and how InfoNCE amounts to entropy estimation via Von Mises Fisher (KDE-vMF) -->\n",
        "\n",
        "**Robust self-supervised architectures**\n",
        "4. Present some of our works in the context of robustness of SSL e.g. [Robust self-supervised learning with Lie groups](https://https://colab.research.google.com/drive/1pKnR0uu1k_SBr8Buz5Rvq4OlMFAJ2_UN#scrollTo=xW-1nTrJMCZ2)\n",
        "\n",
        "\n",
        "**The vision-language revolution**\n",
        "5. The classic OpenAI [CLIP](https://arxiv.org/pdf/2103.00020.pdf) used for image classification\n",
        "  - Then move on to [BLIP](https://arxiv.org/abs/2201.12086) (which has a text generative model)\n",
        "  - *Code assignment*:\n",
        "    - Download BLIP\n",
        "    - Generate caption (BLIP Conditional generation), Answer question (BLIP VQA), image-to-text retrieval\n",
        "    - Compute the likelihood of a caption for a corresponding image\n",
        "    - A look at the model's failures in unusual settings\n",
        "6. Present some of our works on VLMs, again in the context of robustness e. g. Embracing Diversity: Zero-shot Classification Beyond a Single Vector per Class (arxiv out soon) or Polina's project"
      ],
      "metadata": {
        "id": "XBGEXEaO9n1V"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# General introduction: why learning good features? What are good features?\n",
        "* Features that allow to do a variety of tasks without retraining the model\n",
        "* Features that capture high-level, semantic information in order to generalize to unseen settings.\n",
        "* Features that are invariant / robust to noise factors, not useful for downstream tasks.\n"
      ],
      "metadata": {
        "id": "wluoxTlRg94A"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1. Self-supervised learning with InfoNCE: the Contrastive Predictive Coding model\n",
        "Contrastive Predictive Coding leverages (i) contrastive loss with (ii) predictive coding. The learning objective, InfoNCE (Info Noise-Contrastive Estimation) allows us to learn useful representation of input data by encoding shared, high-level information and discarding low-level information and noise.\n",
        "\n"
      ],
      "metadata": {
        "id": "TCzcNAQUC5xz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Intro & intuition\n",
        "Contrastive prediction coding from Oord et al.\n",
        "\n",
        "The task is to learn representations that encode the shared (high-level) information between different samples input data. One of way to do so in an unsupervised manner is to learn to predict missing or future information $x$, from context $c$ (e.g. past observations). However, using powerful conditional generative models to learn $p(x|c)$ often leads to learning complex, low-level information about $x$ that is not useful for the downstream tasks, and may not b optimal for the purpose of extracting shared information between the data and its context. Rather, Oord et al. propose to learn a compact representation of the data which maximally optimize the mutual information between $x$ and $c$, written as\n",
        "$\n",
        "\\begin{equation}\n",
        "I(x;c) = \\sum_{x,c} p(x,c) \\log \\dfrac{p(x|c)}{p(x)}\n",
        "\\end{equation}\n",
        "$\n",
        "\n",
        "The goal is that the model encodes the underlying shared information between different parts of the data while discarding noise and low-level information.Let us derive how to get there with the example from Oord et al."
      ],
      "metadata": {
        "id": "HaHDcKJaf-PX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1qxk7TFOsz8vHJ5UIng3LS88TBdjDw7Iw\" width=\"500\"/>\n",
        "</div>"
      ],
      "metadata": {
        "id": "cWcdJl4NzDwF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ":An encoder model infers compact representations from the observations via $g_{enc}(x_t) = z_t$. An autoregressive model summarizes all previous encodings in a context latent representation $c_t = g_{ar} (z_{\\le t})$.\n",
        "\n",
        "Instead of directly maximizing $I(x;c)$, InfoNCE models a ratio estimator $f(x_{t+k},c_t) \\propto \\dfrac{p(x_{t+k}|c_t)}{p(x_{t+k})}$. We'll see how this leads to maximizing a lower bound of the mutual information.\n"
      ],
      "metadata": {
        "id": "H_NviA5q4aqY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Oord et al. use\n",
        "\\begin{equation}\n",
        "f(x_{t+k},c_t) = \\exp(z_{t+k}^T W_k c_t)\n",
        "\\end{equation}\n",
        "where $z_{t+k}$ is inferred from the encoder and $W_k$ is a linear matrix but a neural network could be used. The encoder $g$, the autoregressive model $g_{ar}$ and $W_k$ must be learnt. How do we do this?\n",
        "\n",
        "From Oord et al., for a given time step $x_{t+k}$ to predict, we have a set $X=\\{x_1, \\dots x_{N}\\}$ of $N$ random samples containing one positive sample from $p(x_{t+k}|c_t)$ and $N-1$ negative samples from a 'proposal' distribution $p(x_{t+k})$ not drawn from the conditional distribution. The InfoNCE objective is defined as\n",
        "\\begin{align}\n",
        "\\text{InfoNCE}_{N} &= - \\mathop{{}\\mathbb{E}}_{X}\\left[\\log \\frac{f_k(x_{t+k}, c_{t})}{\\sum_{x_j \\in X} f_k(x_j, c_{t})}\\right] \\nonumber \\\\\n",
        "& =- \\mathop{{}\\mathbb{E}}_{X}\\left[\\log \\frac{\\exp(z_{t+k}^T W_k^Tc_t)}{\\sum_{x_j \\in X} \\exp(z_{j}^T W_k^Tc_t)}\\right] \\nonumber \\\\\n",
        "& =- \\mathop{{}\\mathbb{E}}_{X}\\left[\\log \\frac{\\exp(z_{t+k}^T \\widehat{z}_{t+k})}{\\sum_{x_j \\in X} \\exp(z_j^T \\widehat{z}_{t+k})}\\right]\n",
        "\\end{align}\n",
        "where $\\widehat{z}_{t+k}= W_k^Tc_t$"
      ],
      "metadata": {
        "id": "m5z5faREWs27"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercice: prove that minimizing InfoNCE_N results in $f(x_{t+k},c_t)$ correctly estimating the ratio $\\dfrac{p(x_{t+k}|c_t)}{p(x_{t+k})}$\n",
        "\n",
        "(from Oord et al.)\n",
        "\n",
        " $\\text{InfoNCE}_{N}$ can be seen as the cross-entropy loss when considering the task of *classifying if the sample $x_i$ is the positive sample and the others are negatives*. Hence the term \"noise contrastive estimation\": distinguish the true sample from the conditional, from the “fake” ones, coming from a noise distribution, here the marginal.\n",
        "\n",
        "Let us write the event that $x_i$ is the true sample drawn from the conditional $p(x_{t+k}|c_t)$ as $[d=i]$. The model prediction is $q(d=i|X,c_t)=\\frac{f_k(x_i, c_{t})}{\\sum_{x_j \\in X} f_k(x_j, c_{t})}$.\n",
        "\n",
        "\n",
        "InfoNCE is minimized when the model prediction $q(d=i|X,c_t)$ equals the true probability for the sample $x_i$ to be the positive sample drawn from $(x_{t+k}|c_{t})$ rather than the proposal distribution $p(x_{t+k})$. Oord et al. derive as follows:\n",
        "\\begin{align}\n",
        "p(d=i| X, c_{t}) &= \\frac{p(x_i|c_{t})\\prod_{l\\neq i}p(x_l)}{\\sum^N_{j=1} p(x_j|c_{t})\\prod_{l\\neq j}p(x_l)} \\nonumber \\\\\n",
        "&=\\frac{\\frac{p(x_i|c_{t})}{p(x_i)}\\prod_{l=1}^{N}p(x_l)}{{\\sum^N_{j=1} \\frac{p(x_j|c_{t})}{p(x_j)}} \\prod_{l=1}^{N}p(x_l)} = \\frac{\\frac{p(x_i|c_{t})}{p(x_i)}}{\\sum^N_{j=1} \\frac{p(x_j|c_{t})}{p(x_j)}}.\n",
        "\\end{align}\n",
        "\n",
        "We see that the optimal value of $f_k(x_{t+k}, c_{t})$ is proportional to $\\frac{p(x_{t+k}|c_{t})}{p(x_{t+k})}$\n",
        "\n",
        "Oord et al. do not explain in details how they come with the first equality, but I think I can write an explanation, with help from [Kratos](https://karlstratos.com/notes/nce.pdf) Section 1.1 on NCE:\n",
        "* The sampling process is: draw uniformly $i \\sim [1,...,N]$. Then for $l=1,...,N$ if $l=i$ then $x_i \\sim p(x_{t+k}|c_{t})$ else $x_l \\sim p(x_{t+k})$. Thus, the joint writes as\n",
        "\n",
        "\\begin{equation}\n",
        "p(d=i,X,c_t) = \\dfrac{1}{N}p(x_i|c_t)\\prod_{l\\neq i}p(x_l)\n",
        "\\end{equation}\n",
        "And the conditional as\n",
        "\n",
        "\\begin{align}\n",
        "p(d=i|X,c_t) &= \\dfrac{\\dfrac{1}{N}p(x_i|c_t)\\prod_{l\\neq i}p(x_l)}{p(X,c_t)} \\\\\n",
        "&= \\dfrac{\\dfrac{1}{N}p(x_i|c_t)\\prod_{l\\neq i}p(x_l)}{\\sum_{j=1}^N p(d=j,X,c_t)} \\\\\n",
        "&= \\dfrac{\\dfrac{1}{N}p(x_i|c_t)\\prod_{l\\neq i}p(x_l)}{\\sum_{j=1}^N \\dfrac{1}{N}p(x_j|c_t)\\prod_{l\\neq j}p(x_l)} \\\\\n",
        "&= \\dfrac{p(x_i|c_t)\\prod_{l\\neq i}p(x_l)}{\\sum_{j=1}^N p(x_j|c_t)\\prod_{l\\neq j}p(x_l)}\n",
        "\\end{align}\n",
        "\n"
      ],
      "metadata": {
        "id": "ihV6pFfqvdF8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## InfoNCE as a MI lower bound\n",
        "Additionally, the mutual information between the variables $c_t$ and $x_{t+k}$ writes as follows (see Oord et al. for the proof):\n",
        "$$\n",
        "I(x_{t+k}, c_t) \\geq \\log(N)-\\text{InfoNCE}_N,\n",
        "$$\n",
        "which becomes tighter as N becomes larger. Hence, minimizing the InfoNCE loss maximizes a lower bound on the mutual information.\n"
      ],
      "metadata": {
        "id": "wvKzQ_vRIyD0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Application of Contrastive Predictive Coding to speaker representation learning and classification\n",
        "### From representation learning unsupervised to supervised classification:\n",
        "\n",
        "Once the unsupervised representation pretraining is done, train a supervised linear head on top of the frozen representation encoder.\n",
        "\n",
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1-0sxR5p3x-n3xREr3wZ-2Uhn32XWrLRa\" width=\"300\"/>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1WBdf_kCMK2PGbl-hbmbwR8XV3yJLh-Wz\" width=\"250\"/>\n",
        "</div>\n"
      ],
      "metadata": {
        "id": "CeMBeTjPhLkF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2. InfoNCE for self-supervised pretrained representations in image classification tasks\n"
      ],
      "metadata": {
        "id": "6vHoE7Hai22-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "CPC was later applied to image classification, with the task of doing next patch prediction from context from previous patches:\n",
        "\n",
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1jTiFOOuZFgOgK9FUZqxioBll9U0mZyMR\" width=\"800\"/>\n",
        "</div>\n",
        "\n",
        "* Each input image is first divided into a set of overlapping patches each of which is encoded with a neural network into a single vector $z_{i,j}$.\n",
        "<!-- * A masked convolutional network is then applied to the  grid of feature vectors. The masks are such that the receptive field of each resulting context vector $c_{i,j}$ only includes feature vectors that lie above it in the image. -->\n",
        "* Similar to the previous example, the prediction task then consists of predicting `future' feature vectors $z_{i+k,j}$ from current context vectors $c_{i,j}$. The predictions are made linearly $\\widehat{z}_{i+k,j} = W_{k}c_{i,j} $.  \n",
        "* Again, InfoNCE can be used: task is to recognize the target $z_{i+k,j}$ among a set of randomly sampled feature vectors $\\{z_l\\}$ from the dataset.\n",
        "\\begin{align}\n",
        "\\text{InfoNCE}_\\textrm{CPC} &= - \\sum_{i,j,k} \\log \\frac{\\exp(z_{i+k,j}^T \\widehat{z}_{i+k,j})}{\\exp(z_{i+k,j}^T \\widehat{z}_{i+k,j})+\\sum_l \\exp(z_l^T \\widehat{z}_{i+k,j})} \\nonumber \\\\\n",
        "&= - \\sum_{i,j,k} \\log \\frac{\\exp(z_{i+k,j}^T \\widehat{z}_{i+k,j})}{\\exp(z_{i+k,j}^T \\widehat{z}_{i+k,j})+\\sum_l \\exp(z_l^T \\widehat{z}_{i+k,j})} \\nonumber \\\\\n",
        "& = - \\mathop{{}\\mathbb{E}}_{X}\\left[\\log \\frac{\\exp(z_{i+k,j}^T \\widehat{z}_{i+k,j})}{ \\sum_{x_l \\in X} \\exp(z_l^T \\widehat{z}_{i+k,j})} \\right]\n",
        "\\end{align}\n",
        "where $X = \\{x_l, x_{i+j,k}\\}$.\n"
      ],
      "metadata": {
        "id": "t9OhigK4ZM4Q"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## SimCLR [Chen et al. ](https://https://arxiv.org/abs/2002.05709)\n",
        "\n",
        "This paper adapts InfoNCE loss in order to learn useful representation of image data. In this case, InfoNCE is used maximize agreement between two representations from two different crops of the same image. This provided impressive results and is to date one of the classic models for unsupervised pretraining on image data.\n"
      ],
      "metadata": {
        "id": "gOcI5PehAjjb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1RfBxXYDr5VglgijSp4aXn_YEvsKNdqtM\" width=\"500\"/>\n",
        "</div>\n",
        "\n",
        "\n",
        "Illustration adapted from the author blog post\n",
        "https://blog.research.google/2020/04/advancing-self-supervised-and-semi.html\n",
        "\n",
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1Kfy68O8aQFIKdEHYJVuJW4y0Evsr6Rpn\" width=\"500\"/>\n",
        "\n",
        "\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=12G-QaWXWUkHIOQd6haJMVfIci-FqEukl\" width=\"500\"/>\n",
        "</div>"
      ],
      "metadata": {
        "id": "QIwC8-pTEAJz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "SimCLR samples a minibatch of $N$ images and augments each image in 2 ways, resulting in $2N$ data points. Positives are crops from the same image, while negatives are the other $2(N-1)$ augmented samples within a minibatch.\n",
        "\n",
        "A CNN model (e.g. ResNet) produces representations $h_i = f(\\tilde{x}_i)$ that are fed to a projection network producing $z_i = g(h_i)$. InfoNCE loss is used to push samples from the same image closer, while pushing away all other augmented samples (negatives). For a pair of augmented samples $i,j$ from the same image, it writes as:\n",
        "\\begin{equation}\n",
        "    \\ell_{i,j} = -\\log \\frac{\\exp(\\text{sim}(z_i, z_j)/\\tau)}{\\sum_{k=1}^{2N} \\mathbf{1}_{k \\neq i}\\exp(\\text{sim}(z_i, z_k)/\\tau)}\n",
        "\\end{equation}\n",
        "where sim is a similarity function (e.g. cosine similarity in practice) and $\\tau$ denotes a temperature parameter. Note that SimCLR does not uses the positive in the denominator. The final loss is computed across all positive pairs, both $(i, j)$ and $(j, i)$ in a minibatch."
      ],
      "metadata": {
        "id": "KUu7ceoLIIig"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Causal interpretation\n",
        "One way to see the augmentation process is to consider a latent variable model by assuming the latent space ($z$ space) partitions into two parts: *content* (e.g. class of the object) which is untouched by augmentation and *style*, which is allowed to change, see [Kügelgen et al](https://arxiv.org/pdf/2106.04619.pdf). Using a causal interpretation of augmentations as soft style intervention, [Kügelgen et al](https://arxiv.org/pdf/2106.04619.pdf) show that training self-supervised architectures with data augmentations as in SimCLR *provably separates the content component from the style component*. Therefore, if we consider classification, it is really easier for a linear head on top of the representation to extract *only* the content, i.e. the class.\n",
        "\n",
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1vmBHIzbdiJVx-UjsweQSXVqCKuj2Uqtw\" width=\"250\"/>\n",
        "\n",
        "Our work [Eastwood et al.](https://openreview.net/forum?id=KVqyyPvK9H) extends it to show that every component of the style part (e.g. color, rotation angle) can similarly be decomposed in the latent space."
      ],
      "metadata": {
        "id": "W7wT4Xq4EEjQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Self-supervised models advantages\n",
        "* The results are close to fully supervised architecture.\n",
        "* The advantages of self-supervised (or unsupervised) representation learning:\n",
        "  * Allows to learn from *wide corpus of unlabelled data*.\n",
        "  * Ability to transfer learning (e.g. from ImageNet to Pascal VOC).\n",
        "\n",
        "#### SimCLR impressive performance on image classification\n",
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1WVz-QwiJsnCCa-ZZCfssg1qDVKbalmMt\" width=\"600\"/>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1OqK8P_a74MPihPJsQkG5KVR0wldFS7o4\" width=\"400\"/>\n",
        "</div>"
      ],
      "metadata": {
        "id": "Rnwis5dwZvqQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Contrastive loss general form\n",
        "\n",
        "Following Wang et Isola, we will write the InfoNCE over all samples in a more general form assuming:\n",
        "* $p_{data}$ the data distribution\n",
        "* $p_{pos}$ the distribution of positive pairs:\n",
        "* $f$ an encoder model mapping data to normalized feature vectors ($z$ in our previous notes)\n",
        "\\begin{equation}\n",
        "\\text{InfoNCE}(f, \\tau, M) = \\mathop{{}\\mathbb{E}}_{(x,y) \\sim p_{pos}\\\\\\{x^-_k\\}_{k=1}^M \\sim p_{data} } \\left[ - \\log \\frac{\\exp(f(x)^Tf(y)/\\tau)}{\\exp(f(x)^T f(y))/\\tau)+ \\sum_{k=1}^N \\exp(f(x)^T f(x^-_k)/\\tau)} \\right]\n",
        "\\end{equation}"
      ],
      "metadata": {
        "id": "0FiJaN_S1GCv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3. A new theoretical analysis of InfoNCE: Wang et Isola alignment and uniformity\n"
      ],
      "metadata": {
        "id": "k9s_CCEOhKO9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In fact, the interpretation of InfoNCE as a mutual information lower bound is inconsistent. Maximizing lower bounds on mutual information with the InfoMax principle can result in bad representations while looser bounds can lead to better representations (see [Tschannen et al.](https://arxiv.org/abs/1907.13625)). [Wang et Isola ](https://https://arxiv.org/pdf/2005.10242.pdf) propose a new view on contrastive learning based on *alignment* and *uniformity*."
      ],
      "metadata": {
        "id": "5xy2-JNok_-P"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1K5NjatcgEQFL2t-GJpqcuskswvF1ZPdb\" width=\"500\"/>\n",
        "</div>"
      ],
      "metadata": {
        "id": "TbPCH1w9k--L"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "They show that the contrastive loss optimizes for alignment and uniformity. Alignment pushes similar samples to have similar features and be close, while uniformity preserves as much information as possible about the data by distributing representation vectors uniformely on the hypersphere. Their main theorem is (we won't go into the proof now):\n",
        "\n",
        "For fixed $\\tau > 0$, as the number of negative samples $M \\rightarrow \\infty$, the contrastive loss converges to\n",
        "\\begin{align}\n",
        "\\lim_{M \\rightarrow \\infty} \\text{InfoNCE}(f,\\tau,M) - \\log(M) &= \\mathop{{}\\mathbb{E}}_{(x,y) \\sim p_{pos}} \\left[ - f(x)^Tf(y)/\\tau \\right] \\nonumber \\\\\n",
        "&+ \\mathop{{}\\mathbb{E}}_{x \\sim p_{data} } \\left[ - \\log \\mathop{{}\\mathbb{E}}_{x^- \\sim p_{data} } \\left[ \\exp(f(x)^T f(x^-))/\\tau \\right] \\right]\n",
        "\\end{align}\n",
        "The first term corresponds to alignment, bringing positive pairs closer. It is always positive and bounded by zero. The second term maximizes pairwise distances. Wang and Isola show that the first term is minimized by a perfectly aligned encoder (positive samples have same encodings) and the second term by a perfectly uniform encoder (samples from $p_{data}$ are uniformely distributed on the hypersphere). Regarding existence of an aligned and uniform encoder, Theorem 2 in Wang and Isola states that if perfectly aligned and uniform encoders exist, they form the exact minimizers of the contrastive loss in the case of a single sample.\n"
      ],
      "metadata": {
        "id": "BLOUAbU7-Ny2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Furthermore, they note that the importance of the normalization to the hypersphere in terms of performance could be explained by formalizing the intuition that connected sets with smooth boundaries can be linearly separated on the hyperphere, as shown by the figure below:\n",
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1xf5ZtTclBN3RUHkwvUI3HZQtsCYA6q5M\" width=\"500\"/>\n",
        "</div>"
      ],
      "metadata": {
        "id": "qARAlo1tEIwe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 4. Robust self-supervised pretraining\n",
        "Introduce our work on [self-supervised learning with Lie groups](https://arxiv.org/abs/2210.13356)"
      ],
      "metadata": {
        "id": "E5RLFjcIig97"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 5. Open-vocabulary classification with VLMs\n",
        "\n"
      ],
      "metadata": {
        "id": "T2qornQAiqsR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## OpenAI's CLIP model\n"
      ],
      "metadata": {
        "id": "bRY5IPKuRAG0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "* Contrastive learning applied to image-caption pairs:\n",
        "  - Pre-training step is to train a multi-modal model on matching images and their correct captions.  \n",
        "  - No need to manually annotate millions of data (costly)\n",
        "<div>\n",
        "<img src=\"https://images.openai.com/blob/fbc4f633-9ad4-4dc2-bd94-0b6f1feee22f/overview-a.svg?width=10&height=10&quality=50\" width=\"500\"/>\n",
        "</div>\n",
        "\n",
        "Training objective would write as:\n",
        "\n",
        "\\begin{align}\n",
        "\\text{InfoNCE} &= \\dfrac{1}{2} (\\text{InfoNCE}_{img} + \\text{InfoNCE}_{text})\\\\\n",
        "&= \\dfrac{1}{2} \\mathop{{}\\mathbb{E}}_{(x,y) \\sim p_{data~pairs}} \\left[ - \\log \\frac{\\exp(f(x)^Tg(y)/\\tau)}{\\exp(f(x)^T g(y))/\\tau)+ \\sum_{y' \\neq y} \\exp(f(x)^T g(y')/\\tau)} \\\\  - \\log \\frac{\\exp(f(x)^Tg(y)/\\tau)}{\\exp(f(x)^T g(y))/\\tau)+ \\sum_{y' \\neq y} \\exp(f(x')^T g(y)/\\tau)} \\right]\n",
        "\\end{align}\n",
        "\n",
        "* Once the model is pre-trained, use the text open-vocabulary to perform efficient zero-shot classification without the need to fine-tune.\n",
        "  - Create text classifier by embedding each class names with template(s) and getting per-class text embedding vector.\n",
        "  - For each image, compare its image embedding similarity with each class text's embedding and pick the highest scoring class.\n",
        "<div>\n",
        "<img src=\"https://images.openai.com/blob/d9d46e4b-6d6a-4f9e-9345-5c6538b1b8c3/overview-b.svg?width=10&height=10&quality=50\" width=\"500\"/>\n",
        "</div>\n"
      ],
      "metadata": {
        "id": "I6mZko4IH3nm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "* A sample of their result from the paper:\n",
        "\n",
        "  *Across a 27 dataset eval suite, a zero-shot CLIP classifier outperforms a fully supervised linear classifier fitted on ResNet-50 features on 16 datasets, including ImageNet.*\n"
      ],
      "metadata": {
        "id": "UIkmAbbCKj_P"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## BLIP model from Salesforce"
      ],
      "metadata": {
        "id": "fjyXK5x_RDKq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "One of the issue with CLIP-like models is that the training data includes noisy captions that only describe part of the image. Plus, CLIP has no generative ability.\n",
        "As a potential improvement, the BLIP model was introduced as multi-task model which can operate in one of the three functionalities (text encoder is shared, apart from the attention modules):\n",
        "* Unimodal encoders: separately encode image and text.\n",
        "* Image-grounded text encoder: encode text conditioned on the image.\n",
        "* Image-grounded text decoder: generate text conditioned on the image.\n",
        "\n",
        "In order to train the modules in the different settings, BLIP uses 3 types of losses:\n",
        "* Contrastive loss (ITC) between image and text modalities, as we've discussed for CLIP, to train in the unimodal setting.\n",
        "* Image-Text Matching Loss (ITM) to train as a image-grounded text encoder. This loss trains the model to predict whether an image-text pair is positive (matched) or negative (unmatched) given their multimodal feature. It can be seen a binary version of the contrastive task.\n",
        "* Language Modeling Loss (LM) to train as an image-grounded text decoder: the loss is text generation (cross-entropy) on the correct caption.\n",
        "\n",
        "BLIP also introduces a captioner and filterer to get cleaner captions but I won't get into these details.\n",
        "\n",
        "<div>\n",
        "<img src=\"https://drive.google.com/uc?export=view&id=1nzh3cyAu7Vfqa6ozdfS4-D1VEEBCoGpK\" width=\"800\"/>\n",
        "</div>\n"
      ],
      "metadata": {
        "id": "wku5T3DZK0HB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Code assignment\n",
        "\n",
        "See https://colab.research.google.com/drive/1ZUl_qKUvSwanj3Sj_XG4JCBUybMZriQP"
      ],
      "metadata": {
        "id": "rae3riJx7_ux"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 6. Limitations of VLMs & future work\n",
        "\n"
      ],
      "metadata": {
        "id": "UfMvhFYAQ1H9"
      }
    }
  ]
}