Funções de Ativação.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Objetivos__:\n",
    "\n",
    "- Implementar as principais funções de ativação\n",
    "- Entender intuitivamente como $w$ e $b$ influenciam nas funções de ativação"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sumário"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Linear](#Linear)\n",
    "- [Sigmoid](#Sigmoid)\n",
    "- [Tanh](#Tanh)\n",
    "- [Rectified Linear Unit (ReLU)](#Rectified-Linear-Unit-(ReLU))\n",
    "- [Leaky ReLU](#Leaky-ReLU)\n",
    "- [Exponential Linear Unit (eLU)](#Exponential-Linear-Unit-(eLU))\n",
    "- [Tabela das Funções de Ativação](#Tabela-das-Funções-de-Ativação)\n",
    "- [Referências](#Referências)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import ipywidgets as wg\n",
    "from ipywidgets import interactive, fixed\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_interactive(w, b, func, ylim=fixed((0, 1)), show_der=False):\n",
    "    plt.figure(0)\n",
    "    \n",
    "    x = np.linspace(-10, 10, num=1000)\n",
    "    z = w*x + b\n",
    "    y = func(z)\n",
    "    \n",
    "    plt.plot(x, y, color='blue')\n",
    "    if show_der:\n",
    "        der = func(z, derivative=True)\n",
    "        y_der_z = der\n",
    "        y_der_x = w*der\n",
    "        plt.plot(x, y_der_z, color='red')\n",
    "        plt.plot(x, y_der_x, color='green')\n",
    "    plt.xlim(-10, 10)\n",
    "    plt.ylim(ylim[0], ylim[1])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$y=x$$\n",
    "\n",
    "$$y^\\prime = 1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactive_plot = interactive(plot_interactive, w=(-2.0, 2.0), b=(-3, 3, 0.5), func=fixed(linear), ylim=fixed((-10, 10)))\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sigmoid "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$y = \\frac{1}{1+e^{-x}}$$\n",
    "\n",
    "$$y^\\prime = y(1-y)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactive_plot = interactive(plot_interactive, w=(-2.0, 2.0), b=(-3, 3, 0.5), func=fixed(sigmoid))\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tanh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$y = \\frac{e^x - e^{-x}}{e^x+e^{-x}}$$\n",
    "\n",
    "$$y^\\prime = 1 - y^2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactive_plot = interactive(plot_interactive, w=(-2.0, 2.0), b=(-3, 3, 0.5), func=fixed(tanh), ylim=fixed((-2, 2)))\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rectified Linear Unit (ReLU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$y = max(0, x)$$\n",
    "\n",
    "$$\\frac{\\partial y}{\\partial x} = \\begin{cases}0 &,\\  x \\leq 0\\\\1 &,\\ x > 0\\end{cases}$$\n",
    "\n",
    "__Obs.__: Lembrando que a derivada da ReLU quando x = 0 não existe matematicamente, mas é convencionalmente definida como 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactive_plot = interactive(plot_interactive, w=(-2.0, 2.0), b=(-3, 3, 0.5), func=fixed(relu), ylim=fixed((-1, 10)))\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Leaky ReLU "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$y = \\begin{cases}\\alpha x &,\\ x \\leq 0\\\\x &,\\ x > 0\\end{cases}$$\n",
    "\n",
    "$$\\frac{\\partial y}{\\partial x} = \\begin{cases}\\alpha &,\\  x \\leq 0\\\\1 &,\\  x > 0\\end{cases}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactive_plot = interactive(plot_interactive, w=(-2.0, 2.0), b=(-3, 3, 0.5), func=fixed(leaky_relu), ylim=fixed((-1, 10)))\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exponential Linear Unit (eLU) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$y = \\begin{cases}\\alpha(e^x -1) &,\\ x \\leq 0\\\\x &,\\ x > 0\\end{cases}$$\n",
    "\n",
    "$$\\frac{\\partial y}{\\partial x} = \\begin{cases}y + \\alpha &,\\  x \\leq 0\\\\1 &,\\  x > 0\\end{cases}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactive_plot = interactive(plot_interactive, w=(-2.0, 2.0), b=(-3, 3, 0.5), func=fixed(elu), ylim=fixed((-2, 10)))\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tabela das Funções de Ativação"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/funcoes_de_ativacao.png\" width='700'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Referências"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Tabela das funções de ativação](https://en.wikipedia.org/wiki/Activation_function)\n",
    "- [Towards Data Science](https://medium.com/towards-data-science/activation-functions-neural-networks-1cbd9f8d91d6)\n",
    "- [Stack Exchange](https://stats.stackexchange.com/questions/115258/comprehensive-list-of-activation-functions-in-neural-networks-with-pros-cons)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "widgets": {
   "state": {
    "568cf18b31b04b14b5b59379918b64e2": {
     "views": [
      {
       "cell_index": 6
      }
     ]
    },
    "58032045773043aeb5dab953be590b44": {
     "views": [
      {
       "cell_index": 30
      }
     ]
    },
    "743ccf7c641c485e9250b2558be149d1": {
     "views": [
      {
       "cell_index": 22
      }
     ]
    },
    "a41e4be364004b0d924d05e304ee3bcb": {
     "views": [
      {
       "cell_index": 26
      }
     ]
    },
    "bbf702b6ba494528a2233c691e2bd6d9": {
     "views": [
      {
       "cell_index": 14
      }
     ]
    },
    "c259af61fd66413783ae3ed1e32ece47": {
     "views": [
      {
       "cell_index": 10
      }
     ]
    },
    "c7f3128cf6d8492795e406ed70fa7e07": {
     "views": [
      {
       "cell_index": 18
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}