Node 21. Tensorflow2_APIs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "polish-offense",
   "metadata": {},
   "source": [
    "## 학습목표\n",
    "\n",
    "- Tensorflow V2의 개요와 특징을 파악한다.\n",
    "- Tensorflow V2의 3가지 주요 API 구성 방식을 이해하고 활용할 수 있다.\n",
    "- GradientTape를 활용해 보고 좀 더 로우 레벨의 딥러닝 구현 방식을 이해한다"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pharmaceutical-globe",
   "metadata": {},
   "source": [
    "## TensorFlow2 API로 모델 구성하기"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nutritional-triumph",
   "metadata": {},
   "source": [
    "### 1) TensorFlow2 Sequential Model\n",
    "\n",
    "> input 1개, output 1개를 전제로 함"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thermal-retro",
   "metadata": {},
   "source": [
    "### 2) TensorFlow2 Functional API\n",
    "> - `keras.Model`을 사용해서 모델을 만든다는 것이 특징이다.  \n",
    "> - 어떻게 보면 Sequential Model은 그냥 `keras.Model` 클래스를 상속바당서 쓰는 것이다.\n",
    "> - 입력과 출력을 규정해서 모델을 설계한다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "occupied-flashing",
   "metadata": {},
   "source": [
    "### 3) TensorFlow2 Subclassing\n",
    "\n",
    "> - `keras.Model`은 `__init__()`이라는 메서드 안에서 레이어 구성을 정의\n",
    "> - `call()` 메소드에서 `forward propagation`을 구현"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "willing-athletics",
   "metadata": {},
   "source": [
    "## Tensorflow2 API로 모델 작성하기: MNIST (1) Sequential API 활용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "pointed-multiple",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "single-infrared",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 10000\n"
     ]
    }
   ],
   "source": [
    "# 데이터 구성부분\n",
    "mnist = keras.datasets.mnist\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train/255.0, x_test/255.0\n",
    "\n",
    "x_train=x_train[...,np.newaxis]\n",
    "x_test=x_test[...,np.newaxis]\n",
    "\n",
    "print(len(x_train), len(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "attractive-winning",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000,)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "photographic-elements",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8),\n",
       " array([5923, 6742, 5958, 6131, 5842, 5421, 5918, 6265, 5851, 5949]))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(y_train, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "reduced-consumer",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "proper-postcard",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[...].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "pharmaceutical-element",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Sequential Model을 구성해주세요.\n",
    "\"\"\"\n",
    "Spec:\n",
    "1. 32개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "2. 64개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "3. Flatten 레이어\n",
    "4. 128개의 아웃풋 노드를 가지고, activation function이 relu인 Fully-Connected Layer(Dense)\n",
    "5. 데이터셋의 클래스 개수에 맞는 아웃풋 노드를 가지고, activation function이 softmax인 Fully-Connected Layer(Dense)\n",
    "\"\"\"\n",
    "\n",
    "# 여기에 모델을 구성해주세요\n",
    "\n",
    "model = keras.Sequential()\n",
    "model.add(keras.layers.Conv2D(32, kernel_size=3, activation='relu'))\n",
    "model.add(keras.layers.Conv2D(64, kernel_size=3, activation='relu'))\n",
    "model.add(keras.layers.Flatten())\n",
    "model.add(keras.layers.Dense(128, activation='relu'))\n",
    "model.add(keras.layers.Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "widespread-alert",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 17s 7ms/step - loss: 0.2043 - accuracy: 0.9381\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0325 - accuracy: 0.9901\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0182 - accuracy: 0.9938\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0095 - accuracy: 0.9969\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0095 - accuracy: 0.9967\n",
      "313/313 - 2s - loss: 0.0492 - accuracy: 0.9884\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.04915982857346535, 0.9883999824523926]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 모델 학습 설정\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=5)\n",
    "\n",
    "model.evaluate(x_test,  y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "disabled-browse",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_2 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 24, 24, 64)        18496     \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 36864)             0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 128)               4718720   \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 4,738,826\n",
      "Trainable params: 4,738,826\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "double-engine",
   "metadata": {},
   "source": [
    "## Tensorflow2 API로 모델 작성하기: MNIST (2) Functional API 활용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "labeled-hollow",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "finite-federation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 10000\n"
     ]
    }
   ],
   "source": [
    "mnist = keras.datasets.mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "\n",
    "x_train=x_train[...,np.newaxis]\n",
    "x_test=x_test[...,np.newaxis]\n",
    "\n",
    "print(len(x_train), len(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "neither-chamber",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28, 1)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "internal-satellite",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, 1, 2, 2)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.arange(16).reshape(4,2,2)[:,np.newaxis,...].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "hidden-catering",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Spec:\n",
    "0. (28X28X1) 차원으로 정의된 Input\n",
    "1. 32개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "2. 64개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "3. Flatten 레이어\n",
    "4. 128개의 아웃풋 노드를 가지고, activation function이 relu인 Fully-Connected Layer(Dense)\n",
    "5. 데이터셋의 클래스 개수에 맞는 아웃풋 노드를 가지고, activation function이 softmax인 Fully-Connected Layer(Dense)\n",
    "\"\"\"\n",
    "\n",
    "# 여기에 모델을 구성해 주세요.\n",
    "\n",
    "# --- Input\n",
    "inputs = keras.layers.Input((28, 28, 1))\n",
    "\n",
    "# --- Conv2D\n",
    "conv2d = keras.layers.Conv2D(32, kernel_size=3, activation='relu')\n",
    "x = conv2d(inputs)\n",
    "\n",
    "# --- Conv2D_1\n",
    "conv2d_1 = keras.layers.Conv2D(64, kernel_size=3, activation='relu')\n",
    "x = conv2d_1(x)\n",
    "\n",
    "# --- Flatten\n",
    "flatten = keras.layers.Flatten()\n",
    "x = flatten(x)\n",
    "\n",
    "# --- Dense\n",
    "dense = keras.layers.Dense(128, activation='relu')\n",
    "x = dense(x)\n",
    "\n",
    "# --- Dense_1\n",
    "dense_1 = keras.layers.Dense(10, activation='softmax')\n",
    "outputs = dense_1(x)\n",
    "\n",
    "# 모델 선언\n",
    "model = keras.Model(inputs=inputs, outputs=outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "emotional-flooring",
   "metadata": {},
   "source": [
    "```python\n",
    "# AIFFEL code\n",
    "inputs = keras.Input(shape=(28, 28, 1))\n",
    "\n",
    "x = keras.layers.Conv2D(32, 3, activation='relu')(inputs)\n",
    "x = keras.layers.Conv2D(64, 3, activation='relu')(x)\n",
    "x = keras.layers.Flatten()(x)\n",
    "x = keras.layers.Dense(128, activation='relu')(x)\n",
    "predictions = keras.layers.Dense(10, activation='softmax')(x)\n",
    "\n",
    "model = keras.Model(inputs=inputs, outputs=predictions)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "round-maryland",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.2136 - accuracy: 0.9324\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0332 - accuracy: 0.9899\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0170 - accuracy: 0.9945\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0116 - accuracy: 0.9958\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0079 - accuracy: 0.9975\n",
      "313/313 - 1s - loss: 0.0415 - accuracy: 0.9903\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.04145710915327072, 0.9902999997138977]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 모델 학습 설정\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=5)\n",
    "\n",
    "model.evaluate(x_test,  y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mental-malpractice",
   "metadata": {},
   "source": [
    "## Tensorflow2 API로 모델 작성하기: MNIST (3) Subclassing 활용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "mental-finish",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "saving-accessory",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 10000\n"
     ]
    }
   ],
   "source": [
    "# 데이터 구성부분\n",
    "mnist = keras.datasets.mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "\n",
    "x_train=x_train[...,np.newaxis]\n",
    "x_test=x_test[...,np.newaxis]\n",
    "\n",
    "print(len(x_train), len(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "indonesian-guitar",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Subclassing을 활용한 Model을 구성해주세요.\n",
    "\"\"\"\n",
    "Spec:\n",
    "0. keras.Model 을 상속받았으며, __init__()와 call() 메서드를 가진 모델 클래스\n",
    "1. 32개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "2. 64개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "3. Flatten 레이어\n",
    "4. 128개의 아웃풋 노드를 가지고, activation function이 relu인 Fully-Connected Layer(Dense)\n",
    "5. 데이터셋의 클래스 개수에 맞는 아웃풋 노드를 가지고, activation function이 softmax인 Fully-Connected Layer(Dense)\n",
    "6. call의 입력값이 모델의 Input, call의 리턴값이 모델의 Output\n",
    "\"\"\"\n",
    "\n",
    "# 여기에 모델을 구성해주세요\n",
    "class CustomModel(keras.Model):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = keras.layers.Conv2D(32, 3, activation='relu')\n",
    "        self.conv2 = keras.layers.Conv2D(64, 3, activation='relu')\n",
    "        self.flatten = keras.layers.Flatten()\n",
    "        self.fc1 = keras.layers.Dense(128, activation='relu')\n",
    "        self.fc2 = keras.layers.Dense(10, activation='softmax')\n",
    "        \n",
    "    def call(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.fc1(x)\n",
    "        x = self.fc2(x)\n",
    "        \n",
    "        return x\n",
    "        \n",
    "model = CustomModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "satellite-meditation",
   "metadata": {},
   "source": [
    "------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rural-andrew",
   "metadata": {},
   "source": [
    "```python\n",
    "# DJ practice\n",
    "class MyModel(keras.Model):\n",
    "    def __init__(self, input_shape=(28,28)):\n",
    "        # 여기서 super.__init__() 필요하고 \n",
    "        self.input_layer = keras.layers.Input(input_shape=input_shape)  # Input 레이어는 써있어도 한번도 call되지 않는단다. 그래서 오류난다.\n",
    "        self.conv2d = keras.layers.Conv2D(32, kernel_size=3, activation='relu')\n",
    "        self.conv2d_1 = keras.layers.Conv2D(64, kernel_size=3, activation='relu')\n",
    "        self.flatten = keras.layers.Flatten()\n",
    "        self.dense = keras.layers.Dense(128, activation='relu')\n",
    "        self.dense_1 = keras.layers.Dense(10, activation='softmax')\n",
    "    \n",
    "    def call(self):\n",
    "        inputs = self.input_layer # Input 삭제되고, call(self, inputs)로 들어가야 한다.\n",
    "        x = self.conv2d(inputs)\n",
    "        x = self.conv2d_1(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.dense(x)\n",
    "        outputs = self.dense_1(x)\n",
    "        \n",
    "        return keras.model(inputs=inputs, outputs=outputs) # 여기는 그냥 outputs\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "optical-abortion",
   "metadata": {},
   "outputs": [],
   "source": [
    "mymodel = MyModel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "national-smooth",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.2174 - accuracy: 0.9326\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0375 - accuracy: 0.9886\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0178 - accuracy: 0.9941\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0122 - accuracy: 0.9960\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0074 - accuracy: 0.9980\n",
      "313/313 - 1s - loss: 0.0416 - accuracy: 0.9896\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.04158053919672966, 0.9896000027656555]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# MyModel 클래스 살짝 고쳐서 해봤는데 됐다.\n",
    "mymodel.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "mymodel.fit(x_train,y_train, epochs=5)\n",
    "mymodel.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "killing-senegal",
   "metadata": {},
   "source": [
    "------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "tutorial-qatar",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.2183 - accuracy: 0.9326\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0333 - accuracy: 0.9897\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0164 - accuracy: 0.9947\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0119 - accuracy: 0.9961\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 14s 7ms/step - loss: 0.0096 - accuracy: 0.9967\n",
      "313/313 - 1s - loss: 0.0485 - accuracy: 0.9886\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.048538122326135635, 0.9886000156402588]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 모델 학습 설정\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=5)\n",
    "\n",
    "model.evaluate(x_test,  y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "precious-structure",
   "metadata": {},
   "source": [
    "## TensorFlow2 API로 모델 작성 및 학습하기: CIFAR-100 (1) Sequential API 활용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "intensive-access",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "caring-testament",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50000 10000\n"
     ]
    }
   ],
   "source": [
    "# 데이터 구성부분\n",
    "cifar100 = keras.datasets.cifar100\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = cifar100.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "print(len(x_train), len(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "clean-cricket",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,\n",
       "        17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,\n",
       "        34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,\n",
       "        51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67,\n",
       "        68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84,\n",
       "        85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]),\n",
       " array([500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500,\n",
       "        500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500,\n",
       "        500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500,\n",
       "        500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500,\n",
       "        500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500,\n",
       "        500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500,\n",
       "        500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500,\n",
       "        500, 500, 500, 500, 500, 500, 500, 500, 500]))"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(y_train, return_counts=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "every-quest",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "module 'tensorflow.keras.layers' has no attribute 'MaxPooling'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-64-04ca9d906096>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSequential\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mConv2D\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m16\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mactivation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'relu'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mMaxPooling\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mConv2D\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m32\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkernel_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mactivation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'relu'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mMaxPooling\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: module 'tensorflow.keras.layers' has no attribute 'MaxPooling'"
     ]
    }
   ],
   "source": [
    "# Sequential Model을 구성해주세요.\n",
    "\"\"\"\n",
    "Spec:\n",
    "1. 16개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "2. pool_size가 2인 MaxPool 레이어\n",
    "3. 32개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "4. pool_size가 2인 MaxPool 레이어\n",
    "5. 256개의 아웃풋 노드를 가지고, activation function이 relu인 Fully-Connected Layer(Dense)\n",
    "6. 데이터셋의 클래스 개수에 맞는 아웃풋 노드를 가지고, activation function이 softmax인 Fully-Connected Layer(Dense)\n",
    "\"\"\"\n",
    "\n",
    "# 여기에 모델을 구성해주세요\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "focal-thunder",
   "metadata": {},
   "source": [
    "```python\n",
    "model = keras.Sequential()\n",
    "model.add(keras.layers.Conv2D(16, kernel_size=3, activation='relu'))\n",
    "model.add(keras.layers.MaxPooling(2)) #<--- MaxPool2D((2,2)) 란다\n",
    "model.add(keras.layers.Conv2D(32, kernel_size=3, activation='relu'))\n",
    "model.add(keras.layers.MaxPooling(2))\n",
    "# 여기도 Flatten()이 들어간다. 안 들어가면 어떻게 될까..?\n",
    "model.add(keras.layers.Dense(256, activation='relu'))\n",
    "model.add(keras.layers.Dense(100, activation='softmax'))\n",
    "```\n",
    "\n",
    "**1) Flatten이 없을 때 어떻게 되는지 실험**  \n",
    "**2) Flatten 있는데 마지막 노드 갯수를 class 수랑 안 맞추면 어떻게 되는지 실험**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "falling-spending",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_noflat = keras.Sequential()\n",
    "model_noflat.add(keras.layers.Conv2D(16, kernel_size=3, activation='relu'))\n",
    "model_noflat.add(keras.layers.MaxPool2D((2,2)))\n",
    "model_noflat.add(keras.layers.Conv2D(32, kernel_size=3, activation='relu'))\n",
    "model_noflat.add(keras.layers.MaxPool2D((2,2)))\n",
    "model_noflat.add(keras.layers.Dense(256, activation='relu'))\n",
    "model_noflat.add(keras.layers.Dense(100, activation='softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "hungry-malaysia",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n"
     ]
    },
    {
     "ename": "InvalidArgumentError",
     "evalue": " logits and labels must have the same first dimension, got logits shape [1152,100] and labels shape [32]\n\t [[node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at <ipython-input-67-5ee36d46b7dd>:5) ]] [Op:__inference_train_function_82044]\n\nFunction call stack:\ntrain_function\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-67-5ee36d46b7dd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m                     metrics=['accuracy'])\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mmodel_noflat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m   1098\u001b[0m                 _r=1):\n\u001b[1;32m   1099\u001b[0m               \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1100\u001b[0;31m               \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1101\u001b[0m               \u001b[0;32mif\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_sync\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1102\u001b[0m                 \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    826\u001b[0m     \u001b[0mtracing_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexperimental_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    827\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mtrace\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTrace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_name\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mtm\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 828\u001b[0;31m       \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    829\u001b[0m       \u001b[0mcompiler\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"xla\"\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_experimental_compile\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m\"nonXla\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    830\u001b[0m       \u001b[0mnew_tracing_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexperimental_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    886\u001b[0m         \u001b[0;31m# Lifting succeeded, so variables are initialized and we can run the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    887\u001b[0m         \u001b[0;31m# stateless function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 888\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateless_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    889\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    890\u001b[0m       \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfiltered_flat_args\u001b[0m \u001b[0;34m=\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2941\u001b[0m        filtered_flat_args) = self._maybe_define_function(args, kwargs)\n\u001b[1;32m   2942\u001b[0m     return graph_function._call_flat(\n\u001b[0;32m-> 2943\u001b[0;31m         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access\n\u001b[0m\u001b[1;32m   2944\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2945\u001b[0m   \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m   1917\u001b[0m       \u001b[0;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1918\u001b[0m       return self._build_call_outputs(self._inference_function.call(\n\u001b[0;32m-> 1919\u001b[0;31m           ctx, args, cancellation_manager=cancellation_manager))\n\u001b[0m\u001b[1;32m   1920\u001b[0m     forward_backward = self._select_forward_and_backward_functions(\n\u001b[1;32m   1921\u001b[0m         \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m    558\u001b[0m               \u001b[0minputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    559\u001b[0m               \u001b[0mattrs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 560\u001b[0;31m               ctx=ctx)\n\u001b[0m\u001b[1;32m    561\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    562\u001b[0m           outputs = execute.execute_with_cancellation(\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m     58\u001b[0m     \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     59\u001b[0m     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 60\u001b[0;31m                                         inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m     61\u001b[0m   \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     62\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mInvalidArgumentError\u001b[0m:  logits and labels must have the same first dimension, got logits shape [1152,100] and labels shape [32]\n\t [[node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at <ipython-input-67-5ee36d46b7dd>:5) ]] [Op:__inference_train_function_82044]\n\nFunction call stack:\ntrain_function\n"
     ]
    }
   ],
   "source": [
    "model_noflat.compile(optimizer='adam',\n",
    "                    loss='sparse_categorical_crossentropy',\n",
    "                    metrics=['accuracy'])\n",
    "\n",
    "model_noflat.fit(x_train, y_train, epochs=5)\n",
    "\n",
    "model.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "processed-angel",
   "metadata": {},
   "source": [
    "#### # 그럼 flatten 추가해보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "stuffed-glenn",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_noflat = keras.Sequential()\n",
    "model_noflat.add(keras.layers.Conv2D(16, kernel_size=3, activation='relu'))\n",
    "model_noflat.add(keras.layers.MaxPool2D((2,2)))\n",
    "model_noflat.add(keras.layers.Conv2D(32, kernel_size=3, activation='relu'))\n",
    "model_noflat.add(keras.layers.MaxPool2D((2,2)))\n",
    "model_noflat.add(keras.layers.Flatten()) # 추가됨\n",
    "model_noflat.add(keras.layers.Dense(256, activation='relu'))\n",
    "model_noflat.add(keras.layers.Dense(100, activation='softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "european-ferry",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.1194 - accuracy: 0.4491\n",
      "Epoch 2/2\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 1.9885 - accuracy: 0.4746\n",
      "313/313 - 1s - loss: 2.5948 - accuracy: 0.3636\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[2.594791889190674, 0.3635999858379364]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_noflat.compile(optimizer='adam',\n",
    "                    loss='sparse_categorical_crossentropy',\n",
    "                    metrics=['accuracy'])\n",
    "\n",
    "model_noflat.fit(x_train, y_train, epochs=2)\n",
    "\n",
    "model_noflat.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "governing-virgin",
   "metadata": {},
   "source": [
    "#### # softmax 전에 클래스 수와 다른 output 개수를 설정할 경우"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "appointed-royalty",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_wrong = keras.Sequential()\n",
    "model_wrong.add(keras.layers.Conv2D(16, kernel_size=3, activation='relu'))\n",
    "model_wrong.add(keras.layers.MaxPool2D((2,2)))\n",
    "model_wrong.add(keras.layers.Conv2D(32, kernel_size=3, activation='relu'))\n",
    "model_wrong.add(keras.layers.MaxPool2D((2,2)))\n",
    "model_wrong.add(keras.layers.Flatten()) # 추가됨\n",
    "model_wrong.add(keras.layers.Dense(256, activation='relu'))\n",
    "model_wrong.add(keras.layers.Dense(200, activation='softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "laughing-sixth",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.7935 - accuracy: 0.3053\n",
      "Epoch 2/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.5773 - accuracy: 0.3495\n",
      "Epoch 3/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.4189 - accuracy: 0.3822\n",
      "Epoch 4/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.2619 - accuracy: 0.4168\n",
      "Epoch 5/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.1342 - accuracy: 0.4407\n",
      "Epoch 6/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.0076 - accuracy: 0.4710\n",
      "Epoch 7/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 1.9053 - accuracy: 0.4953\n",
      "Epoch 8/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 1.8163 - accuracy: 0.5146\n",
      "Epoch 9/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 1.7275 - accuracy: 0.5341\n",
      "Epoch 10/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 1.6445 - accuracy: 0.5551\n",
      "313/313 - 1s - loss: 2.7971 - accuracy: 0.3531\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[2.7971274852752686, 0.3531000018119812]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_wrong.compile(optimizer='adam',\n",
    "                    loss='sparse_categorical_crossentropy',\n",
    "                    metrics=['accuracy'])\n",
    "\n",
    "model_wrong.fit(x_train, y_train, epochs=10)\n",
    "\n",
    "model_wrong.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fatal-parallel",
   "metadata": {},
   "source": [
    "> 아예 작동 안 할 줄 알았는데 작동하긴 한다. 손실함수 계산할 때 y랑 y의 갯수가 다를텐데...  \n",
    "> broadcasting 된건가... 싶다.  \n",
    "> 만약 그런 거라면 제대로 된 모델보다 학습이 느리다고만 결론지으면 될 것 같다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "natural-portsmouth",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.Sequential([\n",
    "    keras.layers.Conv2D(16, 3, activation='relu'),\n",
    "    keras.layers.MaxPool2D((2,2)),\n",
    "    keras.layers.Conv2D(32, 3, activation='relu'),\n",
    "    keras.layers.MaxPool2D((2,2)),\n",
    "    keras.layers.Flatten(),\n",
    "    keras.layers.Dense(256, activation='relu'),\n",
    "    keras.layers.Dense(100, activation='softmax')\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "joint-change",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 3.9830 - accuracy: 0.0987\n",
      "Epoch 2/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.9619 - accuracy: 0.2733\n",
      "Epoch 3/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.6432 - accuracy: 0.3343\n",
      "Epoch 4/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.4319 - accuracy: 0.3802\n",
      "Epoch 5/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.2556 - accuracy: 0.4149\n",
      "313/313 - 1s - loss: 2.6405 - accuracy: 0.3466\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[2.640547037124634, 0.3465999960899353]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=5)\n",
    "\n",
    "model.evaluate(x_test,  y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "remarkable-pasta",
   "metadata": {},
   "source": [
    "> 음... 비슷하다... 뭐지...? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "committed-spiritual",
   "metadata": {},
   "source": [
    "#### # output 개수 완전 틀리게 해보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "separated-luxury",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 4.2012 - accuracy: 0.0867\n",
      "Epoch 2/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 3.1275 - accuracy: 0.2364\n",
      "Epoch 3/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.7714 - accuracy: 0.3043\n",
      "Epoch 4/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.5556 - accuracy: 0.3521\n",
      "Epoch 5/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.3940 - accuracy: 0.3828\n",
      "Epoch 6/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.2434 - accuracy: 0.4160\n",
      "Epoch 7/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.1194 - accuracy: 0.4428\n",
      "Epoch 8/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.0202 - accuracy: 0.4693\n",
      "Epoch 9/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 1.9159 - accuracy: 0.4907\n",
      "Epoch 10/10\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 1.8342 - accuracy: 0.5061\n",
      "313/313 - 1s - loss: 2.7279 - accuracy: 0.3552\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[2.727922201156616, 0.35519999265670776]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_wrong = keras.Sequential()\n",
    "model_wrong.add(keras.layers.Conv2D(16, kernel_size=3, activation='relu'))\n",
    "model_wrong.add(keras.layers.MaxPool2D((2,2)))\n",
    "model_wrong.add(keras.layers.Conv2D(32, kernel_size=3, activation='relu'))\n",
    "model_wrong.add(keras.layers.MaxPool2D((2,2)))\n",
    "model_wrong.add(keras.layers.Flatten()) # 추가됨\n",
    "model_wrong.add(keras.layers.Dense(256, activation='relu'))\n",
    "model_wrong.add(keras.layers.Dense(1000, activation='softmax')) # 100개인데 1000개로\n",
    "\n",
    "model_wrong.compile(optimizer='adam',\n",
    "                    loss='sparse_categorical_crossentropy',\n",
    "                    metrics=['accuracy'])\n",
    "\n",
    "model_wrong.fit(x_train, y_train, epochs=10)\n",
    "\n",
    "model_wrong.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "powerful-muscle",
   "metadata": {},
   "source": [
    "> 일단 broadcasting 되어서 계산은 되는 것 같은데,  \n",
    "> 학습이 되고 accuracy가 올라간다는 것은 결국 나머지 900개의 노드에서 거의 0에 가까운 의미없는 값들이 출력되기 때문에 무시되는 셈이 아닌가 싶다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exempt-valuation",
   "metadata": {},
   "source": [
    "## Tensorflow2 API로 모델 작성 및 학습하기: CIFAR-100 (2) Functional API 활용\n",
    "- 데이터는 위에서 불러왔으므로 건너 뛴다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "smaller-miracle",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Functional API를 활용한 Model을 구성해주세요.\n",
    "\"\"\"\n",
    "Spec:\n",
    "0. (32X32X3) 차원으로 정의된 Input\n",
    "1. 16개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "2. pool_size가 2인 MaxPool 레이어\n",
    "3. 32개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "4. pool_size가 2인 MaxPool 레이어\n",
    "5. 256개의 아웃풋 노드를 가지고, activation function이 relu인 Fully-Connected Layer(Dense)\n",
    "6. 데이터셋의 클래스 개수에 맞는 아웃풋 노드를 가지고, activation function이 softmax인 Fully-Connected Layer(Dense)\n",
    "\"\"\"\n",
    "\n",
    "# 여기에 모델을 구성해주세요\n",
    "\n",
    "# --- Input\n",
    "inputs = keras.layers.Input(shape=(32, 32, 3))\n",
    "\n",
    "# --- Conv2D\n",
    "conv2d = keras.layers.Conv2D(16, kernel_size=3, activation='relu')\n",
    "x = conv2d(inputs)\n",
    "\n",
    "# --- MaxPool\n",
    "maxpool2d = keras.layers.MaxPool2D((2,2))\n",
    "x = maxpool2d(x)\n",
    "\n",
    "# --- Conv2D_1\n",
    "conv2d_1 = keras.layers.Conv2D(32, kernel_size=3, activation='relu')\n",
    "x = conv2d_1(x)\n",
    "\n",
    "# --- MaxPool\n",
    "maxpool2d_1 = keras.layers.MaxPool2D((2,2))\n",
    "x = maxpool2d_1(x)\n",
    "\n",
    "# --- Flatten\n",
    "flatten = keras.layers.Flatten()\n",
    "x = flatten(x)\n",
    "\n",
    "# --- Dense\n",
    "dense = keras.layers.Dense(256, activation='relu')\n",
    "x = dense(x)\n",
    "\n",
    "# --- Dense_1\n",
    "dense_1 = keras.layers.Dense(100, activation='softmax')\n",
    "outputs = dense_1(x)\n",
    "\n",
    "model = keras.Model(inputs=inputs, outputs=outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "composite-heather",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 4.0295 - accuracy: 0.0940\n",
      "Epoch 2/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.9991 - accuracy: 0.2663\n",
      "Epoch 3/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.6588 - accuracy: 0.3350\n",
      "Epoch 4/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.4387 - accuracy: 0.3787\n",
      "Epoch 5/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.2734 - accuracy: 0.4081\n",
      "313/313 - 1s - loss: 2.6318 - accuracy: 0.3426\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[2.6317834854125977, 0.3425999879837036]"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=5)\n",
    "\n",
    "model.evaluate(x_test,  y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "effective-shelter",
   "metadata": {},
   "source": [
    "## Tensorflow2 API로 모델 작성 및 학습하기: CIFAR-100 (3) Subclassing 활용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "metropolitan-printer",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Subclassing을 활용한 Model을 구성해주세요.\n",
    "\"\"\"\n",
    "Spec:\n",
    "0. keras.Model 을 상속받았으며, __init__()와 call() 메서드를 가진 모델 클래스\n",
    "1. 16개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "2. pool_size가 2인 MaxPool 레이어\n",
    "3. 32개의 채널을 가지고, 커널의 크기가 3, activation function이 relu인 Conv2D 레이어\n",
    "4. pool_size가 2인 MaxPool 레이어\n",
    "5. 256개의 아웃풋 노드를 가지고, activation function이 relu인 Fully-Connected Layer(Dense)\n",
    "6. 데이터셋의 클래스 개수에 맞는 아웃풋 노드를 가지고, activation function이 softmax인 Fully-Connected Layer(Dense)\n",
    "7. call의 입력값이 모델의 Input, call의 리턴값이 모델의 Output\n",
    "\"\"\"\n",
    "\n",
    "# 여기에 모델을 구성해주세요\n",
    "\n",
    "class MyModel(keras.Model):\n",
    "    def __init__(self):\n",
    "        super(MyModel, self).__init__()\n",
    "        self.conv2d = keras.layers.Conv2D(16, 3, activation='relu')\n",
    "        self.maxp = keras.layers.MaxPool2D((2,2))\n",
    "        self.conv2d_1 = keras.layers.Conv2D(32, 3, activation='relu')\n",
    "        self.maxp_1 = keras.layers.MaxPool2D(2,2)\n",
    "        self.flatten = keras.layers.Flatten()\n",
    "        self.dense = keras.layers.Dense(256, activation='relu')\n",
    "        self.dense_1 = keras.layers.Dense(100, activation='softmax')\n",
    "    \n",
    "    def call(self, inputs):\n",
    "        x = self.conv2d(inputs)\n",
    "        x = self.maxp(x)\n",
    "        x = self.conv2d_1(x)\n",
    "        x = self.maxp_1(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.dense(x)\n",
    "        outputs = self.dense_1(x)\n",
    "        \n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "accurate-highway",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 4.5576 - accuracy: 0.0215\n",
      "Epoch 2/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 4.0788 - accuracy: 0.0773\n",
      "Epoch 3/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 3.7740 - accuracy: 0.1307\n",
      "Epoch 4/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 3.5970 - accuracy: 0.1640\n",
      "Epoch 5/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 3.4008 - accuracy: 0.1981\n",
      "313/313 - 1s - loss: 3.3886 - accuracy: 0.2024\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[3.3886260986328125, 0.20239999890327454]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = MyModel()\n",
    "sgd = keras.optimizers.SGD(momentum=0.2, nesterov=True)\n",
    "model.compile(optimizer=sgd,\n",
    "             loss='sparse_categorical_crossentropy',\n",
    "             metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=5)\n",
    "model.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "configured-parish",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 4.2382 - accuracy: 0.0579\n",
      "Epoch 2/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 3.2162 - accuracy: 0.2184\n",
      "Epoch 3/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.7883 - accuracy: 0.3054\n",
      "Epoch 4/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.5138 - accuracy: 0.3575\n",
      "Epoch 5/5\n",
      "1563/1563 [==============================] - 5s 3ms/step - loss: 2.2861 - accuracy: 0.4091\n",
      "313/313 - 1s - loss: 2.7217 - accuracy: 0.3271\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[2.721714496612549, 0.32710000872612]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = MyModel()\n",
    "sgd = keras.optimizers.SGD(momentum=0.9, nesterov=True)\n",
    "model.compile(optimizer=sgd,\n",
    "             loss='sparse_categorical_crossentropy',\n",
    "             metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=5)\n",
    "model.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dramatic-routine",
   "metadata": {},
   "source": [
    "## GradientTape의 활용"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "understanding-stevens",
   "metadata": {},
   "source": [
    "### Automatic differentiation - GradientTape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "hungarian-proof",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "# 데이터 구성부분\n",
    "cifar100 = keras.datasets.cifar100\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = cifar100.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "print(len(x_train), len(x_test))\n",
    "\n",
    "# 모델 구성부분\n",
    "class CustomModel(keras.Model):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = keras.layers.Conv2D(16, 3, activation='relu')\n",
    "        self.maxpool1 = keras.layers.MaxPool2D((2,2))\n",
    "        self.conv2 = keras.layers.Conv2D(32, 3, activation='relu')\n",
    "        self.maxpool2 = keras.layers.MaxPool2D((2,2))\n",
    "        self.flatten = keras.layers.Flatten()\n",
    "        self.fc1 = keras.layers.Dense(256, activation='relu')\n",
    "        self.fc2 = keras.layers.Dense(100, activation='softmax')\n",
    "\n",
    "    def call(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.fc1(x)\n",
    "        x = self.fc2(x)\n",
    "\n",
    "        return x\n",
    "\n",
    "model = CustomModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "danish-uzbekistan",
   "metadata": {},
   "source": [
    "> 모델 설계에는 변함 없으나, compile과 fit 부분을 다르게 해볼 것"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "periodic-mozambique",
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_func = tf.keras.losses.SparseCategoricalCrossentropy()\n",
    "optimizer = tf.keras.optimizers.Adam()\n",
    "\n",
    "# tf.GradientTape()를 활용한 train_step\n",
    "def train_step(features, labels):\n",
    "    with tf.GradientTape() as tape:\n",
    "        predictions = model(features)\n",
    "        loss = loss_func(labels, predictions)\n",
    "        gradients = tape.gradient(loss, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "loaded-sodium",
   "metadata": {},
   "source": [
    "> `tape.gradient()`로 매 step마다 발생하는 gradient를 추출하고,  \n",
    "> `model.trainable_variables`를 통해 업데이트해야 할 파라미터를 주고 받고  \n",
    "> `optimizer.apply_gradients()`를 통해 업데이트한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "useful-martin",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0: last batch loss = 2.6041\n",
      "Epoch 1: last batch loss = 1.9978\n",
      "Epoch 2: last batch loss = 1.7482\n",
      "Epoch 3: last batch loss = 1.5230\n",
      "Epoch 4: last batch loss = 1.2091\n",
      "It took 94.82491636276245 seconds\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "def train_model(batch_size=32):\n",
    "    start = time.time()\n",
    "    for epoch in range(5):\n",
    "        x_batch = []\n",
    "        y_batch = []\n",
    "        for step, (x, y) in enumerate(zip(x_train, y_train)):\n",
    "            x_batch.append(x)\n",
    "            y_batch.append(y)\n",
    "            if step % batch_size == batch_size-1:\n",
    "                loss = train_step(np.array(x_batch, dtype=np.float32), np.array(y_batch, dtype=np.float32))\n",
    "                x_batch = []\n",
    "                y_batch = []\n",
    "        print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))\n",
    "    print(\"It took {} seconds\".format(time.time() - start))\n",
    "\n",
    "train_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "fundamental-kernel",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 6s 6s/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.319"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# evaluation\n",
    "prediction = model.predict(x_test, batch_size=x_test.shape[0], verbose=1)\n",
    "temp = sum(np.squeeze(y_test) == np.argmax(prediction, axis=1))\n",
    "temp/len(y_test)  # Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "champion-domestic",
   "metadata": {},
   "source": [
    "> gradients를 사용 안해도 되는 evaluation 단계는 그냥 `model.predict()` 메소드 사용"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "olive-generation",
   "metadata": {},
   "source": [
    "-------------------------------------------------------------\n",
    "### GradientTape 관련 연습"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "congressional-portal",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'my_model_11/conv2d_38/kernel:0' shape=(3, 3, 3, 16) dtype=float32, numpy=\n",
       " array([[[[ 1.56246386e-02, -6.42082334e-01,  2.73936719e-01,\n",
       "            3.96442622e-01, -1.07740767e-01, -5.81818461e-01,\n",
       "           -1.58907518e-01,  2.01875895e-01,  6.01437548e-03,\n",
       "            5.67011118e-01, -1.92512676e-01, -4.17991787e-01,\n",
       "            5.58841944e-01,  1.33760393e-01,  6.14034772e-01,\n",
       "           -2.35914677e-01],\n",
       "          [-3.31133723e-01,  1.18974410e-01, -2.76962012e-01,\n",
       "            4.48547453e-01,  5.04576527e-02, -1.18975437e+00,\n",
       "            6.84822440e-01, -1.00665577e-01,  1.31117716e-01,\n",
       "           -2.97998726e-01, -1.22719444e-01,  1.28715605e-01,\n",
       "           -3.91524702e-01,  8.70915013e-04,  6.09937429e-01,\n",
       "           -3.30402702e-01],\n",
       "          [ 2.48610839e-01, -5.01197018e-02,  2.56127179e-01,\n",
       "            3.45690966e-01, -5.91540597e-02, -8.35236430e-01,\n",
       "           -1.34304777e-01, -8.68157074e-02, -1.54668421e-01,\n",
       "           -1.13675542e-01,  9.92290825e-02, -3.57805461e-01,\n",
       "           -5.75351194e-02,  6.45677030e-01,  4.26436901e-01,\n",
       "           -8.05018067e-01]],\n",
       " \n",
       "         [[-1.23785652e-01, -3.96398425e-01,  3.78056437e-01,\n",
       "           -1.62492871e-01, -1.12557337e-01, -5.12586653e-01,\n",
       "           -7.58265778e-02, -3.98031697e-02, -2.03756869e-01,\n",
       "            3.71239096e-01, -1.06170915e-01, -3.81760329e-01,\n",
       "            7.89626598e-01, -3.22670281e-01,  1.00888252e+00,\n",
       "           -5.03578484e-02],\n",
       "          [-1.20860718e-01,  1.12540424e-01, -9.52902809e-03,\n",
       "           -3.06750864e-01,  1.80905998e-01, -1.13877177e+00,\n",
       "            7.36609876e-01, -4.31297533e-02,  2.48189755e-02,\n",
       "           -1.61815733e-02, -1.73144072e-01, -4.08671275e-02,\n",
       "           -4.60117579e-01, -5.78894205e-02,  1.11045456e+00,\n",
       "           -2.42992401e-01],\n",
       "          [ 6.64278641e-02, -5.73216472e-03,  3.10243666e-01,\n",
       "           -6.76335022e-02,  7.31501654e-02, -8.17662358e-01,\n",
       "           -1.76828012e-01, -1.87746599e-01,  1.92390680e-01,\n",
       "           -2.90464073e-01, -5.78610897e-02, -2.29406729e-02,\n",
       "           -2.83599347e-01,  5.84075630e-01,  8.96639645e-01,\n",
       "           -5.73357761e-01]],\n",
       " \n",
       "         [[ 3.55321348e-01, -2.46955231e-01,  3.62543523e-01,\n",
       "           -3.83783653e-02,  8.99761617e-02,  2.16428563e-01,\n",
       "            4.46065292e-02, -2.18807101e-01, -3.36208522e-01,\n",
       "            2.97556370e-01,  1.02371141e-01, -1.23425573e-01,\n",
       "            4.59444702e-01, -2.93548137e-01,  5.81979513e-01,\n",
       "           -1.93211347e-01],\n",
       "          [-3.47323939e-02, -1.83765646e-02, -1.60177842e-01,\n",
       "           -3.88183564e-01, -5.28198928e-02,  3.68427783e-02,\n",
       "            7.35235751e-01, -1.72343507e-01, -1.97315514e-01,\n",
       "           -6.31227195e-02, -2.11109087e-01,  2.24332377e-01,\n",
       "           -4.93152410e-01, -3.46458286e-01,  3.89178872e-01,\n",
       "           -8.05045143e-02],\n",
       "          [ 7.01162815e-02,  8.53049010e-02,  4.77854192e-01,\n",
       "           -3.93302292e-01,  3.99192460e-02,  1.55266076e-01,\n",
       "           -1.35816529e-01, -2.39306077e-01, -2.97249496e-01,\n",
       "           -1.26997203e-01, -8.44738185e-02,  2.62924641e-01,\n",
       "            1.88086793e-01,  3.33459616e-01,  5.69393873e-01,\n",
       "           -7.57251203e-01]]],\n",
       " \n",
       " \n",
       "        [[[-2.42308363e-01, -7.27272809e-01,  5.36894202e-01,\n",
       "            6.73298955e-01,  4.83072968e-03, -2.08082050e-01,\n",
       "           -5.12907028e-01,  1.22805022e-01,  2.28703365e-01,\n",
       "            4.70568150e-01,  1.15255853e-02, -6.42832160e-01,\n",
       "            7.21626043e-01, -1.64204970e-01,  8.93480405e-02,\n",
       "            2.95816779e-01],\n",
       "          [-3.16053957e-01, -2.13168830e-01, -3.02926123e-01,\n",
       "            9.22194600e-01, -1.79029275e-02, -8.06991458e-01,\n",
       "            5.13930917e-02, -1.61855578e-01,  2.30820388e-01,\n",
       "           -3.99273455e-01,  2.87459791e-02, -7.02519864e-02,\n",
       "           -3.84575635e-01, -2.48262256e-01,  1.31629601e-01,\n",
       "            1.83546484e-01],\n",
       "          [ 2.00320840e-01, -1.51232079e-01,  9.88149792e-02,\n",
       "            7.64514565e-01,  7.60781169e-02, -1.50068432e-01,\n",
       "           -8.79846036e-01, -1.10532992e-01,  1.24631889e-01,\n",
       "           -2.83003211e-01, -1.83538884e-01, -4.95767772e-01,\n",
       "           -3.57860595e-01,  5.16976416e-01,  1.17884807e-01,\n",
       "           -1.73828572e-01]],\n",
       " \n",
       "         [[-1.21066332e-01, -3.14049721e-01,  7.15327919e-01,\n",
       "           -1.42142907e-01, -3.07788961e-02,  8.81527364e-01,\n",
       "           -5.89752905e-02, -1.76933646e-01,  4.66678031e-02,\n",
       "            3.99423301e-01,  6.44816533e-02, -5.83119512e-01,\n",
       "            1.16018569e+00, -1.64912060e-01,  1.50793403e-01,\n",
       "            6.09325878e-02],\n",
       "          [-2.46290624e-01,  2.36442700e-01, -2.26657614e-01,\n",
       "           -2.37145796e-01,  4.35746871e-02, -3.69094983e-02,\n",
       "            6.96052790e-01,  9.39874128e-02,  2.51628179e-02,\n",
       "           -1.77890554e-01, -5.31317405e-02, -5.34693412e-02,\n",
       "           -7.39233196e-01, -1.98830873e-01,  2.03683764e-01,\n",
       "           -1.74797513e-02],\n",
       "          [-2.24550635e-01, -3.64947319e-02,  1.34308115e-01,\n",
       "           -1.44661888e-01,  2.44431585e-01,  6.52732432e-01,\n",
       "           -9.18463230e-01,  5.52068129e-02,  9.11586583e-02,\n",
       "           -3.70363295e-01, -8.69150907e-02, -3.16422701e-01,\n",
       "           -6.14068151e-01,  6.40759051e-01,  4.34856623e-01,\n",
       "           -5.54363847e-01]],\n",
       " \n",
       "         [[-1.46217465e-01, -1.71318457e-01,  5.80951035e-01,\n",
       "           -5.15908360e-01,  8.50586221e-02,  1.13941479e+00,\n",
       "            5.00862300e-01, -6.14181124e-02, -1.38723835e-01,\n",
       "            8.98071229e-02, -1.04799785e-01,  6.52125999e-02,\n",
       "            5.96692085e-01, -4.67858434e-01, -2.44991750e-01,\n",
       "            2.35845715e-01],\n",
       "          [-2.76848078e-01,  2.72795707e-01, -2.91954219e-01,\n",
       "           -8.15355003e-01,  6.07838258e-02,  6.01234019e-01,\n",
       "            1.28062594e+00, -7.25172758e-02, -3.00725669e-01,\n",
       "           -2.88073927e-01, -2.39789441e-01,  3.72493923e-01,\n",
       "           -5.04121661e-01, -1.85459927e-01, -4.43890333e-01,\n",
       "            1.90727159e-01],\n",
       "          [ 3.81089337e-02,  7.41713271e-02,  7.53425509e-02,\n",
       "           -6.30108953e-01,  3.76880690e-02,  8.00913572e-01,\n",
       "           -1.80314511e-01, -4.67390977e-02, -2.39625022e-01,\n",
       "           -2.43246898e-01,  6.62144274e-02,  5.01536094e-02,\n",
       "           -2.84300834e-01,  4.37721252e-01, -1.52757183e-01,\n",
       "           -3.56276006e-01]]],\n",
       " \n",
       " \n",
       "        [[[-3.42462748e-01, -5.10113716e-01,  5.09438634e-01,\n",
       "            5.03117859e-01,  3.29380073e-02, -9.22084004e-02,\n",
       "           -6.68611407e-01, -2.40828753e-01, -2.38455966e-01,\n",
       "            3.52147192e-01, -1.83305562e-01, -7.34045267e-01,\n",
       "            1.50555536e-01, -1.75698474e-01, -7.92084992e-01,\n",
       "            3.32987964e-01],\n",
       "          [-3.79138827e-01,  1.58297405e-01, -6.82291567e-01,\n",
       "            5.84498465e-01, -5.54600693e-02, -5.83324969e-01,\n",
       "           -1.46340609e-01, -1.09432247e-02, -1.43529132e-01,\n",
       "           -2.94351220e-01, -1.81427270e-01, -1.02373235e-01,\n",
       "           -2.43846834e-01, -2.41407946e-01, -7.68460274e-01,\n",
       "            4.19868916e-01],\n",
       "          [ 1.70752555e-01, -3.48242462e-01, -1.40418531e-02,\n",
       "            6.22205853e-01,  3.71903344e-03,  2.75670648e-01,\n",
       "           -8.81667316e-01,  1.80091131e-02, -1.40531212e-01,\n",
       "           -5.21282494e-01, -1.51802674e-01, -1.95361823e-01,\n",
       "           -2.21091975e-02,  1.38099506e-01, -5.51532745e-01,\n",
       "            2.70542562e-01]],\n",
       " \n",
       "         [[-2.30547011e-01,  6.87902868e-02,  6.29313529e-01,\n",
       "            9.34984833e-02, -1.93542894e-02,  6.94422662e-01,\n",
       "           -2.40867838e-01, -1.08280167e-01,  4.67504561e-01,\n",
       "            3.37798297e-01, -7.92519227e-02, -4.44923609e-01,\n",
       "            2.86500335e-01, -5.63054085e-01, -1.10230374e+00,\n",
       "            2.87184328e-01],\n",
       "          [-5.57666123e-01,  4.16889280e-01, -4.94148582e-01,\n",
       "            1.31440699e-01,  1.59162849e-01, -5.14244325e-02,\n",
       "            3.06284308e-01,  7.24266544e-02,  4.63434339e-01,\n",
       "           -4.34776813e-01, -1.13236038e-02,  1.69545129e-01,\n",
       "           -4.50654864e-01, -6.32913709e-02, -1.29102564e+00,\n",
       "            4.54860300e-01],\n",
       "          [-6.02334291e-02, -2.50284486e-02,  1.09843098e-01,\n",
       "            2.07322493e-01,  9.87427533e-02,  5.92406511e-01,\n",
       "           -9.13023770e-01,  4.33392562e-02,  5.34165800e-01,\n",
       "           -3.01481903e-01, -2.16416582e-01, -1.01964939e-02,\n",
       "           -4.32333946e-02,  5.33429444e-01, -9.28481519e-01,\n",
       "            9.20776948e-02]],\n",
       " \n",
       "         [[-2.36825705e-01,  4.46282625e-01,  4.36677009e-01,\n",
       "           -5.97547054e-01, -1.02934465e-01,  3.73798102e-01,\n",
       "            3.46656404e-02, -3.89472172e-02, -1.53952330e-01,\n",
       "           -3.45400535e-02, -2.01227248e-01,  1.48513451e-01,\n",
       "            1.03410557e-01, -3.35393071e-01, -3.27341914e-01,\n",
       "            3.50171715e-01],\n",
       "          [-5.14490306e-01,  7.11908519e-01, -1.77883625e-01,\n",
       "           -5.33924043e-01,  1.20068111e-01,  2.31532683e-03,\n",
       "            7.43604004e-01,  2.74026371e-03, -2.37980083e-01,\n",
       "           -2.94420600e-01, -1.80212650e-02,  5.87992549e-01,\n",
       "           -4.17186886e-01,  6.36466444e-02, -6.12090945e-01,\n",
       "            1.99181840e-01],\n",
       "          [-2.30996802e-01,  2.81659067e-01, -2.32514907e-02,\n",
       "           -5.33858538e-01, -1.72596797e-02,  2.92739958e-01,\n",
       "            6.66036606e-02, -7.76007771e-02, -2.73453951e-01,\n",
       "           -1.61727533e-01, -1.40414551e-01,  1.33176118e-01,\n",
       "            1.66761711e-01,  3.37600589e-01, -4.04081613e-01,\n",
       "            9.96043459e-02]]]], dtype=float32)>,\n",
       " <tf.Variable 'my_model_11/conv2d_38/bias:0' shape=(16,) dtype=float32, numpy=\n",
       " array([ 0.56910485, -0.10233934, -1.5814111 ,  0.05392105, -0.8873773 ,\n",
       "         0.09338906,  0.09586036, -0.02435095,  0.04688086, -0.29603288,\n",
       "        -0.00686779,  1.0601481 ,  0.24128169,  0.07310938,  0.00192663,\n",
       "         0.11753972], dtype=float32)>,\n",
       " <tf.Variable 'my_model_11/conv2d_39/kernel:0' shape=(3, 3, 16, 32) dtype=float32, numpy=\n",
       " array([[[[ 1.92423701e-01, -2.14550301e-01, -3.26433554e-02, ...,\n",
       "            1.90232947e-01, -2.67507493e-01,  5.49081899e-02],\n",
       "          [ 2.14946293e-03, -5.58381900e-03,  3.36158425e-02, ...,\n",
       "           -5.13701849e-02, -1.60038024e-01, -1.46736920e-01],\n",
       "          [ 2.38375545e-01,  2.73701221e-01, -2.61665076e-01, ...,\n",
       "           -5.29847264e-01,  1.81587234e-01, -3.09150964e-01],\n",
       "          ...,\n",
       "          [ 3.53755385e-01,  2.60437846e-01,  2.97213420e-02, ...,\n",
       "           -8.04604590e-03, -3.36965948e-01,  1.92925245e-01],\n",
       "          [ 1.17141739e-01,  3.39447111e-01, -1.16510972e-01, ...,\n",
       "           -4.54041988e-01,  4.97799963e-02, -1.64909780e-01],\n",
       "          [-7.61198923e-02, -7.67648593e-02, -2.85748214e-01, ...,\n",
       "            1.73490822e-01, -2.05412790e-01, -2.40574643e-01]],\n",
       " \n",
       "         [[ 1.06550142e-01, -9.97368321e-02, -1.08209148e-01, ...,\n",
       "            2.57034421e-01, -1.22342929e-01,  2.35923324e-02],\n",
       "          [-9.73831713e-02, -1.62487879e-01,  1.78247362e-01, ...,\n",
       "           -2.12998763e-01, -3.08687568e-01, -8.44930783e-02],\n",
       "          [ 5.42950928e-02,  1.11866467e-01, -3.73856872e-01, ...,\n",
       "            1.78497806e-02, -8.52812156e-02, -1.62189737e-01],\n",
       "          ...,\n",
       "          [-9.92744509e-03,  2.39477381e-01,  1.61483034e-01, ...,\n",
       "           -5.48900783e-01, -1.25964925e-01,  1.72751069e-01],\n",
       "          [-3.56098972e-02,  1.96560487e-01, -3.48943621e-01, ...,\n",
       "            5.30974627e-01,  1.22657023e-01,  1.54209197e-01],\n",
       "          [-1.06959350e-01, -2.48876717e-02, -1.21870354e-01, ...,\n",
       "            1.06930602e-02, -2.20581740e-01, -3.87532890e-01]],\n",
       " \n",
       "         [[ 1.53101042e-01,  9.89418402e-02,  5.70336655e-02, ...,\n",
       "           -4.60881405e-02,  1.27528965e-01, -3.27139646e-02],\n",
       "          [ 1.76992193e-02, -1.11608125e-01, -3.41748029e-01, ...,\n",
       "           -1.85201749e-01, -3.03072363e-01,  4.62151878e-02],\n",
       "          [ 3.84862833e-02, -3.66064161e-02, -3.27593595e-01, ...,\n",
       "            1.37849867e-01, -3.27692300e-01, -1.99934766e-01],\n",
       "          ...,\n",
       "          [ 2.59243160e-01,  1.61204427e-01,  4.85789105e-02, ...,\n",
       "            2.37901464e-01,  8.47605914e-02,  2.09555387e-01],\n",
       "          [-9.61644764e-05, -3.30986142e-01, -2.34415025e-01, ...,\n",
       "           -1.39076754e-01, -1.39407078e-02,  1.46404669e-01],\n",
       "          [ 5.55106550e-02, -1.56918451e-01, -1.96944594e-01, ...,\n",
       "           -2.36728519e-01, -2.11129859e-01, -5.19075930e-01]]],\n",
       " \n",
       " \n",
       "        [[[-1.29456827e-02,  7.00692609e-02, -8.51406604e-02, ...,\n",
       "            6.39390871e-02, -2.18096271e-01,  6.05113320e-02],\n",
       "          [-5.15727550e-02,  6.19620755e-02,  5.21354452e-02, ...,\n",
       "           -4.10627900e-03, -2.33693570e-01, -6.70252144e-02],\n",
       "          [-3.75560187e-02,  6.67616278e-02, -2.61249334e-01, ...,\n",
       "           -1.13493703e-01, -1.69056416e-01, -1.58928156e-01],\n",
       "          ...,\n",
       "          [-4.84124161e-02,  4.73163389e-02,  8.97312686e-02, ...,\n",
       "            5.20033896e-01, -4.22339886e-02,  8.56923908e-02],\n",
       "          [-8.27246755e-02,  3.88354987e-01, -1.50951818e-01, ...,\n",
       "           -3.12975675e-01,  2.55557805e-01, -3.60202938e-01],\n",
       "          [ 9.46728736e-02, -2.22612083e-01, -2.75282953e-02, ...,\n",
       "           -2.00314164e-01, -1.51756644e-01,  1.05173491e-01]],\n",
       " \n",
       "         [[ 9.16282237e-02,  7.46623380e-03, -4.19377871e-02, ...,\n",
       "            2.22168773e-01, -1.13378270e-02,  1.25954404e-01],\n",
       "          [-1.20199084e-01, -1.46326989e-01,  1.77109331e-01, ...,\n",
       "           -1.89013109e-01,  2.91181896e-02,  8.05806890e-02],\n",
       "          [-1.43396944e-01, -4.59934399e-02, -2.29375362e-01, ...,\n",
       "           -2.08302677e-01, -3.06594282e-01, -4.06794697e-02],\n",
       "          ...,\n",
       "          [-4.51934516e-01,  7.29354797e-03, -1.02436222e-01, ...,\n",
       "           -2.72308946e-01,  1.73163116e-02, -1.97752733e-02],\n",
       "          [-7.98156187e-02, -7.67804275e-04,  7.02122599e-03, ...,\n",
       "           -3.40130091e-01,  5.33123165e-02, -1.55285329e-01],\n",
       "          [ 2.40721673e-01, -1.72725052e-01, -1.20736435e-01, ...,\n",
       "            2.38260865e-01, -3.10496449e-01, -6.89610019e-02]],\n",
       " \n",
       "         [[ 1.71135664e-01, -2.22320016e-02,  8.77066627e-02, ...,\n",
       "            2.50914603e-01,  1.98627710e-01, -1.39369944e-03],\n",
       "          [ 3.42732444e-02, -2.08505452e-01, -2.18429804e-01, ...,\n",
       "           -3.91153932e-01,  1.21848285e-02, -5.07296957e-02],\n",
       "          [-2.01876998e-01,  2.81968027e-01, -2.19639838e-01, ...,\n",
       "            3.11050981e-01, -3.65625441e-01, -1.70973986e-01],\n",
       "          ...,\n",
       "          [-1.05249874e-01, -6.45545835e-04, -8.47528949e-02, ...,\n",
       "           -5.20582020e-01,  8.06060731e-02, -2.37815846e-02],\n",
       "          [-2.77997982e-02, -4.43852842e-01,  5.45291416e-03, ...,\n",
       "            2.87957013e-01, -2.57231332e-02,  7.64388815e-02],\n",
       "          [-4.22930997e-03,  2.45148484e-02, -3.32866758e-02, ...,\n",
       "            1.75641775e-02, -2.80825824e-01, -1.57640517e-01]]],\n",
       " \n",
       " \n",
       "        [[[ 4.53497563e-03,  2.12312713e-01, -5.09413369e-02, ...,\n",
       "            1.60848826e-01,  9.74554941e-02, -1.03807691e-02],\n",
       "          [ 3.95504199e-02,  7.37338960e-02, -1.17302850e-01, ...,\n",
       "            2.07689360e-01, -2.53571514e-02, -1.75185427e-01],\n",
       "          [-7.34209195e-02, -1.75332785e-01, -4.53206688e-01, ...,\n",
       "            2.22376734e-01, -2.74587661e-01, -7.05243088e-03],\n",
       "          ...,\n",
       "          [ 1.06705345e-01, -6.49357513e-02,  5.28859906e-03, ...,\n",
       "            6.09413743e-01,  1.79274693e-01, -2.79700041e-01],\n",
       "          [-9.13402736e-02,  1.24051936e-01, -6.17359057e-02, ...,\n",
       "            2.53277391e-01,  4.07797366e-01, -8.66338462e-02],\n",
       "          [-1.58073291e-01, -2.01051578e-01, -1.62004203e-01, ...,\n",
       "           -5.19326746e-01, -1.08289599e-01,  4.64291841e-01]],\n",
       " \n",
       "         [[-2.86425743e-02,  1.76288575e-01,  2.11339034e-02, ...,\n",
       "           -4.18833531e-02,  2.16098994e-01,  7.92490840e-02],\n",
       "          [-5.42181656e-02,  1.62865371e-01, -8.17971006e-02, ...,\n",
       "           -6.20934814e-02,  1.00880101e-01,  7.09183738e-02],\n",
       "          [-2.34009236e-01,  1.86088920e-01, -3.93427342e-01, ...,\n",
       "            8.74169916e-02, -5.32302320e-01, -3.80787477e-02],\n",
       "          ...,\n",
       "          [-2.49874935e-01,  1.07989930e-01,  4.32951725e-04, ...,\n",
       "            4.35328931e-01,  1.59168944e-01, -5.86745203e-01],\n",
       "          [-8.59518498e-02, -1.17593538e-02,  2.03142669e-02, ...,\n",
       "           -2.84372896e-01,  1.43279776e-01,  5.28550297e-02],\n",
       "          [ 4.42231484e-02, -9.54567865e-02, -1.65137216e-01, ...,\n",
       "            1.09886125e-01, -1.84797615e-01,  4.38289791e-01]],\n",
       " \n",
       "         [[ 2.76966363e-01,  7.81612471e-02, -6.51404113e-02, ...,\n",
       "            3.96650098e-02, -4.26433049e-02,  9.88452286e-02],\n",
       "          [ 1.05228074e-01,  2.60101501e-02, -1.17843352e-01, ...,\n",
       "           -6.27651960e-02,  2.80556351e-01, -1.53626531e-01],\n",
       "          [-5.26370993e-03,  1.77401409e-01, -4.00835395e-01, ...,\n",
       "           -2.46250536e-04, -1.06569409e-01, -1.43967465e-01],\n",
       "          ...,\n",
       "          [ 8.79352912e-02,  1.63492516e-01, -1.38960585e-01, ...,\n",
       "           -1.89878687e-01,  7.78920129e-02, -4.73879039e-01],\n",
       "          [-2.31545735e-02,  1.52663186e-01,  1.04664780e-01, ...,\n",
       "           -2.95899481e-01, -5.91674745e-01,  2.90356636e-01],\n",
       "          [-1.50512695e-01,  5.83798578e-03, -9.65668559e-02, ...,\n",
       "            2.11389646e-01, -5.37874214e-02,  1.21754192e-01]]]],\n",
       "       dtype=float32)>,\n",
       " <tf.Variable 'my_model_11/conv2d_39/bias:0' shape=(32,) dtype=float32, numpy=\n",
       " array([-0.168254  , -0.7980571 , -0.14285737,  0.12969868, -0.64854896,\n",
       "         0.5586203 , -0.9368539 ,  0.3136387 ,  1.3287005 , -0.15487061,\n",
       "        -0.6489443 ,  0.25432813, -1.1076782 , -0.3068435 , -0.5730089 ,\n",
       "        -0.76862055,  0.26741207, -0.32308403, -0.10645503,  0.58872074,\n",
       "        -0.06347223, -0.03096142,  0.04121513, -0.40449283, -0.04960252,\n",
       "        -0.2377945 ,  0.01517412,  0.00610766,  1.1769758 , -0.06650961,\n",
       "        -0.12079563, -0.00912976], dtype=float32)>,\n",
       " <tf.Variable 'my_model_11/dense_36/kernel:0' shape=(1152, 256) dtype=float32, numpy=\n",
       " array([[ 0.16568829, -0.13495822, -0.14087507, ..., -0.17293033,\n",
       "         -0.09941444,  0.00399691],\n",
       "        [-0.20137887, -0.0405804 , -0.1612296 , ..., -0.10637722,\n",
       "         -0.03548782,  0.14131936],\n",
       "        [-0.0860404 ,  0.11097732, -0.0945105 , ..., -0.11600018,\n",
       "          0.11977794, -0.05666319],\n",
       "        ...,\n",
       "        [ 0.00796608, -0.04988248,  0.01056121, ..., -0.1769474 ,\n",
       "          0.04854688, -0.14254634],\n",
       "        [-0.05005151, -0.06629809,  0.18981373, ...,  0.23970103,\n",
       "          0.0896008 ,  0.15122765],\n",
       "        [-0.0711858 , -0.01944438, -0.30242372, ...,  0.13389003,\n",
       "          0.07153423,  0.13853687]], dtype=float32)>,\n",
       " <tf.Variable 'my_model_11/dense_36/bias:0' shape=(256,) dtype=float32, numpy=\n",
       " array([ 0.24235466,  0.2952515 , -0.04796787, -0.08890828,  0.13667142,\n",
       "         0.25488952,  0.43180028, -0.25883678, -0.1393318 , -0.10725636,\n",
       "         0.39798856, -0.06098454,  0.26953346,  0.5385933 , -0.44375685,\n",
       "        -0.16057655, -0.04308696, -0.11405196, -0.14139153, -0.03656471,\n",
       "        -0.09984464, -0.08946136,  0.24530526, -0.09606063, -0.1322679 ,\n",
       "        -0.23808245,  0.20944373, -0.12363166, -0.31778654,  0.20876965,\n",
       "        -0.04916516,  0.03118645, -0.004993  , -0.06680187, -0.15509184,\n",
       "         0.23678374,  0.4391437 , -0.03281859,  0.01005218, -0.05785966,\n",
       "        -0.29712957,  0.04764744,  0.08148647, -0.07169758, -0.10860447,\n",
       "         0.03153949, -0.10770924, -0.01395796, -0.49739817, -0.40939876,\n",
       "         0.36655733,  0.21124147,  0.34814757, -0.07056335, -0.43977207,\n",
       "        -0.02925504,  0.08228628,  0.17847887, -0.01607569, -0.1626353 ,\n",
       "        -0.10946446,  0.02697202, -0.10953265, -0.03868067, -0.17086746,\n",
       "         0.03534521, -0.03626616, -0.04405564, -0.08865533,  0.17384319,\n",
       "         0.33300123,  0.08993671,  0.00179645, -0.02131632, -0.10501324,\n",
       "         0.04757755,  0.37799916,  0.09582434,  0.05792109, -0.13706869,\n",
       "         0.03037324,  0.43072328,  0.3238105 , -0.01104   , -0.06109275,\n",
       "        -0.07388194,  0.46299753, -0.02411274, -0.09135192,  0.27593362,\n",
       "         0.09866822, -0.16809192, -0.06706136, -0.40290377,  0.10378062,\n",
       "        -0.12164132,  0.15508069,  0.26818448,  0.04196982, -0.03255515,\n",
       "        -0.22149432, -0.1319614 , -0.45437178, -0.03274579, -0.15169597,\n",
       "        -0.11012233,  0.20058559, -0.08454743,  0.2517596 , -0.16789494,\n",
       "        -0.11633328,  0.1052869 ,  0.18357392, -0.14071873, -0.02537457,\n",
       "        -0.06687897, -0.07445078,  0.13689822, -0.16471428,  0.05727581,\n",
       "        -0.15886441,  0.09185291, -0.11885054, -0.08039785, -0.25170428,\n",
       "         0.5320529 ,  0.02043545,  0.04584684, -0.10888596, -0.08124221,\n",
       "         0.10014903, -0.08600902, -0.1036937 ,  0.33917192, -0.04814829,\n",
       "        -0.06672537, -0.09927049,  0.38202402, -0.29947197, -0.22473036,\n",
       "        -0.39119062, -0.16177458, -0.0112192 , -0.11354151,  0.03445577,\n",
       "         0.11654393, -0.04623572, -0.16272835, -0.11382527,  0.20292607,\n",
       "        -0.05071486,  0.13822636, -0.00637978, -0.0183422 ,  0.01579419,\n",
       "        -0.06316671,  0.27770275, -0.02343089,  0.09724094, -0.05765278,\n",
       "         0.2571769 ,  0.30848533,  0.08674128, -0.1786712 , -0.03324411,\n",
       "        -0.10059787, -0.09167175, -0.07789429, -0.15449922,  0.00714417,\n",
       "         0.6022505 , -0.03072871,  0.1443142 ,  0.09836893, -0.01505679,\n",
       "         0.21590601,  0.1465639 , -0.12906586,  0.22300011,  0.28246683,\n",
       "         0.02425781,  0.1926672 , -0.23520921,  0.34622487, -0.1332389 ,\n",
       "        -0.09690542, -0.00742675, -0.04651741, -0.01843935,  0.12518153,\n",
       "         0.24674532, -0.20329279, -0.00136904, -0.03986407, -0.2924921 ,\n",
       "         0.08604877, -0.13077879, -0.14175692, -0.04446891, -0.26966345,\n",
       "         0.18192005, -0.05521575,  0.02555881, -0.12528448, -0.20268142,\n",
       "        -0.05567075, -0.16571386, -0.04039868,  0.00848727, -0.08331817,\n",
       "         0.23434305,  0.11055528, -0.00850952, -0.05953522, -0.13838913,\n",
       "         0.01652332,  0.22010088,  0.09354903,  0.13939357,  0.14761144,\n",
       "        -0.05188628,  0.06745175, -0.08307455, -0.01801958, -0.05918142,\n",
       "         0.04577738,  0.07678546,  0.14580622,  0.10148383, -0.07878329,\n",
       "         0.3598247 , -0.1146336 , -0.1007519 , -0.0489885 ,  0.02000883,\n",
       "        -0.09453411, -0.2703188 ,  0.12610899,  0.19043769,  0.08693936,\n",
       "         0.13467339,  0.08951784,  0.02672005, -0.04704709, -0.18239963,\n",
       "         0.10610332,  0.20260096,  0.03947033,  0.24717781, -0.09998918,\n",
       "        -0.00438909,  0.3742636 ,  0.10321278, -0.0902221 , -0.09084942,\n",
       "        -0.13767338], dtype=float32)>,\n",
       " <tf.Variable 'my_model_11/dense_37/kernel:0' shape=(256, 100) dtype=float32, numpy=\n",
       " array([[ 0.03170383, -0.12860191, -0.06252094, ..., -0.26023152,\n",
       "          0.15726833, -0.33807108],\n",
       "        [-0.33302662, -0.2688529 , -0.19904943, ..., -0.07618555,\n",
       "         -0.4072146 ,  0.10492098],\n",
       "        [-0.24538305, -0.33073542, -0.04322355, ..., -0.07421663,\n",
       "         -0.16360308,  0.13886322],\n",
       "        ...,\n",
       "        [-0.01837303,  0.01176972, -0.00325429, ...,  0.00677959,\n",
       "         -0.12531267, -0.23670638],\n",
       "        [-0.2328478 , -0.06942023, -0.29163796, ..., -0.01293722,\n",
       "         -0.10516892,  0.15363157],\n",
       "        [-0.08848779, -0.38610226, -0.1973502 , ..., -0.8024932 ,\n",
       "         -0.2395402 ,  0.20475924]], dtype=float32)>,\n",
       " <tf.Variable 'my_model_11/dense_37/bias:0' shape=(100,) dtype=float32, numpy=\n",
       " array([-0.15574838, -0.06921016,  0.31068707,  0.16537862,  0.35311446,\n",
       "         0.16307244,  0.14795142, -0.13693078,  0.37465227, -0.18011765,\n",
       "        -0.04954363, -0.26519412, -0.0675621 , -0.03906521, -0.27751803,\n",
       "        -0.19961643, -0.03104587, -0.2758399 ,  0.14263232, -0.01863366,\n",
       "        -0.42183584,  0.14419584, -0.47579283,  0.26400435, -0.35158312,\n",
       "         0.06045587, -0.07508044,  0.32189208, -0.11680581, -0.10530201,\n",
       "         0.25176522,  0.16890664,  0.05503001,  0.21465191,  0.19487661,\n",
       "         0.01963011,  0.07505315,  0.08173823,  0.08809772, -0.40182215,\n",
       "        -0.346531  , -0.08787865,  0.1860175 ,  0.1429034 ,  0.1644544 ,\n",
       "         0.28725287, -0.05385001, -0.05921935, -0.20280628, -0.229536  ,\n",
       "         0.17791295, -0.2218011 , -0.28325114, -0.42150098,  0.11716691,\n",
       "         0.40071955, -0.11253374, -0.10722569, -0.20862055,  0.08165265,\n",
       "        -0.49604976, -0.20532878, -0.59462535,  0.26936024,  0.2315256 ,\n",
       "         0.18334892,  0.08269902, -0.01460798, -0.27155674, -0.25704253,\n",
       "        -0.22837628, -0.08091663,  0.11347406, -0.00480489,  0.28294164,\n",
       "        -0.51760626, -0.06657667,  0.23300548,  0.08081631,  0.03141642,\n",
       "         0.5654784 ,  0.23056634, -0.27833924, -0.47297233, -0.06950496,\n",
       "        -0.32198927, -0.201736  ,  0.00501115,  0.27323425,  0.17809246,\n",
       "         0.21040596, -0.03265874, -0.44807592,  0.09379514, -0.00861188,\n",
       "        -0.38156867,  0.0301103 ,  0.1629667 , -0.2121743 , -0.4371028 ],\n",
       "       dtype=float32)>]"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.trainable_variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "cognitive-longitude",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__class__',\n",
       " '__del__',\n",
       " '__delattr__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__enter__',\n",
       " '__eq__',\n",
       " '__exit__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__le__',\n",
       " '__lt__',\n",
       " '__module__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__setattr__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__',\n",
       " '__weakref__',\n",
       " '_created_eagerly',\n",
       " '_persistent',\n",
       " '_pop_tape',\n",
       " '_push_tape',\n",
       " '_recording',\n",
       " '_tape',\n",
       " '_tf_api_names',\n",
       " '_tf_api_names_v1',\n",
       " '_watch_accessed_variables',\n",
       " '_watched_variables',\n",
       " 'batch_jacobian',\n",
       " 'gradient',\n",
       " 'jacobian',\n",
       " 'reset',\n",
       " 'stop_recording',\n",
       " 'watch',\n",
       " 'watched_variables']"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(tf.GradientTape())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "above-surveillance",
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.GradientTape() as tape:\n",
    "    anyth = tape.reset()\n",
    "    b = 2\n",
    "anyth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "concrete-aquarium",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "distinct-temple",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NoneType"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(anyth)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "residential-metabolism",
   "metadata": {},
   "source": [
    "> - with 함수 안에서 할당한 변수가 밖에서도 계속 유지되는지(global scope로 되는지) 확인하고 싶었다.  \n",
    "> - 그렇게 되는 걸 확인할 수 있다.  \n",
    "> - 여기서 anyth는 Nonetype인데, 그래서인지 output이 없다."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}