0_OpenAIgymInterface.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "# In google collab, uncomment this:\n",
    "# !wget https://bit.ly/2FMJP5K -O setup.py && bash setup.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " # OpenAI Gym #\n",
    "\n",
    " We're gonna spend several next weeks learning algorithms that solve decision processes. We are then in need of some interesting decision problems to test our algorithms.\n",
    "\n",
    "That's where OpenAI gym comes into play. It's a python library that wraps many classical decision problems including robot control, videogames and board games.\n",
    "\n",
    "So here's how it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observation space: Box(2,)\nAction space: Discrete(3)\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "from gym import wrappers\n",
    "from time import time\n",
    "\n",
    "env = gym.make(\"MountainCar-v0\")\n",
    "env = wrappers.Monitor(env, './videos/' + str(time()))\n",
    "\n",
    "# plt.imshow(env.render('rgb_array')) \n",
    "# plt.show()\n",
    "\n",
    "print(\"Observation space:\", env.observation_space)\n",
    "print(\"Action space:\", env.action_space)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gym interface ##\n",
    "\n",
    "The three main methods of an environment are\n",
    "\n",
    "* **reset()** - reset envronment to initial state, retrun firs observation\n",
    "* **render()** - show current environment state (a more colorful version :) )\n",
    "* **step(a)** commit action **a** and return (new observation, reward, is done, info)\n",
    "    * *new observation* - an observation right after commiting the action **a**\n",
    "    * *reward* - a number representing your reward for commiting action **a**\n",
    "    * *is done* - True if the MDP has just finished, False if still in progress\n",
    "    * *info* - some auxilary stuff about what just happened. Ignore it for now.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Play with it\n",
    "\n",
    "Below is the code that drives the car to the right.\n",
    "\n",
    "However, it doesn't reach the flag at the far right due to gravity.\n",
    "\n",
    "**Your task** is to fix it. Find a strategy that reaches the flag.\n",
    "\n",
    "You're not required to build any sophisticated algorithms for now, feel free to hard-code :)\n",
    "\n",
    "Hint: your action at each step should depend either on t or on s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# def test(args, env, agent):\n",
    "#     if args.record:\n",
    "#         if 'env' in vars(args):\n",
    "#             env = wrappers.Monitor(env, './videos/' + args.env + str(time()) + '/')\n",
    "#         else:\n",
    "#             env = wrappers.Monitor(env, './videos/' + str(time()) + '/')\n",
    "#     test_rewards = []\n",
    "#     test_start = time()\n",
    "#     test_steps = 0\n",
    "#     for iteration in range(1, 1 + args.n_test_iter):\n",
    "#         state = env.reset()\n",
    "#         iter_rewards = 0.0\n",
    "#         done = False\n",
    "#         while not done:\n",
    "#             test_steps += 1\n",
    "#             action, _ = agent.forward(state)\n",
    "#             state, reward, done, _ = env.step(action)\n",
    "#             iter_rewards += reward\n",
    "#         test_rewards.append(iter_rewards)\n",
    "#     print_stats('Test', test_rewards, args.n_test_iter,\n",
    "#                 time() - test_start, test_steps, 0, agent)\n",
    "#     return test_rewards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initial observation code: [-0.42701333  0.        ]\ntaking action 2 (right)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new observation code: [-4.26727625e-01  2.85703169e-04]\nreward: -1.0\nis game over?: False\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- reached peak\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- reached peak\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "# create env manually to set time limit. Please don't change this.\n",
    "TIME_LIMIT = 250\n",
    "env = gym.wrappers.TimeLimit(env, max_episode_steps=TIME_LIMIT + 1)\n",
    "\n",
    "obs0 = env.reset()\n",
    "print(\"initial observation code:\", obs0)\n",
    "# Note: in MountainCar, observation is just two numbers: car position and velocity\n",
    "\n",
    "print(\"taking action 2 (right)\")\n",
    "new_obs, reward, is_done, _ = env.step(2)\n",
    "# Note: as you can see, the car has moved to the right slightly (around 0.0005)\n",
    "\n",
    "\n",
    "print(\"new observation code:\", new_obs)\n",
    "print(\"reward:\", reward)\n",
    "print(\"is game over?:\", is_done)\n",
    "\n",
    "actions = {'left': 0, 'stop': 1, 'right': 2}\n",
    "\n",
    "# prepare \"display\"\n",
    "#% matplotlib inline\n",
    "from IPython.display import clear_output\n",
    "peak_cnt = 0\n",
    "prev_vel = obs0[1]\n",
    "prev_coord = obs0[0]\n",
    "for t in range(TIME_LIMIT):\n",
    "\n",
    "    # change the line below to reach the flag\n",
    "    s, r, done, _ = env.step(actions['right'])\n",
    "  #  print(s)\n",
    "    if (s[1] > 0 or s[0]< -1):\n",
    "        s, r, done, _ = env.step(actions['right'])\n",
    "    else:\n",
    "        s, r, done, _ = env.step(actions['left'])\n",
    "        if prev_vel*s[1] < 0:\n",
    "            peak_cnt += 1\n",
    "            print(\"--- reached peak\")\n",
    "    prev_vel = s[1]\n",
    "    # draw game image on display\n",
    "    # clear_output(True)\n",
    "    # plt.imshow(env.render('rgb_array'))\n",
    "\n",
    "    if done:\n",
    "        print(\"Well done!\")\n",
    "        break\n",
    "else:\n",
    "    print(\"Time limit exceeded. Try again.\");\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You solved it!\n"
     ]
    }
   ],
   "source": [
    "assert s[0] > 0.47\n",
    "print(\"You solved it!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}