3.search_space.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Search Space and Decorator\n",
    "\n",
    "This tutorial explains the supported search spaces and how to use them, including\n",
    "simple search spaces (Int, Real and Categorical) and nested search spaces\n",
    "(Categorical, List, Dict).\n",
    "AutoGluon also enables search spaces in user-defined objects using the decorator\n",
    "`ag.obj` and user-defined functions using the decorator `ag.func`.\n",
    "\n",
    "## Search Space\n",
    "\n",
    "### Simple Search Space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import autogluon as ag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Integer Space :class:`autogluon.space.Int`\n",
    "\n",
    "An integer will be chosen between lower and upper value during the\n",
    "searcher sampleing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Int: lower=0, upper=10\n"
     ]
    }
   ],
   "source": [
    "a = ag.space.Int(lower=0, upper=10)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get default value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change default value, which is the first configuration that a random searcher\n",
    ":class:`autogluon.searcher.RandomSearcher` will try:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "a = ag.space.Int(lower=0, upper=10, default=2)\n",
    "print(a.default)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pick a random value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.rand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Real Space :class:`autogluon.space.Real`\n",
    "\n",
    "An real number will be chosen between lower and upper value during the\n",
    "searcher sampleing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Real: lower=0.0001, upper=0.01\n"
     ]
    }
   ],
   "source": [
    "b = ag.space.Real(lower=1e-4, upper=1e-2)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Real space in log scale:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Real: lower=0.0001, upper=0.01\n"
     ]
    }
   ],
   "source": [
    "c = ag.space.Real(lower=1e-4, upper=1e-2, log=True)\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Categorical Space :class:`autogluon.space.Categorical`\n",
    "\n",
    "Categorical Space will chooce one choice from all the possible values during\n",
    "the searcher sampling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical['Monday', 'Tuesday', 'Wednesday']\n"
     ]
    }
   ],
   "source": [
    "d = ag.space.Categorical('Monday', 'Tuesday', 'Wednesday')\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nested Search Space\n",
    "\n",
    "#### Categorical Space :class:`autogluon.space.Categorical`\n",
    "\n",
    "Categorical Space can also be used as a nested search space.\n",
    "See example at NestedExampleObj_.\n",
    "\n",
    "\n",
    "#### List Space :class:`autogluon.space.List`\n",
    "\n",
    "List Space returns a list of sampled results.\n",
    "\n",
    "In this example, the first element of the list is a Int Space sampled\n",
    "from 0 to 3, and the second element is a Categorical Space sampled\n",
    "from the choices of `'alpha'` and `'beta'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List[Int: lower=0, upper=3, Categorical['alpha', 'beta']]\n"
     ]
    }
   ],
   "source": [
    "f = ag.space.List(\n",
    "        ag.space.Int(0, 3),\n",
    "        ag.space.Categorical('alpha', 'beta'),\n",
    "    )\n",
    "print(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get one example configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3, 'alpha']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.rand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dict Space :class:`autogluon.space.Dict`\n",
    "\n",
    "Dict Space returns a dict of sampled results.\n",
    "\n",
    "Similar to List Space, the resulting configuraton of Dict is\n",
    "a dict. In this example, the value of `'key1'` is sampled from\n",
    "a Categorical Space with the choices of `'alpha'` and `'beta'`,\n",
    "and the value of `'key2'` is sampled from an Int Space between\n",
    "0 and 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dict{'key1': Categorical['alpha', 'beta'], 'key2': Int: lower=0, upper=3, 'key3': 'constant'}\n"
     ]
    }
   ],
   "source": [
    "g = ag.space.Dict(\n",
    "        key1=ag.space.Categorical('alpha', 'beta'),\n",
    "        key2=ag.space.Int(0, 3),\n",
    "        key3='constant'\n",
    "    )\n",
    "print(g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get one example configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'key1': 'beta', 'key2': 0, 'key3': 'constant'}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g.rand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decorators for Searchbale Object and Customized Training Scripts\n",
    "\n",
    "In this section, we show how to insert search space into customized objects and\n",
    "training functions.\n",
    "\n",
    "#### Searchable space in customized class :func:`autogluon.obj`\n",
    "\n",
    "In AutoGluon searchable object can be returned by a user defined class with a decorator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoGluonObject -- MyObj\n"
     ]
    }
   ],
   "source": [
    "@ag.obj(\n",
    "    name=ag.space.Categorical('auto', 'gluon'),\n",
    "    static_value=10,\n",
    "    rank=ag.space.Int(2, 5),\n",
    ")\n",
    "class MyObj:\n",
    "    def __init__(self, name, rank, static_value):\n",
    "        self.name = name\n",
    "        self.rank = rank\n",
    "        self.static_value = static_value\n",
    "    def __repr__(self):\n",
    "        repr = 'MyObj -- name: {}, rank: {}, static_value: {}'.format(\n",
    "                self.name, self.rank, self.static_value)\n",
    "        return repr\n",
    "h = MyObj()\n",
    "print(h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get one example random object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MyObj -- name: gluon, rank: 3, static_value: 10"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h.rand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ".. _NestedExampleObj:\n",
    "\n",
    "We can also use it within a Nested Space such as :class:`autogluon.space.Categorical`.\n",
    "In this example, the resulting nested space will be sampled from"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical[Dict{'obj1': '1', 'obj2': Categorical['a', 'b']}, AutoGluonObject -- MyObj]\n"
     ]
    }
   ],
   "source": [
    "nested = ag.space.Categorical(\n",
    "        ag.space.Dict(\n",
    "                obj1='1',\n",
    "                obj2=ag.space.Categorical('a', 'b'),\n",
    "            ),\n",
    "        MyObj(),\n",
    "    )\n",
    "\n",
    "print(nested)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get an example output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyObj -- name: gluon, rank: 4, static_value: 10\n",
      "{'obj1': '1', 'obj2': 'a'}\n",
      "MyObj -- name: auto, rank: 4, static_value: 10\n",
      "{'obj1': '1', 'obj2': 'b'}\n",
      "{'obj1': '1', 'obj2': 'a'}\n"
     ]
    }
   ],
   "source": [
    "for _ in range(5):\n",
    "    result = nested.rand\n",
    "    assert (isinstance(result, dict) and result['obj2'] in ['a', 'b']) or hasattr(result, 'name')\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Searchable space in customized function :func:`autogluon.obj`\n",
    "\n",
    "We can also insert searchable space in a customized function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoGluonObject\n"
     ]
    }
   ],
   "source": [
    "@ag.func(\n",
    "    framework=ag.space.Categorical('mxnet', 'pytorch'),\n",
    ")\n",
    "def myfunc(framework):\n",
    "    return framework\n",
    "i = myfunc()\n",
    "print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also make them inside a nested space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Dict{'a': Real: lower=0, upper=10, 'obj1': AutoGluonObject -- MyObj, 'obj2': AutoGluonObject},)\n"
     ]
    }
   ],
   "source": [
    "j =ag.space.Dict(\n",
    "        a=ag.Real(0, 10),\n",
    "        obj1=MyObj(),\n",
    "        obj2=myfunc(),\n",
    "    ),\n",
    "print(j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Customized Train Script using :func:`autogluon.args`\n",
    "\n",
    "Train_func is where to put your model training script, which takes in various keyword `args` as its hyperparameters and reports the performance of the trained model using the provided `reporter`. Here, we show a dummy train_func that simply prints these objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "@ag.args(\n",
    "    a=ag.space.Int(1, 10),\n",
    "    b=ag.space.Real(1e-3, 1e-2),\n",
    "    c=ag.space.Real(1e-3, 1e-2, log=True),\n",
    "    d=ag.space.Categorical('a', 'b', 'c', 'd'),\n",
    "    e=ag.space.Bool(),\n",
    "    f=ag.space.List(\n",
    "            ag.space.Int(1, 2),\n",
    "            ag.space.Categorical(4, 5),\n",
    "        ),\n",
    "    g=ag.space.Dict(\n",
    "            a=ag.Real(0, 10),\n",
    "            obj=MyObj(),\n",
    "        ),\n",
    "    h=ag.space.Categorical('test', MyObj()),\n",
    "    i = myfunc(),\n",
    ")\n",
    "def train_fn(args, reporter):\n",
    "    print('args: {}'.format(args))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Searcher and Sample A Configuration\n",
    "\n",
    "#### Create a searcher and sample configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': 7, 'b': 0.005761854804661527, 'c': 0.009150367971074462, 'd.choice': 1, 'e': 0, 'f.0': 2, 'f.1.choice': 1, 'g.a': 0.9288773976854581, 'g.obj.name.choice': 0, 'g.obj.rank': 5, 'h.1.name.choice': 0, 'h.1.rank': 5, 'h.choice': 0, 'i.framework.choice': 0}\n"
     ]
    }
   ],
   "source": [
    "searcher = ag.searcher.RandomSearcher(train_fn.cs)\n",
    "config = searcher.get_config()\n",
    "print(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run one training job with the sampled configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "args: {'a': 7, 'b': 0.005761854804661527, 'c': 0.009150367971074462, 'd': 'b', 'e': 0, 'f': [2, 5], 'g': {'a': 0.9288773976854581, 'obj': MyObj -- name: auto, rank: 5, static_value: 10}, 'h': 'test', 'i': 'mxnet', '_default_config': {}}\n"
     ]
    }
   ],
   "source": [
    "train_fn(train_fn.args, config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exit AutoGluon:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "ag.done()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_autogluon",
   "language": "python",
   "name": "conda_autogluon"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}