CleanUp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Learn.co-CleanUp\" data-toc-modified-id=\"Learn.co-CleanUp-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Learn.co CleanUp</a></span><ul class=\"toc-item\"><li><span><a href=\"#1)-Fork-the-Repo-and-Rename-it.\" data-toc-modified-id=\"1)-Fork-the-Repo-and-Rename-it.-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>1) Fork the Repo and Rename it.</a></span></li><li><span><a href=\"#2)-Download-the-HTML-of-Learn.co\" data-toc-modified-id=\"2)-Download-the-HTML-of-Learn.co-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>2) Download the HTML of Learn.co</a></span></li><li><span><a href=\"#3)-Run-Some-Cells\" data-toc-modified-id=\"3)-Run-Some-Cells-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>3) Run Some Cells</a></span><ul class=\"toc-item\"><li><span><a href=\"#RegEx\" data-toc-modified-id=\"RegEx-1.3.1\"><span class=\"toc-item-num\">1.3.1&nbsp;&nbsp;</span>RegEx</a></span></li></ul></li><li><span><a href=\"#Personalize-the-URLs\" data-toc-modified-id=\"Personalize-the-URLs-1.4\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>Personalize the URLs</a></span></li><li><span><a href=\"#Section-Names\" data-toc-modified-id=\"Section-Names-1.5\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span>Section Names</a></span></li></ul></li><li><span><a href=\"#Interacting-with-the-OS\" data-toc-modified-id=\"Interacting-with-the-OS-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Interacting with the OS</a></span><ul class=\"toc-item\"><li><span><a href=\"#Helper-Function\" data-toc-modified-id=\"Helper-Function-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Helper Function</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learn.co CleanUp\n",
    "\n",
    "There is two reasons to do this:\n",
    "\n",
    "**A)** You want a nice `github` repo where you can quickly look through all your lessons. Who doesn't want that?\n",
    "\n",
    "**B)** Your local file structure is a mess. You saved some stuff over here. You saved some stuff over there. Then you went in a new direction with a naming scheme.\n",
    "\n",
    "`git` has its own idea of nested file structure. It uses _**submodules**_ to reference different repos. The following code automates the process of adding all the lessons you have done to a single `github` repo. \n",
    "\n",
    "A side effect of adding a `submodule` to a repo is that `git` clones that file to your local machine again, creating a set of redundant files. Once you are done with this process, the choice is yours of what files you wish to keep locally. If you are inclined to keep the the organizational structure introduced here, feel free. Otherwise, just delete the recently cloned files.\n",
    "\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Fork the Repo and Rename it.\n",
    "\n",
    "The first step fork this repo and renamed it. It will act as a table of contents for all of the Learn.co lessons.\n",
    "\n",
    "\n",
    "<img src = imgs/fork.png style=\"height: 500px; width:700px; resize:both\">\n",
    "\n",
    "<br>\n",
    "\n",
    "Now title your repo as you wish.\n",
    "\n",
    "<img src = imgs/rename.png style=\"height:500px\">\n",
    "\n",
    "<br>\n",
    "\n",
    "Now clone this file to your machine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Download the HTML of Learn.co \n",
    "\n",
    "When logged in from main landing page of Learn.co, download the HTML of this page. Save this file in the repo we created during Step 1.\n",
    "\n",
    "<img src=imgs/html_save.png>\n",
    "\n",
    "<br>\n",
    "\n",
    "Congratulations, the hard work is done."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Run Some Cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:47:40.815256Z",
     "start_time": "2019-07-04T20:47:40.715080Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import time\n",
    "import webbrowser\n",
    "import subprocess, shlex\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:47:41.632633Z",
     "start_time": "2019-07-04T20:47:41.570232Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('Learn - Data Science Career v1.1.html','r') as f:  #Make sure this file matches the name of HTML file you just saved.\n",
    "    html = f.read()\n",
    "soup = BeautifulSoup(html, \"html.parser\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:50:47.319902Z",
     "start_time": "2019-07-04T20:50:47.308148Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chunk = str(soup('script',{'type' : 'text/javascript'}))\n",
    "#chunk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The information we want is in a javascript block. Uncomment above to see what it is.  \n",
    "Instead of bringing in any more libraries, we will tackle this problem with our `regex` knowhow. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RegEx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:50:50.568874Z",
     "start_time": "2019-07-04T20:50:50.564380Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pattern = re.compile(r\"learn-co-curriculum/d.*?(?=\\\")\") #make a pattern object\n",
    "repo_names = pattern.findall(chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the cell below to what we collected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:50:55.891674Z",
     "start_time": "2019-07-04T20:50:55.881639Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['learn-co-curriculum/dsc-0-00-01',\n",
       " 'learn-co-curriculum/dsc-0-00-04-blogging',\n",
       " 'learn-co-curriculum/dsc-1-01-02-introduction-summary',\n",
       " 'learn-co-curriculum/dsc-1-01-03-problems-ds-can-solve',\n",
       " 'learn-co-curriculum/dsc-1-01-04-the-data-science-process',\n",
       " 'learn-co-curriculum/dsc-1-01-05-setting-up-environment',\n",
       " 'learn-co-curriculum/dsc-1-01-06-working-with-lessons-on-learn',\n",
       " 'learn-co-curriculum/dsc-1-01-07-working-with-lessons-on-learn-lab',\n",
       " 'learn-co-curriculum/dsc-1-01-08-your-first-data-science-codealong',\n",
       " 'learn-co-curriculum/dsc-1-01-09-variable-assignment']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repo_names[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Personalize the URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:52:55.442974Z",
     "start_time": "2019-07-04T20:52:55.439889Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cohort = 'online-ds-sp-000' #Enter your cohort as a string\n",
    "github = 'https://github.com/josephbrockw/' #Enter your github URL. Be sure to include a trailing backslash"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:52:58.104324Z",
     "start_time": "2019-07-04T20:52:58.087843Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mod_full_urls = {'mod_1': [],\n",
    "                 'mod_2': [],\n",
    "                 'mod_3': [],\n",
    "                 'mod_4': [],}\n",
    "\n",
    "for name in repo_names:\n",
    "    name = name.lstrip('learn-co-curriculum/')\n",
    "    \n",
    "    if name.startswith('dsc-1-') or name.startswith('dsc-00-') or name.startswith('dsc-01-'):\n",
    "        mod_full_urls['mod_1'].append(github + name + '-' + cohort)\n",
    "    \n",
    "    elif name.startswith('dsc-2-'):\n",
    "        mod_full_urls['mod_2'].append(github + name + '-' + cohort)\n",
    "        \n",
    "    elif name.startswith('dsc-3-'):\n",
    "        mod_full_urls['mod_3'].append(github + name + '-' + cohort)\n",
    "        \n",
    "    elif name.startswith('dsc-4-') or name.startswith(f'dsc-04-'):\n",
    "        mod_full_urls['mod_4'].append(github + name + '-' + cohort)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section Names\n",
    "\n",
    "I provided a list of section names, keep in mind this is for the V1.1 circ. If you go to scrapping the new list, be warned of inconsistencies in naming schemes. Some cleaning is required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:53:34.223688Z",
     "start_time": "2019-07-04T20:53:34.208464Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "section_list = ['01-getting-started-with-data-science',\n",
    " '02-importing-and-statistical-analysis-of-data',\n",
    " '03-working-with-pandas',\n",
    " '04-data-cleaning-in-pandas',\n",
    " '05-sql-and-relational-databases',\n",
    " '06-object-oriented-programming',\n",
    " '07-oop-continued',\n",
    " '08-numpy-and-foundations-of-probability-and-combinatorics',\n",
    " '09-statistical-distributions',\n",
    " '10-introduction-to-linear-regression',\n",
    " '11-multiple-regression-and-model-validation',\n",
    " '12-a-complete-data-science-project-using-multiple-regression',\n",
    " '13-linear-algebra',\n",
    " '14-calculus-cost-function-and-gradient-descent',\n",
    " '15-an-introduction-to-orms',\n",
    " '16-working-with-json-and-xml',\n",
    " '17-accessing-data-through-apis',\n",
    " '18-html-css-and-web-scraping',\n",
    " '19-distributions-and-sampling',\n",
    " '20-hypothesis-and-ab-testing',\n",
    " '21-combinatorics-continued-and-maximum-likelihood-estimation',\n",
    " '22-bayesian-classification',\n",
    " '23-resampling-and-monte-carlo-simulation',\n",
    " '24-extensions-to-linear-models',\n",
    " '25-time-series-visualization-and-testing-for-trends',\n",
    " '26-time-series-modeling',\n",
    " '27-distance-metrics-and-k-nearest-neighbors',\n",
    " '28-graph-theory',\n",
    " '29-introduction-to-logistic-regression',\n",
    " '30-logistic-regression-in-depth-mle-and-gradient-descent',\n",
    " '31-decision-trees',\n",
    " '32-ensemble-methods',\n",
    " '33-support-vector-machines',\n",
    " '34-dimensionality-reduction-with-pca',\n",
    " '35-clustering',\n",
    " '36-building-a-machine-learning-pipeline',\n",
    " '37-foundations-of-natural-language-processing-nlp',\n",
    " '38-big-data-in-pyspark',\n",
    " '39-developing-a-recommendation-system-in-pyspark',\n",
    " '40-introduction-to-deep-learning',\n",
    " '41-multi-layer-perceptrons',\n",
    " '42-regularization-and-optimization',\n",
    " '43-introduction-to-convolutional-neural-networks',\n",
    " '44-convolutional-neural-networks-continued',\n",
    " '45-deep-nlp-word-embeddings',\n",
    " '46-deep-nlp-sequence-models']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interacting with the OS\n",
    "\n",
    "___\n",
    "___\n",
    "\n",
    "Below are two functions. The first function takes a Learn URL, identifies the section is belongs to, and returns the full section name.  \n",
    "\n",
    "The second function is the workhorse of this notebook. It takes in two arguments: `mod_number` as an integer and `url_dict` which is the dictionary we created earlier. The function then creates the proper directories for both the module and section, then adds the lessons as submodules. Finally it will return a list of URLs of lessons that were not cloned.  \n",
    "\n",
    "This process works only work with one module at a time and takes a few minutes to run. An output will display for each of the URLs being added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:53:55.467331Z",
     "start_time": "2019-07-04T20:53:55.451534Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def section_name(url):   #Takes in the URL and returns the section as a string\n",
    "    \n",
    "    name = None          #Resets the name variable\n",
    "    \n",
    "    to_remove = len(github) + 4     #Github names are variable. The 4 is removing \"dsc-\" \n",
    "    to_check = url[to_remove:]\n",
    "    \n",
    "    if to_check.startswith('0'):    #Checks for the leading zero and removes it, if that is the case\n",
    "        to_check = to_check[1:]\n",
    "    \n",
    "    section_num = to_check[2:4]     #Isolates the section number as a string\n",
    "    \n",
    "    for section in section_list:    #Loops through section list\n",
    "        \n",
    "        if section.startswith(section_num) == True:\n",
    "            name = section\n",
    "            break\n",
    "        \n",
    "    if name == None:\n",
    "        name = 'Project'\n",
    "    \n",
    "    return name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T20:53:56.819893Z",
     "start_time": "2019-07-04T20:53:56.747641Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_submodule(mod_number, url_dict):\n",
    "    \n",
    "    valid = [1,2,3,4]\n",
    "    if mod_number not in valid:\n",
    "        raise ValueError(f\"results: mod_number must be either {valid}.\")\n",
    "        \n",
    "    to_fork = []\n",
    "    mod_name = 'Module_' + str(mod_number)                          \n",
    "    mod_dict = mod_full_urls[f'mod_' + str(mod_number)]             \n",
    "    \n",
    "    subprocess.run(f'mkdir {mod_name}', shell=True)               #Makes the module directory\n",
    "    time.sleep(1)                                                 #Don't use (shell=True) lightly #https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess\n",
    "\n",
    "\n",
    "    \n",
    "    for url in mod_dict:\n",
    "        name = section_name(url)\n",
    "        subprocess.run(f'mkdir {name}', shell=True, cwd = f'{mod_name}')\n",
    "        time.sleep(1)\n",
    "        \n",
    "        \n",
    "        command = f'git submodule add {url}'                        #Iterate over the URL dictionary we created before and adds them as submodules.\n",
    "        kwargs = {}\n",
    "        kwargs['stdout'] = subprocess.PIPE\n",
    "        kwargs['stderr'] = subprocess.PIPE\n",
    "        proc = subprocess.Popen(shlex.split(command), **kwargs, cwd = f'{mod_name}/{name}')\n",
    "        (stdout_str, stderr_str) = proc.communicate()\n",
    "        return_code = proc.wait()\n",
    "        #print (stdout_str)\n",
    "        #print (stderr_str)\n",
    "    \n",
    "    \n",
    "        to_check = stderr_str.decode('utf-8')                     #Changing the terminal output from btye to str\n",
    "        print(to_check)                                           #Prints status updates --optional\n",
    "        pattern = 'fatal'                                         #Making a pattern to loop through and find all the non exsistant URLs\n",
    "        if to_check.find(pattern) > 0:\n",
    "            to_fork.append(url)\n",
    "                \n",
    "    commands = [\"git commit -m 'adding a submodule'\", \"git push\"]    #Pushing all the changes\n",
    "    for command in commands:\n",
    "        subprocess.Popen(shlex.split(command))\n",
    "    \n",
    "    print('Go create the following')\n",
    "    print(to_fork)\n",
    "    return(to_fork)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T23:01:59.902876Z",
     "start_time": "2019-07-04T22:56:02.318419Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-01-introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-02-introduction-to-nlp-with-nltk-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-03-introduction-to-regular-expressions-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-04-regular-expressions-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-05-feature-engineering-for-text-data-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-06-corpus-statistics-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-07-context-free-grammars-and-pos-tagging-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-08-context-free-grammars-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-09-text-classification-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-10-text-classification-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/37-foundations-of-natural-language-processing-nlp/dsc-4-37-11-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-01-introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-02-intrduction-to-big-data-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-03-parallel-and-distributed-computing-with-mapreduce-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-04-big-data-analytics-apache-spark-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-05-installing-configuring-spark-with-docker-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-06-understanding-sparkcontext-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-07-resilient-distributed-datasets-rdds-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-08-rdd-transformations-and-actions-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-09-rdd-transformations-and-actions-ii-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/38-big-data-in-pyspark/dsc-4-38-12-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/39-developing-a-recommendation-system-in-pyspark/dsc-4-39-01-introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/39-developing-a-recommendation-system-in-pyspark/dsc-4-39-02-intro-to-recommendation-systems-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/39-developing-a-recommendation-system-in-pyspark/dsc-4-39-03-collaborative-filtering-singular-value-decomposition-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/39-developing-a-recommendation-system-in-pyspark/dsc-4-39-04-singular-value-decomposition-numpy-scipy-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/39-developing-a-recommendation-system-in-pyspark/dsc-4-39-05-matrix-factorization-als-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/39-developing-a-recommendation-system-in-pyspark/dsc-4-39-06-building-recommendation-system-als-pyspark-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/39-developing-a-recommendation-system-in-pyspark/dsc-4-39-07-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/40-introduction-to-deep-learning/dsc-4-40-01-introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/40-introduction-to-deep-learning/dsc-04-40-02-introduction-to-neural-networks-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/40-introduction-to-deep-learning/dsc-04-40-03-introduction-to-neural-networks-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/40-introduction-to-deep-learning/dsc-04-40-04-deeper-neural-networks-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/40-introduction-to-deep-learning/dsc-04-40-05-deeper-neural-networks-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/40-introduction-to-deep-learning/dsc-4-40-06-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/41-multi-layer-perceptrons/dsc-4-41-01-Introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/41-multi-layer-perceptrons/dsc-04-41-02-deep-networks-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/41-multi-layer-perceptrons/dsc-04-41-03-deep-networks-lab-building-an-image-classifier-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/41-multi-layer-perceptrons/dsc-04-41-04-introduction-to-keras-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/41-multi-layer-perceptrons/dsc-04-41-05-keras-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/41-multi-layer-perceptrons/dsc-4-41-06-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/42-regularization-and-optimization/dsc-4-42-01-Introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/42-regularization-and-optimization/dsc-04-42-02-tuning-neural-networks-with-regularization-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/42-regularization-and-optimization/dsc-04-42-03-tuning-neural-networks-with-regularization-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/42-regularization-and-optimization/dsc-04-42-04-normalization-and-parameter-tuning-of-neural-networks-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/42-regularization-and-optimization/dsc-04-42-05-normalization-and-tuning-neural-networks-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/42-regularization-and-optimization/dsc-04-42-06-tuning-and-optimizing-neural-networks-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/42-regularization-and-optimization/dsc-4-42-07-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/43-introduction-to-convolutional-neural-networks/dsc-4-43-01-introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/43-introduction-to-convolutional-neural-networks/dsc-04-43-02-convolutional-neural-networks-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/43-introduction-to-convolutional-neural-networks/dsc-04-43-03-convolutional-neural-networks-code-along-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/43-introduction-to-convolutional-neural-networks/dsc-04-43-04-building-a-cnn-from-scratch-online-ds-sp-000'...\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/43-introduction-to-convolutional-neural-networks/dsc-04-43-05-visualizing-intermediate-activations-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/43-introduction-to-convolutional-neural-networks/dsc-04-43-06-visualizing-intermediate-activations-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/43-introduction-to-convolutional-neural-networks/dsc-4-43-07-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/44-convolutional-neural-networks-continued/dsc-4-44-01-introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/44-convolutional-neural-networks-continued/dsc-04-44-02-using-pretrained-networks-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/44-convolutional-neural-networks-continued/dsc-04-44-03-using-pretrained-networks-codealong-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/44-convolutional-neural-networks-continued/dsc-04-44-04-image-classifcation-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/44-convolutional-neural-networks-continued/dsc-4-44-05-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/45-deep-nlp-word-embeddings/dsc-4-45-01-introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/45-deep-nlp-word-embeddings/dsc-04-45-02-word-embeddings-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/45-deep-nlp-word-embeddings/dsc-04-45-03-using-word2vec-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/45-deep-nlp-word-embeddings/dsc-04-45-04-generating-word-embeddings-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/45-deep-nlp-word-embeddings/dsc-04-45-05-classification-with-word-embeddings-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/45-deep-nlp-word-embeddings/dsc-04-45-06-classification-with-word-embeddings-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/45-deep-nlp-word-embeddings/dsc-4-45-07-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/46-deep-nlp-sequence-models/dsc-4-46-01-introduction-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/46-deep-nlp-sequence-models/dsc-04-46-02-sequence-model-use-cases-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/46-deep-nlp-sequence-models/dsc-04-46-03-understanding-recurrent-neural-networks-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/46-deep-nlp-sequence-models/dsc-04-46-04-LSTMs-and-GRUs-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/46-deep-nlp-sequence-models/dsc-04-46-05-LSTMs-and-GRUs-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/46-deep-nlp-sequence-models/dsc-04-46-06-bidirectional-sequence-models-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/46-deep-nlp-sequence-models/dsc-04-46-07-bidirectional-sequence-models-lab-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/46-deep-nlp-sequence-models/dsc-4-46-08-section-recap-online-ds-sp-000'...\n",
      "\n",
      "Cloning into '/Users/joe/Projects/flatiron_lessons/Module_4/Project/dsc-4-final-project-online-ds-sp-000'...\n",
      "\n",
      "Go create the following\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "to_fork = add_submodule(4, mod_full_urls)   #Input the module number and the url dictionary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper Function\n",
    "After running the above cell, there may be some Learn lessons that you didn't fork: Section Recaps, Introductions, etc.  \n",
    "\n",
    "I have included a helper function to open all the URLS that need to be cloned before the above needs to be run again. Once they have been forked, just rerun the above cell to add them as `submodules`.\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T23:02:13.210644Z",
     "start_time": "2019-07-04T23:02:13.207669Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#to_fork"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T23:02:13.516691Z",
     "start_time": "2019-07-04T23:02:13.512196Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def let_there_be_tabs(url_list):\n",
    "    learn = 'https://github.com/learn-co-students/'\n",
    "    for url in url_list:\n",
    "#         print(url.replace(github, learn))\n",
    "        webbrowser.open(url.replace(github, learn))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment below to run the function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-04T23:02:14.450538Z",
     "start_time": "2019-07-04T23:02:14.447380Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "let_there_be_tabs(to_fork)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}