Chapter04_CrawlingModels.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getPage(url):\n",
    "    \"\"\"\n",
    "    Utilty function used to get a Beautiful Soup object from a given URL\n",
    "    \"\"\"\n",
    "\n",
    "    session = requests.Session()\n",
    "    headers = {\"User-Agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36\",\n",
    "               \"Accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"}\n",
    "    try:\n",
    "        req = session.get(url, headers=headers)\n",
    "    except requests.exceptions.RequestException:\n",
    "        return None\n",
    "    bs = BeautifulSoup(req.text, \"html.parser\")\n",
    "    return bs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with different website layouts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: Delivering inclusive urban access: 3 uncomfortable truths\n",
      "URL: https://www.brookings.edu/blog/future-development/2018/01/26/delivering-inclusive-urban-access-3-uncomfortable-truths/\n",
      "\n",
      "\n",
      "The past few decades have been filled with a deep optimism about the role of cities and suburbs across the world. These engines of economic growth host a majority of world population, are major drivers of economic innovation, and have created pathways to opportunities for untold amounts of people.\t\n",
      "Authors\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Jeffrey Gutman\n",
      "Senior Fellow - Global Economy and Development\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Adie Tomer\n",
      "Fellow - Metropolitan Policy Program\n",
      "\n",
      " Twitter\n",
      "AdieTomer\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "But all is not well within our so-called Urban Century. Rapid urbanization, rising gentrification, concentrated poverty, and shortages of basic infrastructure have combined to create spatial inequity in cities and suburbs across the globe. The challenges of housing, moving, and employing so many people have led to longer travel times, rising housing costs, and unsustainable public spending. Moreover, policymakers are questioning traditional policies and approaches.\n",
      "The past couple years, we’ve led a project at Brookings—Moving to Access—that responds to these spatial challenges by promoting the idea of connecting people to opportunities as a new foundational principle for 21st century urban development. This principle of accessibility is meant to be a corollary to the natural questions we ask ourselves everyday about the communities where we live: Is this the best location to access employment? Are there nearby schools and health services? Is there a market in the neighborhood? How can I get from here to there? Such choices are valid for those with sufficient income. But what about those with more limited resources and thus choices in terms of affordable housing and affordable transport?\n",
      "While economists, planners, and engineers have promoted accessibility for decades, the concept is more often found in textbooks than formal urban policies. In the first stage of this project, we worked with a team of experts to determine what has stalled practical implementation of appropriate policies and practices? “Delivering Inclusive Access,” a report of this initial work, offers a synthesis of what we found and where we believe researchers, policymakers, and practitioners can take this work next. The paper found three central challenges.\n",
      "The fallacy of the single indicator\n",
      "The current transport regime’s approach to measurement is one of outward elegance: The dominant pursuit is speed, and the primary way to measure it is congestion (or what slows us down). Many have come to label this approach a pursuit of “mobility.” It is seen through different, but often singular, measures of how congestion affects a specific roadway. Such singular measures are easily interpreted by policymakers and civil society and can be translated directly into economic analysis of related investments through timesavings. They also conveniently serve such purposes as the internationally agreed-upon Sustainable Development Goals. Yet they actually don’t answer the fundamental question of who can reach where, in how much time, and at what cost.\n",
      "\n",
      "\n",
      "Related Content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Cities & Regions\n",
      "Delivering inclusive access\n",
      "\n",
      "Jeffrey Gutman, Adie Tomer, Joseph Kane, Nirav Patel, and Ranjitha Shivaram\n",
      "August 2017\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Cities & Regions\n",
      "Measuring performance: Accessibility metrics in metropolitan regions around the world\n",
      "\n",
      "Geneviève Boisjoly and Ahmed El-Geneidy\n",
      "August 2017\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Cities & Regions\n",
      "Is better access key to inclusive cities?\n",
      "\n",
      "Jeffrey Gutman and Nirav Patel\n",
      "Wednesday, October 5, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Accessibility measures can answer those questions, but not through any one measure. First, the variable social, economic, and political contexts related to access mean searching for a single magical indicator is counterintuitive. For example, a wealthy, automobile-centric region like Dallas, Texas, may have very different measurable goals than a denser, poorer region like Dar es Salaam, Tanzania. Second, academic literature is now rife with such complex measures that it could be difficult to communicate their methodology and results with practitioners. The development of a suite of indicators could offer a menu for policymakers and practitioners to judge accessibility based on local objectives, local conditions, and local capacity.\n",
      "The danger of excessive localization\n",
      "Decentralization and empowering local communities is fast becoming a mantra of governance experts across the world, from development practitioners at institutions like the World Bank to city-focused theorists. And for good reason: delegating policy design and fiscal authority directly to the local level helps ensure policies and practices respond to local needs and desires. Yet as urban areas spillover into contiguous and often numerous municipalities, local independence can introduce certain challenges, especially relating to social and environmental externalities. When it comes to transportation and land development, interests of one municipality are often different from its neighbors. And these divergent development goals can exacerbate accessibility challenges within growing regions, spreading people, housing jobs, and other activities further from one another.\n",
      "Addressing spatial inequities in land use and real estate markets require a broader approach to horizontal governance. While there are examples of metropolitan transport authorities, there is less willingness to consider metropolitan or horizontal governance of land use and fiscal policies. For example, should housing be coordinated across an entire region?\n",
      "Countries with a more centralized top down approach to governance, such as France and Germany, have greater ability to formulate metropolitan governance than more decentralized countries such as the U.S. This is not to say there is a one-size-fits-all approach, but there is an opportunity to test different solutions within different governance contexts, comparing how effective each model is to promote spatial inclusivity.\n",
      "The finance community is missing in action\n",
      "Financing is a central topic in infrastructure circles. As maintenance bills from the automobile era come due, populations continue to grow, and fiscal budgets are tight, how can urban areas afford to build enough infrastructure to support future economic growth? In response, new approaches are evolving in fiscal instruments, such as value capture and private-public partnerships. Missing in these discussions, however, are the implications for inclusive access.\n",
      "We conducted a multi-decade review of past academic literature on access and found that there is no clear substantive discussion of accessibility from a fiscal perspective. While urban transport and land use professionals clearly recognize their interrelationship in achieving inclusive accessibility, at least in theory, the fiscal and finance professionals generally ignore the implications of their instruments with regard to inclusivity. The multilateral development banks and their economic evaluations have ignored the distributive impacts until very recently. And the efforts of some countries to incorporate measures through multi-criteria analysis have had limited impact.\n",
      "This gap must be resolved in any effort toward inclusive urban development. There is little doubt that fiscal approaches must carefully assess who ultimately pays and that alternative finance instruments should be adapted to foster access for all.\n",
      "Going forward\n",
      "Our research confirms that there are enormous opportunities to advance accessibility theory into practice. At this point, what is desperately needed is to launch a range of case studies that deal with these issues and challenges under different geographic, governance, and economic contexts. The good news is that many initiatives are already underway, and more robust communication channels and technology can support such efforts. In Chicago, researchers created an online platform to visually explore accessibility by location. In Bogota, researchers evaluated how affordability is a key principle of access. And in Cairo and Kigali, researchers used open tools to achieve new insights for accessibility. Sharing the results of these case studies could lead to a new level of cross-disciplinary approaches to improve accessibility and lessen the effects of spatial inequity.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: The Men Who Want to Live Forever\n",
      "URL: https://www.nytimes.com/2018/01/25/opinion/sunday/silicon-valley-immortality.html\n",
      "\n",
      "Would you like to live forever? Some billionaires, already invincible in every other way, have decided that they also deserve not to die. Today several biotech companies, fueled by Silicon Valley fortunes, are devoted to “life extension” — or as some put it, to solving “the problem of death.”\n",
      "It’s a cause championed by the tech billionaire Peter Thiel, the TED Talk darling Aubrey de Gray, Google’s billion-dollar Calico longevity lab and investment by Amazon’s Jeff Bezos. The National Academy of Medicine, an independent group, recently dedicated funding to “end aging forever.”\n",
      "As the longevity entrepreneur Arram Sabeti told The New Yorker: “The proposition that we can live forever is obvious. It doesn’t violate the laws of physics, so we can achieve it.” Of all the slightly creepy aspects to this trend, the strangest is the least noticed: The people publicly championing life extension are mainly men.\n",
      "Not all of them, of course. In 2009, Elizabeth Blackburn received the Nobel Prize for her work on telomeres, protein caps on chromosomes that may be a key to understanding aging. Cynthia Kenyon, the vice president for aging research at Calico, studied life extension long before it was cool; her former protégée, Laura Deming, now runs a venture capital fund for the cause. But these women are focused on curbing age-related pathology, a concept about as controversial as cancer research. They do not appear thirsty for the Fountain of Youth.\n",
      "Professor Blackburn’s new book on telomeres couldn’t be clearer. “Does our research show that by maintaining your telomeres you will live into your hundreds?” it says. “No. Everyone’s cells become old and eventually we die.” Ms. Kenyon once described her research’s goal as “to just have a healthy life and then turn out the lights.” Even Ms. Deming, a 23-year-old prodigy who worked in Ms. Kenyon’s lab at age 12, points out that “aging is innately important to us.”\n",
      "Few of these experts come close to matching the gaudy statements of the longevity investor and “biohacker” Dave Asprey, who has told journalists, “I decided that I was just not going to die.” Or those of Brian Hanley, a microbiologist who has tested an anti-aging gene therapy he developed on himself, who claimed: “There’s a bunch of things that will need to be done to achieve life spans into at least hundreds of years. But we’ll get there.” Or of the 74-year-old fashion mogul Peter Nygard, who during a promotional clip receives injections of his own stem cells to reverse his aging while declaring: “Ponce de León had the right idea. He was just too early. That was then. This is now.”\n",
      "I came across Mr. Nygard’s ode to human endurance three years ago while beginning research on a novel about a woman who can’t die, and watching that video allowed me to experience something close to life extension. As Mr. Nygard compared himself to Leonardo da Vinci and Benjamin Franklin while dancing with a bevy of models — or as a voice-over explained, “living a life most can only dream of” — nine minutes of YouTube expanded into a vapid eternity, where time melted into a vortex of solipsism.\n",
      "At that time I was immersed in caring for my four young children, and this paean to everlasting youth seemed especially stupid. I recall thinking that if this was eternal life, death didn’t seem that bad.\n",
      "But now, as powerful men have begun falling like dominoes under accusations of sexual assault, that video with its young women clustered around an elderly multimillionaire has haunted me anew. As I recall my discomfort with the proclamations of longevity-driven men who hope to achieve “escape velocity,” I think of the astonishing hubris of the Harvey Weinsteins of the world, those who saw young women’s bodies as theirs for the taking.\n",
      "Much has been said about why we allowed such behavior to go unchecked. What has remained unsaid, because it is so obvious, is what would make someone so shameless in the first place: These people believed they were invincible. They saw their own bodies as entirely theirs and other people’s bodies as at their disposal; apparently nothing in their lives led them to believe otherwise.\n",
      "Historically, this is a mistake that few women would make, because until very recently, the physical experience of being a woman entailed exactly the opposite — and not only because women have to hold their keys in self-defense while walking through parking lots at night. It’s only very recently that women have widely participated in public life, but it’s even more recently that men have been welcome, or even expected, to provide physical care for vulnerable people.\n",
      "Only for a nanosecond of human history have men even slightly shared what was once exclusively a woman’s burden: the relentless daily labor of caring for another person’s body, the life-preserving work of cleaning feces and vomit, the constant cycle of cooking and feeding and blanketing and bathing, whether for the young, the ill or the old. For nearly as long as there have been humans, being a female human has meant a daily nonoptional immersion in the fragility of human life and the endless effort required to sustain it.\n",
      "Obviously not everyone who provides care for others is a saint. But engaging in that daily devotion, or even living with its expectation, has enormous potential to change a person. It forces one to constantly imagine the world from someone else’s point of view: Is he hungry? Maybe she’s tired. Is his back hurting him? What is she trying to say?\n",
      "The most obvious cure for today’s gender inequities is to put more women in power. But if we really hope to create an equal society, we will also need more men to care for the powerless — more women in the boardroom, but also more men at the nurses’ station and the changing table, immersed in daily physical empathy. If that sounds like an evolutionary impossibility, well, it doesn’t violate the laws of physics, so we can achieve it. It is surely worth at least as much investment as defeating death.\n",
      "Perhaps it takes the promise of immortality to inspire the self-absorbed to invest in unsexy work like Alzheimer’s research. If so, we may all one day bless the inane death-defiance as a means to a worthy end.\n",
      "But men who hope to live forever might pause on their eternal journey to consider the frightening void at invincibility’s core. Death is the ultimate vulnerability. It is the moment when all of us must confront exactly what so many women have known all too well: You are a body, only a body, and nothing more.\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "\n",
    "class Content:\n",
    "    def __init__(self, url, title, body):\n",
    "        self.url = url\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "\n",
    "\n",
    "def getPage(url):\n",
    "    req = requests.get(url)\n",
    "    return BeautifulSoup(req.text, 'html.parser')\n",
    "\n",
    "\n",
    "def scrapeNYTimes(url):\n",
    "    bs = getPage(url)\n",
    "    title = bs.find(\"h1\").text\n",
    "    lines = bs.find_all(\"p\", {\"class\": \"story-content\"})\n",
    "    body = '\\n'.join([line.text for line in lines])\n",
    "    return Content(url, title, body)\n",
    "\n",
    "\n",
    "def scrapeBrookings(url):\n",
    "    bs = getPage(url)\n",
    "    title = bs.find(\"h1\").text\n",
    "    body = bs.find(\"div\", {\"class\", \"post-body\"}).text\n",
    "    return Content(url, title, body)\n",
    "\n",
    "\n",
    "url = 'https://www.brookings.edu/blog/future-development/2018/01/26/delivering-inclusive-urban-access-3-uncomfortable-truths/'\n",
    "content = scrapeBrookings(url)\n",
    "print('Title: {}'.format(content.title))\n",
    "print('URL: {}\\n'.format(content.url))\n",
    "print(content.body)\n",
    "\n",
    "url = 'https://www.nytimes.com/2018/01/25/opinion/sunday/silicon-valley-immortality.html'\n",
    "content = scrapeNYTimes(url)\n",
    "print('Title: {}'.format(content.title))\n",
    "print('URL: {}\\n'.format(content.url))\n",
    "print(content.body)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Content:\n",
    "    \"\"\"\n",
    "    Common base class for all articles/pages\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, url, title, body):\n",
    "        self.url = url\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "\n",
    "    def print(self):\n",
    "        \"\"\"\n",
    "        Flexible printing function controls output\n",
    "        \"\"\"\n",
    "        print(\"URL: {}\".format(self.url))\n",
    "        print(\"TITLE: {}\".format(self.title))\n",
    "        print(\"BODY:\\n{}\".format(self.body))\n",
    "\n",
    "\n",
    "class Website:\n",
    "    \"\"\" \n",
    "    Contains information about website structure\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "class Crawler:\n",
    "\n",
    "    def getPage(self, url):\n",
    "        try:\n",
    "            req = requests.get(url)\n",
    "        except requests.exceptions.RequestException:\n",
    "            return None\n",
    "        return BeautifulSoup(req.text, 'html.parser')\n",
    "\n",
    "    def safeGet(self, pageObj, selector):\n",
    "        \"\"\"\n",
    "        Utilty function used to get a content string from a Beautiful Soup\n",
    "        object and a selector. Returns an empty string if no object\n",
    "        is found for the given selector\n",
    "        \"\"\"\n",
    "        selectedElems = pageObj.select(selector)\n",
    "        if selectedElems is not None and len(selectedElems) > 0:\n",
    "            return '\\n'.join([elem.get_text() for elem in selectedElems])\n",
    "        return ''\n",
    "\n",
    "    def parse(self, site, url):\n",
    "        \"\"\"\n",
    "        Extract content from a given page URL\n",
    "        \"\"\"\n",
    "        bs = self.getPage(url)\n",
    "        if bs is not None:\n",
    "            title = self.safeGet(bs, site.titleTag)\n",
    "            body = self.safeGet(bs, site.bodyTag)\n",
    "            if title != '' and body != '':\n",
    "                content = Content(url, title, body)\n",
    "                content.print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title is:\n",
      "Learning Python, 5th Edition \n",
      "Body is:\n",
      "\n",
      "Get a comprehensive, in-depth introduction to the core Python language with this hands-on book. Based on author Mark Lutz’s popular training course, this updated fifth edition will help you quickly write efficient, high-quality code with Python. It’s an ideal way to begin, whether you’re new to programming or a professional developer versed in other languages. \n",
      "\n",
      "Complete with quizzes, exercises, and helpful illustrations,  this easy-to-follow, self-paced tutorial gets you started with both Python 2.7 and 3.3— the latest releases in the 3.X  and 2.X lines—plus all other releases in common use today. You’ll also learn some advanced language features that recently have become more common in Python code.\n",
      "\n",
      "Explore Python’s major built-in object types such as numbers, lists, and dictionaries \n",
      "Create and process objects with Python statements, and learn Python’s general syntax model\n",
      "Use functions to avoid code redundancy and package code for reuse\n",
      "Organize statements, functions, and other tools into larger components with modules \n",
      "Dive into classes: Python’s object-oriented programming tool for structuring code\n",
      "Write large programs with Python’s exception-handling model and development tools\n",
      "Learn advanced Python tools, including decorators, descriptors, metaclasses, and Unicode processing\n",
      "\n",
      "\n",
      "URL: http://shop.oreilly.com/product/0636920028154.do\n",
      "TITLE: Learning Python, 5th Edition \n",
      "BODY:\n",
      "\n",
      "Get a comprehensive, in-depth introduction to the core Python language with this hands-on book. Based on author Mark Lutz’s popular training course, this updated fifth edition will help you quickly write efficient, high-quality code with Python. It’s an ideal way to begin, whether you’re new to programming or a professional developer versed in other languages. \n",
      "\n",
      "Complete with quizzes, exercises, and helpful illustrations,  this easy-to-follow, self-paced tutorial gets you started with both Python 2.7 and 3.3— the latest releases in the 3.X  and 2.X lines—plus all other releases in common use today. You’ll also learn some advanced language features that recently have become more common in Python code.\n",
      "\n",
      "Explore Python’s major built-in object types such as numbers, lists, and dictionaries \n",
      "Create and process objects with Python statements, and learn Python’s general syntax model\n",
      "Use functions to avoid code redundancy and package code for reuse\n",
      "Organize statements, functions, and other tools into larger components with modules \n",
      "Dive into classes: Python’s object-oriented programming tool for structuring code\n",
      "Write large programs with Python’s exception-handling model and development tools\n",
      "Learn advanced Python tools, including decorators, descriptors, metaclasses, and Unicode processing\n",
      "\n",
      "\n",
      "title is:\n",
      "EPA chief wants scientists to debate climate on TV\n",
      "Body is:\n",
      "WASHINGTON (Reuters) - The U.S. Environmental Protection Agency is in the early stages of launching a debate about climate change that could air on television – challenging scientists to prove the widespread view that global warming is a serious threat, the head of the agency said. The move comes as the administration of President Donald Trump seeks to roll back a slew of Obama-era regulations limiting carbon dioxide emissions from fossil fuels, and begins a withdrawal from the Paris Climate Agreement - a global pact to stem planetary warming through emissions cuts. “There are lots of questions that have not been asked and answered (about climate change),” EPA Administrator Scott Pruitt told Reuters in an interview late on Monday.  “Who better to do that than a group of scientists... getting together and having a robust discussion for all the world to see,” he added without explaining how the scientists would be chosen. Asked if he thought the debate should be televised, Pruitt said: “I think so. I think so. I mean, I don’t know yet, but you want this to be open to the world. You want this to be on full display. I think the American people would be very interested in consuming that. I think they deserve it.” Pruitt, one of the most controversial figures in the Trump administration, has repeatedly expressed doubts about climate change – one of the main points of contention in his narrow confirmation by the Senate.  While acknowledging the planet is warming, Pruitt says he questions the gravity of the problem and the need for regulations that require companies to take costly measures to reduce their carbon footprint. “It is a question about how much we contribute to it. How do we measure that with precision? And by the way, are we on an unsustainable path? And is it causing an existential threat?” he said in the interview. Since taking up his role at EPA, he has emerged as one of the more prolific Trump cabinet appointees, taking steps to undo more than two dozen regulations, and influencing Trump’s decision to pull the United States from the Paris climate change deal, agreed by nearly 200 countries in 2015. Pruitt rejected global criticism of the United States for pulling out of the climate deal, which Trump has said would have cost America trillions of dollars without benefit. Environmental Protection Agency Administrator Scott Pruitt waves after an interview for Reuters at his office in Washington, U.S., July 10, 2017. REUTERS/Yuri Gripas“We have nothing to be apologetic about,” Pruitt said. “It was absolutely a decision of courage and fortitude and truly represented an America First strategy with respect to how we are leading on this issue.” Pruitt said the United States had already cut its carbon output to the lowest levels in nearly 25 years without mandates, thanks mainly to increased use of natural gas - which burns cleaner than coal. “RED TEAM, BLUE TEAM” TACTICS Slideshow (4 Images)Pruitt said his desire for the agency to host an ongoing climate change debate was inspired by two articles published in April – one in the Wall Street Journal by theoretical physicist Steve Koonin, who served as undersecretary of energy under Obama – and one by conservative columnist Brett Stephens in the New York Times.  Koonin’s article made the case that climate science should use the “red team-blue team” methodology used by the national security community to test assumptions. Stephens’ article criticized claims of complete certainty in climate science, saying that it “traduces the spirit of science.” Pruitt said scientists should not scoff at the idea of participating in these debates. Related CoverageTranscript of Reuters interview with EPA Administrator Scott Pruitt“If you’re going to win and if you’re so certain about it, come and do your deal. They shouldn’t be scared of the debate and discussion,” he said. Pruitt said debate is not necessarily aimed at undermining the 2009 “endangerment finding,” the scientific determination that carbon dioxide harms human health that formed the basis for the Democratic Obama administration’s regulation of greenhouse gases. He said there may be a legal basis to challenge the finding but would prefer Congress weigh in on the matter. In the interview, Pruitt added that he intended to deal “very aggressively” with automakers that use devices to cheat emissions tests, and would also seek to boost accountability for companies to clean up polluted sites under the Superfund program.  He said EPA was also not ready to decide yet on a change proposed by Trump’s special adviser Carl Icahn to the U.S. biofuels program, that would shift the burden of blending biofuels like ethanol into gasoline away from refiners to companies further down the supply chain. Editing by Richard Valdmanis and Marguerita ChoyOur Standards:The Thomson Reuters Trust Principles.\n",
      "URL: http://www.reuters.com/article/us-usa-epa-pruitt-idUSKBN19W2D0\n",
      "TITLE: EPA chief wants scientists to debate climate on TV\n",
      "BODY:\n",
      "WASHINGTON (Reuters) - The U.S. Environmental Protection Agency is in the early stages of launching a debate about climate change that could air on television – challenging scientists to prove the widespread view that global warming is a serious threat, the head of the agency said. The move comes as the administration of President Donald Trump seeks to roll back a slew of Obama-era regulations limiting carbon dioxide emissions from fossil fuels, and begins a withdrawal from the Paris Climate Agreement - a global pact to stem planetary warming through emissions cuts. “There are lots of questions that have not been asked and answered (about climate change),” EPA Administrator Scott Pruitt told Reuters in an interview late on Monday.  “Who better to do that than a group of scientists... getting together and having a robust discussion for all the world to see,” he added without explaining how the scientists would be chosen. Asked if he thought the debate should be televised, Pruitt said: “I think so. I think so. I mean, I don’t know yet, but you want this to be open to the world. You want this to be on full display. I think the American people would be very interested in consuming that. I think they deserve it.” Pruitt, one of the most controversial figures in the Trump administration, has repeatedly expressed doubts about climate change – one of the main points of contention in his narrow confirmation by the Senate.  While acknowledging the planet is warming, Pruitt says he questions the gravity of the problem and the need for regulations that require companies to take costly measures to reduce their carbon footprint. “It is a question about how much we contribute to it. How do we measure that with precision? And by the way, are we on an unsustainable path? And is it causing an existential threat?” he said in the interview. Since taking up his role at EPA, he has emerged as one of the more prolific Trump cabinet appointees, taking steps to undo more than two dozen regulations, and influencing Trump’s decision to pull the United States from the Paris climate change deal, agreed by nearly 200 countries in 2015. Pruitt rejected global criticism of the United States for pulling out of the climate deal, which Trump has said would have cost America trillions of dollars without benefit. Environmental Protection Agency Administrator Scott Pruitt waves after an interview for Reuters at his office in Washington, U.S., July 10, 2017. REUTERS/Yuri Gripas“We have nothing to be apologetic about,” Pruitt said. “It was absolutely a decision of courage and fortitude and truly represented an America First strategy with respect to how we are leading on this issue.” Pruitt said the United States had already cut its carbon output to the lowest levels in nearly 25 years without mandates, thanks mainly to increased use of natural gas - which burns cleaner than coal. “RED TEAM, BLUE TEAM” TACTICS Slideshow (4 Images)Pruitt said his desire for the agency to host an ongoing climate change debate was inspired by two articles published in April – one in the Wall Street Journal by theoretical physicist Steve Koonin, who served as undersecretary of energy under Obama – and one by conservative columnist Brett Stephens in the New York Times.  Koonin’s article made the case that climate science should use the “red team-blue team” methodology used by the national security community to test assumptions. Stephens’ article criticized claims of complete certainty in climate science, saying that it “traduces the spirit of science.” Pruitt said scientists should not scoff at the idea of participating in these debates. Related CoverageTranscript of Reuters interview with EPA Administrator Scott Pruitt“If you’re going to win and if you’re so certain about it, come and do your deal. They shouldn’t be scared of the debate and discussion,” he said. Pruitt said debate is not necessarily aimed at undermining the 2009 “endangerment finding,” the scientific determination that carbon dioxide harms human health that formed the basis for the Democratic Obama administration’s regulation of greenhouse gases. He said there may be a legal basis to challenge the finding but would prefer Congress weigh in on the matter. In the interview, Pruitt added that he intended to deal “very aggressively” with automakers that use devices to cheat emissions tests, and would also seek to boost accountability for companies to clean up polluted sites under the Superfund program.  He said EPA was also not ready to decide yet on a change proposed by Trump’s special adviser Carl Icahn to the U.S. biofuels program, that would shift the burden of blending biofuels like ethanol into gasoline away from refiners to companies further down the supply chain. Editing by Richard Valdmanis and Marguerita ChoyOur Standards:The Thomson Reuters Trust Principles.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title is:\n",
      "Idea to Retire: Old methods of policy education\n",
      "Idea to Retire: Old methods of policy education\n",
      "Body is:\n",
      "\n",
      "Public policy and public affairs schools aim to train competent creators and implementers of government policy. While drawing on the principles that gird our economic and political systems to provide a well-rounded education, like law schools and business schools, policy schools provide professional training. They are quite distinct from graduate programs in political science or economics which aim to train the next generation of academics. As professional training programs, they add value by imparting both the skills which are relevant to current employers, and skills which we know will be relevant as organizations and societies evolve. \n",
      "The relevance of the skills that policy programs impart to address problems of today and tomorrow bears further discussion. We are living through an era in which societies are increasingly interconnected. The wide-scale adoption of devices such as the smartphone is having a profound impact on our culture, communities, and economy. The use of social and digital media and associated means of communication enabled by mobile devices is changing the tone, content, and geographic scope of our conversations, modifying how information is generated and consumed, and changing the very nature of citizen engagement. \n",
      "Information technology-based platforms provisioned by private providers such as Facebook, Google, Uber, and Lyft maintain information about millions of citizens and enable services such as transportation that were mediated in the past solely by the public sector. Surveillance for purposes of public safety via large-scale deployment of sensors also raises fundamental questions about information privacy. From technology-enabled global delivery of work to displacement and replacement of categories of work, some studies estimate that up to 47 percent of U.S. employment might be at risk of computerization with an attendant rise in income inequality. These technology-induced changes will affect every policy domain. How should policy programs best prepare students to address societal challenges in this world that is being transformed by technology? We believe the answer lies in educating students to be “men and women of intelligent action.” \n",
      "A model of policy education\n",
      "We begin with a skills-based model of policy education. These four essential skills address the general problems policy practitioners frequently face:\n",
      "\n",
      "Design skills to craft policy ideas \n",
      "Analytical skills to make smart ex ante decisions \n",
      "Interpersonal experience to manage policy implementation  \n",
      "Evaluative skills to assess outcomes ex post and correct course if necessary\n",
      "\n",
      "These skills make up the policy analysis toolkit required to be data driven practitioner of “intelligent action” in any policy domain. This toolkit needs to be supplemented by an understanding of how technology is transforming societal challenges, enabling new solutions, or disrupting existing regulatory regimes. This understanding is essential to policy formulation and implementation. \n",
      "Pillar 1: Design skills\n",
      "As with engineering, where design precedes analysis, this first pillar seeks to educate students in thinking creatively about problems in order to devise and develop policy ideas. Using ideas derived from design, divergent and convergent thinking principles are employed to generate, explore, and arrive at a candidate set of solutions. Using Uber as an example, an approach to identify and explore the key policy issues such as convenience, costs, driver working hours, and insurance would involve interviewing and observing both incumbent taxi drivers and Uber drivers. This in turn would lead to a set of alternatives that deserve further and careful consideration.  Using these skills, candidate designs and choices that are generated can be evaluated using the policy analytic toolkit. \n",
      "Pillar 2: Analytical skills\n",
      "\n",
      "\n",
      "Related\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "The Blockchain: What It Is and Why It Matters\n",
      "\n",
      "Mohit Kaushal and Sheel Tyle\n",
      "Tuesday, January 13, 2015\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "How technology is changing manufacturing\n",
      "\n",
      "Darrell M. West\n",
      "Thursday, June 2, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "Rural and urban America divided by broadband access\n",
      "\n",
      "Darrell M. West and Jack Karsten\n",
      "Monday, July 18, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "At Carnegie Mellon, we are often cited in media and interrogated by peers on our approach to analytical and technology skills education. Curiosity about which skills are the “right” skills to teach policy practitioners are common, but we believe this is the wrong approach. We instead begin from the premise that policy or management decisions should be grounded in evidence.  We then determine the skills required to assemble the types of evidence that will likely be available to policy makers in the future.  In increasingly instrumented environments where citizens and infrastructure produce continuous streams of data, making sense of it all will require a somewhat different set of skills. We believe that a grounding in micro-economics, operations research, statistics, and program evaluation (aka causal inference) to be an essential core to policy programs. \n",
      "New coursework will teach students to work with multi-variable data and machine learning with an emphasis on prediction. This material ought to be part of the required coursework in statistics given the importance of prediction in many policy implementation settings. Along the same lines, the ability to work with unstructured data (especially text) and data visualization will become increasingly relevant to all students, not just those students who want to specialize in data analytics. Finally, knowledge of data manipulation and analysis languages such as Python and R for analytic work will be important because data often has to be massaged and cleansed prior to analysis. An important task for programs will be to determine the competencies expected of graduates. \n",
      "Pillar 3: Interpersonal experiences\n",
      "The third pillar of the skills-based model is interpersonal experience, where the practiced habits of good communication and steady negotiation developed with a sound understanding of organizations, their design and their behaviors. We label these purposely as experiences rather than skills because we believe they are best practiced either in the real-world or in simulated real-world settings. It is also in this pillar where practitioners learn the knowledge necessary to become credible experts in their domain. We believe that in addition to core coursework in the area, a supplementary curriculum which provides students with opportunities to gain these experiences is an essential component of our educational model.\n",
      "Pillar 4: Evaluative skills\n",
      "\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Constitution 3.0\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tEdited by Jeffrey Rosen and Benjamin Wittes \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The Need for Speed\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert E. Litan and Hal J. Singer \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "After the Breakup\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert W. Crandall \n",
      "2010\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The ability to carefully diagnose the effectiveness of policy or management interventions is the fourth pillar of our model. It is insufficient to create and execute policy without measurement, and this is where both careful thought to the fundamental issues of measurement and evaluation become important. The ability to make objective judgments on the benefits, liabilities, and unintended consequences of prior policies is the goal of this set of skills. Here, sound statistical and econometric training with an understanding of the principles of causal inference is essential. In addition, program evaluation skills such as cost-benefit and financial analysis help practitioners round out their evaluation skills by considering both non-monetary and economic impacts.\n",
      "What should be retired?\n",
      "A skills-based approach might replace certain aspects of existing policy training.  This depends on a number of factors specific to each institution, but three generally applicable observations are clear. First, real-world experiences are a powerful way to encode domain learning as well as project management skills. Through project-based work, students can learn about institutional contexts in specific policy domains and political processes such as budgeting. Second, team-based projects allow students to learn and apply principles of management and organizational behavior. At Carnegie Mellon, we refer to these as “systems synthesis” projects, since they require students to adopt a systemic point of view and to synthesize a number of skills in their policy analysis toolkit. Third, interpersonal skills training can be practiced through activities such as weekend negotiation exercises, hackathons, and speaker series. These activities can be highly intentional and fashioned to reinforce skills rather than as a recess from the “real work” of classroom training. Since students complete graduate programs in such a short time, counseling them to focus on outcomes from day one will allow them to choose a reinforcing set of coursework and real-world experiences. \n",
      "In summary, we argue for a model of policy education that views practitioners as future problem solvers. Good policy education must consider the ways in which problems will present themselves, and the ways in which answers will obscure themselves. Rigorous training grounded in the analysis of available evidence and buoyed by real-world interpersonal experiences is a sound approach to relevant, durable policy training.\n",
      " \n",
      "Authors\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "R\n",
      "\n",
      "\n",
      "\n",
      "Ramayya Krishnan\n",
      "Ramayya Krishnan is the dean of H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University where he is the W.W. Cooper and Ruth F. Cooper Professor of Management Science and Information Systems.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "J\n",
      "\n",
      "\n",
      "\n",
      "Jon Nehlsen\n",
      "Jon Nehlsen is senior director of external relations at H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Read other essays in the Ideas to Retire blog series here.\n",
      "\n",
      "URL: https://www.brookings.edu/blog/techtank/2016/03/01/idea-to-retire-old-methods-of-policy-education/\n",
      "TITLE: Idea to Retire: Old methods of policy education\n",
      "Idea to Retire: Old methods of policy education\n",
      "BODY:\n",
      "\n",
      "Public policy and public affairs schools aim to train competent creators and implementers of government policy. While drawing on the principles that gird our economic and political systems to provide a well-rounded education, like law schools and business schools, policy schools provide professional training. They are quite distinct from graduate programs in political science or economics which aim to train the next generation of academics. As professional training programs, they add value by imparting both the skills which are relevant to current employers, and skills which we know will be relevant as organizations and societies evolve. \n",
      "The relevance of the skills that policy programs impart to address problems of today and tomorrow bears further discussion. We are living through an era in which societies are increasingly interconnected. The wide-scale adoption of devices such as the smartphone is having a profound impact on our culture, communities, and economy. The use of social and digital media and associated means of communication enabled by mobile devices is changing the tone, content, and geographic scope of our conversations, modifying how information is generated and consumed, and changing the very nature of citizen engagement. \n",
      "Information technology-based platforms provisioned by private providers such as Facebook, Google, Uber, and Lyft maintain information about millions of citizens and enable services such as transportation that were mediated in the past solely by the public sector. Surveillance for purposes of public safety via large-scale deployment of sensors also raises fundamental questions about information privacy. From technology-enabled global delivery of work to displacement and replacement of categories of work, some studies estimate that up to 47 percent of U.S. employment might be at risk of computerization with an attendant rise in income inequality. These technology-induced changes will affect every policy domain. How should policy programs best prepare students to address societal challenges in this world that is being transformed by technology? We believe the answer lies in educating students to be “men and women of intelligent action.” \n",
      "A model of policy education\n",
      "We begin with a skills-based model of policy education. These four essential skills address the general problems policy practitioners frequently face:\n",
      "\n",
      "Design skills to craft policy ideas \n",
      "Analytical skills to make smart ex ante decisions \n",
      "Interpersonal experience to manage policy implementation  \n",
      "Evaluative skills to assess outcomes ex post and correct course if necessary\n",
      "\n",
      "These skills make up the policy analysis toolkit required to be data driven practitioner of “intelligent action” in any policy domain. This toolkit needs to be supplemented by an understanding of how technology is transforming societal challenges, enabling new solutions, or disrupting existing regulatory regimes. This understanding is essential to policy formulation and implementation. \n",
      "Pillar 1: Design skills\n",
      "As with engineering, where design precedes analysis, this first pillar seeks to educate students in thinking creatively about problems in order to devise and develop policy ideas. Using ideas derived from design, divergent and convergent thinking principles are employed to generate, explore, and arrive at a candidate set of solutions. Using Uber as an example, an approach to identify and explore the key policy issues such as convenience, costs, driver working hours, and insurance would involve interviewing and observing both incumbent taxi drivers and Uber drivers. This in turn would lead to a set of alternatives that deserve further and careful consideration.  Using these skills, candidate designs and choices that are generated can be evaluated using the policy analytic toolkit. \n",
      "Pillar 2: Analytical skills\n",
      "\n",
      "\n",
      "Related\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "The Blockchain: What It Is and Why It Matters\n",
      "\n",
      "Mohit Kaushal and Sheel Tyle\n",
      "Tuesday, January 13, 2015\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "How technology is changing manufacturing\n",
      "\n",
      "Darrell M. West\n",
      "Thursday, June 2, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "Rural and urban America divided by broadband access\n",
      "\n",
      "Darrell M. West and Jack Karsten\n",
      "Monday, July 18, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "At Carnegie Mellon, we are often cited in media and interrogated by peers on our approach to analytical and technology skills education. Curiosity about which skills are the “right” skills to teach policy practitioners are common, but we believe this is the wrong approach. We instead begin from the premise that policy or management decisions should be grounded in evidence.  We then determine the skills required to assemble the types of evidence that will likely be available to policy makers in the future.  In increasingly instrumented environments where citizens and infrastructure produce continuous streams of data, making sense of it all will require a somewhat different set of skills. We believe that a grounding in micro-economics, operations research, statistics, and program evaluation (aka causal inference) to be an essential core to policy programs. \n",
      "New coursework will teach students to work with multi-variable data and machine learning with an emphasis on prediction. This material ought to be part of the required coursework in statistics given the importance of prediction in many policy implementation settings. Along the same lines, the ability to work with unstructured data (especially text) and data visualization will become increasingly relevant to all students, not just those students who want to specialize in data analytics. Finally, knowledge of data manipulation and analysis languages such as Python and R for analytic work will be important because data often has to be massaged and cleansed prior to analysis. An important task for programs will be to determine the competencies expected of graduates. \n",
      "Pillar 3: Interpersonal experiences\n",
      "The third pillar of the skills-based model is interpersonal experience, where the practiced habits of good communication and steady negotiation developed with a sound understanding of organizations, their design and their behaviors. We label these purposely as experiences rather than skills because we believe they are best practiced either in the real-world or in simulated real-world settings. It is also in this pillar where practitioners learn the knowledge necessary to become credible experts in their domain. We believe that in addition to core coursework in the area, a supplementary curriculum which provides students with opportunities to gain these experiences is an essential component of our educational model.\n",
      "Pillar 4: Evaluative skills\n",
      "\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Constitution 3.0\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tEdited by Jeffrey Rosen and Benjamin Wittes \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The Need for Speed\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert E. Litan and Hal J. Singer \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "After the Breakup\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert W. Crandall \n",
      "2010\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The ability to carefully diagnose the effectiveness of policy or management interventions is the fourth pillar of our model. It is insufficient to create and execute policy without measurement, and this is where both careful thought to the fundamental issues of measurement and evaluation become important. The ability to make objective judgments on the benefits, liabilities, and unintended consequences of prior policies is the goal of this set of skills. Here, sound statistical and econometric training with an understanding of the principles of causal inference is essential. In addition, program evaluation skills such as cost-benefit and financial analysis help practitioners round out their evaluation skills by considering both non-monetary and economic impacts.\n",
      "What should be retired?\n",
      "A skills-based approach might replace certain aspects of existing policy training.  This depends on a number of factors specific to each institution, but three generally applicable observations are clear. First, real-world experiences are a powerful way to encode domain learning as well as project management skills. Through project-based work, students can learn about institutional contexts in specific policy domains and political processes such as budgeting. Second, team-based projects allow students to learn and apply principles of management and organizational behavior. At Carnegie Mellon, we refer to these as “systems synthesis” projects, since they require students to adopt a systemic point of view and to synthesize a number of skills in their policy analysis toolkit. Third, interpersonal skills training can be practiced through activities such as weekend negotiation exercises, hackathons, and speaker series. These activities can be highly intentional and fashioned to reinforce skills rather than as a recess from the “real work” of classroom training. Since students complete graduate programs in such a short time, counseling them to focus on outcomes from day one will allow them to choose a reinforcing set of coursework and real-world experiences. \n",
      "In summary, we argue for a model of policy education that views practitioners as future problem solvers. Good policy education must consider the ways in which problems will present themselves, and the ways in which answers will obscure themselves. Rigorous training grounded in the analysis of available evidence and buoyed by real-world interpersonal experiences is a sound approach to relevant, durable policy training.\n",
      " \n",
      "Authors\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "R\n",
      "\n",
      "\n",
      "\n",
      "Ramayya Krishnan\n",
      "Ramayya Krishnan is the dean of H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University where he is the W.W. Cooper and Ruth F. Cooper Professor of Management Science and Information Systems.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "J\n",
      "\n",
      "\n",
      "\n",
      "Jon Nehlsen\n",
      "Jon Nehlsen is senior director of external relations at H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Read other essays in the Ideas to Retire blog series here.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title is:\n",
      "Oil Boom Gives the U.S. a New Edge in Energy and Diplomacy\n",
      "Body is:\n",
      "HOUSTON — A substantial rise in oil prices in recent months has led to a resurgence in American oil production, enabling the country to challenge the dominance of Saudi Arabia and dampen price pressures at the pump.\n",
      "The success has come in the face of efforts by Saudi Arabia and its oil allies to undercut the shale drilling spree in the United States. Those strategies backfired and ultimately ended up benefiting the oil industry.\n",
      "Overcoming three years of slumping prices proved the resiliency of the shale boom. Energy companies and their financial backers were able to weather market turmoil — and the maneuvers of the global oil cartel — by adjusting exploration and extraction techniques.\n",
      "After a painful shakeout in the industry that included scores of bankruptcies and a significant loss of jobs, a steadier shale-drilling industry is arising, anchored by better-financed companies.\n",
      "With the price of West Texas intermediate crude above $65 a barrel, a level not seen in almost three years, the United States is becoming a dominant producer. It is able to outflank competitors in supplying growing global markets, particularly China and India, while slashing imports from the Middle East and North Africa.\n",
      "This year, the United States is expected to surpass Saudi Arabia and to rival Russia as the world’s leader, with record output of over 10 million barrels a day, according to the International Energy Agency.\n",
      "“This is a 180-degree turn for the United States and the impacts are being felt around the world,” said Daniel Yergin, the economic historian and author of “The Prize: The Epic Quest for Oil, Money and Power.” “This not only contributes to U.S. energy security but also contributes to world energy security by bringing new supplies to the world.”\n",
      "At the same time, the United States is becoming a major exporter of natural gas, another outgrowth of the shale revolution, undercutting Russian energy dominance over Eastern Europe.\n",
      "The improving energy picture comes as the Trump administration is attempting to increase offshore drilling and loosen other regulations on fossil fuel development. But just as the surge in oil and gas production in shale fields during the Barack Obama administration had little to do with Washington, the current rise is the result of private companies responding to global markets.\n",
      "Shale fields can be developed relatively quickly and at modest costs relative to the giant projects, whether on land or offshore, that were once favored by big oil companies. That makes it easier to turn investment spigots on or off to adjust to market fluctuations. Companies like Exxon Mobil and Chevron are putting increasing amounts of capital in shale fields, particularly in West Texas and New Mexico.\n",
      "The results go far beyond the economic, offering Washington strategic weapons once unthinkable. The United States and its allies now have a supply cushion at a time when political turmoil in Venezuela, Libya and Nigeria is threatening to interrupt flows to markets.\n",
      "Only a few years ago, such threats — along with a recent pipeline failure in the North Sea and storms in the Gulf of Mexico — would have sent the price of crude soaring. Instead, the rise has been muted, and gasoline at the pump remains below $2.60 a gallon across most of the United States.\n",
      "The new energy power also relieves pressure on Washington to act militarily if tensions between Iran and Saudi Arabia break out into war. And it gives Washington the leeway to apply sanctions on other producers — as it has in Russia, and may in Iran or Venezuela — with far less risk to the global economy.\n",
      "It is a striking contrast to the 1970s, when Arab oil boycotts forced motorists to line up for blocks to fill their tanks and the economy went into a tailspin. Even more recently, during the presidency of George W. Bush, domestic oil output was declining so rapidly that the country set a course to replace oil with biofuels like ethanol.\n",
      "Many environmentalists argue that by increasing oil and gas supplies and lowering prices for consumers, shale drilling is extending the life of fossil fuels to the detriment of the environment and the development of cleaner energy.\n",
      "The shale drilling revolution has remade the global energy market, with imports from members of the Organization of the Petroleum Exporting Countries plunging by 20 percent from late 2016 to late 2017. At the same time, exports rose by hundreds of thousands of barrels a day.\n",
      "Nothing like the current situation was foreseen in late 2014, when rising domestic production began weighing on global oil prices.\n",
      "In response, Saudi Arabia led OPEC in a new direction. Instead of throttling back to support prices as the cartel had done so often, it left the market alone and even increased production for a time.\n",
      "Prices fell below $40 a barrel, as the Saudis and their allies hoped to drive American operations out of business by making shale drilling uneconomical. American exploration quickly dropped, but the price squeeze made companies more innovative in the use of drilling technologies, robotics and sensors to maximize output and reduce costs.\n",
      "While scores of smaller companies went out of business, the survivors lengthened horizontal wells to yield more oil, and used clever hedging and drilling strategies to maximize profits even when prices slumped.\n",
      "The response surprised the global oil community. OPEC, Russia and allied producing countries changed course and began cutting back again in 2016.\n",
      "“OPEC missed the point,” said René Ortiz, a former OPEC secretary general and former Ecuadorean energy minister. “They thought they could recover the U.S. market by bringing the prices down. Now the U.S. has gained the leading position in the world oil market regardless of what OPEC does.”\n",
      "“This displacement of Saudi oil, Nigerian oil, Libyan oil and Venezuelan oil,” Mr. Ortiz concluded, “was never anticipated.”\n",
      "A week ago, OPEC leaders met in Oman to discuss a probable extension of production cuts into 2019 to support prices. Their biggest obstacle is the United States.\n",
      "Technological advances unlocking oil from tight rocks like shale has led to a drilling frenzy enabling a doubling of output in a decade, transforming unlikely places like North Dakota and New Mexico into world class petroleum hubs. Pipelines are being built across Texas to serve ports where oil can be pumped onto tankers headed for China, India and other markets.\n",
      "Domestic production last year averaged 9.3 million barrels a day, and the Energy Department projects that the figure will climb to 10.3 million barrels a day this year, surpassing the record set in 1970. In the meantime, since a 40-year export ban was lifted in 2015, exports of American oil have risen to roughly two million barrels a day — more than many OPEC members.\n",
      "The department projects an additional increase in domestic production of 500,000 barrels a day in 2019.\n",
      "Concerns over climate change as well as the growing popularity of electric cars and the eventual aging of the best shale fields will probably curb production and demand over the next few decades. But in the short term, the boom has changed the landscape.\n",
      "The Energy Department projects that the recent surge will hold the price of Brent crude, the global benchmark, to $60 a barrel in 2018 and $61 a barrel in 2019 — a modest increase from $54 last year. (The Brent price rose above $70 a barrel this month, but few analysts see a return to $100-a-barrel oil.)\n",
      "The emerging order in the energy realm is a stable balance of power. Saudi Arabia, which essentially runs OPEC, has put a floor under the oil price — probably around $50 a barrel — with its limits on output and exports over the last four years. But now the United States, by the sheer force of its production, the supremacy of its technology, and an unmatched pipeline, refinery and storage structure, has put a ceiling to the price.\n",
      "Experts note that when oil climbs to $60 a barrel and higher, as it has lately, a drilling rush commences — the national rig count has climbed by over a third in the last year — promising to refill domestic and even global energy inventories. Only a major war or other disruption is likely to send prices soaring.\n",
      "“We have all suffered these depressed prices over the last two years and we are excited to see the new prices and we will respond accordingly,” said Harald Jordan, vice president for engineering at Peak Energy, a Colorado-based producer. “You will see rig activity continue to increase.”\n",
      "URL: https://www.nytimes.com/2018/01/28/business/energy-environment/oil-boom.html\n",
      "TITLE: Oil Boom Gives the U.S. a New Edge in Energy and Diplomacy\n",
      "BODY:\n",
      "HOUSTON — A substantial rise in oil prices in recent months has led to a resurgence in American oil production, enabling the country to challenge the dominance of Saudi Arabia and dampen price pressures at the pump.\n",
      "The success has come in the face of efforts by Saudi Arabia and its oil allies to undercut the shale drilling spree in the United States. Those strategies backfired and ultimately ended up benefiting the oil industry.\n",
      "Overcoming three years of slumping prices proved the resiliency of the shale boom. Energy companies and their financial backers were able to weather market turmoil — and the maneuvers of the global oil cartel — by adjusting exploration and extraction techniques.\n",
      "After a painful shakeout in the industry that included scores of bankruptcies and a significant loss of jobs, a steadier shale-drilling industry is arising, anchored by better-financed companies.\n",
      "With the price of West Texas intermediate crude above $65 a barrel, a level not seen in almost three years, the United States is becoming a dominant producer. It is able to outflank competitors in supplying growing global markets, particularly China and India, while slashing imports from the Middle East and North Africa.\n",
      "This year, the United States is expected to surpass Saudi Arabia and to rival Russia as the world’s leader, with record output of over 10 million barrels a day, according to the International Energy Agency.\n",
      "“This is a 180-degree turn for the United States and the impacts are being felt around the world,” said Daniel Yergin, the economic historian and author of “The Prize: The Epic Quest for Oil, Money and Power.” “This not only contributes to U.S. energy security but also contributes to world energy security by bringing new supplies to the world.”\n",
      "At the same time, the United States is becoming a major exporter of natural gas, another outgrowth of the shale revolution, undercutting Russian energy dominance over Eastern Europe.\n",
      "The improving energy picture comes as the Trump administration is attempting to increase offshore drilling and loosen other regulations on fossil fuel development. But just as the surge in oil and gas production in shale fields during the Barack Obama administration had little to do with Washington, the current rise is the result of private companies responding to global markets.\n",
      "Shale fields can be developed relatively quickly and at modest costs relative to the giant projects, whether on land or offshore, that were once favored by big oil companies. That makes it easier to turn investment spigots on or off to adjust to market fluctuations. Companies like Exxon Mobil and Chevron are putting increasing amounts of capital in shale fields, particularly in West Texas and New Mexico.\n",
      "The results go far beyond the economic, offering Washington strategic weapons once unthinkable. The United States and its allies now have a supply cushion at a time when political turmoil in Venezuela, Libya and Nigeria is threatening to interrupt flows to markets.\n",
      "Only a few years ago, such threats — along with a recent pipeline failure in the North Sea and storms in the Gulf of Mexico — would have sent the price of crude soaring. Instead, the rise has been muted, and gasoline at the pump remains below $2.60 a gallon across most of the United States.\n",
      "The new energy power also relieves pressure on Washington to act militarily if tensions between Iran and Saudi Arabia break out into war. And it gives Washington the leeway to apply sanctions on other producers — as it has in Russia, and may in Iran or Venezuela — with far less risk to the global economy.\n",
      "It is a striking contrast to the 1970s, when Arab oil boycotts forced motorists to line up for blocks to fill their tanks and the economy went into a tailspin. Even more recently, during the presidency of George W. Bush, domestic oil output was declining so rapidly that the country set a course to replace oil with biofuels like ethanol.\n",
      "Many environmentalists argue that by increasing oil and gas supplies and lowering prices for consumers, shale drilling is extending the life of fossil fuels to the detriment of the environment and the development of cleaner energy.\n",
      "The shale drilling revolution has remade the global energy market, with imports from members of the Organization of the Petroleum Exporting Countries plunging by 20 percent from late 2016 to late 2017. At the same time, exports rose by hundreds of thousands of barrels a day.\n",
      "Nothing like the current situation was foreseen in late 2014, when rising domestic production began weighing on global oil prices.\n",
      "In response, Saudi Arabia led OPEC in a new direction. Instead of throttling back to support prices as the cartel had done so often, it left the market alone and even increased production for a time.\n",
      "Prices fell below $40 a barrel, as the Saudis and their allies hoped to drive American operations out of business by making shale drilling uneconomical. American exploration quickly dropped, but the price squeeze made companies more innovative in the use of drilling technologies, robotics and sensors to maximize output and reduce costs.\n",
      "While scores of smaller companies went out of business, the survivors lengthened horizontal wells to yield more oil, and used clever hedging and drilling strategies to maximize profits even when prices slumped.\n",
      "The response surprised the global oil community. OPEC, Russia and allied producing countries changed course and began cutting back again in 2016.\n",
      "“OPEC missed the point,” said René Ortiz, a former OPEC secretary general and former Ecuadorean energy minister. “They thought they could recover the U.S. market by bringing the prices down. Now the U.S. has gained the leading position in the world oil market regardless of what OPEC does.”\n",
      "“This displacement of Saudi oil, Nigerian oil, Libyan oil and Venezuelan oil,” Mr. Ortiz concluded, “was never anticipated.”\n",
      "A week ago, OPEC leaders met in Oman to discuss a probable extension of production cuts into 2019 to support prices. Their biggest obstacle is the United States.\n",
      "Technological advances unlocking oil from tight rocks like shale has led to a drilling frenzy enabling a doubling of output in a decade, transforming unlikely places like North Dakota and New Mexico into world class petroleum hubs. Pipelines are being built across Texas to serve ports where oil can be pumped onto tankers headed for China, India and other markets.\n",
      "Domestic production last year averaged 9.3 million barrels a day, and the Energy Department projects that the figure will climb to 10.3 million barrels a day this year, surpassing the record set in 1970. In the meantime, since a 40-year export ban was lifted in 2015, exports of American oil have risen to roughly two million barrels a day — more than many OPEC members.\n",
      "The department projects an additional increase in domestic production of 500,000 barrels a day in 2019.\n",
      "Concerns over climate change as well as the growing popularity of electric cars and the eventual aging of the best shale fields will probably curb production and demand over the next few decades. But in the short term, the boom has changed the landscape.\n",
      "The Energy Department projects that the recent surge will hold the price of Brent crude, the global benchmark, to $60 a barrel in 2018 and $61 a barrel in 2019 — a modest increase from $54 last year. (The Brent price rose above $70 a barrel this month, but few analysts see a return to $100-a-barrel oil.)\n",
      "The emerging order in the energy realm is a stable balance of power. Saudi Arabia, which essentially runs OPEC, has put a floor under the oil price — probably around $50 a barrel — with its limits on output and exports over the last four years. But now the United States, by the sheer force of its production, the supremacy of its technology, and an unmatched pipeline, refinery and storage structure, has put a ceiling to the price.\n",
      "Experts note that when oil climbs to $60 a barrel and higher, as it has lately, a drilling rush commences — the national rig count has climbed by over a third in the last year — promising to refill domestic and even global energy inventories. Only a major war or other disruption is likely to send prices soaring.\n",
      "“We have all suffered these depressed prices over the last two years and we are excited to see the new prices and we will respond accordingly,” said Harald Jordan, vice president for engineering at Peak Energy, a Colorado-based producer. “You will see rig activity continue to increase.”\n"
     ]
    }
   ],
   "source": [
    "crawler = Crawler()\n",
    "\n",
    "siteData = [\n",
    "    ['O\\'Reilly Media', 'http://oreilly.com', 'h1', 'section#product-description'],\n",
    "    ['Reuters', 'http://reuters.com', 'h1', 'div.StandardArticleBody_body_1gnLA'],\n",
    "    ['Brookings', 'http://www.brookings.edu', 'h1', 'div.post-body'],\n",
    "    ['New York Times', 'http://nytimes.com', 'h1', 'p.story-content']\n",
    "]\n",
    "websites = []\n",
    "for row in siteData:\n",
    "    websites.append(Website(row[0], row[1], row[2], row[3]))\n",
    "\n",
    "crawler.parse(websites[0], 'http://shop.oreilly.com/product/0636920028154.do')\n",
    "crawler.parse(\n",
    "    websites[1], 'http://www.reuters.com/article/us-usa-epa-pruitt-idUSKBN19W2D0')\n",
    "crawler.parse(\n",
    "    websites[2],\n",
    "    'https://www.brookings.edu/blog/techtank/2016/03/01/idea-to-retire-old-methods-of-policy-education/')\n",
    "crawler.parse(\n",
    "    websites[3], \n",
    "    'https://www.nytimes.com/2018/01/28/business/energy-environment/oil-boom.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawling through sites with search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Content:\n",
    "    \"\"\"Common base class for all articles/pages\"\"\"\n",
    "\n",
    "    def __init__(self, topic, url, title, body):\n",
    "        self.topic = topic\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "        self.url = url\n",
    "\n",
    "    def print(self):\n",
    "        \"\"\"\n",
    "        Flexible printing function controls output\n",
    "        \"\"\"\n",
    "        print(\"New article found for topic: {}\".format(self.topic))\n",
    "        print(\"URL: {}\".format(self.url))\n",
    "        print(\"TITLE: {}\".format(self.title))\n",
    "        print(\"BODY:\\n{}\".format(self.body))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Website:\n",
    "    \"\"\"Contains information about website structure\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, searchUrl, resultListing, resultUrl, absoluteUrl, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.searchUrl = searchUrl\n",
    "        self.resultListing = resultListing\n",
    "        self.resultUrl = resultUrl\n",
    "        self.absoluteUrl = absoluteUrl\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GETTING INFO ABOUT: python\n",
      "New article found for topic: python\n",
      "TITLE: JAKARTA (Reuters) - An Indonesian farmer has been found dead inside a 7-meter (23-foot) python after being reported missing on his failure to return home from work on a palm plantation. Village officials cut open the swollen body of the snake in a graphic video taken by a resident of Mamuju, on the eastern island of Sulawesi, the site of the incident. The victim’s legs,  encased in rubber boots, emerge as the snake is pulled apart. Family members and neighbors of the 26-year-old victim, Akbar, had launched a search when he failed to come home for more than 24 hours, a resident of the area told Reuters. A 7-metre (23-foot) python is being cut open to reveal a young Indonesian man in the village of Salobiru, in a remote part of the West Sulawesi province, Indonesia in this still image taken from video on March 26, 2017. Courtesy of Andi Fathir/via REUTERS TV “We saw a python that couldn’t move properly and it’s belly was swollen,” said Asdin Rudi Fathir, 43. “Residents and officials decided to kill the python and were shocked to find a human body inside.”  Relatives identified the victim, who went by only one name, as do many Indonesians.  While large snakes and other animals are common in that part of Sulawesi, attacks on humans are rare, Fathir added. Reporting by Kanupriya Kapoor; Editing by Clarence FernandezOur Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/us-indonesia-python/indonesian-found-dead-after-being-swallowed-by-python-idUSKBN1711E3\n",
      "URL: Indonesian found dead after being swallowed by python\n",
      "New article found for topic: python\n",
      "TITLE: GANYIEL, South Sudan (Thomson Reuters Foundation) - Martha’s fear grows each morning as her toes touch the cold and muddy water of the swamp in a remote part of South Sudan. Yet she steps confidently into the chest-deep marsh, her clothes drenched and her feet sinking into the slippery mud as she holds her schoolbag above her head to keep it dry.  “I know there could be crocodiles, pythons and Guinea worm in here and I‘m scared every day,” said 18-year-old, who is one of thousands who have found safety in Ganyiel, a rebel-held town in the center of the war-torn country. “Some of the children in my village have died in the swamps,” she told the Thomson Reuters Foundation, sitting on a plastic chair outside her classroom in a spare dry dress that she carried with her. Martha has made up her mind that receiving an education is her priority even though the journey takes four hours a day, a quarter of which is spent wading through the treacherous swamp. With 72 percent of children out of school, South Sudan ranks worst in education among all African nations, according to the United Nations children’s agency (UNICEF). One of the most common reasons for non-attendance is the long distance students have to walk to school, it says.  The world’s youngest nation gained independence in 2011, but civil war erupted in late 2013 between soldiers of President Salva Kiir, an ethnic Dinka, and his former vice president, Riek Machar, a Nuer. Tens of thousands have been killed and one-third of the country’s 12 million-strong population has fled their homes, creating Africa’s largest refugee crisis.  School enrollment, which was 42 percent at the start of the war, has plummeted. Only 700,000 school-aged children out of a total of 2.5 million attend classes, UNICEF says.  “With so many children out of school and with a large portion of those currently enrolled unlikely to go beyond primary school, the country will face a serious shortage of qualified people,” its education expert, Vinobajee Gautam, said.  CAUGHT IN THE CROSSFIRE Martha only returned to school last month. She and her siblings had been living in one of the U.N.’s tightly-packed camps for displaced people in the capital, Juba.  “My mother took me and my seven siblings to Juba many years ago, because she wanted us to have access to education,” the teenager explained in perfect English. “When the war broke out, soldiers abducted and killed her. I had to take care of my brothers and sisters and wasn’t able to go to school anymore.” Martha decided to journey northwards for several days by boat to opposition-held Ganyiel in Southern Liech State, so that her father and other relatives who still lived there could help care for her siblings, freeing up her time to study. “I heard that a new school was built in Ganyiel, so I decided that going back home would help our family receive a better education,” she said. Wild animals and waterborne diseases are not the only risks students face on their exhausting journeys to school. “Children from rural areas outside Ganyiel have to live with the fear of being caught in the crossfire of inter-clan fights, revenge killings or even cattle raids,” said Raphael Ndiku of Welthungerhilfe, the German charity that built Martha’s school. With the construction of a new building last year, about 500 new students arrived from remote villages, boosting enrolment by more than half to almost 1,500 children, he said. News of the latest school openings spreads fast. At least one in three schools has been attacked by armed forces since the start of the conflict, according to UNICEF. Many are closed, destroyed or occupied by soldiers or displaced people.  “We have nine classrooms and 37 teachers and use the buildings for primary school students in the mornings and secondary education in the afternoons,” said Ganyiel Mixed School’s headmaster, William Puol. South Sudanese families often make huge personal sacrifices so that their children can learn. “My father made me move to my aunt’s house because he wanted me to get an education,” said 14-year-old James Koang, whose family home was four hours from the nearest school.  “I now walk for less than an hour but I‘m always afraid that wild animals might attack me because I walk alone,” said Koang, the only one of six siblings in education. He has set his mind on attending university.  “If all children could go to school, there would be less fighting in my country,” he said, pointing out that many deadly cattle raids are carried out by youths who did not have the chance to get an education. Being in school gives children some protection from being recruited as child soldiers or forced into other dangerous means of survival, such as selling sex.  “I am sometimes scared of being abducted on my way to school,” Martha said quietly.  “But I want South Sudan to be a strong and peaceful country. Children need to be educated because we are the future. That’s what I think about when I walk to school and it helps me overcome my fear.”  Reporting by Stefanie Glinski. Editing by Katy Migiro. Please credit the Thomson Reuters Foundation, the charitable arm of Thomson Reuters, that covers humanitarian news, women’s rights, trafficking, property rights and climate change. Visit news.trust.org to see more stories.Our Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/us-southsudan-education-children/crocodiles-pythons-and-militias-south-sudans-children-risk-death-for-school-idUSKBN1DR0UP\n",
      "URL: Crocodiles, pythons and militias: South Sudan's children risk death for school\n",
      "New article found for topic: python\n",
      "TITLE: A 20 feet rock python was caught on camera in Junagadh district of India’s western Gujarat state with a swollen stomach after it consumed an antelope on Tuesday (September 20). Residents informed authorities at Girnar Wildlife Sanctuary after they spotted the reptile lying in discomfort in a field. In view of the massive swelling of the python’s stomach, the forest authorities suspect that it gobbled up a full-grown ‘nilgai’ or blue bull. The python - unable to move now - was rescued by the forest personnel and has been put under observation. “We will keep it (python) under observation. We will release it back in the wild once it digests the antelope and the swelling subsides,” said Assistant Conservator of Forest, S.D. Tilala. A blue bull is far larger than an ideal prey for pythons and digesting the mammal could prove to be a great struggle for the reptile. When unable to digest an unusually large prey, pythons are known to regurgitate them.Our Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/us-india-python/python-in-india-demonstrates-huge-appetite-idUSKCN11S04G\n",
      "URL: Python in India demonstrates huge appetite\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: python\n",
      "TITLE: Forest department officials on Saturday (September 24) promised strict action after a python attacked a man when a group was trying to click a photo with it. The incident took place on Friday (September 23) after forest department officials removed the python from the premises of a school in northwestern Rajasthan state. They were posing for pictures with it after successfully capturing it. Deputy Conservator of Forests, K.G.Shrivastav, said the incident was unexpected as all officers were highly trained. “I wanted the snake to be released in my presence, but the officials released it before I could reach the spot. I had no clue that civilians were also present at the spot and something could go wrong as all the officials are highly trained. Clicking selfies with the snake and releasing the photos publicly is an offense under the Wildlife Act and we will issue notice to all involved in this incident,” Shrivastav said. The victim, Ashok Bishnoi, accepted the incident happened due to carelessness on his part. “The moment I received the news I reached the location. I did not realize how close I was to the python and it attacked me. This is entirely my fault,” said the victim, Ashok Bishnoi. Vishnoi added that the mishap happened in a short span of time in spite of the fact that the rescue officials had a strong grip on the reptile. The snake was later released in a forest.Our Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/us-india-python-attack/indian-man-bitten-by-python-whilst-posing-for-selfie-idUSKCN11W1LT\n",
      "URL: Indian man bitten by python whilst posing for selfie\n",
      "New article found for topic: python\n",
      "TITLE: JOHANNESBURG (Reuters) - A Zimbabwean man has been jailed for nine years for eating python, a protected species, a Zimbabwean daily reported on Friday. Archwell Maramba said he had been eating the snake meat for its medicinal value. “I wanted to eat the python to cure my spine and since I ate my first python I have seen remarkable improvement,” NewsDay quoted Maramba as saying during his court case. Acting on a tipoff, police found the 58-year-old drying snakeskin and python meat stashed in his house. Reporting by Helen Nyambura-Mwaura; Editing by Andrew RocheOur Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/us-zimbabwe-python/zimbabwean-jailed-for-nine-years-for-eating-python-meat-idUSKBN0L31PS20150130\n",
      "URL: Zimbabwean jailed for nine years for eating python meat\n",
      "New article found for topic: python\n",
      "TITLE: LONDON (Reuters) - The surviving members of the British comedy troupe Monty Python lost a High Court battle on Friday over tens of thousands of pounds in royalties from their hit Broadway musical “Spamalot”. Mark Forstater, who helped produce the 1975 film “Monty Python and the Holy Grail” on which the stage show was based, said he had not received his fair share of the profits from the spin-off. Despite hearing evidence from three Pythons - Eric Idle, Michael Palin and Terry Jones - the judge, Justice Alastair Norris, sided with the producer. “I have always been adamant I was correct. I have been proved right - justice has prevailed,” Forstater said. Inspired by the film, the musical opened on Broadway in 2005 and has also enjoyed a successful run in Britain. Idle wrote the lyrics and collaborated on most of the music. Forstater, an American based in Britain, argued that he was entitled to one-seventh of the profits from the “Holy Grail” film and any merchandise or spin-offs. His lawyer told the court that for the purposes of profit-sharing, it had been agreed in 1974 that Forstater was “the seventh Python”. However Palin, along with Jones and Idle, who formed Monty Python with John Cleese, Terry Gilliam and Graham Chapman, dismissed this suggestion. “The idea of a seventh Python just doesn’t happen ... I don’t think there was ever any suggestion this man was going to be a ‘seventh Python’,” said Palin, giving evidence in December. In his judgment, Norris said Palin had been a “balanced and trustworthy” witness but had admitted his recollection was “hazy”, while evidence from Jones had been “suffused with a sense that Mr Forstater had done very well out of his brief connection with the Pythons”. “Eric Idle was frank enough to acknowledge that he now disliked Mr Forstater, but he expressed the hope that, in his evidence, he was being honest and that his dislike did not affect his honesty,” Norris said. “He undoubtedly regarded Mr Forstater as ungrateful.” “SOFT LOT” His ruling also made reference to the Pythons’ lack of business acumen, highlighting a diary entry from Palin in 1975. “As we are a soft lot and not at all businesslike, I think it would be in the finest traditions of Python irrationality if we gave Mark an extra 1,000 pounds and a silver tray with some cut-glass sherry glasses and told him to stop writing to us for more money,” Palin wrote. “Beyond that even I am not prepared to go. Oh, all right, some cheese straws to go with the sherry glasses.” The judge added: “As I assess the evidence, the Pythons continued at that point to be ‘a soft lot and not at all businesslike’.” No members of the Python group were present in court to hear the ruling. Final figures will be worked out at later hearings but Forstater told reporters he was entitled to more than 200,000 pounds ($300,000) including interest, the Press Association reported. “There is a sadness, though, about having to face people who were my friends in court,” he said. “The friendship has gone.” Forstater said he did not think the litigation would damage the Pythons’ reputation. “They’re an institution,” he said. “I still think they are very funny.” Editing by Michael Holden and Janet LawrenceOur Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/entertainment-us-britain-python-court/monty-python-lose-spamalot-court-battle-idUSBRE9640KE20130705\n",
      "URL: Monty Python lose 'Spamalot' court battle\n",
      "New article found for topic: python\n",
      "TITLE: ORLANDO, Florida (Reuters) - A python hunting competition starting on Saturday is drawing hundreds of amateurs armed with clubs, machetes and guns to the Florida Everglades, where captured Burmese pythons have exceeded the length of minivans and weighed as much as grown men. Python Challenge 2013, a month-long event sponsored by the Florida Fish and Wildlife Conservation Commission, is open to hunters and non-hunters alike. But the idea of luring weapon-wielding amateurs into the harsh environment of the Everglades has raised some alarms. “I just thought it was as exciting as could be. It’s a once- in-a-lifetime opportunity,” said contestant Ron Polster, a retired salesman from Ohio whose closest encounter with the swamp has been from the highway heading south for the winter. Participants pay a $25 entry fee and take an online training course, which consists mostly of looking at photographs of both the targeted pythons and protected native snakes to learn the difference. The state wildlife agency is offering prizes of $1,500 for the most pythons captured and $1,000 for the longest python. A Burmese python found in Florida last year set records as the largest ever captured in the state at 17-feet, 7-inches. The snake weighed nearly 165 pounds (75 kg). FWC spokeswoman Carli Segelson said the number of registered contestants reached about 500 this week and was growing, with people coming from 32 states. A handout photo released on October 6, 2005 by Everglades National Park shows a dead Burmese python which had swallowed an American alligator. REUTERS/Everglades National Park/HandoutThe stated goal of the competition is to raise awareness of the threat Burmese pythons pose to the Everglades ecosystem. The snakes are native to Southeast Asia and have no known predators in Florida. The contest also serves as a pilot program to determine whether regular hunting competitions can cull the growing population of the invasive species, said Frank Mazzotti, a wildlife expert from the University of Florida who helped create the competition. Python Challenge rules require contestants to kill specimens on the spot in a humane fashion, recommending shooting the snakes precisely through the brain. “I was hoping there would be a lot of machetes and not a lot of guns,” said Polster, the retired salesman. He said he worries “these idiots will be firing all over the place.” Shawn Heflick, star of the National Geographic “Wild” television show “Python Hunters,” told Reuters that despite the formidable size of the snakes, he expects the swamp itself, with its alligators, crocodiles and venomous snakes, to pose a greater threat to the contestants. “You get these people going down there, they’ll get lost, they’ll get dehydrated, they’ll get sucked dry by mosquitoes,” Heflick said. Segelson said the wildlife agency will provide training on the use of GPS devices and on identifying venomous snakes at the kick-off event. In the meantime, she said, contestants should be familiarizing themselves with the Everglades environment, just as they should before entering any other strange territory. Heflick said most of the contestants likely were drawn to the Python Challenge by the romantic mystique of bagging a giant predator. He expects few will last long in the hunt. “The vast majority of them will never see a python. The vast majority of them will probably curtail their hunting very quickly when they figure out there’s a lot of mosquitoes, it’s hot, it’s rather boring sometimes - most of the time really, and I think a lot of them will go home,” Heflick said. Editing by Tom Brown and Dan GreblerOur Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/us-usa-python-hunt/florida-python-hunting-contest-draws-hundreds-idUSBRE9081CL20130110\n",
      "URL: Florida python hunting contest draws hundreds\n",
      "New article found for topic: python\n",
      "TITLE: ORLANDO, Florida (Reuters) - A python hunting competition starting on Saturday is drawing hundreds of amateurs armed with clubs, machetes and guns to the Florida Everglades, where captured Burmese pythons have exceeded the length of minivans and weighed as much as grown men. Python Challenge 2013, a month-long event sponsored by the Florida Fish and Wildlife Conservation Commission, is open to hunters and non-hunters alike. But the idea of luring weapon-wielding amateurs into the harsh environment of the Everglades has raised some alarms. “I just thought it was as exciting as could be. It’s a once- in-a-lifetime opportunity,” said contestant Ron Polster, a retired salesman from Ohio whose closest encounter with the swamp has been from the highway heading south for the winter. Participants pay a $25 entry fee and take an online training course, which consists mostly of looking at photographs of both the targeted pythons and protected native snakes to learn the difference. The state wildlife agency is offering prizes of $1,500 for the most pythons captured and $1,000 for the longest python. A Burmese python found in Florida last year set records as the largest ever captured in the state at 17-feet, 7-inches. The snake weighed nearly 165 pounds (75 kg). FWC spokeswoman Carli Segelson said the number of registered contestants reached about 500 this week and was growing, with people coming from 32 states. The stated goal of the competition is to raise awareness of the threat Burmese pythons pose to the Everglades ecosystem. The snakes are native to Southeast Asia and have no known predators in Florida. The contest also serves as a pilot program to determine whether regular hunting competitions can cull the growing population of the invasive species, said Frank Mazzotti, a wildlife expert from the University of Florida who helped create the competition. Python Challenge rules require contestants to kill specimens on the spot in a humane fashion, recommending shooting the snakes precisely through the brain. “I was hoping there would be a lot of machetes and not a lot of guns,” said Polster, the retired salesman. He said he worries “these idiots will be firing all over the place.” Shawn Heflick, star of the National Geographic “Wild” television show “Python Hunters,” told Reuters that despite the formidable size of the snakes, he expects the swamp itself, with its alligators, crocodiles and venomous snakes, to pose a greater threat to the contestants. “You get these people going down there, they’ll get lost, they’ll get dehydrated, they’ll get sucked dry by mosquitoes,” Heflick said. Segelson said the wildlife agency will provide training on the use of GPS devices and on identifying venomous snakes at the kick-off event. In the meantime, she said, contestants should be familiarizing themselves with the Everglades environment, just as they should before entering any other strange territory. Heflick said most of the contestants likely were drawn to the Python Challenge by the romantic mystique of bagging a giant predator. He expects few will last long in the hunt. “The vast majority of them will never see a python. The vast majority of them will probably curtail their hunting very quickly when they figure out there’s a lot of mosquitoes, it’s hot, it’s rather boring sometimes - most of the time really, and I think a lot of them will go home,” Heflick said. Editing by Tom Brown and Dan GreblerOur Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/us-usa-python-hunt/florida-python-hunting-contest-draws-hundreds-idUSBRE9081CL20130109\n",
      "URL: Florida python hunting contest draws hundreds\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: python\n",
      "TITLE: MIAMI (Reuters) - Engineers in the Everglades stumbled upon a near-record-breaking Burmese python measuring more than 18 feet long during a routine inspection of levees on Tuesday, a water management district spokesman said. The snake, measuring at 18 feet 2 inches, fell short of the state record by 6 inches, according to the Florida Fish and Wildlife Conservation Commission. Last year, a snake collector in the state discovered the largest python on record there, measuring 18 feet 8 inches, commission spokeswoman Katie Johnson said. The pythons, which can grow to more than 20 feet in their native habitat in Southeast Asia, are one of the most problematic invaders of Florida’s sprawling Everglades wetlands. A near record-breaking Burmese Python measuring more than 18-feet long (5.5 meters) is shown in this January 4, 2014 handout photo provided by South Florida Water Management District January 5, 2014 in Everglades National Park near Miami, Florida. REUTERS/South Florida Water Management District/Handout via ReutersThey eat indigenous species and their food sources, fueling concerns that the predator snakes will fundamentally change the ecosystem. The python found on Tuesday was killed, and its corpse was taken to the University of Florida, where it will be measured and studied by scientists trying to combat the species, according to South Florida Water Management District spokesman Gabe Margasak. Officials have said the python population is believed to have grown to as many as 150,000 in the Everglades. The cold-blooded reptiles are often found atop levees, where they lie for hours at a time to warm up under the Florida sun. The snakes, one of the largest species in the world, found a home to their liking in the Everglades when pet owners started using the wetlands as a convenient dumping ground. Reporting by Zachary Fagenson; Editing by Colleen Jenkins and Lisa Von AhnOur Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/us-usa-florida-python/18-foot-burmese-python-found-in-florida-everglades-idUSBREA141OP20140205\n",
      "URL: 18-foot Burmese python found in Florida Everglades\n",
      "New article found for topic: python\n",
      "TITLE: LONDON, Nov 20 (Reuters) - The comic team Monty Python, whose BBC TV series from the 1970s and feature films took their subversive humour and “Dead Parrot” routine around the world, are to reunite for a stage show, British media reported on Wednesday. A news release issued on behalf of the five surviving Pythons, Eric Idle, John Cleese, Terry Gilliam, Michael Palin and Terry Jones, all in their 70s, said that an official announcement would be made on Thursday. But several British newspapers and media outlets reported that the five would be appearing on stage for the first time together since the 1980s. The group was famed for its skits about a man trying to return a dead parrot to a shopkeeper who claimed the bird was “resting” and for poking fun at the establishment, the military and religion. “We’re getting together and putting on a show - it’s real,” Jones told the BBC. “I‘m quite excited about it. I hope it makes us a lot of money. I hope to be able to pay off my mortgage!” On his Twitter account, Palin wrote: “The Python rides again...?” while Idle on Tuesday tweeted: “Only three days to go till the Python Press Conference. Make sure Python fans are alerted to the big forthcoming news event.” The BBC programme “Monty Python’s Flying Circus” was made for television between 1969 and 1974. The Pythons went on to make films including “Monty Python And The Holy Grail” (1975) and “Monty Python’s Life of Brian” (1979). They last performed live together in Monty Python Live At The Hollywood Bowl in 1982. The sixth Python, Graham Chapman, died of cancer at age 48 in 1989.   (Writing by Michael Roddy; Editing by Doina Chiacu)Our Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/britain-python/monty-python-not-dead-after-all-stage-show-planned-idUSL5N0J50QB20131120\n",
      "URL: Monty Python not dead after all - stage show planned\n",
      "GETTING INFO ABOUT: data science\n",
      "New article found for topic: data science\n",
      "TITLE: Dec 20 (Reuters) - Veritone Inc: * VERITONE ACQUIRES ADVANCED DATA SCIENCE AND MACHINE LEARNING FIRM * VERITONE INC - ACQUIRED ADVANCED DATA ANALYTICS SOFTWARE AND RELATED IP ASSETS OF ATIGEO CORP Source text for Eikon: Further company coverage:Our Standards:The Thomson Reuters Trust Principles.\n",
      "BODY:\n",
      "/article/brief-veritone-acquires-advanced-data-sc/brief-veritone-acquires-advanced-data-science-and-machine-learning-firm-idUSASB0BYPZ\n",
      "URL: BRIEF-Veritone Acquires Advanced Data Science And Machine Learning Firm\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    375\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Python 2.7, use buffering of HTTP responses\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 376\u001b[0;31m                 \u001b[0mhttplib_response\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbuffering\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    377\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Python 2.6 and older\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: getresponse() got an unexpected keyword argument 'buffering'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-48-360a87384216>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     55\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"GETTING INFO ABOUT: \"\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mtopic\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mtargetSite\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msites\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 57\u001b[0;31m         \u001b[0mcrawler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtopic\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtargetSite\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-48-360a87384216>\u001b[0m in \u001b[0;36msearch\u001b[0;34m(self, topic, site)\u001b[0m\n\u001b[1;32m     29\u001b[0m                 \u001b[0mbs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetPage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 31\u001b[0;31m                 \u001b[0mbs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetPage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     32\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mbs\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     33\u001b[0m                 \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Something was wrong with that page or URL. Skipping!\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-48-360a87384216>\u001b[0m in \u001b[0;36mgetPage\u001b[0;34m(self, url)\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mgetPage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m             \u001b[0mreq\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRequestException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(url, params, **kwargs)\u001b[0m\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m     \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetdefault\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'allow_redirects'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 67\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'get'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     68\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(method, url, **kwargs)\u001b[0m\n\u001b[1;32m     51\u001b[0m     \u001b[0;31m# cases, and look like a memory leak in others.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     52\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0msessions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 53\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     54\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     55\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m    466\u001b[0m         }\n\u001b[1;32m    467\u001b[0m         \u001b[0msend_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 468\u001b[0;31m         \u001b[0mresp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    469\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    470\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m    595\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    596\u001b[0m         \u001b[0;31m# Resolve redirects if allowed.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 597\u001b[0;31m         \u001b[0mhistory\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mresp\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mresp\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mgen\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mallow_redirects\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    598\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    599\u001b[0m         \u001b[0;31m# Shuffle things around if there's history.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m    595\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    596\u001b[0m         \u001b[0;31m# Resolve redirects if allowed.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 597\u001b[0;31m         \u001b[0mhistory\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mresp\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mresp\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mgen\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mallow_redirects\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    598\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    599\u001b[0m         \u001b[0;31m# Shuffle things around if there's history.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mresolve_redirects\u001b[0;34m(self, resp, req, stream, timeout, verify, cert, proxies, **adapter_kwargs)\u001b[0m\n\u001b[1;32m    193\u001b[0m                 \u001b[0mproxies\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mproxies\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    194\u001b[0m                 \u001b[0mallow_redirects\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 195\u001b[0;31m                 \u001b[0;34m**\u001b[0m\u001b[0madapter_kwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    196\u001b[0m             )\n\u001b[1;32m    197\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m    574\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    575\u001b[0m         \u001b[0;31m# Send the request\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 576\u001b[0;31m         \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    577\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    578\u001b[0m         \u001b[0;31m# Total elapsed time of the request (approximately)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m    374\u001b[0m                     \u001b[0mdecode_content\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    375\u001b[0m                     \u001b[0mretries\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_retries\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 376\u001b[0;31m                     \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    377\u001b[0m                 )\n\u001b[1;32m    378\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, **response_kw)\u001b[0m\n\u001b[1;32m    557\u001b[0m             httplib_response = self._make_request(conn, method, url,\n\u001b[1;32m    558\u001b[0m                                                   \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout_obj\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 559\u001b[0;31m                                                   body=body, headers=headers)\n\u001b[0m\u001b[1;32m    560\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    561\u001b[0m             \u001b[0;31m# If we're going to release the connection in ``finally:``, then\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    376\u001b[0m                 \u001b[0mhttplib_response\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbuffering\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    377\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Python 2.6 and older\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 378\u001b[0;31m                 \u001b[0mhttplib_response\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    379\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mSocketTimeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mBaseSSLError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mSocketError\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    380\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_raise_timeout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout_value\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mread_timeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36mgetresponse\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1329\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1330\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1331\u001b[0;31m                 \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbegin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1332\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1333\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36mbegin\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    295\u001b[0m         \u001b[0;31m# read until we get a non-100 response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    296\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 297\u001b[0;31m             \u001b[0mversion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreason\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    298\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mstatus\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mCONTINUE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    299\u001b[0m                 \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36m_read_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    256\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    257\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 258\u001b[0;31m         \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_MAXLINE\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"iso-8859-1\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    259\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0m_MAXLINE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    260\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mLineTooLong\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"status line\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    584\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    585\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 586\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    587\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    588\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mrecv_into\u001b[0;34m(self, buffer, nbytes, flags)\u001b[0m\n\u001b[1;32m   1000\u001b[0m                   \u001b[0;34m\"non-zero flags not allowed in calls to recv_into() on %s\"\u001b[0m \u001b[0;34m%\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1001\u001b[0m                   self.__class__)\n\u001b[0;32m-> 1002\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnbytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1003\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1004\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnbytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m    863\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Read on closed or unwrapped SSL socket.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    864\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 865\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    866\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mSSLError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    867\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mSSL_ERROR_EOF\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msuppress_ragged_eofs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m    623\u001b[0m         \"\"\"\n\u001b[1;32m    624\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mbuffer\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 625\u001b[0;31m             \u001b[0mv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    626\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    627\u001b[0m             \u001b[0mv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "class Crawler:\n",
    "\n",
    "    def getPage(self, url):\n",
    "        try:\n",
    "            req = requests.get(url)\n",
    "        except requests.exceptions.RequestException:\n",
    "            return None\n",
    "        return BeautifulSoup(req.text, 'html.parser')\n",
    "\n",
    "    def safeGet(self, pageObj, selector):\n",
    "        childObj = pageObj.select(selector)\n",
    "        if childObj is not None and len(childObj) > 0:\n",
    "            return childObj[0].get_text()\n",
    "        return \"\"\n",
    "\n",
    "    def search(self, topic, site):\n",
    "        \"\"\"\n",
    "        Searches a given website for a given topic and records all pages found\n",
    "        \"\"\"\n",
    "        bs = self.getPage(site.searchUrl + topic)\n",
    "        searchResults = bs.select(site.resultListing)\n",
    "        for result in searchResults:\n",
    "            url = result.select(site.resultUrl)[0].attrs[\"href\"]\n",
    "            # Check to see whether it's a relative or an absolute URL\n",
    "            if(site.absoluteUrl):\n",
    "                bs = self.getPage(url)\n",
    "            else:\n",
    "                bs = self.getPage(site.url + url)\n",
    "            if bs is None:\n",
    "                print(\"Something was wrong with that page or URL. Skipping!\")\n",
    "                return\n",
    "            title = self.safeGet(bs, site.titleTag)\n",
    "            body = self.safeGet(bs, site.bodyTag)\n",
    "            if title != '' and body != '':\n",
    "                content = Content(topic, title, body, url)\n",
    "                content.print()\n",
    "\n",
    "\n",
    "crawler = Crawler()\n",
    "\n",
    "siteData = [\n",
    "    ['O\\'Reilly Media', 'http://oreilly.com', 'https://ssearch.oreilly.com/?q=',\n",
    "        'article.product-result', 'p.title a', True, 'h1', 'section#product-description'],\n",
    "    ['Reuters', 'http://reuters.com', 'http://www.reuters.com/search/news?blob=', 'div.search-result-content',\n",
    "        'h3.search-result-title a', False, 'h1', 'div.StandardArticleBody_body_1gnLA'],\n",
    "    ['Brookings', 'http://www.brookings.edu', 'https://www.brookings.edu/search/?s=',\n",
    "        'div.list-content article', 'h4.title a', True, 'h1', 'div.post-body']\n",
    "]\n",
    "sites = []\n",
    "for row in siteData:\n",
    "    sites.append(Website(row[0], row[1], row[2],\n",
    "                         row[3], row[4], row[5], row[6], row[7]))\n",
    "\n",
    "topics = ['python', 'data science']\n",
    "for topic in topics:\n",
    "    print(\"GETTING INFO ABOUT: \" + topic)\n",
    "    for targetSite in sites:\n",
    "        crawler.search(topic, targetSite)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawling Sites through Links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Website:\n",
    "\n",
    "    def __init__(self, name, url, targetPattern, absoluteUrl, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.targetPattern = targetPattern\n",
    "        self.absoluteUrl = absoluteUrl\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag\n",
    "\n",
    "\n",
    "class Content:\n",
    "\n",
    "    def __init__(self, url, title, body):\n",
    "        self.url = url\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "\n",
    "    def print(self):\n",
    "        print(\"URL: {}\".format(self.url))\n",
    "        print(\"TITLE: {}\".format(self.title))\n",
    "        print(\"BODY:\\n{}\".format(self.body))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GETTING https://www.reuters.com\n",
      "GETTING https://www.reuters.com/article/us-usa-trump-5g/trump-national-security-team-sees-building-5g-network-as-option-idUSKBN1FH103\n",
      "URL: https://www.reuters.com/article/us-usa-trump-5g/trump-national-security-team-sees-building-5g-network-as-option-idUSKBN1FH103\n",
      "TITLE: Trump security team sees building U.S. 5G network as option\n",
      "BODY:\n",
      "WASHINGTON (Reuters) - President Donald Trump’s national security team is looking at options to counter the threat of China spying on U.S. phone calls that include the government building a super-fast 5G wireless network, a senior administration official said on Sunday. The official, confirming the gist of a report from Axios.com, said the option was being debated at a low level in the administration and was six to eight months away from being considered by the president himself. The 5G network concept is aimed at addressing what officials see as China’s threat to U.S. cyber security and economic security. The Trump administration has taken a harder line on policies initiated by predecessor Barack Obama on issues ranging from Beijing’s role in restraining North Korea to Chinese efforts to acquire U.S. strategic industries. Earlier this month, AT&T; was forced to scrap a plan to offer its customers handsets built by China’s Huawei after some members of Congress lobbied against the idea with federal regulators, sources told Reuters.  In 2012, Huawei and ZTE Corp were the subject of a U.S. investigation into whether their equipment provided an opportunity for foreign espionage and threatened critical U.S. infrastructure.  Some members of the House intelligence committee remain troubled by security threats posed by Huawei and ZTE, according to a congressional aide. Issues raised in a 2012 committee report about the Chinese firms have “never subsided,” the aide said, adding that there was newer classified intelligence that recently resurfaced those concerns. “We want to build a network so the Chinese can’t listen to your calls,” the senior official told Reuters. “We have to have a secure network that doesn’t allow bad actors to get in. We also have to ensure the Chinese don’t take over the market and put every non-5G network out of business.” Major wireless carriers have spent billions of dollars buying spectrum to launch 5G networks, and it is unclear if the U.S. government would have enough spectrum to build its own 5G network. Furthermore, Accenture has estimated that wireless operators will invest as much as $275 billion in the United States over seven years as they build out 5G. Last year, T-Mobile US Inc spent $8 billion and Dish Network Corp $6.2 billion to win the bulk of broadcast airwaves spectrum for sale in a government auction.  An AT&T; spokesman said they could not comment on something they have not seen, and added: “Thanks to multi-billion dollar investments made by American companies, the work to launch 5G service in the United States is already well down the road.” Later this year, AT&T; is set to be the first to launch mobile 5G service in 12 U.S. locations, the spokesman said. A Verizon spokesman declined to comment. Representatives for Sprint and T-Mobile did not immediately respond to requests for comment.     Another option includes having a 5G network built by a consortium of wireless carriers, the U.S. official said. “We want to build a secure 5G network and we have to work with industry to figure out the best way to do it,” the official said, speaking on condition of anonymity. Axios published documents it said were from a presentation from a National Security Council official. If the government built the network, it would rent access to carriers, Axios said. A looming concern laid out in the presentation was China’s growing presence in the manufacture and operation of wireless networks. A concerted government push could help the U.S. compete on that front, according to the presentation. A 5G network is expected to offer significantly faster speeds, more capacity and shorter response times, which could be utilized for new technologies ranging from self-driving cars to remote surgeries. Telecom companies and their suppliers consider it to be a multibillion-dollar revenue opportunity. Reporting by Steve Holland and Pete Schroeder; Additional reporting by Duston Volz, Suzanne Barlyn and David Shepardson; Editing by Chris Sanders, Peter Cooney and Cynthia OstermanOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-trump-effect-nafta-farmers/u-s-farmers-have-much-to-lose-if-nafta-deal-collapses-idUSKBN1FH0O0\n",
      "URL: https://www.reuters.com/article/us-trump-effect-nafta-farmers/u-s-farmers-have-much-to-lose-if-nafta-deal-collapses-idUSKBN1FH0O0\n",
      "TITLE: U.S. farmers have much to lose if NAFTA deal collapses\n",
      "BODY:\n",
      "CHICAGO/MONTREAL (Reuters) - A collapse of the North American Free Trade Agreement (NAFTA), which U.S. President Donald Trump has threatened to scrap, could create the most profound disruption for U.S. farmers who produce grains, meats and dairy products sold to Canada and Mexico. Blake Erwin, a third-generation American who raises cattle, corn and soybeans in Dixon, Nebraska, said on Saturday that he is not closely monitoring the negotiations, but that he hopes the outcome will support U.S. farmers who are struggling to make a living due to low commodities prices, rising healthcare costs and high property taxes.  “A trade agreement has to be fair for the United States, but we also want to keep those exports going for the farmer,” said Erwin, 34. “We don’t want to mess up any good things we got going.”  Erwin spoke to Reuters over the weekend as U.S., Canadian and Mexican negotiators met in Montreal for the sixth of seven planned rounds of talks to revamp the 1994 pact. U.S. farmers and exporters are fighting to preserve their exports at a time when Canada is finding customers in new markets. They also face strained relations between the United States and Mexico, a major buyer of U.S. corn, wheat, beef, pork and dairy products. “The U.S. is behaving so badly it’s going to create opportunities for Canadian agriculture,” Iowa State University economist Dermot Hayes said last week during a visit to Winnipeg.  Trade flows have already begun to shift. The United States remains the dominant grain supplier to Mexico. Yet Mexico imported 583,000 metric tonnes of corn from Brazil in 2017, a 980 percent jump from the previous year, according to Mexican government trade data. Mexican imports of U.S. soybean meal, used to feed chickens and livestock, fell 29 percent in the first 11 months of 2017, compared with the same period the previous year, according to the U.S. Department of Agriculture.  ‘MORE IMPORTANT THAN PRICE’  Slideshow (17 Images)Trump’s animosity toward Mexico and complaints over trade imbalances have pushed longtime buyers to work with new suppliers and expand existing relationships in South America, the European Union and other regions, trade experts said. “You get partners who build a bond and get real comfortable working together. We’re starting to see that bond becoming more important than price for where countries are buying grains,” said Karl Setzer, risk management team leader for MaxYield Cooperative. Case in point: A rare 30,000-tonne shipment of Brazilian corn steamed its way in November to grain terminals in the state of Veracruz, Mexico, operated by agribusiness heavyweights Cargill Inc [CARG.UL] and Archer Daniels Midland Co (ADM.N).  Despite a steep decline in U.S. corn prices, with stocks sitting at a historic high, the buyer paid a premium for the Brazilian grain - as much as $2 more per tonne, according to trade sources.  A Cargill spokeswoman said the company had no immediate comment. ADM did not respond to requests for comment. Canada last week agreed to join the new version of the Trans-Pacific Partnership, part of a broad effort to court new trade partners. “The tough NAFTA negotiations have convinced Canada that we have to have a number of trading partners, not just one,” said Ron Bonnett, a beef farmer and president of the Canadian Federation of Agriculture.  The revised TPP, known as the Comprehensive and Progressive Agreement for Trans-Pacific Partnership, will reduce tariffs on Canadian pork, beef and wheat to Japan and other markets, in some cases eliminating duties altogether. Darci Vetter, former U.S. chief agriculture negotiator, said if the talks dragged on past March they might not end until next year, making it more challenging to sell American farm products. “Other trade agreements will be implemented, buyers of U.S. products in Mexico and Canada won’t be sure that we are a good long-term bet, and so we’re likely to see our clients react accordingly,” she told a panel on NAFTA in Montreal on Friday. Reporting by P.J. Huffstutter in Chicago and David Ljunggren in Montreal; Writing and additional reporting by Rod Nickel in Winnipeg, Manitoba; Additional reporting by Lucas Jackson in Dixon, Nebraska; Editing by Jim Finkle and Daniel WallisOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-afghanistan-blast/militants-attack-afghan-army-post-near-military-academy-in-capital-idUSKBN1FI07M\n",
      "URL: https://www.reuters.com/article/us-afghanistan-blast/militants-attack-afghan-army-post-near-military-academy-in-capital-idUSKBN1FI07M\n",
      "TITLE: Militants attack Afghan army post near military academy in capital\n",
      "BODY:\n",
      "KABUL (Reuters) - At least four militants attacked an army outpost near one of Afghanistan’s main military academies on Monday and at least one soldier was killed and three wounded, a defense ministry official said. The attack in the western outskirts of the capital, Kabul, came two days after an ambulance bomb in the center of the city killed more than 100 people and just over a week after another attack on the Hotel Intercontinental killed more than 20. Both of those attacks were claimed by the Taliban. Ministry of Defence officials said the militants attacked the outpost near the well-defended Marshal Fahim military academy just before dawn. One of the attackers blew himself up, one had been killed and two were still fighting. One soldier had been killed and three were wounded, said one official, who declined to be identified. Ministry spokesman Dawlat Waziri said three soldiers had been wounded and the clash was still going on. Earlier, resident Mohammad Ehsan said he heard a series of explosions coming from the area beginning at around 5 a.m. and lasting for at least an hour.  Smaller blasts could still be heard at less frequent intervals. In October, a suicide attacker rammed a car full of explosives into a bus carrying cadets from the defense university, which is home to one of Afghanistan’s main officer training schools, killing 15 of them. Reporting by Omar Sobhani; Writing by James Mackenzie; Editing by Robert Birsel and Paul TaitOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-usa-immigration-manchin/democratic-senator-criticizes-pelosis-immigration-comment-idUSKBN1FH0RC\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.reuters.com/article/us-usa-immigration-manchin/democratic-senator-criticizes-pelosis-immigration-comment-idUSKBN1FH0RC\n",
      "TITLE: Democratic senator criticizes Pelosi's immigration comment\n",
      "BODY:\n",
      "WASHINGTON (Reuters) - U.S. Senator Joe Manchin, a moderate Democrat, said on Sunday he thought a new White House immigration plan was a good starting point, and he criticized House Democratic leader Nancy Pelosi for dismissing it as a way to “make America white again.” “We don’t need that type of rhetoric on either side, from Nancy, (Republican House Speaker) Paul Ryan or anybody else,” said Manchin, a West Virginian and a leader of a bipartisan Senate group working on immigration. He spoke on CNN’s “State of the Union” program. Manchin’s comments highlighted differences among Democrats ahead of a Feb. 8 deadline for the U.S. Congress to pass another spending bill and try to reach an immigration agreement that would also protect up to 1.8 million illegal immigrants brought to the United States as children, a group known as “Dreamers,” from deportation. Senior White House officials outlined an immigration plan on Thursday that would offer Dreamers a path to citizenship. The proposal also would curb some legal immigration programs and build a border wall with Mexico. The White House described the language on Dreamers as a major concession to Democrats, but leading Democrats quickly dismissed the plan as a non-starter. FILE PHOTO - House Minority Leader Nancy Pelosi (D-CA) speaks during a news conference after President Donald Trump and the U.S. Congress failed to reach a deal on funding for federal agencies on Capitol Hill in Washington, U.S., January 20, 2018.  REUTERS/Joshua Roberts Pelosi said it held Dreamers “hostage to a hateful anti-immigrant scheme” and accused the administration of Republican President Donald Trump of a campaign “to make America white again.” In a separate interview on Sunday on NBC’s “Meet the Press” program, Manchin said he thought the White House plan was “a good starting point.”   Manchin said the bipartisan group that he and moderate Republican Senator Susan Collins lead would meet on Monday evening to examine the White House immigration outline. “I think we can find a pathway forward; I really do,” Manchin told NBC. Collins, speaking to CBS’s “Face the Nation” on Sunday, said the group hoped to influence a proposal that the Senate’s two whips, Republican John Cornyn and Democrat Dick Durbin, are trying to assemble. “If (Cornyn and Durbin) agree, I have a feeling that that will be a bill that can go all the way to the president’s desk, and that’s our goal,” Collins said.  The group of more than 20 senators from both parties, which has been dubbed the “Common Sense Coalition,” helped to end a three-day U.S. government shutdown last week. Reporting by Susan Cornwell; Editing by Jeffrey Benkoe and Lisa Von AhnOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-usa-trump-russia/republican-senators-urge-congress-to-revisit-bill-to-protect-mueller-idUSKBN1FH0SG\n",
      "URL: https://www.reuters.com/article/us-usa-trump-russia/republican-senators-urge-congress-to-revisit-bill-to-protect-mueller-idUSKBN1FH0SG\n",
      "TITLE: Republican senators urge Congress to revisit bill to protect Mueller\n",
      "BODY:\n",
      "WASHINGTON (Reuters) - The U.S. Congress should revisit proposed legislation to protect special counsel Robert Mueller after President Donald Trump tried to fire him last year while he investigated the Trump campaign’s ties with Russia, two Republican senators said on Sunday. In separate television interviews, Senators Susan Collins and Lindsey Graham expressed dismay at reports the Republican president had told the top White House lawyer to order U.S. Justice Department officials to fire Mueller. “I’ve got legislation protecting Mr. Mueller, and I’d be glad to pass it tomorrow,” Graham told the ABC News “This Week” program.  On CNN’s “State of the Union,” Collins said: “It certainly wouldn’t hurt to put that extra safeguard in place, given the latest stories.”  Tensions over Mueller’s probe are hovering over Trump’s year-old presidency as he prepares to give his first State of the Union Address on Tuesday. The New York Times reported on Thursday that Trump ordered White House counsel Donald McGahn to fire Mueller in June but backed down after McGahn threatened to resign rather than carry out the order. McGahn was “fed up” after Trump’s order, a person familiar with the matter told Reuters. He did not issue an ultimatum directly to the president but told then-White House chief of staff Reince Priebus and then-chief strategist Steve Bannon that he wanted to quit, the source said. FILE PHOTO: Special Counsel Robert Mueller (R) departs after briefing members of the U.S. Senate on his investigation into potential collusion between Russia and the Trump campaign on Capitol Hill in Washington, U.S., June 21, 2017.   REUTERS/Joshua Roberts/File PhotoGraham and three Democratic senators introduced legislation last August that would protect special counsels, including Mueller, by requiring that a panel of federal judges review any action to remove them. The likelihood that such a bill would become law have seemed remote. Republicans in the U.S. House of Representatives have shown little enthusiasm for the idea. “I don’t think there’s a need for legislation right now to protect Mueller,” House Republican leader Kevin McCarthy said on NBC’s “Meet the Press” program. “The president and his team have fully cooperated” with the special counsel, he said. U.S. President Donald Trump looks on during the World Economic Forum (WEF) annual meeting in Davos, Switzerland January 26, 2018. REUTERS/Carlos Barria Republicans hold the majority in both the House and Senate. Mueller is investigating whether Trump associates and the Kremlin colluded during the 2016 presidential election. Russia denies such collusion, and Trump frequently denounces the probe as a “witch hunt.” Both Collins and Graham said they saw no sign that Trump is currently trying to fire Mueller. “I think what happened here is the president had a bad idea,” Collins said. “He talked with his counsel, who explained to an angry and frustrated president why it was a bad idea.” Graham said: “It’s pretty clear to me everyone in the White House knows it’d be the end of President Trump’s presidency if he tried to fire Mr. Mueller.” Reporting by Yasmeen Abutaled and Caren Bohan Writing by Warren Strobel Editing by Lisa Von AhnOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-china-economy-growth/china-eyes-black-swans-gray-rhinos-as-2018-growth-seen-slowing-to-6-5-6-8-percent-media-idUSKBN1FI0AY?il=0\n",
      "URL: https://www.reuters.com/article/us-china-economy-growth/china-eyes-black-swans-gray-rhinos-as-2018-growth-seen-slowing-to-6-5-6-8-percent-media-idUSKBN1FI0AY?il=0\n",
      "TITLE: China eyes black swans, gray rhinos as 2018 growth seen slowing to 6.5-6.8 percent - media\n",
      "BODY:\n",
      "BEIJING (Reuters) - China’s economic growth will likely slow to 6.5-6.8 percent this year, a senior official at the country’s top economic planner wrote in the Beijing Daily on Monday, while warning about the risks of “Black Swan” and “Gray Rhino” events.  Black swans, or unforeseen occurrences, and gray rhinos, or highly possible yet ignored threats, are likely to occur this year with adverse consequences, Fan Hengshan, vice secretary general of the National Development and Reform Commission (NDRC), wrote in a commentary in the state-controlled newspaper.  China’s economy grew 6.9 percent in 2017, the first annual acceleration since 2010. That pace easily beat the government’s 2017 target of around 6.5 percent, welcome news for policymakers looking to curb financial risks and cut corporate debt. “My personal opinion is that economic growth this year is very likely to exceed 6.5 percent, roughly between 6.5 and 6.8 percent,” Fan said.  Analysts polled by Reuters earlier this month also predicted a slowdown to 6.5 percent this year as government-led crackdowns on debt risks and factory pollution drag on overall activity. China will focus on fending off risks this year, particularly risks that will impact regions and cause systemic fluctuations, Fan said.  “To this end, we must remain highly vigilant and enhance our sense of urgency,” Fan said.  Earlier this month, China’s banking regulator chief told the official People’s Daily in an interview that a black swan event could threaten the country’s financial stability, adding that risks, while still manageable, are “complex and serious.” Reporting by Stella Qiu and Ryan Woo; Editing by Kim CoghillOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-australia-defence/australia-to-spend-3-1-billion-to-increase-stake-in-global-arms-exports-idUSKBN1FI08I?il=0\n",
      "URL: https://www.reuters.com/article/us-australia-defence/australia-to-spend-3-1-billion-to-increase-stake-in-global-arms-exports-idUSKBN1FI08I?il=0\n",
      "TITLE: Australia to spend $3.1 billion to increase stake in global arms exports\n",
      "BODY:\n",
      "SYDNEY (Reuters) - Prime Minister Malcolm Turnbull said on Monday Australian military equipment manufacturers will be offered government-backed loans as part of a A$3.8 billion ($3.1 billion) package to become one of the world’s top 10 defense exporters. Australia said in 2016 it would boost defense spending by A$30 billion by 2021, purchasing frigates, armored personnel carriers, strike fighter jets, drones and a fleet of new submarines - many of which would be built at home. The defense industry has struggled to obtain finance from traditional lenders that have been unwilling to fund the arms industry, so Australia has created a A$3.8 billion loan scheme for companies seeking finance to export military equipment.  “Australia is around the 20th largest exporter. Given the size of our defense budget we should be higher up the scale than that,” Turnbull told reporters in Sydney. “The goal is to get into the top 10,” he said. Christopher Pyne, the minister for the defense industry, said Australia would target sales to the United States, Canada, Britain and New Zealand. Australia’s annual defense budget was worth A$34.6 billion this year. The scheme is also meant to arrest a slide in Australia’s manufacturing sector and provide some support for its economy, which has been hampered by record-low wage growth. Australia saw a record number of jobs created in 2017 but its manufacturing sector has shrunk significantly following the end of domestic car manufacturing. Employment in manufacturing peaked in mid-1989 at roughly 1.17 million, or 15 percent of the entire workforce. That shrank to 877,000, or 7 percent, late last year.   Australia has seen a wave of new jobs but companies are not keen on paying employees more, leaving wage growth near record lows in an unwelcome drag on consumer spending and inflation. Australia’s expansion plans come amid increased global demand for military hardware, led by China and Middle East nations, prompting criticism of Canberra from aid agencies who argue Australia could make human rights violations worse if weapons were sold to the wrong buyers. Analysts said Australia would need to significantly expand sales beyond its traditional partners to have any chance of fulfilling its ambition. “There are possibilities, but I doubt U.S. interest especially will go beyond niche capabilities,” said Euan Graham, director of the international security program at Australian think tank the Lowy Institute. Reporting by Colin PackhamOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-afghanistan-blast/militants-attack-afghan-army-post-near-military-academy-in-capital-idUSKBN1FI07M?il=0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.reuters.com/article/us-afghanistan-blast/militants-attack-afghan-army-post-near-military-academy-in-capital-idUSKBN1FI07M?il=0\n",
      "TITLE: Militants attack Afghan army post near military academy in capital\n",
      "BODY:\n",
      "KABUL (Reuters) - At least four militants attacked an army outpost near one of Afghanistan’s main military academies on Monday and at least one soldier was killed and three wounded, a defense ministry official said. The attack in the western outskirts of the capital, Kabul, came two days after an ambulance bomb in the center of the city killed more than 100 people and just over a week after another attack on the Hotel Intercontinental killed more than 20. Both of those attacks were claimed by the Taliban. Ministry of Defence officials said the militants attacked the outpost near the well-defended Marshal Fahim military academy just before dawn. One of the attackers blew himself up, one had been killed and two were still fighting. One soldier had been killed and three were wounded, said one official, who declined to be identified. Ministry spokesman Dawlat Waziri said three soldiers had been wounded and the clash was still going on. Earlier, resident Mohammad Ehsan said he heard a series of explosions coming from the area beginning at around 5 a.m. and lasting for at least an hour.  Smaller blasts could still be heard at less frequent intervals. In October, a suicide attacker rammed a car full of explosives into a bus carrying cadets from the defense university, which is home to one of Afghanistan’s main officer training schools, killing 15 of them. Reporting by Omar Sobhani; Writing by James Mackenzie; Editing by Robert Birsel and Paul TaitOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-usa-election-kochs/fearing-democratic-wave-koch-network-to-spend-big-on-u-s-midterm-elections-idUSKBN1FI07H?il=0\n",
      "URL: https://www.reuters.com/article/us-usa-election-kochs/fearing-democratic-wave-koch-network-to-spend-big-on-u-s-midterm-elections-idUSKBN1FI07H?il=0\n",
      "TITLE: Fearing Democratic wave, Koch network to spend big on U.S. midterm elections\n",
      "BODY:\n",
      "INDIAN WELLS, Calif. (Reuters) - The conservative Koch network spent its annual donor conference celebrating policy victories under President Donald Trump such as the tax overhaul, but the elation was tinged with anxiety over November’s congressional elections that could pose a risk to its agenda. To that end, the network plans to spend what would be unprecedented sums for the Kochs to maintain Republican majorities in the U.S. Senate and House of Representatives, while trying to sell voters on the benefits of the newly passed tax package, according to network officials who briefed reporters on their strategy during the conference this weekend in Indian Wells, California. Historically, the party in power loses seats in congressional elections after a new president’s election. This year, Republican angst is compounded by Trump, whose tumultuous presidency helped galvanize Democratic and independent voters to go to the polls in special and state elections last year. “It’s going to be a very challenging environment,” said Tim Phillips, president of Americans for Prosperity, a grassroots political group that is part of the Koch network. “The left is energized. There’s no question about that.” The network is prepared to spend up to $400 million on the congressional races - a 60 percent increase from its investment in the 2016 election, officials said. Energy magnates Charles and David Koch have long been outsized players in Republican politics, but they never warmed to Trump during the 2016 campaign. With Trump in the White House, however, they have seen several policy goals realized, including the tax legislation and cutting federal regulations. But the Koch network still diverges from the Republican president on issues such as immigration and trade. The Kochs strongly support legislation that would protect “Dreamers” - people brought illegally to the United States as children - from deportation. Two Koch operatives, Daniel Garza and Jorge Lima, were at the White House on Friday to try to help broker a deal with Congress over the Dreamers. The network issued a statement on Friday disagreeing with a proposal in Trump’s immigration blueprint that would set new limits on legal immigration. On trade, Phillips and other Koch operatives are deeply concerned about the administration’s moves to impose tariffs on some imports and ardently support the North American Free Trade Agreement with Canada and Mexico, which is being renegotiated and that Trump has threatened to abandon. ‘GET OUT AND DEFINE YOURSELF’ Senator John Cornyn, the No. 2 Republican in the Senate, was an attendee at the Koch event, a measure of the network’s deep ties to conservatives in Congress and an example of the leverage it would lose should Democrats take control next year. Of the $400 million the network is looking to spend, $20 million will go toward promoting the tax law, which passed Congress in December and included big cuts in corporate tax rates along with tax reductions for many individuals. Polls at the time of passage showed Americans divided about the bill’s merits, but Phillips said he believed that would  change as voters see increased paychecks. Democrats condemned the tax measure as favoring corporations and the rich. Americans for Prosperity is positioned to be a ground-level force in the congressional elections. Its largest presence is in Florida, where Democratic Senator Bill Nelson may face a fight from Republican Governor Rick Scott. The group also has offices in Arizona, Nevada, and Wisconsin, among other states, all of which are expected to see highly competitive Senate races. Phillips said his group would likely stay out of Republican primaries. One variable hanging over the Kochs’ effort is Trump. Republican losses in a special U.S. Senate election in Alabama last month and a governor’s race in Virginia in November were attributed in part to discontent with the president. In 2010, Republicans seized on voter worries about Democratic President Barack Obama’s signature healthcare law, known as Obamacare, to capture the House and thwart Obama’s policy goals. For the Koch network, the challenge will be to avoid having every local race become a referendum on the president, which could propel a Democratic wave. James Davis, vice president of Freedom Partners, another Koch-backed policy group, said strong candidates should be able to differentiate themselves from Trump. “Get out there and define yourself and where you stand on the issues,” Davis said.  During one event, a donor from Nebraska, Gail Werner-Robertson, stood up and addressed Charles Koch directly, urging attendees to contribute more to the midterm effort. “We can’t lose the progress you all have fought so hard for,” she said.  Reporting by James Oliphant; Editing by Caren Bohan and Peter CooneyOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/uk-global-markets/asia-shares-extend-bull-run-dollar-crawls-off-lows-idUSKBN1FI01D?il=0\n",
      "URL: https://www.reuters.com/article/uk-global-markets/asia-shares-extend-bull-run-dollar-crawls-off-lows-idUSKBN1FI01D?il=0\n",
      "TITLE: Asia shares extend bull run, dollar crawls off lows\n",
      "BODY:\n",
      "SYDNEY (Reuters) - Asian shares extended their bull run on Monday amid upbeat corporate earnings and strong global economic growth, while the dollar tried to bounce even as the White House continued to complain of “unfair” trade practices by competitors. MSCI’s broadest index of Asia-Pacific shares outside Japan added 0.4 percent, aiming for a 12th straight session of gains. It is up 8 percent for the year so far. Japan’s Nikkei rose 0.5 percent as the yen eased a little, while South Korea notched a record. Hong Kong’s Hang Seng also rose 0.5 percent. It has been the best performer for the year with a rise of more than 11 percent, followed by Shanghai blue chips with gains of nearly 9 percent, though the latter dipped on Monday. Wall Street has likewise been on a tear. Just last week, the Dow rose 2.08 percent, the S&P; 500 2.22 percent and the Nasdaq 2.31 percent. Quarterly earnings growth for the S&P; 500 is estimated at 13.2 percent, according to Thomson Reuters data, up from 12 percent at the start of the year. Of the 133 companies in the index that have reported, almost 80 percent beat forecasts. Another 36 percent of the S&P; 500 is due to report this week including heavy hitters Apple, Alphabet, Facebook, Microsoft and Amazon. The rush to equities combined with the risk of faster global inflation, has been a major negative for sovereign bonds with yields rising across much of the developed world. Yields on U.S. two-year Treasuries have risen steadily to their highest since 2008 and are fully priced for a rate hike by the Federal Reserve in March. Ten-year yields broke above the range of the last week or so to reach 2.69 percent on Monday, levels last visited in mid-2014. The Fed holds its next meeting on Wednesday, the last for Chair Janet Yellen, and analysts suspect the statement will only cement expectations for a March move. WORDS MATTER     The inexorable increase in Treasury yields has not, however, been enough to rescue the U.S. dollar which sank to three-year lows last week as U.S. officials welcomed a weaker currency. President Donald Trump did try and walk some of that back late in the week but by then the damage had been done. Indeed, in an interview shown on Sunday, Trump threatened to confront the European Union over what he calls “very unfair” trade policy toward the U.S.. “‘Words’ in the world of FX do matter,” said Deutsche Bank strategist George Saravelos. “The U.S. is reengaging with a weak dollar policy similarly to the 1994-95 period.” This was happening while the sum of trade and investment flows into the United States was shrinking. The opposite was happening in the euro zone, where the German export engine was powering an ever-expanding current account surplus. “We continue to target $1.30 in EUR/USD for this year,” Saravelos concluded. The euro did run into a little profit-taking in Asia on Monday which nudged it to back to $1.2393 and away from a three-year peak of $1.2538 last week. The dollar was a fraction firmer on the yen at 108.89, but not far from a four-month trough of 108.28. Against a basket of major currencies, it edged up 0.2 percent to 89.281 having been at the lowest since late 2014. The dollar faces a bevy of U.S. economic reports this week including releases on inflation, manufacturing and payrolls. The currency’s decline has been a boon for many commodities, with gold making a 17-month top last week and last trading at $1,348.10 an ounce. Oil prices had reached their highest in three years and Brent crude futures were holding atop $70 at $70.40 a barrel. U.S. crude futures were up 18 cents at $66.32.  Editing by Shri Navaratnam and Kim CoghillOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-japan-cryptocurrency/japan-to-punish-hacked-cryptocurrency-exchange-coincheck-on-monday-idUSKBN1FI06S?il=0\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-69-4ebc9b5cac68>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     52\u001b[0m \u001b[0mreuters\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mWebsite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Reuters'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'https://www.reuters.com'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'^(/article/)'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'h1'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'div.StandardArticleBody_body_1gnLA'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m \u001b[0mcrawler\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCrawler\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreuters\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m \u001b[0mcrawler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcrawl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-69-4ebc9b5cac68>\u001b[0m in \u001b[0;36mcrawl\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     47\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mabsoluteUrl\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     48\u001b[0m                     \u001b[0mtargetPage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'{}{}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtargetPage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 49\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtargetPage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     50\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     51\u001b[0m                 \u001b[0;31m# self, name, url, targetPattern, absoluteUrl, titleTag, bodyTag\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-69-4ebc9b5cac68>\u001b[0m in \u001b[0;36mparse\u001b[0;34m(self, url)\u001b[0m\n\u001b[1;32m     27\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mparse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m         \u001b[0mbs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetPage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     30\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mbs\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m             \u001b[0mtitle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msafeGet\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtitleTag\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-69-4ebc9b5cac68>\u001b[0m in \u001b[0;36mgetPage\u001b[0;34m(self, url)\u001b[0m\n\u001b[1;32m      9\u001b[0m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"GETTING \"\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m             \u001b[0mreq\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRequestException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Did not get page\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(url, params, **kwargs)\u001b[0m\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m     \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetdefault\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'allow_redirects'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 67\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'get'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     68\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(method, url, **kwargs)\u001b[0m\n\u001b[1;32m     51\u001b[0m     \u001b[0;31m# cases, and look like a memory leak in others.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     52\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0msessions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 53\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     54\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     55\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36m__exit__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m    348\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    349\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__exit__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 350\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    351\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    352\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mprepare_request\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mclose\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    647\u001b[0m         \u001b[0;34m\"\"\"Closes all adapters and as such the session\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    648\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madapters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 649\u001b[0;31m             \u001b[0mv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    650\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    651\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmount\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py\u001b[0m in \u001b[0;36mclose\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    268\u001b[0m         \u001b[0mconnections\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    269\u001b[0m         \"\"\"\n\u001b[0;32m--> 270\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpoolmanager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    271\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    272\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrequest_url\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproxies\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/poolmanager.py\u001b[0m in \u001b[0;36mclear\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     98\u001b[0m         \u001b[0mre\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mused\u001b[0m \u001b[0mafter\u001b[0m \u001b[0mcompletion\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     99\u001b[0m         \"\"\"\n\u001b[0;32m--> 100\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpools\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    102\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mconnection_from_host\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhost\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mport\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscheme\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'http'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/_collections.py\u001b[0m in \u001b[0;36mclear\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     92\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispose_func\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     93\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mvalues\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 94\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispose_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     95\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     96\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/poolmanager.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(p)\u001b[0m\n\u001b[1;32m     64\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnection_pool_kw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconnection_pool_kw\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     65\u001b[0m         self.pools = RecentlyUsedContainer(num_pools,\n\u001b[0;32m---> 66\u001b[0;31m                                            dispose_func=lambda p: p.close())\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__enter__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36mclose\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    410\u001b[0m                 \u001b[0mconn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mold_pool\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mblock\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    411\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 412\u001b[0;31m                     \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    413\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    414\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mEmpty\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "\n",
    "class Crawler:\n",
    "    def __init__(self, site):\n",
    "        self.site = site\n",
    "        self.visited = []\n",
    "\n",
    "    def getPage(self, url):\n",
    "        try:\n",
    "            req = requests.get(url)\n",
    "        except requests.exceptions.RequestException:\n",
    "            return None\n",
    "        return BeautifulSoup(req.text, 'html.parser')\n",
    "\n",
    "    def safeGet(self, pageObj, selector):\n",
    "        selectedElems = pageObj.select(selector)\n",
    "        if selectedElems is not None and len(selectedElems) > 0:\n",
    "            return '\\n'.join([elem.get_text() for elem in selectedElems])\n",
    "        return ''\n",
    "\n",
    "    def parse(self, url):\n",
    "        bs = self.getPage(url)\n",
    "        if bs is not None:\n",
    "            title = self.safeGet(bs, self.site.titleTag)\n",
    "            body = self.safeGet(bs, self.site.bodyTag)\n",
    "            if title != '' and body != '':\n",
    "                content = Content(url, title, body)\n",
    "                content.print()\n",
    "\n",
    "    def crawl(self):\n",
    "        \"\"\"\n",
    "        Get pages from website home page\n",
    "        \"\"\"\n",
    "        bs = self.getPage(self.site.url)\n",
    "        targetPages = bs.findAll('a', href=re.compile(self.site.targetPattern))\n",
    "        for targetPage in targetPages:\n",
    "            targetPage = targetPage.attrs['href']\n",
    "            if targetPage not in self.visited:\n",
    "                self.visited.append(targetPage)\n",
    "                if not self.site.absoluteUrl:\n",
    "                    targetPage = '{}{}'.format(self.site.url, targetPage)\n",
    "                self.parse(targetPage)\n",
    "\n",
    "\n",
    "reuters = Website('Reuters', 'https://www.reuters.com', '^(/article/)',\n",
    "                  False, 'h1', 'div.StandardArticleBody_body_1gnLA')\n",
    "crawler = Crawler(reuters)\n",
    "crawler.crawl()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawling multiple page types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Website:\n",
    "    \"\"\"Common base class for all articles/pages\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Product(Website):\n",
    "    \"\"\"Contains information for scraping a product page\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, productNumber, price):\n",
    "        Website.__init__(self, name, url, TitleTag)\n",
    "        self.productNumberTag = productNumberTag\n",
    "        self.priceTag = priceTag\n",
    "\n",
    "\n",
    "class Article(Website):\n",
    "    \"\"\"Contains information for scraping an article page\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, bodyTag, dateTag):\n",
    "        Website.__init__(self, name, url, titleTag)\n",
    "        self.bodyTag = bodyTag\n",
    "        self.dateTag = dateTag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def parsePage(url):\n",
    "    \n",
    "    if '/ideas/' in url:\n",
    "        \n",
    "\n",
    "oreilly = Website('O\\'Reilly', 'https://oreilly.com', 'h1' '')        "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}