Chapter07_CrawlingModels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with different website layouts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TITLE: Robotic rulemaking\n",
      "URL: https://www.brookings.edu/research/robotic-rulemaking/\n",
      "BODY: \n",
      "As it has rocketed to some 100 million active users in record time, ChatGPT is provoking conversations about the role of artificial intelligence (AI) in drafting written materials such as student exams, news articles, legal pleadings, poems, and more. The chatbot, developed by OpenAI, relies on a large language model (LLM) to respond to user-submitted requests, or “prompts” as they are known. It is an example of generative AI, a technology that upends our understanding of who creates written materials and how they do it, challenging what it means to create, analyze, and express ideas.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Bridget C. E. Dooling\n",
      "\n",
      "\t\t\t\t\tResearch Professor - The George Washington University Regulatory Studies Center \n",
      "\n",
      " Twitter\n",
      "BridgetDooling\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Mark Febrizio\n",
      "\n",
      "\t\t\t\t\tSenior Policy Analyst - The George Washington University Regulatory Studies Center \n",
      "\n",
      " Twitter\n",
      "markfebrizio\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Rulemaking by federal agencies is a very text-intensive process, both in terms of writing the rules themselves, which express not only the law but also the agencies’ rationales for their regulatory choices, as well as public comments which arrive almost exclusively in the form of text. How might generative AI intersect with rulemaking? In this essay, we work through some use cases for generative AI in the rulemaking process, for better and for worse, both for the public and federal agencies.\n",
      "Public comments and generative AI\n",
      "For the public, generative AI might help people structure their information and views into public comments that have a better chance of influencing agency decisions. While agencies usually permit commenters to send in whatever text they want, more sophisticated comments tend to follow a professional format that contains substantive and sometimes highly technical information. A fairly constant worry with respect to public participation in rulemaking is that special interests overtake diffuse interests due to collective action problems. While readers of this series likely follow rulemaking closely, regulation remains esoteric for most people. Even for those aware of the rulemaking process, figuring out the style and content of a comment might seem out of reach. (Brookings published a helpful guide to commenting, by the way.) Scholars and policymakers disagree about the extent to which the public’s awareness of and participation in rulemaking is a problem that needs to be remedied, but at a minimum a tool that helps interested people compose a persuasive comment could be useful.\n",
      "Someone could, for example, prompt a generative AI tool to summarize that person’s position on a proposal and knit it into a comment that looks organized and clear. The prompt could be something like: “Write a comment to the Consumer Product Safety Commission telling them that I support their proposed rule on fireworks.” Even better, the prompt could guide the AI to emphasize specific concepts or reasons for the person’s views: “Write a comment to the Consumer Product Safety Commission telling them that I support their proposed rule on fireworks because fireworks can be traumatic for little kids and pets.” In our experience, ChatGPT can readily create convincing public submissions based on such straightforward prompts. For one Department of Labor proposal, simply requesting that the chatbot produce several paragraphs objecting to the rule resulted in text comparable to a mass comment campaign submission, and the content was quickly inverted by asking for a supportive comment.\n",
      "Generative AI also takes the possibility of “mass” and “malattributed” comments to the next level. Mass comments are “identical and near-duplicate comments” that are often “sponsored by organizations and submitted by group members and supporters to government agencies in response to proposed rules.” A team of researchers from The George Washington University and the Israel Democracy Institute wrote about the political reasons why groups organize these campaigns, likening them to the kind of lobbying activity that happens in Congress and other venues. Whether these mass comment campaigns actually influence agencies is the subject of some debate. The laws governing regulatory decisions generally do not call upon, or allow, the agencies to factor in public opinion; rather, agencies seek substantive and technical information from public comments. And the number of comments received is not a reliable proxy for general public opinion anyway because such submissions are not made by a representative sample of the population—even setting aside the possibility that some of the comments were not sent by real people, a possibility that generative AI increases. So most observers (with Professor Nina Mendelson a notable exception) have been dismissive of the role of mass comment campaigns in agency rulemaking decisions. Yet mass comment campaigns persist, for reasons that political scientists like Devin Judge-Lord are exploring in ongoing research. The reality is that generative AI arrives at a time when mass comment campaigns are a regular, if not frequent, component of rulemaking, so we can expect the two to intersect.\n",
      "\n",
      "\n",
      "Related Content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Technology & Innovation\n",
      "Exploring the impact of language models on cognitive automation\n",
      "\n",
      "Anton Korinek\n",
      "Monday, March 6, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Climate Change\n",
      "Debt-for-adaptation swaps\n",
      "\n",
      "Chetan Hebbale and Johannes Urpelainen\n",
      "Tuesday, March 21, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Courts & Law\n",
      "Judicial deference and the future of regulation\n",
      "\n",
      "James Kunhardt and Anne Joseph O’Connell\n",
      "Thursday, August 18, 2022\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Combining generative AI with mass comment campaigns could lead to mass comments that look less duplicative in nature, as the same idea could be expressed in many different ways with some support from an AI tool. Agencies currently have access to language processing tools that allow them to group comments based on the similarity of their text. This helps agencies meet their burden under the law to consider and respond to all significant comments. More varied comments will strain the current set of tools and likely lead to increased agency resources dedicated to comment analysis. That could further slow the already cumbersome rulemaking process, as agencies figure out how to cope with large and overwhelming volumes of differentiated and ostensibly substantive comments. For advocates looking to gum up the works, this could be an appealing tactic.\n",
      "In response, and following the approach they took in response to mass comment campaigns, agencies might be tempted to spend resources to develop tools to help identify which comments were generated by an AI rather than a human. Such tools are already in production for other purposes. While these tools are not “fully reliable,” they could alert agency staff that a comment they’re reviewing is likely generated by AI. It is not immediately clear how an agency would make use of such information, however, because the Administrative Procedure Act only requires that the commenter is a “person,” and a person could have submitted the comment, no matter who or what drafted it. Perhaps the alert could encourage agency staff to read the comment with some skepticism, but it’s not obvious at this point that such an approach would be reasoned or fair.\n",
      "Generative AI can be viewed as part of an ongoing tit-for-tat for public participation, with commenters deploying more sophisticated commenting methods and agencies attempting to respond with their own technology. Such an arms race is a waste of resources, though, if the end result is a large body of comments that neither represent the views of the general public nor offer novel and reliable information to the agency. More comments do not necessarily lead to better regulatory choices. The “arms race” frame is also troubling as applied to public input in a process intended to welcome it. Whether the type of participation that generative AI facilitates is the right kind of participation is part of what makes it such a provocative development.\n",
      "If generative AI adds to the richness of mass comments, that could be an improvement over many mass comment campaigns which tend to express up-or-down sentiment. Personal stories woven into comments can sometimes shed light on problems that agencies did not anticipate—the question is whether generative AI is poised to actually elucidate such richness or simply fake it. If regulators end up altering rules because of convincing but made-up “facts,” that would certainly be a step backward. In our experience, agency staff work hard to substantiate the information provided to them by public comments rather than accepting them at face value, but it is not implausible to imagine such safeguards breaking down. In that case, the potential for review in the courts offers an important backstop.\n",
      "Generative AI arrives at a time when mass comment campaigns are a regular, if not frequent, component of rulemaking, so we can expect the two to intersect.\n",
      "Taking this analysis one step further, Professor Michael Herz coined the term “malattributed comments” after the spectacle that accompanied the Federal Communication Commission’s (FCC) Net Neutrality rulemaking, in which millions of public comments claimed to be from people who either did not exist or who did not actually send comments. In a study commissioned by the Administrative Conference of the United States (ACUS), researchers (including one of us) concluded that the risks of an agency being misled by malattributed comments are lower than might be expected because of the way agencies evaluate comments.1 Generative AI disturbs this equilibrium because it may help bad actors generate comments that look more persuasive, i.e., comments that seemingly present evidence beyond mere sentiment.\n",
      "By reducing the costs of producing “malattributed” comments, generative AI could lead to a pooling equilibrium—to borrow a concept from game theory that is often applied to insurance markets—where agencies can no longer meaningfully distinguish between valid and malicious comments. Agencies could then be inclined to assume all comments might be “fake” and discount their relevance, weakening public commenting as an avenue for meaningful public input and the formulation of improved policies and, ultimately, making people worse off. That need not come to pass, however. The Administrative Procedure Act does not permit agencies to entirely dismiss all public comments in this manner, nor does it categorically prohibit the public from using AI to aid in comment creation. Agencies also have a track record of collaborating to address novel issues, such as mass comment campaigns, via the eRulemaking program, and we expect that work to continue. Overall, as commenters reach for ways to use generative AI, agencies would be wise to lean on this existing governance structure as they consider potential responses.\n",
      "Agency workflows and generative AI\n",
      "Generative AI also offers some promises and perils for internal agency processes. Beyond malattributed comments, one worry is that flooding rulemaking dockets with a virtually unlimited supply of unique comments would incapacitate government systems and prevent other users from submitting public input. While these fears are alarming, the rulemaking system is fairly robust to a torrent of bot-generated comments. More specifically, Regulations.gov—the site that a majority of agencies use to accept public submissions on rules—already implements several techniques to manage large volumes of comments.\n",
      "First, Regulations.gov employs a CAPTCHA system developed by Google to distinguish between humans and bots. This prevents a computer program from automating comment submissions through the web interface, meaning that bad actors who want to spam the system would need to do so manually. In fact, one paper warning against the risk of bot submissions to rulemakings suggested this very solution. Second, the Regulations.gov Application Programming Interface (API) provides a way for organizations to submit multiple comments in an automated fashion—within certain limits. The API, which is managed by the General Services Administration (GSA), requires adherence to terms of service and uses a key to authenticate post submissions. Accordingly, GSA can also revoke access when it detects malicious activity. Further, submissions are constrained by a limit of 50 per minute or 500 per hour (whichever is reached first). This throttling inhibits malicious users from overwhelming the system before being identified and could be made stricter if necessary. While these safeguards are not foolproof, they provide meaningful protections against incapacitating the comment system with AI-generated text.\n",
      "Another worry is that the government could be misled by AI-generated comments. The text from generative AI can be very convincing, even if it is entirely untrue. This is because LLMs draw from voluminous writings by humans. They are essentially extremely big text prediction models that select the next word, phrase, or punctuation according to sample texts that they have been trained on (i.e., the texts fed to the model as examples for it to learn from); they are not a lookup table or encyclopedia. ChatGPT is not connected to the internet and its training data stop in 2021, but it can still generate plausible analyses of current articles based on information gleaned from prompts and its training data. Even LLMs connected to the internet, such as Bing AI, have a tendency to make up or “hallucinate” information, especially in contexts lacking training data.2 Relatedly, they offer more tailored answers in areas where they have received more training. In this context, public comments on rules are available on Regulations.gov going back many years, making them a rich source of training data.\n",
      "Given the likely pace of development for generative AI, the federal government needs to be prepared to adapt to this intriguing new set of tools.\n",
      "Given this backdrop, someone could prompt a generative AI tool to write a comment that supports or criticizes a rule based on fake scientific data or other technical information.3 Interestingly, the public can already submit fake information to an agency.4 It is currently the agency’s responsibility to wade through public comments and discern which information is or is not reliable; this is part of why it can take agencies months or years to finalize rules once they have been proposed. While the rulemaking process can be criticized for its length, taking time to discern the weight of public comments helps protect against challenges to the rulemaking process’ integrity, whether they come from generative AI or other sources. The possibility raised above of a large number of authentic, substantive, and varied comments does give us some pause because such comments would not violate the terms of service and could therefore spike agency workloads. It may be a challenge to balance the policy goal of a notice-and-comment process that is open to all with the reality of limited agency resources to consider so much information. This could be an area ripe for enhancing executive branch and congressional oversight of agency rulemaking.\n",
      "Generative AI could also help agency staff summarize and respond to comments received on the rules. A strength of LLMs is their ability to process and compose information based on their training data; regulators could use this to their advantage, especially if agencies had access to a model trained on public comments or texts related to the content of a rulemaking. Further, LLMs are most useful when combined with expertise because the information produced by the AI can be verified and supplemented by those with subject matter or “topic” knowledge. For example, an LLM could help regulators summarize comments on the proposal, classify feedback based on predefined categories, and cluster information based on similarities in content, style, or other features. Then, agency staff could provide a rough outline of responses to comments and prompt a generative AI to format them in the style of a rulemaking published in the Federal Register. This workflow would incorporate a more sophisticated set of tools than what agencies currently use to analyze and respond to public comments. While we are not in a position to say whether the federal government will actually invest in generative AI technologies, the capability is there. One pathway would be to fine-tune existing models like OpenAI’s GPT 3.5 (the basis for ChatGPT) for the rulemaking context.5 This could entail an agency customizing an existing LLM to better apply to its rulemaking activity by conducting additional training with relevant texts. For instance, the Environmental Protection Agency (EPA) could fine-tune GPT 3.5 for use by its Office of Air and Radiation by feeding it examples of its responses to comments from prior rules and topic-specific materials on the Clean Air Act.\n",
      "Using AI to support agency workflows, including analyzing public comments, sits in tension with current notions of who is supposed to do the “thinking work” of the government. One of us has written about this recently (with co-author Professor Rachel A. Potter) in the context of regulations that are drafted by government contractors, exploring whether drafting rules should be considered an “inherently governmental function” that is off-limits to contractors. While contractors can serve as vital supplements to agency capacity or expertise, overreliance on contractors can introduce conflicts of interest and other risks into the process. Generative AI offers an interesting twist on this concern. Might generative AI be more conflicted or biased than an outside contractor, less conflicted or biased, or might it simply present entirely different considerations? Because an LLM’s training data inform the way it generates text, using it to draft regulatory material could therefore reinforce the status quo in some circumstances and, in others, help create new connections in human knowledge. Of course, existing approaches to crafting rules are not without their own biases, nor are other tools (e.g., Google searches) that have become commonplace in policymaking. We are only at the beginning of working through these issues as they apply to this essential form of executive branch lawmaking.\n",
      "We limited this essay to generative AI, one of many technologies that intersect with rulemaking comments. Other tools could, for example, help alert people to rules that interest them and help the government catch errors and omissions in their analyses. The technology of rulemaking evolves along with the rest of society, and regulators should consider how to take advantage of the upside of tools like generative AI while minimizing their risks. One thing is sure: Given the likely pace of development for generative AI, the federal government needs to be prepared to adapt to this intriguing new set of tools.\n",
      "\n",
      "body: \n",
      "\n",
      "\n",
      "\n",
      "New York\n",
      "CNN\n",
      "         — \n",
      "    \n",
      "\n",
      "\n",
      "      Twitter’s traditional bird icon was booted and replaced with an image of a Shiba Inu, an apparent nod to dogecoin, the joke cryptocurrency that CEO Elon Musk is being sued over. \n",
      "  \n",
      "\n",
      "      Musk addressed the change Monday afternoon, tweeting, “as promised” above an image of a year-old conversation in which another user suggested that Musk “just buy Twitter” and “change the bird logo to a doge.” \n",
      "  \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "CNN/Adobe Stock\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Elon Musk's Twitter promised a purge of blue check marks. Instead he singled out one account\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "      The doge logo appeared on the site two days after Musk asked a judge to throw out a $258 billion racketeering lawsuit accusing him of running a pyramid scheme to support the dogecoin, according to Reuters.\n",
      "\n",
      "\n",
      "      Lawyers for Musk and Tesla called the lawsuit by dogecoin investors a “fanciful work of fiction” over Musk’s “innocuous and often silly tweets.”\n",
      "  \n",
      "\n",
      "      It wasn’t clear whether the logo change was permanent. Musk has been known to use Twitter to troll both his fans and his critics. \n",
      "  \n",
      "\n",
      "      The price of dogecoin, which is typically volatile, was up more than 20% over the past 24 hours, to about 9 cents. It was trading just under 8 cents Monday morning.\n",
      "  \n",
      "\n",
      "Dogecoin was created December 6, 2013, by a pair of software engineers — as a joke. The name is a nod to the “doge” meme that became popular a decade ago. Its Shiba Inu mascot mimicks that meme: a dog surrounded by a bunch of Comic Sans text in broken English.\n",
      "  \n",
      "\n",
      "TITLE: \n",
      "      Dogecoin jumps after Elon Musk replaces Twitter bird with Shiba Inu\n",
      "    \n",
      "URL: https://www.cnn.com/2023/04/03/investing/dogecoin-elon-musk-twitter/index.html\n",
      "BODY: \n",
      "\n",
      "\n",
      "New York\n",
      "CNN\n",
      "         — \n",
      "    \n",
      "\n",
      "\n",
      "      Twitter’s traditional bird icon was booted and replaced with an image of a Shiba Inu, an apparent nod to dogecoin, the joke cryptocurrency that CEO Elon Musk is being sued over. \n",
      "  \n",
      "\n",
      "      Musk addressed the change Monday afternoon, tweeting, “as promised” above an image of a year-old conversation in which another user suggested that Musk “just buy Twitter” and “change the bird logo to a doge.” \n",
      "  \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "CNN/Adobe Stock\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Elon Musk's Twitter promised a purge of blue check marks. Instead he singled out one account\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "      The doge logo appeared on the site two days after Musk asked a judge to throw out a $258 billion racketeering lawsuit accusing him of running a pyramid scheme to support the dogecoin, according to Reuters.\n",
      "\n",
      "\n",
      "      Lawyers for Musk and Tesla called the lawsuit by dogecoin investors a “fanciful work of fiction” over Musk’s “innocuous and often silly tweets.”\n",
      "  \n",
      "\n",
      "      It wasn’t clear whether the logo change was permanent. Musk has been known to use Twitter to troll both his fans and his critics. \n",
      "  \n",
      "\n",
      "      The price of dogecoin, which is typically volatile, was up more than 20% over the past 24 hours, to about 9 cents. It was trading just under 8 cents Monday morning.\n",
      "  \n",
      "\n",
      "Dogecoin was created December 6, 2013, by a pair of software engineers — as a joke. The name is a nod to the “doge” meme that became popular a decade ago. Its Shiba Inu mascot mimicks that meme: a dog surrounded by a bunch of Comic Sans text in broken English.\n",
      "  \n",
      "\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "from urllib.request import urlopen\n",
    "\n",
    "class Content:\n",
    "    def __init__(self, url, title, body):\n",
    "        self.url = url\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "    \n",
    "    def print(self):\n",
    "        print(f'TITLE: {self.title}')\n",
    "        print(f'URL: {self.url}')\n",
    "        print(f'BODY:\\n {self.body}')\n",
    "\n",
    "def scrapeCNN(url):\n",
    "    bs = BeautifulSoup(urlopen(url))\n",
    "    title = bs.find('h1').text\n",
    "    body = bs.find('div', {'class': 'article__content'}).text\n",
    "    print('body: ')\n",
    "    print(body)\n",
    "    return Content(url, title, body)\n",
    "\n",
    "def scrapeBrookings(url):\n",
    "    bs = BeautifulSoup(urlopen(url))\n",
    "    title = bs.find('h1').text\n",
    "    body = bs.find('div', {'class': 'post-body'}).text\n",
    "    return Content(url, title, body)\n",
    "\n",
    "url = 'https://www.brookings.edu/research/robotic-rulemaking/'\n",
    "content = scrapeBrookings(url)\n",
    "content.print()\n",
    "\n",
    "url = 'https://www.cnn.com/2023/04/03/investing/dogecoin-elon-musk-twitter/index.html'\n",
    "content = scrapeCNN(url)\n",
    "content.print()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Content:\n",
    "    \"\"\"\n",
    "    Common base class for all articles/pages\n",
    "    \"\"\"\n",
    "    def __init__(self, url, title, body):\n",
    "        self.url = url\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "\n",
    "    def print(self):\n",
    "        \"\"\"\n",
    "        Flexible printing function controls output\n",
    "        \"\"\"\n",
    "        print(f'URL: {self.url}')\n",
    "        print(f'TITLE: {self.title}')\n",
    "        print(f'BODY:\\n{self.body}')\n",
    "\n",
    "class Website:\n",
    "    \"\"\" \n",
    "    Contains information about website structure\n",
    "    \"\"\"\n",
    "    def __init__(self, name, url, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "class Crawler:\n",
    "    def getPage(url):\n",
    "        try:\n",
    "            html = urlopen(url)\n",
    "        except Exception:\n",
    "            return None\n",
    "        return BeautifulSoup(html, 'html.parser')\n",
    "\n",
    "    def safeGet(bs, selector):\n",
    "        \"\"\"\n",
    "        Utilty function used to get a content string from a Beautiful Soup\n",
    "        object and a selector. Returns an empty string if no object\n",
    "        is found for the given selector\n",
    "        \"\"\"\n",
    "        selectedElems = bs.select(selector)\n",
    "        if selectedElems is not None and len(selectedElems) > 0:\n",
    "            return '\\n'.join([elem.get_text() for elem in selectedElems])\n",
    "        return ''\n",
    "\n",
    "    def getContent(website, path):\n",
    "        \"\"\"\n",
    "        Extract content from a given page URL\n",
    "        \"\"\"\n",
    "        url = website.url+path\n",
    "        bs = Crawler.getPage(url)\n",
    "        if bs is not None:\n",
    "            title = Crawler.safeGet(bs, website.titleTag)\n",
    "            body = Crawler.safeGet(bs, website.bodyTag)\n",
    "            return Content(url, title, body)\n",
    "        return Content(url, '', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.oreilly.com/library/view/web-scraping-with/9781491910283\n",
      "TITLE: Web Scraping with Python\n",
      "BODY:\n",
      "\n",
      "\n",
      "Book description\n",
      "Learn web scraping and crawling techniques to access unlimited data from any web source in any format. With this practical guide, you’ll learn how to use Python scripts and web APIs to gather and process data from thousands—or even millions—of web pages at once.Ideal for programmers, security professionals, and web administrators familiar with Python, this book not only teaches basic web scraping mechanics, but also delves into more advanced topics, such as analyzing raw data or using scrapers for frontend website testing. Code samples are available to help you understand the concepts in practice.\n",
      "Show and hide more\n",
      "\n",
      "Publisher resources\n",
      "View/Submit Errata\n",
      "Download Example Code\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "siteData = [\n",
    "    ['O\\'Reilly Media', 'https://www.oreilly.com', 'h1', 'div.title-description'],\n",
    "    ['Reuters', 'https://www.reuters.com', 'h1', 'div.ArticleBodyWrapper'],\n",
    "    ['Brookings', 'https://www.brookings.edu', 'h1', 'div.post-body'],\n",
    "    ['CNN', 'https://www.cnn.com', 'h1', 'div.article__content']\n",
    "]\n",
    "websites = []\n",
    "for name, url, title, body in siteData:\n",
    "    websites.append(Website(name, url, title, body))\n",
    "\n",
    "Crawler.getContent(websites[0], '/library/view/web-scraping-with/9781491910283').print()\n",
    "Crawler.getContent(\n",
    "    websites[1], '/article/us-usa-epa-pruitt-idUSKBN19W2D0').print()\n",
    "Crawler.getContent(\n",
    "    websites[2],\n",
    "    '/blog/techtank/2016/03/01/idea-to-retire-old-methods-of-policy-education/').print()\n",
    "Crawler.getContent(\n",
    "    websites[3], \n",
    "    '/2023/04/03/investing/dogecoin-elon-musk-twitter/index.html').print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawling through sites with search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Content:\n",
    "    \"\"\"Common base class for all articles/pages\"\"\"\n",
    "\n",
    "    def __init__(self, topic, url, title, body):\n",
    "        self.topic = topic\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "        self.url = url\n",
    "\n",
    "    def print(self):\n",
    "        \"\"\"\n",
    "        Flexible printing function controls output\n",
    "        \"\"\"\n",
    "        print(f'New article found for topic: {self.topic}')\n",
    "        print(f'URL: {self.url}')\n",
    "        print(f'TITLE: {self.title}')\n",
    "        print(f'BODY:\\n{self.body}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Website:\n",
    "    \"\"\"Contains information about website structure\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, searchUrl, resultListing, resultUrl, absoluteUrl, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.searchUrl = searchUrl\n",
    "        self.resultListing = resultListing\n",
    "        self.resultUrl = resultUrl\n",
    "        self.absoluteUrl = absoluteUrl\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.reuters.com/search/news?blob=python\n",
      "http://reuters.com/article/idUSKCN11S04G\n",
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSKCN11S04G\n",
      "TITLE: Python in India demonstrates huge appetite\n",
      "BODY:\n",
      "By 1 Min ReadA 20 feet rock python was caught on camera in Junagadh district of India’s western Gujarat state with a swollen stomach after it consumed an antelope on Tuesday (September 20).Residents informed authorities at Girnar Wildlife Sanctuary after they spotted the reptile lying in discomfort in a field.In view of the massive swelling of the python’s stomach, the forest authorities suspect that it gobbled up a full-grown ‘nilgai’ or blue bull.The python - unable to move now - was rescued by the forest personnel and has been put under observation.“We will keep it (python) under observation. We will release it back in the wild once it digests the antelope and the swelling subsides,” said Assistant Conservator of Forest, S.D. Tilala.A blue bull is far larger than an ideal prey for pythons and digesting the mammal could prove to be a great struggle for the reptile.When unable to digest an unusually large prey, pythons are known to regurgitate them.\n",
      "http://reuters.com/article/idUSKBN0L31PS20150130\n",
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSKBN0L31PS20150130\n",
      "TITLE: Zimbabwean jailed for nine years for eating python meat\n",
      "BODY:\n",
      "By Reuters Staff1 Min ReadJOHANNESBURG (Reuters) - A Zimbabwean man has been jailed for nine years for eating python, a protected species, a Zimbabwean daily reported on Friday.Archwell Maramba said he had been eating the snake meat for its medicinal value.“I wanted to eat the python to cure my spine and since I ate my first python I have seen remarkable improvement,” NewsDay quoted Maramba as saying during his court case.Acting on a tipoff, police found the 58-year-old drying snakeskin and python meat stashed in his house.Reporting by Helen Nyambura-Mwaura; Editing by Andrew RocheOur Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSBRE9081CL20130110\n",
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSBRE9081CL20130110\n",
      "TITLE: Florida python hunting contest draws hundreds\n",
      "BODY:\n",
      "By Barbara Liston4 Min ReadORLANDO, Florida (Reuters) - A python hunting competition starting on Saturday is drawing hundreds of amateurs armed with clubs, machetes and guns to the Florida Everglades, where captured Burmese pythons have exceeded the length of minivans and weighed as much as grown men.Researchers at the Florida Museum of Natural History in Gainesville, Florida examine a 17-foot-7-inch Burmese python which was captured in Everglades National Park in this handout picture taken on August 10, 2012 and released on August 14, 2012. A python hunting competition starting January 12, 2013 is drawing hundreds of amateurs armed with clubs, machetes and guns to the Florida Everglades, where captured Burmese pythons have exceeded the length of minivans and weighed as much as grown men. REUTERS/Kristen Grace/Florida Museum of Natural History at University of Florida/HandoutPython Challenge 2013, a month-long event sponsored by the Florida Fish and Wildlife Conservation Commission, is open to hunters and non-hunters alike.But the idea of luring weapon-wielding amateurs into the harsh environment of the Everglades has raised some alarms.“I just thought it was as exciting as could be. It’s a once- in-a-lifetime opportunity,” said contestant Ron Polster, a retired salesman from Ohio whose closest encounter with the swamp has been from the highway heading south for the winter.Participants pay a $25 entry fee and take an online training course, which consists mostly of looking at photographs of both the targeted pythons and protected native snakes to learn the difference.The state wildlife agency is offering prizes of $1,500 for the most pythons captured and $1,000 for the longest python.A Burmese python found in Florida last year set records as the largest ever captured in the state at 17-feet, 7-inches. The snake weighed nearly 165 pounds (75 kg).FWC spokeswoman Carli Segelson said the number of registered contestants reached about 500 this week and was growing, with people coming from 32 states.The stated goal of the competition is to raise awareness of the threat Burmese pythons pose to the Everglades ecosystem. The snakes are native to Southeast Asia and have no known predators in Florida.The contest also serves as a pilot program to determine whether regular hunting competitions can cull the growing population of the invasive species, said Frank Mazzotti, a wildlife expert from the University of Florida who helped create the competition.Python Challenge rules require contestants to kill specimens on the spot in a humane fashion, recommending shooting the snakes precisely through the brain.“I was hoping there would be a lot of machetes and not a lot of guns,” said Polster, the retired salesman. He said he worries “these idiots will be firing all over the place.”Shawn Heflick, star of the National Geographic “Wild” television show “Python Hunters,” told Reuters that despite the formidable size of the snakes, he expects the swamp itself, with its alligators, crocodiles and venomous snakes, to pose a greater threat to the contestants.“You get these people going down there, they’ll get lost, they’ll get dehydrated, they’ll get sucked dry by mosquitoes,” Heflick said.Segelson said the wildlife agency will provide training on the use of GPS devices and on identifying venomous snakes at the kick-off event. In the meantime, she said, contestants should be familiarizing themselves with the Everglades environment, just as they should before entering any other strange territory.Heflick said most of the contestants likely were drawn to the Python Challenge by the romantic mystique of bagging a giant predator. He expects few will last long in the hunt.“The vast majority of them will never see a python. The vast majority of them will probably curtail their hunting very quickly when they figure out there’s a lot of mosquitoes, it’s hot, it’s rather boring sometimes - most of the time really, and I think a lot of them will go home,” Heflick said.Editing by Tom Brown and Dan GreblerOur Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSBRE9081CL20130109\n",
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSBRE9081CL20130109\n",
      "TITLE: Florida python hunting contest draws hundreds\n",
      "BODY:\n",
      "By Barbara Liston4 Min ReadORLANDO, Florida (Reuters) - A python hunting competition starting on Saturday is drawing hundreds of amateurs armed with clubs, machetes and guns to the Florida Everglades, where captured Burmese pythons have exceeded the length of minivans and weighed as much as grown men.Python Challenge 2013, a month-long event sponsored by the Florida Fish and Wildlife Conservation Commission, is open to hunters and non-hunters alike.But the idea of luring weapon-wielding amateurs into the harsh environment of the Everglades has raised some alarms.“I just thought it was as exciting as could be. It’s a once- in-a-lifetime opportunity,” said contestant Ron Polster, a retired salesman from Ohio whose closest encounter with the swamp has been from the highway heading south for the winter.Participants pay a $25 entry fee and take an online training course, which consists mostly of looking at photographs of both the targeted pythons and protected native snakes to learn the difference.The state wildlife agency is offering prizes of $1,500 for the most pythons captured and $1,000 for the longest python.A Burmese python found in Florida last year set records as the largest ever captured in the state at 17-feet, 7-inches. The snake weighed nearly 165 pounds (75 kg).FWC spokeswoman Carli Segelson said the number of registered contestants reached about 500 this week and was growing, with people coming from 32 states.The stated goal of the competition is to raise awareness of the threat Burmese pythons pose to the Everglades ecosystem. The snakes are native to Southeast Asia and have no known predators in Florida.The contest also serves as a pilot program to determine whether regular hunting competitions can cull the growing population of the invasive species, said Frank Mazzotti, a wildlife expert from the University of Florida who helped create the competition.Python Challenge rules require contestants to kill specimens on the spot in a humane fashion, recommending shooting the snakes precisely through the brain.“I was hoping there would be a lot of machetes and not a lot of guns,” said Polster, the retired salesman. He said he worries “these idiots will be firing all over the place.”Shawn Heflick, star of the National Geographic “Wild” television show “Python Hunters,” told Reuters that despite the formidable size of the snakes, he expects the swamp itself, with its alligators, crocodiles and venomous snakes, to pose a greater threat to the contestants.“You get these people going down there, they’ll get lost, they’ll get dehydrated, they’ll get sucked dry by mosquitoes,” Heflick said.Segelson said the wildlife agency will provide training on the use of GPS devices and on identifying venomous snakes at the kick-off event. In the meantime, she said, contestants should be familiarizing themselves with the Everglades environment, just as they should before entering any other strange territory.Heflick said most of the contestants likely were drawn to the Python Challenge by the romantic mystique of bagging a giant predator. He expects few will last long in the hunt.“The vast majority of them will never see a python. The vast majority of them will probably curtail their hunting very quickly when they figure out there’s a lot of mosquitoes, it’s hot, it’s rather boring sometimes - most of the time really, and I think a lot of them will go home,” Heflick said.Editing by Tom Brown and Dan GreblerOur Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSKBN1OD2CM\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSKBN1OD2CM\n",
      "TITLE: UK woman illegally imported python-skin products\n",
      "BODY:\n",
      "By Reuters Staff2 Min ReadLONDON (Reuters) - A British woman who illegally imported and sold fashion accessories made from python skin was convicted on Friday, London police said.Stephanie Scolaro, 26, was involved in the illegal import of a parcel containing 10 python-skin hats and two bags which was seized by customs in 2016 at Leipzig airport in Germany, Southwark Crown Court had heard.An investigation began after London police’s Wildlife Crime Unit was alerted to the incident.The enquiry found that Scolaro operated an online company named ‘SS-Python.com’, where she sold python-skin hats, bags, chokers and mobile phone covers.More illicit python snake products were subsequently found in Scolaro’s central London home, and for sale at three shops in London.“Pythons are one of many species protected under CITES, an international treaty to protect endangered plants and animals,” said Sarah Bailey of the Wildlife Crime Unit.“Any trade outside of the legal framework could threaten the conservation status of the remaining wild population and contribute to its decline in the wild,” she added in a statement.Scolaro, who pleaded guilty, will be sentenced on Jan. 21.Reporting by Amy O’Brien; editing by Stephen AddisonOur Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSBRE9640KE20130705\n",
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSBRE9640KE20130705\n",
      "TITLE: Monty Python lose 'Spamalot' court battle\n",
      "BODY:\n",
      "By Li-mei Hoang4 Min ReadLONDON (Reuters) - The surviving members of the British comedy troupe Monty Python lost a High Court battle on Friday over tens of thousands of pounds in royalties from their hit Broadway musical “Spamalot”.Monty Python members Eric Idle (R) and Terry Jones return to the High Court after a lunch break in central London December 4, 2012. REUTERS/Andrew WinningMark Forstater, who helped produce the 1975 film “Monty Python and the Holy Grail” on which the stage show was based, said he had not received his fair share of the profits from the spin-off.Despite hearing evidence from three Pythons - Eric Idle, Michael Palin and Terry Jones - the judge, Justice Alastair Norris, sided with the producer.“I have always been adamant I was correct. I have been proved right - justice has prevailed,” Forstater said.Inspired by the film, the musical opened on Broadway in 2005 and has also enjoyed a successful run in Britain. Idle wrote the lyrics and collaborated on most of the music.Forstater, an American based in Britain, argued that he was entitled to one-seventh of the profits from the “Holy Grail” film and any merchandise or spin-offs.His lawyer told the court that for the purposes of profit-sharing, it had been agreed in 1974 that Forstater was “the seventh Python”.However Palin, along with Jones and Idle, who formed Monty Python with John Cleese, Terry Gilliam and Graham Chapman, dismissed this suggestion.“The idea of a seventh Python just doesn’t happen ... I don’t think there was ever any suggestion this man was going to be a ‘seventh Python’,” said Palin, giving evidence in December.In his judgment, Norris said Palin had been a “balanced and trustworthy” witness but had admitted his recollection was “hazy”, while evidence from Jones had been “suffused with a sense that Mr Forstater had done very well out of his brief connection with the Pythons”.“Eric Idle was frank enough to acknowledge that he now disliked Mr Forstater, but he expressed the hope that, in his evidence, he was being honest and that his dislike did not affect his honesty,” Norris said.“He undoubtedly regarded Mr Forstater as ungrateful.”“SOFT LOT”His ruling also made reference to the Pythons’ lack of business acumen, highlighting a diary entry from Palin in 1975.“As we are a soft lot and not at all businesslike, I think it would be in the finest traditions of Python irrationality if we gave Mark an extra 1,000 pounds and a silver tray with some cut-glass sherry glasses and told him to stop writing to us for more money,” Palin wrote.“Beyond that even I am not prepared to go. Oh, all right, some cheese straws to go with the sherry glasses.”The judge added: “As I assess the evidence, the Pythons continued at that point to be ‘a soft lot and not at all businesslike’.”No members of the Python group were present in court to hear the ruling.Final figures will be worked out at later hearings but Forstater told reporters he was entitled to more than 200,000 pounds ($300,000) including interest, the Press Association reported.“There is a sadness, though, about having to face people who were my friends in court,” he said. “The friendship has gone.”Forstater said he did not think the litigation would damage the Pythons’ reputation. “They’re an institution,” he said. “I still think they are very funny.”Editing by Michael Holden and Janet LawrenceOur Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSBREA141OP20140205\n",
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSBREA141OP20140205\n",
      "TITLE: 18-foot Burmese python found in Florida Everglades\n",
      "BODY:\n",
      "By Reuters Staff2 Min ReadSlideshow ( 2 images )MIAMI (Reuters) - Engineers in the Everglades stumbled upon a near-record-breaking Burmese python measuring more than 18 feet long during a routine inspection of levees on Tuesday, a water management district spokesman said.The snake, measuring at 18 feet 2 inches, fell short of the state record by 6 inches, according to the Florida Fish and Wildlife Conservation Commission.Last year, a snake collector in the state discovered the largest python on record there, measuring 18 feet 8 inches, commission spokeswoman Katie Johnson said.The pythons, which can grow to more than 20 feet in their native habitat in Southeast Asia, are one of the most problematic invaders of Florida’s sprawling Everglades wetlands.They eat indigenous species and their food sources, fueling concerns that the predator snakes will fundamentally change the ecosystem.The python found on Tuesday was killed, and its corpse was taken to the University of Florida, where it will be measured and studied by scientists trying to combat the species, according to South Florida Water Management District spokesman Gabe Margasak.Officials have said the python population is believed to have grown to as many as 150,000 in the Everglades. The cold-blooded reptiles are often found atop levees, where they lie for hours at a time to warm up under the Florida sun.The snakes, one of the largest species in the world, found a home to their liking in the Everglades when pet owners started using the wetlands as a convenient dumping ground.Reporting by Zachary Fagenson; Editing by Colleen Jenkins and Lisa Von AhnOur Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSKCN11W1LT\n",
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSKCN11W1LT\n",
      "TITLE: Indian man bitten by python whilst posing for selfie\n",
      "BODY:\n",
      "By 2 Min ReadForest department officials on Saturday (September 24) promised strict action after a python attacked a man when a group was trying to click a photo with it.The incident took place on Friday (September 23) after forest department officials removed the python from the premises of a school in northwestern Rajasthan state. They were posing for pictures with it after successfully capturing it.Deputy Conservator of Forests, K.G.Shrivastav, said the incident was unexpected as all officers were highly trained.“I wanted the snake to be released in my presence, but the officials released it before I could reach the spot. I had no clue that civilians were also present at the spot and something could go wrong as all the officials are highly trained. Clicking selfies with the snake and releasing the photos publicly is an offense under the Wildlife Act and we will issue notice to all involved in this incident,” Shrivastav said.The victim, Ashok Bishnoi, accepted the incident happened due to carelessness on his part.“The moment I received the news I reached the location. I did not realize how close I was to the python and it attacked me. This is entirely my fault,” said the victim, Ashok Bishnoi.Vishnoi added that the mishap happened in a short span of time in spite of the fact that the rescue officials had a strong grip on the reptile.The snake was later released in a forest.\n",
      "http://reuters.com/article/idUSKBN1711E3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSKBN1711E3\n",
      "TITLE: Indonesian found dead after being swallowed by python\n",
      "BODY:\n",
      "By Reuters Staff2 Min ReadSlideshow ( 2 images )JAKARTA (Reuters) - An Indonesian farmer has been found dead inside a 7-meter (23-foot) python after being reported missing on his failure to return home from work on a palm plantation.Village officials cut open the swollen body of the snake in a graphic video taken by a resident of Mamuju, on the eastern island of Sulawesi, the site of the incident. The victim’s legs, encased in rubber boots, emerge as the snake is pulled apart.Family members and neighbors of the 26-year-old victim, Akbar, had launched a search when he failed to come home for more than 24 hours, a resident of the area told Reuters.“We saw a python that couldn’t move properly and it’s belly was swollen,” said Asdin Rudi Fathir, 43. “Residents and officials decided to kill the python and were shocked to find a human body inside.”Relatives identified the victim, who went by only one name, as do many Indonesians.While large snakes and other animals are common in that part of Sulawesi, attacks on humans are rare, Fathir added.Reporting by Kanupriya Kapoor; Editing by Clarence FernandezOur Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSL5N0J50QB20131120\n",
      "New article found for topic: python\n",
      "URL: http://reuters.com/article/idUSL5N0J50QB20131120\n",
      "TITLE: Monty Python not dead after all - stage show planned\n",
      "BODY:\n",
      "By Reuters Staff2 Min ReadLONDON, Nov 20 (Reuters) - The comic team Monty Python, whose BBC TV series from the 1970s and feature films took their subversive humour and “Dead Parrot” routine around the world, are to reunite for a stage show, British media reported on Wednesday.A news release issued on behalf of the five surviving Pythons, Eric Idle, John Cleese, Terry Gilliam, Michael Palin and Terry Jones, all in their 70s, said that an official announcement would be made on Thursday.But several British newspapers and media outlets reported that the five would be appearing on stage for the first time together since the 1980s.The group was famed for its skits about a man trying to return a dead parrot to a shopkeeper who claimed the bird was “resting” and for poking fun at the establishment, the military and religion.“We’re getting together and putting on a show - it’s real,” Jones told the BBC.“I’m quite excited about it. I hope it makes us a lot of money. I hope to be able to pay off my mortgage!”On his Twitter account, Palin wrote: “The Python rides again...?” while Idle on Tuesday tweeted: “Only three days to go till the Python Press Conference. Make sure Python fans are alerted to the big forthcoming news event.”The BBC programme “Monty Python’s Flying Circus” was made for television between 1969 and 1974.The Pythons went on to make films including “Monty Python And The Holy Grail” (1975) and “Monty Python’s Life of Brian” (1979).They last performed live together in Monty Python Live At The Hollywood Bowl in 1982. The sixth Python, Graham Chapman, died of cancer at age 48 in 1989. (Writing by Michael Roddy; Editing by Doina Chiacu)Our Standards: The Thomson Reuters Trust Principles.\n",
      "https://www.reuters.com/search/news?blob=data%20science\n",
      "http://reuters.com/article/idUSL4N33D1H6\n",
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSL4N33D1H6\n",
      "TITLE: CORRECTED-Data science firm Tredence raises $175 mln from PE firm Advent\n",
      "BODY:\n",
      "By Reuters Staff2 Min Read(Corrects paragraph 3 to add dropped word “not”)BENGALURU, Dec 23 (Reuters) - Tredence, a data science and artificial intelligence company, has raised $175 million from private equity firm Advent International, the company said on Friday.The Series B round values Tredence, which has Indian founders but is headquartered in San Jose, at $500 million, the Economic Times newspaper reported, citing Tredence chief executive and co-founder, Shub Bhowmick.Tredence, which gets 80% of its revenue from North America, did not immediately respond to Reuters’ request for comment.Advent will take a minority stake in Tredence and join its board, while Chicago Pacific Founders, a PE firm that invested in Tredence in December 2020, will remain a shareholder, Tredence said in a statement.The company was started by Bhowmick, Sumit Mehra and Shashank Dubey in 2013 and raised $30 million in its Series A funding round in late 2020.It has over 1,800 employees and offices in San Jose, Foster City, Chicago, London, Toronto and Bengaluru. (Reporting by Nandan Mandayam in Bengaluru; Editing by Savio D’Souza)Our Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSL1N36702Q\n",
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSL1N36702Q\n",
      "TITLE: Biden to discuss risks of AI in Tuesday meeting with science advisers\n",
      "BODY:\n",
      "By Jeff Mason2 Min ReadWASHINGTON, April 4 (Reuters) - U.S. President Joe Biden will discuss the “risks and opportunities” that artificial intelligence poses for people, society and national security during a meeting with science and technology advisers at the White House on Tuesday, an official said.Biden, a Democrat, is scheduled to meet with the President’s Council of Advisors on Science and Technology (PCAST) on the same day that his predecessor, former President Donald Trump, surrenders in New York over charges stemming from a probe into hush money paid to a porn star.Biden has declined to comment on Trump’s legal woes, and Democratic strategists say his focus on governing will create a politically advantageous split screen of sorts as his former rival, a Republican, deals with his legal challenges.“The president will discuss the importance of protecting rights and safety to ensure responsible innovation and appropriate safeguards,” a White House official said in a statement ahead of Biden’s meeting.“He will call on Congress to pass bipartisan privacy legislation to protect kids and limit personal data tech companies collect on all of us.”Artificial intelligence or AI is becoming a hot topic for policy makers.The tech ethics group Center for Artificial Intelligence and Digital Policy has asked the U.S. Federal Trade Commission month to stop OpenAI from issuing new commercial releases of GPT-4, which has wowed and appalled users with its human-like abilities to generate written responses to requests.Democratic U.S. Senator Chris Murphy has urged society to pause as it considers the ramifications of AI.Last year the Biden administration released a blueprint “Bill of Rights” to help ensure users’ rights are protected as technology companies design and develop AI systems. (Reporting by Jeff Mason; Editing by Stephen Coates)Our Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSL1N33A11T\n",
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSL1N33A11T\n",
      "TITLE: SPECIAL REPORT-Why detransitioners are crucial to the science of gender care\n",
      "BODY:\n",
      "By Robin Respaut, Chad Terhune, Michelle Conlin39 Min Read(Takes sidebar USA-TRANSYOUTH/STUDIES)TORONTO, Dec 22 (Reuters) - For years, Dr Kinnon MacKinnon, like many people in the transgender community, considered the word “regret” to be taboo.MacKinnon, a 37-year-old transgender man and assistant professor of social work at York University here, thought it was offensive to talk about people who transitioned, later regretted their decision, and detransitioned. They were too few in number, he figured, and any attention they got reinforced to the public the false impression that transgender people were incapable of making sound decisions about their treatment.“This doesn’t even really happen,” MacKinnon recalled thinking as he listened to an academic presentation on detransitioners in 2017. “We’re not supposed to be talking about this.”MacKinnon, whose academic career has focused on sexual and gender minority health, assumed that nearly everyone who detransitioned did so because they lacked family support or couldn’t bear the discrimination and hostility they encountered – nothing to do with their own regret. To learn more about this group for a new study, he started interviewing people.In the past year, MacKinnon and his team of researchers have talked to 40 detransitioners in the United States, Canada and Europe, many of them having first received gender-affirming medical treatment in their 20s or younger. Their stories have upended his assumptions.Many have said their gender identity remained fluid well after the start of treatment, and a third of them expressed regret about their decision to transition from the gender they were assigned at birth. Some said they avoided telling their doctors about detransitioning out of embarrassment or shame. Others said their doctors were ill-equipped to help them with the process. Most often, they talked about how transitioning did not address their mental health problems.In his continuing search for detransitioners, MacKinnon spent hours scrolling through TikTok and sifting through online forums where people shared their experiences and found comfort from each other. These forays opened his eyes to the online abuse detransitioners receive – not just the usual anti-transgender attacks, but members of the transgender community telling them to “shut up” and even sending death threats.“I can’t think of any other examples where you’re not allowed to speak about your own healthcare experiences if you didn’t have a good outcome,” MacKinnon told Reuters.The stories he heard convinced him that doctors need to provide detransitioners the same supportive care they give to young people to transition, and that they need to inform their patients, especially minors, that detransitioning can occur because gender identity may change. A few months ago, he decided to organize a symposium to share his findings and new perspective with other researchers, clinicians, and patients and their families.Not everyone was willing to join the discussion. A Canadian health provider said it couldn’t participate, citing recent threats to hospitals offering youth gender care. An LGBTQ advocacy group refused to promote the event. MacKinnon declined to identify either, telling Reuters he didn’t want to single them out. Later, after he shared his findings on Twitter, a transgender person denounced his work as “transphobia.”He expected his research would be a hard sell even to many of the 100 or so people from Canada, the United States and elsewhere who accepted his invitation. “I need your help,” he told the crowd that assembled in November in a York University conference room for the daylong session. “My perspectives have changed significantly. But I recognize that for many of you, you may find yourselves feeling much like I did back in 2017 – challenged, apprehensive, maybe fearful.”FIGHTING WORDSIn the world of gender-affirming care, as well as in the broader transgender community, few words cause more discomfort and outright anger than “detransition” and “regret.” That’s particularly true among medical practitioners in the United States and other countries who provide treatment to rising numbers of minors seeking to transition.They insist, as MacKinnon once did, that detransitioning is too rare to warrant much attention, citing their own experiences with patients and extant research to support their view. When someone does detransition, they say, it’s almost never because of regret, but rather, a response to the hardship of living in a society where transphobia still runs rampant.“These patients are not returning in droves” to detransition, said Dr Marci Bowers, a transgender woman, gender surgeon and president of the World Professional Association for Transgender Health (WPATH), an international group that sets guidelines for transgender care. Patients with regret “are very rare,” she told Reuters. “Highest you’ll find is 1% or 1.5% of any kind of regret.”Doctors and many transgender people say that focusing on isolated cases of detransitioning and regret endangers hard-won gains for broader recognition of transgender identity and a rapid increase in the availability of gender care that has helped thousands of minors. They argue that as youth gender care has become highly politicized in the United States and other countries, opponents of that care are able to weaponize rare cases of detransition in their efforts to limit or end it altogether, even though major medical groups deem it safe and potentially life-saving.“Stories with people who have a lot of anger and regret” about transitioning are over-represented in the media, and they don’t reflect “what we are seeing in the clinics,” said Dr Jason Rafferty, a pediatrician and child psychiatrist at Hasbro Children’s Hospital in Providence, Rhode Island. He also helped write the American Academy of Pediatrics’ policy statement in support of gender-affirming care. Detransitioning is a “very invalidating term for a lot of people who are trans and gender-diverse,” Rafferty said.Some people do detransition, however, and some do so because of regret. The incidence of regret could be as low as clinicians like Bowers say, or it could be much higher. But as Reuters found, hard evidence on long-term outcomes for the rising numbers of people who received gender treatment as minors is very weak.Dr Laura Edwards-Leeper, a clinical psychologist in Oregon who treats transgender youths and a co-author of WPATH’s new Standards of Care for adolescents and children, said MacKinnon’s work represents some of the most extensive research to date on the reasons for detransitioning and the obstacles patients face. She said the vitriol he has encountered illustrates one reason so few clinicians and researchers are willing to broach the subject.“People are terrified to do this research,” she said.For this article, Reuters spoke to 17 people who began medical transition as minors and said they now regretted some or all of their transition. Many said they realized only after transitioning that they were homosexual, or they always knew they were lesbian or gay but felt, as adolescents, that it was safer or more desirable to transition to a gender that made them heterosexual. Others said sexual abuse or assault made them want to leave the gender associated with that trauma. Many also said they had autism or mental health issues such as bipolar disorder that complicated their search for identity as teenagers.Echoing what MacKinnon has found in his work, nearly all of these young people told Reuters that they wished their doctors or therapists had more fully discussed these complicating factors before allowing them to medically transition.No large-scale studies have tracked people who received gender care as adolescents to determine how many remained satisfied with their treatment as they aged and how many eventually regretted transitioning. The studies that have been done have yielded a wide range of findings, and even the most rigorous of them have severe limitations. Some focus on people who began treatment as adults, not adolescents. Some follow patients for only a short period of time, while others lose track of a significant number of patients.“There’s a real need for more long-term studies that track patients for five years or longer,” MacKinnon said. “Many detransitioners talk about feeling good during the first few years of their transition. After that, they may experience regret.”In October, Dutch researchers reported results of what they billed as the largest study to date of continuation of care among transgender youths. In a review of prescription drug records, they found that 704, or 98%, of 720 adolescents who started on puberty blockers before taking hormones had continued with treatment after four years on average. The researchers couldn’t tell from the records why the 16 had discontinued treatment.Gender-care professionals and transgender-rights advocates hailed the 98% figure as evidence that regret is rare. However, the authors cautioned that the result may not be replicated elsewhere because the adolescents studied had undergone comprehensive assessments, lasting a year on average, before being recommended for treatment. This slower, methodical approach is uncommon at many U.S. gender clinics, where patient evaluations are typically done much faster and any delay in treatment, or “gatekeeping,” is often believed to put youth at risk of self-harm because of their distress from gender dysphoria.Dr Marianne van der Loos, the Dutch study’s lead author, is a physician at Amsterdam University Medical Center’s Center for Expertise on Gender Dysphoria, a pioneer in gender care for adolescents. “It’s important to have evidence-based medicine instead of expert opinion or just opinion at all,” van der Loos said.Reliable evidence of the frequency of detransition and regret is important because, as MacKinnon, van der Loos and other researchers say, it could be used to help ensure that adolescent patients receive the best possible care.A basic tenet of modern medical science is to examine outcomes, identify potential mistakes, and, when deemed necessary, adjust treatment protocols to improve results for patients. For example, only after large international studies analyzing outcomes for thousands of patients did researchers establish that implanted coronary artery stents were no better than medication for treating most cases of heart disease.Stronger data on outcomes, including the circumstances that make regret more likely, would also help transgender teens and their parents make better-informed decisions as they weigh the benefits and risks of treatments with potentially irreversible effects.“We cannot carry on in this field that involves permanently changing young people’s bodies if we don’t fully understand what we’re doing and learn from those we fail,” said Edwards-Leeper, the clinical psychologist and WPATH member. “We need to take responsibility as a medical and mental-health community to see all the outcomes,” she said in an interview.As Reuters reported in October, thousands of families in the U.S. have been weighing these difficult choices amid soaring numbers of children diagnosed with gender dysphoria, the distress experienced when a person’s gender identity doesn’t align with their gender assigned at birth. They have had to do so based on scant scientific evidence of the long-term safety and efficacy of gender-affirming treatment for minors.Concern about how to cope with the growing waiting lists at gender clinics that treat minors has divided experts. Some urge caution to ensure that only adolescents deemed well-suited to treatment after thorough evaluation receive it. Others argue that any delay in treatment prolongs a child’s distress and puts them at risk of self-harm.DETRANSITION DEFINEDDetransitioning can mean many things. For those who transitioned socially, it may entail another change in name, preferred pronouns, and dress and other forms of identity expression. For those who also received medical treatment, detransitioning typically includes halting the hormone therapy they otherwise would receive for years.Nor do all people who stop treatment regret transitioning, according to interviews with detransitioners, doctors and researchers. Some end hormone therapy when they have achieved physical changes with which they are comfortable. Some are unhappy with the side effects of hormones, such as male pattern baldness, acne or weight gain. And some are unable to cope with the longstanding social stigma and discrimination of being transgender.Doctors and detransitioners also described the challenging physical and emotional consequences of the process. For example, patients who had their ovaries or testes removed no longer produce the hormones that match their gender assigned at birth, risking bone-density loss and other effects unless they take those hormones the rest of their lives. Some may undergo years of painful and expensive procedures to undo changes to their bodies caused by the hormones they took to transition. Those who had mastectomies may later undergo breast reconstruction surgery. As parents, they may regret losing the ability to lactate. Detransitioners also may need counseling to cope with the process and any lingering regret.The impact can be social, too. In a study published last year in the Journal of Homosexuality, a researcher in Germany surveyed 237 people who had socially or medically transitioned and later detransitioned, half of them having transitioned as minors. Many respondents reported a loss of support from the LGBTQ community and friends, negative experiences with medical professionals, difficulty in finding a therapist familiar with detransition and the overall isolation after detransition.“Many respondents described experiences of outright rejection from LGBT+ spaces due to their decision to detransition,” wrote Elie Vandenbussche, the study’s author, a detransitioner and at the time a student at Rhine-Waal University of Applied Sciences. “It seems reasonable to suspect that this loss of support experienced by detransitioners must have serious implications on their psychological well-being.”In its new Standards of Care, released in September, WPATH cited Vandenbussche’s paper and a few others on detransitioning and continuation of care among younger patients. “Some adolescents may regret the steps they have taken,” the WPATH guidelines say. “Therefore, it is important to present the full range of possible outcomes when assisting transgender adolescents.”However, Bowers, WPATH’s president, is among several gender-care specialists who say patients are ultimately responsible for choices they make about treatment, even as minors. They should not be “blaming the clinician or the people who helped guide them,” she said. “They need to own that final step.”WPATH’s guidelines acknowledge the lack of research on long-term outcomes for youth who didn’t undergo comprehensive assessments, saying that the “emerging evidence base indicates a general improvement in the lives of transgender adolescents” who receive treatment after careful evaluation. “Further, rates of reported regret during the study monitoring periods are low,” the guidelines say.Specific treatment protocols for detransitioning are hard to find. WPATH’s guidelines don’t provide detailed advice to clinicians on treating patients who detransition. The Endocrine Society’s guidelines for gender-affirming care, published in 2017, don’t address the issue, either. The “question of discontinuing hormone treatment is beyond the scope covered by the current guideline,” an Endocrine Society spokeswoman said.Some doctors think they – and patients – would benefit from more guidance. “We have guidelines to guide us in providing transition-related care, initiating hormones and managing them long-term. Equally as important would be having guidelines in deprescribing hormones in the safest way possible,” said Dr Mari-Lynne Sinnott, a doctor who attended MacKinnon’s symposium. She runs one of the only family medical practices in Newfoundland focused on gender-diverse people, who make up about half of her 1,500 patients.“SURE OF MY IDENTITY”Max Lazzara’s childhood in Minneapolis, Minnesota, was chaotic, with divorce, “moving around a lot, some emotionally abusive stuff at home,” she said. Her mother worked full-time, so Lazzara did most of the cooking, cleaning and caring for her little brother. She began to cut and burn herself as a means of coping and had tried to commit suicide three times before she entered high school, according to Lazzara and her medical records, which cite a history of bipolar disorder.“The life of a woman was bleak to me,” Lazzara told Reuters. “I worried that I would have to get married to a man someday and have a baby. I wanted to run far away from that.”In early 2011, when Lazzara was 14, she started questioning her gender identity. After discovering forums on Tumblr where young people described their transitions, she felt like something snapped into place. “I thought, ‘Wow, this could explain why my whole life felt wrong.’”During the summer of that year, Lazzara changed her name and began experimenting with presenting as more masculine. It felt good to cut her hair and wear gender-neutral or men’s clothing. She took medications and received therapy to treat bipolar disorder. But it wasn’t enough to alleviate her distress. In April 2012, Lazzara was admitted to the hospital at the University of Minnesota after a fourth suicide attempt.Three weeks later, she sought care at the university’s Center for Sexual Health, where she was diagnosed with gender identity disorder. Lazzara told the clinic she was “sure of my identity,” according to her medical records. She wanted hormones and surgeries, the records show, including a mastectomy, a hysterectomy, and liposuction to slim her legs and hips. She was horrified at her body, could not look down in the shower and felt “absolute dread at the time of menstrual cycle,” the records note.“I felt so strongly. I thought nothing would change my mind,” Lazzara told Reuters.Clinicians at the university warned families that their children were suicidal “because they are born in the wrong bodies,” Lazzara’s mother, Lisa Lind, told Reuters. “I thought, ‘I’ll do whatever it takes, so she doesn’t kill herself.’”Lazzara started taking testosterone in the fall of 2012, at age 16. She was still binding her breasts – so tightly, she said, that her ribs deformed. After a man groped her on the street, she decided to have breast-removal surgery, tapping the college fund her grandmother had left for her to cover the nearly $10,000 cost.Initially, Lazzara was happy with her transition. She liked the changes from taking testosterone – the redistribution of fat away from her hips, the lower voice, the facial hair – and she was spared the sexist cat-calling that her female friends endured. “I felt like I was growing into something I wanted to be,” Lazzara said.But her mental health continued to deteriorate. She attempted suicide twice more, at ages 17 and 20, landing in the hospital both times. Her depression worsened after a friend sexually abused her. She became dependent on prescription anti-anxiety medication and developed a severe eating disorder.During the summer of 2020, Lazzara was spiraling. She realized she no longer believed in her gender identity, but “I didn’t see a way forward.”That October, Lazzara was working as a janitor in an office building in the Seattle area when she caught her reflection in a bathroom mirror. For the first time, she said, she saw herself as a woman. “I had not allowed myself to have that thought before,” she said. It was shocking but also clarifying, she said, and “a peaceful feeling came over me.”Then she began to ponder her sexuality. In middle school, she had crushes on girls. After her transition, she identified as a transgender man who was bisexual. Now, she realized, she was a lesbian.Lazzara stopped taking testosterone. She later asked her doctor in the Seattle area for advice, but he seemed unsure about how to proceed. She found a new doctor and recently sought laser hair removal on her face.Lazzara told Reuters she now realizes that gender treatment was not appropriate for her and that it took a toll on her physical and mental health. “I do wish my doctors had said to me, ‘It’s OK to feel disconnected from your body. It’s OK to like girls. It’s OK to be gender non-conforming.’”Her original gender-care providers at the University of Minnesota declined to comment. In a statement, the university’s medical school said “gender-affirming care involves a carefully thought-out care plan between a patient and their multidisciplinary team of providers.”Lazzara recently found the before-and-after pictures of her torso on the website of the surgeon who performed her mastectomy in 2013. She had given him permission to post the images because he was proud of the outcome. Seeing her body as it once was stunned her. “I saw my breasts before I got them removed. That’s my 16-year-old body,” she said. “I had no ability at that age to be in my own body in my own way.”Since revealing she detransitioned, Lazzara said, many in the online transgender community who embraced her a decade ago have distanced themselves from her, and she has received hateful messages on social media. Now, when she sees someone come out online as detransitioned, she sends them a private message of support. “I know how lonely and alienating it can be,” she said.“SHUT UP,” DETRANSITIONERTransgender people are frequently subjected to harassment, abuse and threats online. And as Lazzara’s experience shows, so are detransitioners. In recent posts on TikTok, users took turns telling detransitioners to “shut up,” and mocked, attacked and blamed them for perpetuating harm on the transgender community. Diana Salameh, a transgender woman, film director and comedian from Mississippi, posted a TikTok video on Oct. 1 to “all the so-called transgender detransitioners out there.”Detransitioners “are just giving fuel to the fire to the people who think that no trans person should exist,” she said in the video. “You people who jumped the gun, made wrong decisions that you should actually feel embarrassed for, but you want to blame somebody else.” In closing, she said, “I think you all need to sit down and shut the fuck up!”Salameh told Reuters she posted the video because detransitioners spread the false idea “that nobody can be happy after transition,” and right-wing opponents of youth gender care are using their stories “to fuel their agendas.”Earlier this year, K.C. Miller, a 22-year-old in Pennsylvania who was assigned female at birth, began wrestling with how she felt about her medical transition.Miller initially sought treatment for gender dysphoria when she was 16 from the adolescent gender clinic at Children’s Hospital of Philadelphia. In September 2017, Miller met with Dr Linda Hawkins, a counselor and co-founder of the hospital’s gender clinic, for the first of two 90-minute visits. During that session, Miller told Hawkins she had wanted to be a Boy Scout as a kid and “always felt like a tomboy,” according to Hawkins’ notes in Miller’s medical records, reviewed by Reuters. Miller also told Reuters that as a young girl she was attracted to other girls, but didn’t feel she could pursue those relationships because her family’s church didn’t accept homosexuality.Miller’s case had further complications. Hawkins noted that Miller had an extensive history of sexual abuse by a family member starting at age 4, and that as a result, Miller had already been diagnosed with anxiety and post-traumatic stress disorder. Miller had been admitted to a psychiatric hospital for 10 days because of suicidal thoughts in late 2016.While in the hospital, Miller told her mother she wished she wasn’t a girl “because then the abuse would not have happened,” Hawkins wrote. Elsewhere in the records, Hawkins noted that “Mom expresses concern that the desire to be male and not female may be a trauma response.”Miller, her mother and Hawkins met again seven weeks later. Miller had continued to have suicidal thoughts. She had taken medication for depression and anxiety and was working with a therapist, Hawkins noted. By the end of that second visit, Hawkins concluded that, “in spite of” Miller’s trauma from abuse, the 16-year-old “has been insistent, persistent and consistent” in thinking of herself as male.Hawkins referred Miller to a local gender clinic to receive testosterone. Miller got a mastectomy about six months later.But medical treatment didn’t offer the relief she sought. Her body started to change due to the hormones, yet Miller didn’t feel better. Instead, she cycled through bouts of depression. She passed as a young man, but “something felt off. It felt like I was putting on an act.”Then Miller began reading the stories posted online by young detransitioners. Parts of their experiences resonated with her. “I absolutely would not have done this if I could go back and do it again,” Miller told Reuters. “I would have worked through therapy and would be living my life as a lesbian.”Miller said Hawkins should have done a more thorough evaluation of all of Miller’s mental health issues and shouldn’t have recommended treatment so quickly.Her mother, who asked not to be identified to protect her privacy, told Reuters that providers assured her that Miller’s distress was related to her gender identity and that gender-affirming care would reduce the risk of suicide.A spokesman for Children’s Hospital of Philadelphia declined to comment, citing patient privacy.Sitting in her car in early October, Miller let out years of frustration in a video posted on Twitter. She told viewers she felt she looked too masculine to detransition. She described how testosterone thinned her hair. “I don’t see me personally being able to come back from what’s happened,” she said in the video.The video went viral, registering nearly four million views within days and igniting an avalanche of comments. Two days after Miller’s post, Alejandra Caraballo, a transgender woman, LGBTQ-rights advocate and clinical instructor at Harvard Law School’s Cyberlaw Clinic, wrote on Twitter: “The detransition grift where you complain about transitioning not making you look like a greek god but you also aren’t actually detransitioning yet because you don’t feel like your birth gender and you follow a bunch of anti-trans reactionaries that want all trans people gone.”Caraballo told Reuters she reacted to Miller’s video because those types of detransition stories are “outlier examples being used by many on the anti-trans side to undermine access to gender-affirming care. They aren’t representative of detransitioners on the whole.”In other posts and direct messages, some transgender people Miller had once idolized made fun of her appearance and criticized her decisions. One person made a death threat.A few weeks later, Miller said she stopped taking testosterone, began to feel suicidal and sought psychiatric care. She uses female pronouns among friends, but still presents as a man in public.In its Standards of Care, WPATH says many detransitioners “expressed difficulties finding help during their detransition process and reported their detransition was an isolating experience during which they did not receive either sufficient or appropriate support.”In May, Dr Jamison Green, a transgender man, author and former president of WPATH, said he was encouraged when about 30 medical professionals attended an online WPATH seminar he and other gender-care specialists helped lead. The session was intended to help providers better serve detransitioners and other patients with an evolving gender identity.“I wish people in the transgender community would be less judgmental about people who change their mind,” Green said. “Transgender people, especially when they are newer to the community, can be really brutal to people for not conforming. I really think it’s harmful for everybody.”WORD SEARCH PITFALLSEver since the first clinic to offer gender care to minors in the United States opened in Boston 15 years ago, none of the leading providers have published any systematic, long-term studies tracking outcomes for all patients.In 2015, the National Institutes of Health funded a study to examine outcomes for about 400 transgender youth treated at four U.S. children’s hospitals, including the gender clinic at Boston Children’s Hospital. Researchers have said they are looking at “continuation of care.” However, long-term results are years away.That has left a small assortment of studies to guide clinicians in this emerging field of medicine. The results of these studies suggest a wide range of possibilities for rates of detransitioning, from less than 1% to 25%. The research provides even less certainty about the incidence of regret among patients who received medical treatment as minors. And the studies have serious drawbacks.Two of the largest ones, which found that 2% or less of people who transitioned experienced regret, focused on Europeans who primarily initiated treatment as adults. Experts caution that the results, because of the differences in maturity and life experiences between adults and adolescents, may have limited relevance as an indicator of outcomes for minors.Researchers acknowledge that studies that follow patients for only a short time may underestimate detransition and regret because evidence indicates some people may not reach that point until as long as a decade after treatment began. Some studies also lose track of patients – a recurring challenge as minors age out of pediatric clinics and have to seek care elsewhere.Even the choice of search terms can trip up researchers, as apparently happened in a study published in May by Kaiser Permanente, a large integrated health system based in Oakland, California.That study examined 209 patients who underwent gender-affirming mastectomies as minors between 2013 and 2020 in Kaiser’s northern California region. Its authors searched the patients’ medical records for words such as “regret,” “dissatisfaction,” “unsatisfied” and “unhappy” as indicators of regret. They didn’t look for the term “detransition,” according to the study.Their search yielded two patients who had expressed regret, or less than 1% of the group studied. The two patients, identified as nonbinary, had top surgery at age 16, and expressed regret within a year and a half.Reuters found two other patients in the region covered by the study who don’t match those characteristics and whom the Kaiser researchers apparently missed. Both have been outspoken about their detransitions.One is Max Robinson, who was 16 when she sought gender care at Kaiser in 2012. Her pediatric endocrinologist prescribed a puberty blocker and later testosterone.The doctor monitored Robinson’s hormone levels, wrote numerous letters to help Robinson change her legal gender from female to male, and recommended a plastic surgeon in San Francisco, Robinson’s medical records show. “I have no reservations recommending Max as a well adjusted candidate for breast reduction,” the Kaiser endocrinologist wrote to the surgeon in May 2013. Max had the surgery six weeks later, when she was 17.After the surgery, Robinson felt better. But within a year, her mental health issues, including anxiety and depression, had escalated, medical records show.In November 2015, three years after starting testosterone and two years after her surgery, Robinson told the Kaiser physician she was now seeing that she wasn’t interested in taking hormones any longer. “I’m no longer going to be using testosterone, so I don’t need further appointments or for those prescriptions to be active,” she wrote to the doctor. Two months later, she asked Kaiser to provide a letter confirming her detransition so she could change her legal records back to female. Kaiser obliged.“The whole experience alienated me from my doctors,” she told Reuters.Robinson began to speak publicly about her decision to detransition and in 2021 published “Detransition: Beyond Before and After,” a book in which she details her own process of medical transition and detransition.The other patient was Chloe Cole. According to a letter of intent to sue that her lawyers sent to Kaiser in November, Cole was 13 when a Kaiser doctor in 2018 put her on a puberty blocker, followed a few weeks later by testosterone, for her gender-affirming treatment.At 15, Cole told Reuters, she also wanted top surgery. In an interview, she and her father said the doctors at Kaiser readily agreed, though he wanted to wait until she was older.“They were so adamant,” he said. He recalled the doctors telling him: “‘At this age, they definitely know what their gender is.’” The father asked not to be named out of concern that speaking publicly might jeopardize his employment. Detransition, he said, “wasn’t really discussed as a possibility.”In June 2020, a Kaiser surgeon performed a mastectomy on Cole, according to the letter of intent to sue. That was a month before her 16th birthday. Less than a year later, Cole said, she began to realize she regretted her surgery and medically transitioning in general after a discussion in school about breastfeeding and pregnancy.Cole said that when she discussed her decision to detransition with her gender-care specialist at Kaiser, “I could tell that I made her upset that I was so regretful,” Cole said in an interview. Eventually, the doctor offered to recommend a surgeon for breast reconstruction, Cole said, “but that’s something I’ve decided to not go through with.”Cole has begun speaking out publicly in support of measures to end gender-affirming care for minors, appearing often on conservative media and with politicians who back such bans.In the letter of intent, Cole’s lawyers said Kaiser’s treatment “represents gross negligence and an egregious breach of the standard of care.”Steve Shivinsky, a spokesman for Kaiser Permanente, declined to comment on the care provided to Cole and Robinson or whether they were included in the study, citing patient privacy.In a statement, he said Kaiser’s “clinicians are deeply interested in the outcomes of the care we provide and the individual’s state of health and wellbeing before, during and beyond their gender transition.” For adolescents seeking gender-affirming care, he said, “the decision always rests with the patient and their parents and, in every case, we respect the patients’ and their families’ informed decision to choose one form of care over another.”The Kaiser researchers followed up with patients in their study an average of 2.1 years after surgery. “The time to develop postoperative regret and/or dissatisfaction remains unknown and may be difficult to discern given that regret is quite rare,” the researchers wrote.A CHANGE OF PERSPECTIVEMacKinnon, the assistant professor of social work, grew up as what he calls “a gender-nonconforming tomboy” in a small Nova Scotia town. After getting his degree in social work, he medically transitioned at 24 when he started taking testosterone. “It was a very slow build,” MacKinnon said of his transition. He didn’t identify as transgender as a child.As a young researcher in Toronto, MacKinnon was drawn to work that exposed the barriers transgender people face in getting medical care and navigating daily life, interviewing clinicians and patients about their experiences. More recently, he turned his attention to detransition and regret.In August 2021, MacKinnon published a paper in which he and his co-authors wrote that there was “scant evidence that detransition is a negative phenomenon” for patients that would justify limiting access to gender-affirming treatment. That conclusion angered many of the detransitioners he would later need to win over.Michelle Alleva, a 34-year-old detransitioner in Canada, criticized MacKinnon’s study in a blog post as another effort by gender-care supporters to whitewash the pain of regret and assuage clinicians’ fears of malpractice lawsuits. Another detransitioner complained on Twitter that the word “regret” was put in quotes in the paper, undermining its legitimacy in her opinion.Still skeptical that regret was a significant issue, MacKinnon in the autumn of 2021 embarked on his latest study and began talking to more people about their decisions to detransition. In July, he published a paper based on formal interviews with 28 of the more than 200 detransitioners he and his colleagues have found.A third expressed either strong or partial regret about their transition. Some said their transitions should have proceeded more slowly, with more therapy. Others expressed regret about the lasting impact on their bodies. Some said their mental health needs weren’t adequately addressed before transitioning. “They felt like their consent wasn’t informed because they didn’t initially understand what was going on that might have explained their feelings and suffering,” MacKinnon told Reuters.The patients’ stories brought MacKinnon round to the view that the gender-care community needs to address regret, adjust treatment to reduce its incidence, and provide better support for detransitioners. “Some of what I’ve learned about detransitioners is identifying cracks in the gender-affirming care system, particularly for young people,” he said.In September, MacKinnon presented his findings to a small but attentive crowd at WPATH’s annual conference in Montreal. A few weeks later, he shared his research more widely on Twitter. “We need to listen to and learn from the experiences of detransitioners, not silence them,” he wrote.Some people applauded his work. Others criticized it. Robyn D., who identified as “quietly trans,” replied on Twitter: “Transphobia disguised as academic opinion is the most poisonous of them all.” She didn’t respond to requests for comment from Reuters.At his November symposium, MacKinnon didn’t encounter the blowback from clinicians that he had expected. In fact, he accepted an invitation from one to speak about detransition at her medical practice.Alleva, who had criticized MacKinnon’s earlier study, was also there, one of the scores of detransitioners MacKinnon and his colleagues have talked to. She medically transitioned 12 years ago and then detransitioned in 2020 after a mastectomy, a hysterectomy and years of testosterone. She had refused to participate in his research because she didn’t trust MacKinnon, but over the summer, they began talking.“He reminded me of my old trans friends who I don’t speak with anymore,” Alleva said. “He actually listened to me.”Edited by Michele Gershberg and John Blanton.Our Standards: The Thomson Reuters Trust Principles.\n",
      "http://reuters.com/article/idUSL8N2YT2PB\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSL8N2YT2PB\n",
      "TITLE: Science offers basis for national climate damage claims -study\n",
      "BODY:\n",
      "\n",
      "http://reuters.com/article/idUSKBN2ON108\n",
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSKBN2ON108\n",
      "TITLE: Science offers basis for national climate damage claims -study\n",
      "BODY:\n",
      "\n",
      "http://reuters.com/article/idUSKBN2P20SJ\n",
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSKBN2P20SJ\n",
      "TITLE: Exclusive: Cassava Sciences faces U.S. criminal probe tied to Alzheimer's drug, sources say\n",
      "BODY:\n",
      "\n",
      "http://reuters.com/article/idUSL1N2ZN2GC\n",
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSL1N2ZN2GC\n",
      "TITLE: Long COVID risk up for unvaccinated children; at-home antibody test shows promise\n",
      "BODY:\n",
      "\n",
      "http://reuters.com/article/idUSL1N2Z61E5\n",
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSL1N2Z61E5\n",
      "TITLE: Exclusive: Cassava Sciences faces U.S. criminal probe tied to Alzheimer's drug, sources say\n",
      "BODY:\n",
      "\n",
      "http://reuters.com/article/idUSL1N2ZE21A\n",
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSL1N2ZE21A\n",
      "TITLE: Omicron better at invading young noses than other variants; smell loss may predict memory issues\n",
      "BODY:\n",
      "\n",
      "http://reuters.com/article/idUSL1N2Z71GW\n",
      "New article found for topic: data%20science\n",
      "URL: http://reuters.com/article/idUSL1N2Z71GW\n",
      "TITLE: Experimental chewing gum reduces Omicron in saliva; sexual dysfunction, hair loss among long COVID symptoms\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/search/?s=python\n",
      "https://www.brookings.edu/blog/up-front/2022/03/17/housing-finance-reform-the-path-forward-gets-rolling/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/blog/up-front/2022/03/17/housing-finance-reform-the-path-forward-gets-rolling/\n",
      "TITLE: Housing finance reform: The path forward gets rolling\n",
      "Housing finance reform: The path forward gets rolling\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/blog/techtank/2023/02/07/building-guardrails-for-chatgpt/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/blog/techtank/2023/02/07/building-guardrails-for-chatgpt/\n",
      "TITLE: Building guardrails for ChatGPT\n",
      "Building guardrails for ChatGPT\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/research/how-open-source-software-shapes-ai-policy/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/research/how-open-source-software-shapes-ai-policy/\n",
      "TITLE: How open-source software shapes AI policy\n",
      "How open-source software shapes AI policy\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/techstream/how-generative-ai-impacts-democratic-engagement/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/techstream/how-generative-ai-impacts-democratic-engagement/\n",
      "TITLE: How generative AI impacts democratic engagement\n",
      "BODY:\n",
      "\n",
      "The exterior of the U.S. Capitol is seen at sunset on January 22, 2020. (REUTERS/Sarah Silbiger)\n",
      "\n",
      "\n",
      "Within two months of its launch last fall, the popular chatbot ChatGPT had reached an estimated 100 million monthly users—making it the fastest growing consumer application in history. Now that its parent company OpenAI has released a new version of the chatbot’s underlying language model, anyone with access to the app will soon be able not only to write poetry and practice journalism, but even ace the LSAT and GRE. Like most leading-edge language applications today, ChatGPT relies on a machine learning algorithm known as a transformer to generate probability distributions over words and then recognize, translate, predict, or generate text. The quality of that text is often deeply impressive—so much so that even relatively restrained tech publications have concluded ChatGPT may be “coming for your job.”\n",
      "\n",
      "\n",
      "Yet large language models may disrupt far more than just the economy. They also appear poised to challenge democracy too. At issue is not just the risk of automated misinformation campaigns, but the threat to traditional forms of democratic engagement. Democracy depends in part on how responsive those who govern are to the preferences of the governed: from public comments on proposed agency rules (“submit your comments and let your voice be heard,” exhorts regulations.gov) to legislators’ refashioned electronic mail boxes, citizens have long offered feedback to policymakers via the written word. By making it trivial to produce large quantities of credible text, language models threaten to weaken the signal those words provide. As the New Yorker recently observed, ChatGPT “will strain a political system in peril.”\n",
      "\n",
      "\n",
      "The threat is far from an idle one. In 2017, the Federal Communications Commission (FCC) invited public commentary regarding its proposed changes to net neutrality regulations. An orchestrated astroturf campaign opposed to net neutrality soon flooded the comment line, generating more than 8 million comments, while a single college student in favor of net neutrality used an automated script to generate seven million comments of his own. The government foiled the attempt only because of the inadequacy of the technology: since only 6% of the 21.7 million comments were unique and seven comments alone accounted for 38% of all submissions, the manipulation was easy to detect. By contrast, language models capable of automating unique submissions at scale will not be as easy to uncover.\n",
      "\n",
      "\n",
      "To gauge the extent of the problem, we ran an experiment on the impact of language models on the efficacy of congressional correspondence. More specifically, we examined whether language models could distort the legislative agenda by drafting limitless original emails to members of Congress. Using both human writers and GPT-3, OpenAI’s predecessor to the GPT-4 system it announced last week, we randomly sent email to over 7,000 state legislators around the country and then compared response rates. The difference in response rates, which scholars use as a proxy for government responsiveness to constituent concerns, was just 2% overall, with no differences in response rates for three of the policy issues we analyzed. The findings suggest that legislators are potentially susceptible to the misuse of these technologies.\n",
      "\n",
      "\n",
      "Research design and methodology\n",
      "\n",
      "\n",
      "To investigate the possible misuse of language models like ChatGPT, we conducted a field experiment in 2020 using OpenAI’s then-state-of-the-art language model, GPT-3. We selected six policy issues—gun control, reproductive rights, education, health care, policing, and taxes—and asked a number of undergraduate research assistants to write right and left-leaning advocacy letters. We then trained GPT-3 on the student letters and asked GPT-3 to write new letters advancing a right-wing and left-wing position on each issue. For example, here’s the first paragraph of a right-wing gun control letter that one of our research assistants wrote:\n",
      "\n",
      "\n",
      "\n",
      "I was recently appalled by an interview on the News with the co-founder of Students for Safer Schools. He insisted that it was a natural consequence of our current gun control laws to allow active shooters to commit mass murder in schools. I am writing this letter in order to implore you to oppose any bill to ban or curtail the rights of gun owners in America.\n",
      "\n",
      "\n",
      "\n",
      "I was recently appalled by an interview on the News with the co-founder of Students for Safer Schools. He insisted that it was a natural consequence of our current gun control laws to allow active shooters to commit mass murder in schools. I am writing this letter in order to implore you to oppose any bill to ban or curtail the rights of gun owners in America.\n",
      "\n",
      "\n",
      "Based on this student sample, we asked GPT-3 to “write a right-wing letter on gun control.”  Here’s the first paragraph of the AI-written letter:\n",
      "\n",
      "\n",
      "\n",
      "I hope this letter finds you well. I would first like to thank you for the job you’ve done in representing us in this time of national turmoil. The pandemic and social unrest sweeping the nation right now have made life difficult for all, and many citizens, myself included, fear that our Second Amendment right to firearm ownership may become more and more of a necessity in daily life.\n",
      "\n",
      "\n",
      "\n",
      "I hope this letter finds you well. I would first like to thank you for the job you’ve done in representing us in this time of national turmoil. The pandemic and social unrest sweeping the nation right now have made life difficult for all, and many citizens, myself included, fear that our Second Amendment right to firearm ownership may become more and more of a necessity in daily life.\n",
      "\n",
      "\n",
      "Next, we created a program on Python to alternately send either human or AI-written emails to roughly 7,200 state legislators. All told, we sent a total of 32,398 emails or approximately 5 per legislator.\n",
      "\n",
      "\n",
      "We then compared response rates to human and AI-generated emails as a measure of how credible each type of email was. Given the competing demands on a legislator’s time, the decision to answer a constituent letter implies a calculus about the importance of responsiveness to that individual or issue. Previous studies seeking to understand whether legislators discriminate on the basis of constituent race or ethnicity have sent emails from Hispanic, Black, or Muslim aliases and looked for differential response rates as evidence of discrimination. In our case, we judged that a busy legislator would not respond to a message that was clearly robotic or nonsensical, but rather would reply preferentially to real constituents.\n",
      "\n",
      "\n",
      "Our analysis suggested that legislators found the AI-generated text almost as credible as the human-written messages. On two issues – gun control and health policy – we found virtually identical response rates for human versus machine emails (See Figure 1). And on a third, education policy, we observed a higher response rate for the machine-generated emails, though the difference is not statistically significant. This suggests that on these issues GPT-3 succeeded in producing content that was almost indistinguishable in the eyes of state legislative offices from human content. By contrast, legislators were less responsive to machine-generated communications on three issues: policing; reproductive rights; and taxes. However, even on these issues legislators responded to GPT-3-generated emails at rates only 4% lower than to hand-written emails.\n",
      "\n",
      "\n",
      "Figure 1: Differential Response Rates (GPT-3 – Human Emails) by Policy Area\n",
      "\n",
      "\n",
      "Note: I-bars present 95% confidence intervals around each difference in means.\n",
      "\n",
      "\n",
      "While our findings suggest legislators routinely failed to distinguish real from AI-manufactured communications, follow-up exchanges with legislators surfaced some helpful tips for guarding against manipulation. Even the best language models can produce bland, generic, or sometimes factually inaccurate content, or can “hallucinate” and become repetitive. For example, one email started, “My name is Rebecca Johnson, and I am a single father raising a daughter.” This raised a red flag for at least one legislator in our sample. Further, the machine sometimes struggled to write ideologically consistent letters, particularly from a right-wing perspective on the issue of gun control. For instance, one AI-generated letter began, “I am a rifle owner and hunter”—but then suggested that the author took a turn and argued that owning a firearm was a constitutional right.\n",
      "\n",
      "\n",
      "Even if these inconsistencies occur in a small fraction of the outputs, avoiding them requires a human-in-the-loop that is at odds with a nefarious actor’s goal of automating malice. The same antidote to these low-quality outputs also undermines the potential misuse of these tools. Of course, as the technology improves—such as through reinforcement learning from human feedback, or RLHF—those types of outputs will become less prevalent. Nevertheless, for now legislators or staffers should look for these hallmarks—or even better, the absence of telltale residency markers like specifics about the district or state.\n",
      "\n",
      "\n",
      "Policy implications of generative AI\n",
      "\n",
      "\n",
      "What then is a legislator to do?  Mail has always been an important way that politicians from local representatives to presidents have kept tabs on public opinion. Even in an era of ubiquitous polling, many have viewed mail as informative of the concerned public on a given issue—the effort it takes to compose a letter or note has historically served as a strong signal of the salience of a given issue or policy proposal, particularly when multiple constituents touch on the same points. However, language models threaten to weaken that signal. As malicious actors leverage them in a bid to influence the behavior of policymakers and regulators, elected and non-elected officials alike will have strong incentives to be skeptical of the information they can glean about public preferences from both constituent feedback and correspondence.\n",
      "\n",
      "\n",
      "Fortunately, legislators have other means of keeping their finger on the pulse of public opinion. With enough resources, they can obtain district polling data on key issues; alternately, they can hold more frequent and more targeted town meetings to get a direct earful from those most animated about an issue. As the capacity for astroturfing increases, legislators may have to rely more heavily on these other sources of information about constituency preferences—and discount the sheer volume of text that they previously took as a signal of public attitudes.\n",
      "\n",
      "\n",
      "In addition, regulators and legislators may also soon be able to more readily detect auto-generated text. For one, the same algorithms underlying models like GPT-3 can also be adapted to identify machine-generated text; researchers have shown that the accuracy of machine-based detection of generated text can reach accuracy rates of close to 90%. For another, OpenAI, Anthropic, and other labs responsible for leading-edge language models are all now experimenting with statistically watermarking the text they produce. As consensus standards and tools around watermarking emerge, legislators and regulators may be able to filter their correspondence through software that identifies auto-generated content. Finally, as language models improve, agency and congressional staff will need to rely on greater digital literacy to understand the risks they pose.\n",
      "\n",
      "\n",
      "Recent election cycles have made it clear that malicious actors are both willing and able to leverage digital applications to subvert American democracy. With the advent of powerful new language models, those actors now have a potent new weapon in their arsenal. Lawmakers and regulators will need to work proactively to guard against the risks they introduce.\n",
      "\n",
      "\n",
      "Sarah Kreps is a Non-Resident Senior Fellow in the Brookings Foreign Policy Program and the Brookings Artificial Intelligence and Emerging Technology Initiative, as well as the Director of the Tech Policy Institute at Cornell University.\n",
      "\n",
      "\n",
      "Douglas Kriner is the Clinton Rossiter Professor in American Institutions and faculty director of the Institute of Politics and Global Affairs at Cornell University.\n",
      "\n",
      "\n",
      "Microsoft, a major partner and investor in OpenAI, provides financial support to the Brookings Institution. The findings, interpretations, and conclusions posted in this piece are solely those of the authors and are not influenced by any donation.\n",
      "\n",
      "https://www.brookings.edu/techstream/lessons-from-the-meme-war-in-ukraine/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/techstream/lessons-from-the-meme-war-in-ukraine/\n",
      "TITLE: Lessons from the meme war in Ukraine\n",
      "BODY:\n",
      "\n",
      "Illustration of a mini replica of tanks and soldiers, seen in front of the Ukrainian flag in Krakow, Poland. (Photo by Artur Widak/NurPhoto)\n",
      "\n",
      "\n",
      "In 1927, the political scientist Harold Lasswell wrote about political propaganda as “the management of collective attitudes by the manipulation of significant symbols.” Underlying Lasswell’s work were two sets of insights. One is that the mass public played a key role in political outcomes, such as success and failure in war. Second, that those public attitudes could also be manipulated. Scaling to the mass-level, however, required simplicity. This included the use of symbols and slogans that were memorable, such that they could frame “pictures”—or, cognitive shortcuts—that the public recalled when engaging elected officials to shape certain policies.\n",
      "\n",
      "\n",
      "Nowhere has the use of propaganda been more ubiquitous than in war, especially because acquiescence or resistance is based on public sentiment and behavior. In World War II, Hollywood produced films that “created a communal viewing experience unlike any during World War I” intended to maintain resolve for the war. These films capitalized on the public’s predisposition to understand social life in terms of in- and out-groups, which shapes how people often interpret foreign policies, including the use of force.\n",
      "\n",
      "\n",
      "In contemporary conflict, those symbols have increasingly taken the form of memes, defined as a “piece of media that is repurposed to deliver a cultural, social, or political expression, mainly through humor.” Online users have attempted to counter the Islamic State by creating memes satirizing the group’s barbarism, especially on specific “Troll ISIS Days.” Lebanese Internet-users have ruthlessly mocked Hezbollah’s leader, Hassan Nasrallah, with memes.\n",
      "\n",
      "\n",
      "Memes have also been a staple of the war in Ukraine, providing a valuable window into key questions about how actors use memes for political purposes in war. Who is the audience, what is the message, and what events drive the production of these memes?\n",
      "\n",
      "\n",
      "To explore these and related questions, we compiled an original dataset of memes posted by Ukrainians throughout the war. The memes were all taken from Reddit, a popular social media website that allows users to comment in discussion forums based on shared interests. Overall, our analysis of Ukrainians’ use of memes points to several findings that shed new light on how other countries may use memes during conflict. First, memes are not used in isolation from a particular military operation on the battlefield, such as an offensive or counter-offensive. Rather, they are concurrent and complementary to these military efforts, suggesting that they are meant to play a supporting role. Second, memes do not seem intended to directly influence diplomacy, but may further diplomatic efforts indirectly by bolstering popular support for the war. Third, memes target a diverse array of audiences, including Ukrainian citizens, expatriate audiences abroad, and Russians, especially soldiers’ families. This suggests that those creating and posting memes assume that success is a function of both domestic resolve as well as foreign material support.\n",
      "\n",
      "\n",
      "Our empirical approach: Data, method, and descriptive trends\n",
      "\n",
      "\n",
      "We analyzed Ukraine-related memes generated between the beginning of the war on February 24, 2022, and January 14, 2023. Although the Internet is rife with memes, we focused on Reddit, using the PRAW library in Python, which uses Reddit’s Application Programming Interface to request data from the website. To optimize the library, we used it against a handful of subreddits, which limited our scraping to three subreddits used almost exclusively by Ukrainians: (1) /r/ukrmemes, (2) /r/UkraineMem, and (3) /r/ukraine22memes. This approach yielded 1,365 visual memes, along with metadata for each meme, including its title, author’s screen name, date and time of dissemination, and popularity. The latter is a score that Reddit publishes with every post to help users gauge how much attention it is getting, and is the difference between the number of “upvotes” and “downvotes” a post receives. A greater number of upvotes, therefore, indicates the post is more popular with Reddit users. Finally, we also randomly sampled from our data to confirm that our machine-coding process was reliable.1 \n",
      "\n",
      "\n",
      "The graph below illustrates how the Ukrainian meme war has evolved over time. By plotting meme frequency and popularity against The New York Times’ key monthly developments in the war, we are able to show how the intensity and public uptake of memes correlates with political and military milestones throughout the war.\n",
      "\n",
      "\n",
      "This figure, made from data provided by Reddit, reflects the meme score (left y-axis) and frequency (right y-axis) from February 24, 2022, to January 14, 2023. For ease of interpretation, we put dates in a day/month/year format (click to expand).\n",
      "\n",
      "\n",
      "The graph above reveals several patterns. First, the use of memes surged during Russia’s invasion of Ukraine and has steadily declined throughout the war. This trend suggests that the meme campaign is likely more informal or crowd-sourced than centralized among Ukrainian officials and leaders. If this were a coordinated, top-down effort, we would expect to see more continuity over time, especially in terms of frequency. As a bottom-up effort, the declines point to the possibility of war fatigue among the population.\n",
      "\n",
      "\n",
      "Second, the frequency and popularity of memes correspond to major ebbs and flows in the Ukraine war, with spikes that clearly relate to major offensives and battles. More specifically, memes seem to play a supporting role designed to amplify tactical effects on the battlefield rather than precede them as a separate form of psychological operations. This finding is in line with what U.S. Army General Christopher Cavoli, the Supreme Allied Commander-Europe, observed at a recent security forum in Sweden: “hard power is a reality.” In short, memes are meant not to replace hard power on the battlefield but instead to provide a psychological dividend to the successful application of hard power.\n",
      "\n",
      "\n",
      "Third, even as memes mirror major milestones in the war, they do not seem to correspond closely with important diplomatic and political events. Among these include the announcement on May 18, 2022, that Sweden and Finland submitted accession documents to join the North Atlantic Treaty Organization, as well as Ukrainian President Volodymyr Zelenskyy’s visit to the White House in late December 2022. This may suggest that memes are hitched to some immediately observable effects, which are typically more pronounced on the battlefield rather than diplomacy, considering negotiations often occur over a longer time horizon.\n",
      "\n",
      "\n",
      "A typology of memes\n",
      "\n",
      "\n",
      "Given the visually evocative nature of an image, fully understanding the role played by memes in the Ukrainian conflict requires more than just empirical analysis. Examining a subset of specific memes tells a more complete story of what some analysts have referred to as “memetic warfare,” or the use of memes to control the narrative. Accordingly, we randomly selected a handful of memes that piece together a story about messages and audiences. \n",
      "\n",
      "\n",
      "Our analysis suggests that a rich typology of memes has emerged during the war. For instance, expatriates in places like the United States often generate memes to establish a sense of solidarity and support abroad. The meme below illustrates this purpose while capitalizing on a clear reference to the popular U.S. television show, “The Office.”  Published on December 14, 2022, the Ukrainian is translated as “Glory to Ukraine!” suggesting a pro-Ukrainian sentiment. Such sentiment may be important to sustain public support for the war, especially in the United States where Congress passed the “Ukraine Democracy Defense Lend-Lease Act of 2022” on May 9. Following the precedent of World War II, this act facilitates the transfer of arms and equipment to the Ukrainian government to enable its military operations against Russia.\n",
      "\n",
      "\n",
      "\n",
      " I live in DC, so this was from a while ago when I went for a walk around the national mall. The couple were going to a small solidarity demonstration outside the White House from      ukrainememes  \n",
      "\n",
      "\n",
      "\n",
      "Memes can also serve as a commentary on a military operation. The meme below shows a Russian billboard that reads “Kherson—Forever Russia.” The bottom panel then mocks this claim with a SpongeBob SquarePants cartoon, suggesting that Ukrainians believe Russia’s attempt to occupy Kherson is as ridiculous as a childrens’ cartoon. This meme was posted on November 9, 2022, just as the Ukrainians were in the process of recapturing Kherson, which culminated several days later.\n",
      "\n",
      "\n",
      "\n",
      " „Kherson, forever Russia!“ from      ukrainememes  \n",
      "\n",
      "\n",
      "\n",
      "Similarly, the meme below is from April 20, 2022, days after the Ukrainians had used a drone to sink the Russian warship Moskva, the flagship ship of the Russian Navy’s Black Sea Fleet. It portrays a tough-looking Putin on a ship at the bottom of the sea. The artist appears to be Hajo de Reijger, a freelance illustrator who writes for Dutch newspapers and is based in Amsterdam, but the meme clearly has universal accessibility in highlighting how damaging the sinking of the Moskva was to Russian President Vladimir Putin’s image as a competent military leader.\n",
      "\n",
      "\n",
      "\n",
      " Battleship Moskva – Amsterdam (Netherlands), Hajo de Reijger (20/4/2022) from      UkrainianMemes  \n",
      "\n",
      "\n",
      "\n",
      "Lastly, some memes are in both Ukrainian and English, suggesting they are designed to shape the attitudes of domestic and international audiences. The following meme, also posted on June 11, 2022, carries the following title: “Putin lacks imagination—У Путина нет воображения.” This statement highlights Putin’s targeting of civilians during the war, contrasting this immoral and illegal use of force with other countries’ investment in children, a message that could be intended for an audience in Russia, for example soldiers’ parents.\n",
      "\n",
      "\n",
      "\n",
      " Putin lacks imagination -У Путина нет воображения from      memes  \n",
      "\n",
      "\n",
      "\n",
      "The implications\n",
      "\n",
      "\n",
      "What insights can we draw from this early analysis of memes in Ukraine? In a sense, the most powerful argument that can be made in defense of memes is theoretical. Countries have always relied on psychological operations conducted through visual expression. They do so because they think that the visual medium is effective for persuasion. In a domestic context, the supporters of political candidates place yard signs around communities because they think people can be persuaded by these visual insinuations, and research suggests that those resources are not wasted. These signs influence vote shares.\n",
      "\n",
      "\n",
      "Our initial investigation suggests that memes can play a similar role. They offer a simple and digestible way to communicate with supporters, both at home and abroad. Propping up support among citizens is important, if not integral, to the war effort in Ukraine. Ukrainians are at the literal tip of the spear; if they do not resist, the country will fail. The psychological persuasion and support of expatriates is also crucial because they live in countries that materially support the Ukrainian military.\n",
      "\n",
      "\n",
      "Though our research reflects a mix of memes intended to connect with these audiences, we also find that memes, at least in Ukraine, also play a supporting role to combat operations. As we show, the frequency and popularity of memes track closely with key military milestones in the war and mostly function informally as a way to highlight battlefield successes. To best capitalize on the psychological dividends that memes promise, however, officials should better integrate them into their overall wartime strategies, perhaps through a “Meme Warfare Center,” as recommended by other analysts. The relative decline of memes throughout the Ukrainian war raises another set of questions. Why, if their messages have such powerful potential, have memes declined over time? Do memes operate in a similar fashion as the news cycle where internet-users simply move on to the next viral social issue after a matter of time? Have meme creators concluded that the images are not effective? Has another form of online persuasion, if any, taken their place? Does the use of different social media platforms, including Facebook, LinkedIn, Telegram, and Twitter, shape the frequency and popularity of memes, as well as the anticipated effects? Scholars should study these and related questions to better understand if—and how—memes can play a role in the psychology, and perhaps even outcomes, of war.\n",
      "\n",
      "\n",
      "Sarah Kreps is the John L. Wetherill Professor of Government at Cornell University and the director of the Cornell Brooks School Tech Policy Institute.\n",
      "\n",
      "\n",
      "Paul Lushenko is a lieutenant colonel in the U.S. Army and deputy director of the Cornell Brooks School Tech Policy Institute.\n",
      "\n",
      "\n",
      "Keith Carter is a lieutenant colonel in the U.S. Army and director of the defense and strategic studies program at the United States Military Academy.\n",
      "\n",
      "\n",
      "Acknowledgments: The authors would like to thank Alexey Elkin for excellent research assistance, as well as Matthew Evangelista, Bryn Rosenfeld, Aleksandar Vladicic, and an anonymous reviewer for feedback on this research.\n",
      "\n",
      "\n",
      "The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the United States Department of the Army, Department of Defense, or Government.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Note 1. We manually reviewed a cross-section of the memes to check the accuracy of our scraping technique. Specifically, we built an algorithm to randomly select a percentage of memes from our database. This process resulted in a subset of 34 memes—approximately 3% of our database—from which we ensured they were, in fact, memes. Our hand-coding recorded an 85% fidelity rate on the machine-coding, suggesting that the volume of memes may be somewhat lower than the overall number of hits we gathered but we have no reason to think that this affects the broader patterns we observe.\n",
      "\n",
      "https://www.brookings.edu/research/preventing-pandemics-through-biodiversity-conservation-and-smart-wildlife-trade-regulation/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/research/preventing-pandemics-through-biodiversity-conservation-and-smart-wildlife-trade-regulation/\n",
      "TITLE: Preventing pandemics through biodiversity conservation and smart wildlife trade regulation\n",
      "Preventing pandemics through biodiversity conservation and smart wildlife trade regulation\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/blog/techtank/2017/11/16/leveraging-the-disruptive-power-of-artificial-intelligence-for-fairer-opportunities/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/blog/techtank/2017/11/16/leveraging-the-disruptive-power-of-artificial-intelligence-for-fairer-opportunities/\n",
      "TITLE: Leveraging the disruptive power of artificial intelligence for fairer opportunities\n",
      "Leveraging the disruptive power of artificial intelligence for fairer opportunities\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/blog/up-front/2015/12/21/the-hutchins-center-explains-budgeting-for-aging-america/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/blog/up-front/2015/12/21/the-hutchins-center-explains-budgeting-for-aging-america/\n",
      "TITLE: The Hutchins Center Explains: Budgeting for aging America\n",
      "The Hutchins Center Explains: Budgeting for aging America\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/research/an-atlanta-organizations-mission-to-bring-racial-equity-to-the-tech-ecosystem/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/research/an-atlanta-organizations-mission-to-bring-racial-equity-to-the-tech-ecosystem/\n",
      "TITLE: An Atlanta organization’s mission to bring racial equity to the tech ecosystem\n",
      "An Atlanta organization’s mission to bring racial equity to the tech ecosystem\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/blog/the-avenue/2014/08/06/the-silicon-valley-wage-premium/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/blog/the-avenue/2014/08/06/the-silicon-valley-wage-premium/\n",
      "TITLE: The Silicon Valley Wage Premium\n",
      "The Silicon Valley Wage Premium\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/blog/techtank/2016/03/01/idea-to-retire-old-methods-of-policy-education/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/blog/techtank/2016/03/01/idea-to-retire-old-methods-of-policy-education/\n",
      "TITLE: Idea to Retire: Old methods of policy education\n",
      "Idea to Retire: Old methods of policy education\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/research/institutionalizing-data-analysis-in-german-federal-governance/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/research/institutionalizing-data-analysis-in-german-federal-governance/\n",
      "TITLE: Institutionalizing Data Analysis in German Federal Governance\n",
      "Institutionalizing Data Analysis in German Federal Governance\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/opinions/skills-success-and-why-your-choice-of-college-matters/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/opinions/skills-success-and-why-your-choice-of-college-matters/\n",
      "TITLE: Skills, success, and why your choice of college matters\n",
      "Skills, success, and why your choice of college matters\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/research/making-waves-in-india-media-and-the-covid-19-pandemic/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/research/making-waves-in-india-media-and-the-covid-19-pandemic/\n",
      "TITLE: Making waves in India: Media and the COVID-19 pandemic\n",
      "Making waves in India: Media and the COVID-19 pandemic\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/essay/building-skills-for-life-how-to-expand-and-improve-computer-science-education-around-the-world/\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/essay/building-skills-for-life-how-to-expand-and-improve-computer-science-education-around-the-world/\n",
      "TITLE: \n",
      "\t\t\t\tBUILDING SKILLS FOR LIFE\t\t\t\n",
      "BODY:\n",
      "\n",
      "This report makes the case for expanding computer science education in primary and secondary schools around the world, and outlines the key challenges standing in the way. Through analysis of regional and national education systems at various stages of progress in implementing computer science education programs, the report offers transferable lessons learned across a wide range of settings with the aim that all students—regardless of income level, race, or sex—can one day build foundational skills necessary for thriving in the 21st century.\n",
      "\n",
      "\n",
      "Download the full report\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Introduction\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Access to education has expanded around the world since the late 1990s through the combined efforts of governments, bilateral and multilateral agencies, donors, civil society, and the private sector, yet education quality has not kept pace. Even before the COVID-19 pandemic led to school closures around the world, all young people were not developing the broad suite of skills they need to thrive in work, life, and citizenship (Filmer, Langthaler, Stehrer, & Vogel, 2018).\n",
      "\n",
      "\n",
      "The impact of the pandemic on education investment, student learning, and longer-term economic outcomes threatens not only to dial back progress to date in addressing this learning crisis in skills development but also to further widen learning gaps within and between countries. Beyond the immediate and disparate impacts of COVID-19 on students’ access to quality learning, the global economic crisis it has precipitated will shrink government budgets, potentially resulting in lower education investment and impacting the ability to provide quality education (Vegas, 2020). There is also a concern that as governments struggle to reopen schools and/or provide sufficient distance-learning opportunities, many education systems will focus on foundational skills, such as literacy and numeracy, neglecting a broader set of skills needed to thrive in a rapidly changing, technologically-advanced world.\n",
      "\n",
      "\n",
      "Among these broader skills, knowledge of computer science (CS) is increasingly relevant. CS is defined as “the study of computers and algorithmic processes, including their principles, their hardware and software designs, their [implementation], and their impact on society” (Tucker, 2003).1 CS skills enable individuals to understand how technology works, and how best to harness its potential to improve lives. The goal of CS education is to develop computational thinking skills, which refer to the “thought processes involved in expressing solutions as computational steps or algorithms that can be carried out by a computer” (K-12 Computer Science Framework Steering Committee, 2016). CS education is also distinct from computer or digital literacy, in that it is more concerned with computer design than with computer use. For example, coding is a skill one would learn in a CS course, while creating a document or slideshow presentation using an existing program is a skill one would learn in a computer or digital literacy course.\n",
      "\n",
      "\n",
      "Research has shown that students benefit from CS education by increasing college enrollment rates and developing problem-solving abilities (Brown & Brown, 2020; Salehi et al., 2020). Research has also shown that lessons in computational thinking improve student response inhibition, planning, and coding skills (Arfé et al., 2020). Importantly, CS skills pay off in the labor market through higher likelihood of employment and better wages (Hanson & Slaughter, 2016; Nager & Atkinson, 2016). As these skills take preeminence in the rapidly changing 21st century, CS education promises to significantly enhance student preparedness for the future of work and active citizenship.\n",
      "\n",
      "\n",
      "The benefits of CS education extend beyond economic motivations. Given the increasing integration of technology into many aspects of daily life in the 21st century, a functional knowledge of how computers work—beyond the simple use of applications—will help all students.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Why expand CS education?\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "By this point, many countries have begun making progress toward offering CS education more universally for their students. The specific reasons for offering it will be as varied as the countries themselves, though economic arguments often top the list of motivations. Other considerations beyond economics, however, are also relevant, and we account for the most common of these varied motives here.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The economic argument\n",
      "\n",
      "\n",
      "At the macroeconomic level, previous research has suggested that countries with more workers with ICT (information, communications, and technology) skills will have higher economic growth through increases in productivity (Maryska, Doucek, & Kunstova, 2012; Jorgenson & Vu, 2016). Recent global data indicate that there is a positive relationship between the share of a country’s workforce with ICT skills and its economic growth. For example, using data from the Organisation for Economic Cooperation and Development (OECD), we find that countries with a higher share of graduates from an ICT field tend to have higher rates of per capita GDP (Figure 1). The strength of the estimated relationship here is noteworthy: A one percentage point increase in the share of ICT graduates correlates with nearly a quarter percentage point increase in recent economic growth, though we cannot determine the causal nature of this relationship (if any). Nonetheless, this figure supports the common view that economic growth follows from greater levels of investment in technological education.\n",
      "\n",
      "\n",
      "FIGURE 1\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "At the microeconomic level, CS skills pay off for individuals—both for those who later choose to specialize in CS and those who do not. Focusing first on the majority of students who pursue careers outside of CS, foundational training in CS is still beneficial. Technology is becoming more heavily integrated across many industrial endeavors and academic disciplines—not just those typically included under the umbrella of science, technology, engineering, and mathematics (STEM) occupations. Careers from law to manufacturing to retail to health sciences all use computing and data more intensively now than in decades past (Lemieux, 2014). For example, using data from Germany, researchers showed that higher education programs in CS compared favorably against many other fields of study, producing a relatively high return on investment for lower risk (Glocker and Storck, 2014). Notably, completing advanced training in CS is not necessary to attain these benefits; rather, even short introductions to foundational skills in CS can increase young students’ executive functions (Arfe et al., 2020). Further, those with CS training develop better problem-solving abilities compared to those with more general training in math and sciences, suggesting that CS education offers unique skills not readily developed in other more common subjects (Salehi et al., 2020).\n",
      "\n",
      "\n",
      "For those who choose to pursue advanced CS studies, specializing in CS pays off both in employment opportunities and earnings. For example, data from the U.S. show workers with CS skills are less likely to be unemployed than workers in other occupations (Figure 2). Moreover, the average earnings for workers with CS skills are higher than for workers in other occupations (Figure 3). These results are consistent across multiple studies using U.S. data (Carnevale et al., 2013; Altonji et al., 2012) and international data (Belfield et al., 2019; Hastings et al., 2013; Kirkeboen et al., 2016). Further, the U.S. Bureau of Labor Statistics has projected that the market for CS professionals will continue to grow at twice the speed of the rest of the labor market between 2014 and 2024 (National Academies of Sciences, 2018).\n",
      "\n",
      "\n",
      "FIGURE 2\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "FIGURE 3\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "A common, though inaccurate, perception about the CS field is that anybody with a passion for technology can succeed without formal training. There is a nugget of truth in this view, as many leaders of major technology companies including Bill Gates, Elon Musk, Mark Zuckerberg, and many others have famously risen to the top of the field despite not having bachelor’s degrees in CS. Yet, it is a fallacy to assume that these outliers are representative of most who are successful in the field. This misconception could lead observers to conclude that investments in universal CS education are, at best, ineffective: providing skills to people who would learn them on their own regardless, and spending resources on developing skills in people who will not use them. However, such conclusions are not supported by empirical evidence. Rather, across STEM disciplines, including CS, higher levels of training and educational attainment lead to stronger employment outcomes, on average, than those with lesser levels of training in the same fields (Altonji et al., 2016; Altonji and Zhong, 2021).\n",
      "\n",
      "\n",
      "The inequality argument\n",
      "\n",
      "\n",
      "Technology—and particularly unequal access to its benefits—has been a key driver of social and economic inequality within countries. That is, those with elite social status or higher wealth have historically gotten access to technology first for their private advantages, which tends to reinforce preexisting social and economic inequalities. Conversely, providing universal access to CS education and computing technologies can enable those with lower access to technological resources the opportunity to catch up and, consequently, mitigate these inequalities. Empirical studies have shown how technological skills or occupations, in particular, have reduced inequalities between groups or accelerated the assimilation of immigrants (Hanson and Slaughter, 2017; DeVol, 2016).\n",
      "\n",
      "\n",
      "Technology and CS education are likewise frequently considered critical in narrowing income gaps between developed and developing countries. This argument can be particularly compelling for low-income countries, as global development gaps will only be expected to widen if low-income countries’ investments in these domains falter while high-income countries continue to move ahead. Rather, strategic and intensive technological investment is frequently seen as a key strategy for less-developed countries to leapfrog stages of economic development to quickly catch up to more advanced countries (Fong, 2009; Lee, 2019).\n",
      "\n",
      "\n",
      "CS skills enable adaptation in a quickly changing world, and adaptability is critical to progress in society and the economy. Perhaps there is no better illustration of the ability to thrive and adapt than from the COVID-19 pandemic. The pandemic has forced closures of many public spaces across the globe, though those closures’ impacts have been disproportionately felt across workers and sectors. Workers with the skills and abilities to move their job functions online have generally endured the pandemic more comfortably than those without those skills. And even more importantly, the organizations and private companies that had the human capacity to identify how technology could be utilized and applied to their operations could adapt in the face of the pandemic, while those without the resources to pivot their operations have frequently been forced to close in the wake of pandemic-induced restrictions. Thus, the pandemic bestowed comparative benefits on those with access to technology, the skills to use it, and the vision to recognize and implement novel applications quickly, while often punishing those with the least access and resources (OECD, 2021).\n",
      "\n",
      "\n",
      "Failing to invest in technology and CS education may result in constrained global competitiveness, leaving governments less able to support its citizens. We recognize that efforts to expand CS education will demand time and money of public officials and school leaders, often in the face of other worthy competing demands. Though the contemporary costs may even seem prohibitive in some cases, the costs of inaction (while less immediately visible) are also real and meaningful in most contexts.\n",
      "\n",
      "\n",
      "Beyond economics\n",
      "\n",
      "\n",
      "We expect the benefits of CS education to extend beyond economic motivations, as well. Many household activities that were previously performed in real life are now often performed digitally, ranging from banking, shopping, travel planning, and socializing. A functional knowledge of how computers work—beyond the simple use of applications—should benefit all students as they mature into adults given the increasing integration of technology into many aspects of daily life in the 21st century. For example, whether a person wants to find a job or a romantic partner, these activities frequently occur through the use of technology, and understanding how matching algorithms work make for more sophisticated technology users in the future. Familiarity with CS basic principles can provide users more flexibility in the face of constant innovation and make them less vulnerable to digital security threats or predators (Livingstone et al., 2011). Many school systems now provide lessons in online safety for children, and those lessons will presumably be more effective if children have a foundational understanding of how the internet works.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Global advances in expanding CS education\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "To better understand what is needed to expand CS education, we first took stock of the extent to which countries around the world have integrated CS education into primary and secondary schools, and how this varied by region and income level. We also reviewed the existing literature on integrating CS into K-12 education to gain a deeper understanding of the key barriers and challenges to expanding CS education globally. Then, we selected jurisdictions at various stages of progress in implementing CS education programs in from multiple regions of the world and income levels, and drafted in-depth case studies on the origins, key milestones, barriers, and challenges of CS expansion.\n",
      "\n",
      "\n",
      "Progress in expanding CS education across the globe\n",
      "\n",
      "\n",
      "As shown in Figure 4, the extent to which CS education is offered in primary and secondary schools varies across the globe. Countries with mandatory CS education are geographically clustered in Eastern Europe and East Asia. Most states and provinces in the U.S. and Canada offer CS on a school-to-school basis or as an elective course. Multiple countries in Western Europe offer CS education as a cross-curricular topic integrated into other subjects. Latin America and Central and Southeast Asia have the most countries that have announced CS education programs or pilot projects. Countries in Africa and the Middle East have integrated the least amount of CS education into school curricula. Nevertheless, the number of countries piloting programs or adopting CS curricula indicate a global trend of more education systems integrating the subject into their curriculum.\n",
      "\n",
      "\n",
      "FIGURE 4\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "As expected, students living in higher-income countries generally have better access to CS education. As Figure 5 shows, 43 percent of high-income countries require students to learn CS education in primary and/or secondary schools. Additionally, high-income countries also offer CS as an elective course to the largest share of the population. A further 35 percent of high-income countries offer CS on a school-to-school basis while not making it mandatory for all schools. Interestingly, upper-middle income countries host the largest share of students (62 percent) who are required to learn CS at any point in primary or secondary schools. Presumably, many upper-middle income countries likely have national economic development strategies focused on expanding tech-related jobs, and thus see the need to expand the labor force with CS skills. By contrast, only 5 percent of lower-middle income countries require CS during primary or secondary school, while 58 percent may offer CS education on a school-to-school basis.\n",
      "\n",
      "\n",
      "FIGURE 5\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Key barriers and challenges to expand CS education globally\n",
      "\n",
      "\n",
      "To expand quality CS education, education systems must overcome enormous challenges. Many countries do not have enough teachers who are qualified to teach CS, and even though there is growing interest among students to pursue CS, relatively few students pursue more advanced training like CS testing certifications (Department for Education, 2019) or CS undergraduate majors compared to other STEM fields like engineering or biology (Hendrickson, 2019). This is especially true for girls and underrepresented minorities, who generally have fewer opportunities to develop an interest in CS and STEM more broadly (Code.org & CSTA, 2018). Our review of the literature identified four key challenges to expanding CS education.:\n",
      "\n",
      "\n",
      "1. Providing access to ICT infrastructure to students and educators\n",
      "\n",
      "\n",
      "Student access to ICT infrastructure, including both personal access to computing devices and an internet connection, is critical to a robust CS education. Without this infrastructure, students cannot easily integrate CS skills into their daily lives, and they will have few opportunities to experiment with new approaches on their own.\n",
      "\n",
      "\n",
      "However, some initiatives have succeeded by introducing elements of CS education in settings without adequate ICT infrastructure. For example, many educators use alternative learning strategies like CS Unplugged to teach CS and computational thinking when computers are unavailable (Bell & Vahrenhold, 2018). One study shows that analog lessons can help primary school students develop computational thinking skills (Harris, 2018). Even without laptops or desktop computers, it is still possible for teachers to use digital tools for computational thinking. In South Africa, Professor Jean Greyling of Nelson Mandela University Computing Sciences co-created Tanks, a game that uses puzzle pieces and a mobile application to teach coding to children (Ellis, 2021). This is an especially useful concept, as many households and schools in South Africa and other developing countries have smartphones and access to analog materials but do not have access to personal computers or broadband connectivity (McCrocklin, 2021).\n",
      "\n",
      "\n",
      "Taking a full CS curriculum to scale, however, requires investing in adequate access to ICT infrastructure for educators and students (Lockwood & Cornell, 2013). Indeed, as discussed in Section 3, our analysis of numerous case studies indicates that ICT infrastructure in schools provides a critical foundation to expand CS education.\n",
      "\n",
      "\n",
      "2. Ensuring qualified teachers through teacher preparation and professional development\n",
      "\n",
      "\n",
      "Many education systems encounter shortages of qualified CS teachers, contributing to a major bottleneck in CS expansion. A well-prepared and knowledgeable teacher is the most important component for instruction in commonly taught subjects (Chetty et al. 2014 a,b; Rivkin et al., 2005). We suspect this is no different for CS, though major deficiencies in the necessary CS skills among the teacher workforce are evident. For example, in a survey of preservice elementary school teachers in the United States, only 10 percent responded that they understood the concept of computational thinking (Campbell & Heller, 2019). Until six years ago, 75 percent of teachers in the U.S. incorrectly considered “creating documents or presentations on the computer” as a topic one would learn in a CS course (Google & Gallup, 2015), demonstrating a poor understanding of the distinction between CS and computer literacy. Other case studies, surveys, and interviews have found that teachers in India, Saudi Arabia, the U.K., and Turkey self-report low confidence in their understanding of CS (Ramen et al., 2015; Alfayez & Lambert, 2019; Royal Society, 2017; Gülbahar & Kalelioğlu, 2017). Indeed, developing the necessary skills and confidence levels for teachers to offer effective CS instruction remains challenging.\n",
      "\n",
      "\n",
      "To address these challenges, school systems have introduced continuous professional development (PD), postgraduate certification programs, and CS credentials issued by teacher education degree programs. PD programs are common approaches, as they utilize the existing teacher workforce to fill the needs for special skills, rather than recruiting specialized teachers from outside the school system. For example, the British Computing Society created 10 regional university-based hubs to lead training activities, including lectures and meetings, to facilitate collaboration as part of the network of excellence (Dickens, 2016; Heintz et al., 2016; Royal Society, 2017). Most hubs involve multi-day seminars and workshops meant to familiarize teachers with CS concepts and provide ongoing support to help teachers as they encounter new challenges in the classroom. Cutts et al. (2017) further recommend teacher-led PD groups so that CS teachers can form collaborative professional networks. Various teacher surveys have found these PD programs in CS helpful (Alkaria & Alhassan, 2017; Goode et al., 2014). Still, more evidence is needed on the effectiveness of PD programs in CS education specifically (Hill, 2009).\n",
      "\n",
      "\n",
      "Less commonly, some education systems have worked with teacher training institutions to introduce certification schemes so teachers can signal their unique qualifications in CS to employers. This signal can make teacher recruitment more transparent and incentivize more teachers to pursue training. This approach does require, though, an investment in developing CS education faculty at the teacher training institution, which may be a critical bottleneck in many places (Delyser et al., 2018). Advocates of the approach have recommended that school systems initiate certification schemes quickly and with a low bar at first, followed by improvement over time (Code.org, 2017; Lang et al., 2013; Sentance & Csizmadia, 2017). Short-term recommendations include giving temporary licenses to teachers who meet minimum content and knowledge requirements. Long-term recommendations, on the other hand, encourage preservice teachers to take CS courses as part of their teaching degree programs or in-service teachers to take CS courses as part of their graduate studies to augment their skillset.2 Upon completing these courses, teachers would earn a full CS endorsement or certificate.\n",
      "\n",
      "\n",
      "3. Fostering student engagement and interest in CS education\n",
      "\n",
      "\n",
      "Surveys from various countries suggest that despite a clear economic incentive, relatively few K-12 students express interest in pursuing advanced CS education. For example, 3 out of 4 U.S. students in a recent survey declared no interest in pursuing a career in computer science. And the differences by gender are notable: Nearly three times as many male students (33 percent) compared to female students (12 percent) expressed interest in pursuing a computer science career in the future (Google & Gallup, 2020).\n",
      "\n",
      "\n",
      "Generally, parents view CS education favorably but also hold distinct misconceptions. For instance, more than 80 percent of U.S. parents surveyed in a Google and Gallup (2016) study reported that they think CS is as important as any other discipline. Nevertheless, the same parents indicated biases around who should take CS courses: 57 percent of parents think that one needs to be “very smart” to learn CS (Google & Gallup, 2015). Researchers have equated this kind of thinking to the idea that some people could be inherently gifted or inept at CS, a belief that could discourage some students from developing an interest or talent in CS (McCartney, 2017). Contrary to this belief, Patitsas et al. (2019) found that only 5.8 percent of university-level exam distributions were multimodal, indicating that most classes did not have a measurable divide between those who were inherently gifted and those who were not. This signals that CS is no more specialized to specific groups of students than any other subject.\n",
      "\n",
      "\n",
      "Fostering student engagement, however, does not equate to developing a generation of programmers. Employment projections suggest the future demand for workers with CS skills will likely outpace supply in the absence of promoting students’ interest in the field. Yet, no countries expand access to CS education with the expectation of turning all students into computer programmers. Forcing students into career paths that are unnatural fits for their interests and skill levels result in worse outcomes for students at the decision margins (Kirkeboen et al., 2016). Rather, current engagement efforts both expose students to foundational skills that help navigate technology in 21st century life and provide opportunities for students to explore technical fields.\n",
      "\n",
      "\n",
      "A lack of diversity in CS education not only excludes some people from accessing high-paying jobs, but it also reduces the number of students who would enter and succeed in the field (Du & Wimmer, 2019). Girls and racial minorities have been historically underrepresented in CS education (Sax et al., 2016). Research indicates that the diversity gap is not due to innate talent differences among demographic groups (Sullivan & Bers, 2012; Cussó-Calabuig et al., 2017), but rather a disparity of access to CS content (Google & Gallup 2016; Code.org & CSTA, 2018; Du & Wimmer, 2019), widely held cultural perceptions, and poor representation of women and underrepresented minorities (URMs) among industry leaders and in media depictions (Google & Gallup, 2015; Ayebi-Arthur, 2011; Downes & Looker, 2011).\n",
      "\n",
      "\n",
      "To help meet the demand for CS professionals, government and philanthropic organizations have implemented programs that familiarize students with CS. By increasing student interest among K-12 students who may eventually pursue CS professions, these strategies have the potential to address the well documented lack of diversity in the tech industry (Harrison, 2019; Ioannou, 2018).For example, some have used short, one-time lessons in coding to reduce student anxiety around CS. Of these lessons, perhaps the best known is Hour of Code, designed by Code.org. In multiple surveys, students indicated more confidence after exposure to this program (Phillips & Brooks, 2017; Doukaki et al., 2013; Lang et al., 2016). It is not clear, however, whether these programs make students more likely to consider semester-long CS courses (Phillips & Brooks, 2017; Lang et al., 2016).\n",
      "\n",
      "\n",
      "Other initiatives create more time-intensive programs for students. The U.S. state of Georgia, for example, implemented a program involving after-school, weekend, and summer workshops over a six-year period. Georgia saw an increase in participation in the Advanced Placement (AP) CS exam during the duration of the program, especially among girls and URMs (Guzdial et al., 2014). Other states have offered similar programs, setting up summer camps and weekend workshops in universities to help high school students become familiar with CS (Best College Reviews, 2021). These initiatives, whether one-off introductions to CS or time-intensive programs, typically share the explicit goal of encouraging participation in CS education among all students, and especially girls and URMs.\n",
      "\n",
      "\n",
      "Yet, while studies indicate that Hour of Code and summer camps might improve student enthusiasm for CS, they do not provide the kind of rigorous impact assessment one would need to make a definitive conclusion of their effectiveness. They do not use a valid control group, meaning that there is no like-for-like comparison to students who are similar except for no exposure to the program. It is not clear that the increase in girls and URMs taking CS would not have happened if it were not for Georgia’s after-school clubs.\n",
      "\n",
      "\n",
      "4. Generating and using evidence on curriculum and core competencies, instructional methods, and assessment\n",
      "\n",
      "\n",
      "There is no one-size-fits-all CS curriculum for all education systems, schools, or classrooms. Regional contexts, school infrastructure, prior access, and exposure to CS need to be considered when developing CS curricula and competencies (Ackovska et al., 2015). Some CS skills, such as programming language, require access to computer infrastructure that may be absent in some contexts (Lockwood & Cornell, 2013). Rather than prescribing a curriculum, the U.S. K-12 Computer Science Framework Steering Committee (2016) recommends foundational CS concepts and competencies for education systems to consider. This framework encourages curriculum developers and educators to create learning experiences that extend beyond the framework to encompass student interests and abilities.\n",
      "\n",
      "\n",
      "There is increasing consensus around what core CS competencies students should master when they complete primary and secondary education. Core competencies that students may learn by the end of primary school include:\n",
      "\n",
      "\n",
      "abstraction—creating a model to solve a problem;generalization—remixing and reusing resources that were previously created;decomposition—breaking a complex task into simpler subtasks;algorithmic thinking—defining a series of steps for a solution, putting instructions in the correct sequence, and formulating mathematical and logical expressions;programming—understanding how to code a solution using the available features and syntax of a programming language or environment; anddebugging—recognizing when instructions do not correspond to actions and then removing or fixing errors (Angeli, 2016).\n",
      "\n",
      "\n",
      "Competencies that secondary school students may learn in CS courses include:\n",
      "\n",
      "\n",
      "logical and abstract thinking;representations of data, including various kinds of data structures;problem-solving by designing and programming algorithms using digital devices;performing calculations and executing programs;collaboration; and,ethics such as privacy and data security (Syslo & Kwiatkowska, 2015).\n",
      "\n",
      "\n",
      "Several studies have described various methods for teaching CS core competencies. Integrated development environments are recommended especially for teaching coding skills (Florez et al., 2017; Saez-Lopez et al., 2016).2 These environments include block-based programming languages that encourage novice programmers to engage with programming, in part by alleviating the burden of syntax on learners (Weintrop & Wilensky, 2017; Repenning, 1993). Others recommended a variety of teaching methods that blend computerized lessons with offline activities (Taub et al. 2009; Curzon et al., 2009, Ackovska et al., 2015). This approach is meant to teach core concepts of computational thinking while keeping students engaged in physical, as well as digital, environments (Nishida et al., 2009). CS Unplugged, for example, provides kinesthetic lesson plans that include games and puzzles that teach core CS concepts like decomposition and algorithmic thinking.\n",
      "\n",
      "\n",
      "Various studies have also attempted to measure traditional lecture-based instruction for CS (Alhassan 2017; Cicek & Taspinar, 2016).3 These studies, however, rely on small sample sizes wherein the experiment and control group each comprised of individual classes. More rigorous research is required to understand the effectiveness of teaching strategies for CS.\n",
      "\n",
      "\n",
      "No consensus has emerged on the best ways to assess student competency in core CS concepts (So et al., 2019; Djambong & Freiman, 2016). Though various approaches to assessment are widely available—including classical cognitive tests, standardized tests in digital environments, and CS Unplugged activity tests—too many countries have yet to introduce regular assessments that may evaluate various curricula or instructional methods in CS. While several assessments have been developed for CS and CT at various grade levels as part of various research studies, there have been challenges to broader use. This is due to either a lack of large-scale studies using these assessments or diversity in programming environments used to teach programming and CS or simply a lack of interest in using objective tests of learning (as opposed to student projects and portfolios).\n",
      "\n",
      "\n",
      "Fortunately, a growing number of organizations are developing standardized tests in CS and computational thinking. For example, the International Computer and Information Literacy Study included examinations in computational thinking in 2018 that had two 25-minute modules, where students were asked to develop a sequence of tasks in a program that related to a unified theme (Fraillon et al., 2020). The OECD’s PISA will also include questions in 2021 to assess computational thinking across countries (Schleicher & Partovi, 2019). The AP CS exam has also yielded useful comparisons that have indirectly evaluated CS teacher PD programs (Brown & Brown, 2019).\n",
      "\n",
      "\n",
      "In summary, the current evidence base provides little consensus on the specific means of scaling a high-quality CS education and leaves wide latitude for experimentation. Consequently, in this report we do not offer prescriptions on how to expand CS education, even while arguing that expanding access to it generally is beneficial for students and the societies that invest in it. Given the current (uneven) distribution of ICT infrastructure and CS education resources, high-quality CS education may be at odds with expanded access. While we focus on ensuring universal access first, it is important to recognize that as CS education scales both locally and globally, the issues of curricula, pedagogies, instructor quality, and evaluation naturally become more pressing.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Lessons from education systems that have introduced CS education\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Based on the available literature discussed in the previous section, we selected education systems that have implemented CS education programs and reviewed their progress through in-depth case studies. Intentionally, we selected jurisdictions at various levels of economic development, at different levels of progress in expanding CS education, and from different regions of the world. They include Arkansas (U.S.), British Columbia (Canada), Chile, England, Italy, New Brunswick (Canada), Poland, South Africa, South Korea, Thailand, and Uruguay. For each case, we reviewed the historical origins for introducing CS education and the institutional arrangements involved in CS education’s expansion. We also analyzed how the jurisdictions addressed the common challenges of ensuring CS teacher preparation and qualification, fostering student demand for CS education (especially among girls and URMs), and how they developed curriculum, identified core competencies, promoted effective instruction, and assessed students’ CS skills. In this section, we draw lessons from these case studies, which can be downloaded and read in full at the bottom of this page.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Figure 6 presents a graphical representation summarizing the trajectories of the case study jurisdictions as they expanded CS education. Together, the elements in the figure provide a rough approximation of how CS education has expanded in recent years in each case. For example, when South Korea focused its efforts on universal CS education in 2015, basic ICT infrastructure and broadband connectivity were already available in all schools and two CS education expansion policies had been previously implemented. Its movement since 2015 is represented purely in the vertical policy action space, as it moved up four intervals on the index. Uruguay, conversely, started expanding its CS education program t a lower level both in terms of ICT infrastructure (x-axis) and existing CS policies (y-axis). Since starting CS expansion efforts in 2007, though, it has built a robust ICT infrastructure in its school systems and implemented 4 of 7 possible policy actions.\n",
      "\n",
      "\n",
      "FIGURE 6\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Figure 6 suggests that first securing access to ICT infrastructure and broadband connectivity allows systems to dramatically improve access to and the quality of CS education. Examples include England, British Columbia, South Korea, and Arkansas. At the same time, Figure 6 suggests that systems that face the dual challenge of expanding ICT infrastructure and broadband connectivity and scaling the delivery of quality CS education, such as Chile, South Africa, Thailand, and Uruguay, may require more time and/or substantial investment to expand quality CS education to match the former cases. Even though Chile, Thailand, and especially Uruguay have made impressive progress since their CS education expansion efforts began, they continue to lag a few steps behind those countries that started with established ICT infrastructure in place.\n",
      "\n",
      "\n",
      "Our analysis of these case studies surfaced six key lessons (Figure 7) for governments wishing to take CS education to scale in primary and secondary schools, which we discuss in further detail below.\n",
      "\n",
      "\n",
      "FIGURE 7\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "1. Expanding tech-based jobs is a powerful lever for expanding CS education\n",
      "\n",
      "\n",
      "In several of the case studies, economic development strategies were the underlying motivation to introduce or expand CS education. For example, Thailand’s 2017 20-year Strategic Plan marked the beginning of CS education in that country. The 72-page document, approved by the Thai Cabinet and Parliament, explained how Thailand could become a more “stable, prosperous, and sustainable” country and proposed to reform the education curriculum to prepare students for future labor demands (20-year National Strategy comes into effect, 2018). Similarly, Arkansas’s Governor Hutchinson made CS education a key part of his first campaign in 2014 (CS for All, n.d.), stating that “Through encouraging computer science and technology as a meaningful career path, we will produce more graduates prepared for the information-based economy that represents a wide-open job market for our young people” (Arkansas Department of Education, 2019).\n",
      "\n",
      "\n",
      "Uruguay’s Plan Ceibal, named after the country’s national flowering tree, was likewise introduced in 2007 as a presidential initiative to incorporate technology in education and help close a gaping digital divide in the country. The initiative’s main objectives were to promote digital inclusion, graduate employability, a national digital culture, higher-order thinking skills, gender equity, and student motivation (Jara, Hepp, & Rodriguez, 2018)\n",
      "\n",
      "\n",
      "Last, in 2018, the European Commission issued the Digital Education Action Plan that enumerated key digital skills for European citizens and students, including CS and computational thinking (European Commission, 2018). The plan encouraged young Europeans to understand the algorithms that underpin the technologies they use on a regular basis. In response to the plan, Italy’s 2018 National Indications and New Scenarios report included a discussion on the importance of computational thinking and the potential role of educational gaming and robotics in enhancing learning outcomes (Giacalone, 2019). Then, in 2019, the Italian Ministry of Education and the Parliament approved a legislative motion to include CS and computational thinking in primary school curricula by 2022 (Orizzontescuola, 2019).\n",
      "\n",
      "\n",
      "In some cases, the impetus to expand CS education came more directly from demands from key stakeholders, including industry and parents. For example, British Columbia’s CS education program traces back to calls from a growing technology industry (Doucette, 2016). In 2016, the province’s technology sector employed 86,000 people—more than the mining, forestry, and oil and gas sectors combined, with high growth projections (Silcoff, 2016). The same year, leaders of the province’s technology companies revealed in interviews that access to talent had become their biggest concern (KPMG, 2016). According to a 2016 B.C. Technology Association report, the province needed 12,500 more graduates in CS from tertiary institutions between 2015 and 2021 to fill unmet demand in the labor market (Orton, 2018). The economic justification for improving CS education in the province was clear.\n",
      "\n",
      "\n",
      "Growing parental demand helped create the impetus for changes to the CS curriculum in Poland. According to Kozlowski (2016), Polish parents perceive CS professions as some of the most desirable options for their children. And given the lack of options for CS education in schools, parents often seek out extracurricular workshops for their children to encourage them to develop their CS skills (Panskyi, Rowinska, & Biedron, 2019). The lack of in-school CS options for students created the push for curricular reforms to expand CS in primary and secondary schools. As former Minister of Education Anna Zalewska declared, Polish students “cannot afford to waste time on [the] slow, arduous task of building digital skills outside school [ and] only school education can offer systematic teaching of digital skills” (Szymański, 2016).\n",
      "\n",
      "\n",
      "2. ICT in schools provides the foundation to expand CS education\n",
      "\n",
      "\n",
      "Previous efforts to expand access to devices, connectivity, or basic computer literacy in schools provided a starting point in several jurisdictions to expand CS education. For example, the Uruguayan government built its CS education program after implementing expansive one-to-one computing projects, which made CS education affordable and accessible. In England, an ICT course was implemented in schools in the mid-1990s. These dedicated hours during the school day for ICT facilitated the expansion of CS education in the country.\n",
      "\n",
      "\n",
      "The Chilean Enlaces program, developed in 1992 as a network of 24 universities, technology companies, and other organizations (Jara, Hepp, & Rodriguez, 2018; Sánchez & Salinas, 2008) sought to equip schools with digital tools and train teachers in their use (Severin, 2016). It provided internet connectivity and digital devices that enabled ICT education to take place in virtually all of Chile’s 10,000 public and subsidized private schools by 2008 (Santiago, Fiszbein, Jaramillo, & Radinger, 2017; Severin et al., 2016). Though Enlaces yielded few observable effects on classroom learning or ICT competencies (Sánchez & Salinas, 2008), the program provided the infrastructure needed to begin CS education initiatives years later.\n",
      "\n",
      "\n",
      "While a history of ICT expansion can serve as a base for CS education, institutional flexibility to transform traditional ICT projects into CS education is crucial. The Chilean Enlaces program’s broader institutional reach resulted in a larger bureaucracy, slower implementation of new programs, and greater dependence on high-level political agendas (Severin, 2016). As a result, the program’s inflexibility prevented it from taking on new projects, placing the onus on the Ministry of Education to take the lead in initiating CS education. In Uruguay, Plan Ceibal’s initial top-down organizational structure enabled relatively fast implementation of the One Laptop per Child program, but closer coordination with educators and education authorities may have helped to better integrate education technology into teaching and learning. More recently, Plan Ceibal has involved teachers and school leaders more closely when introducing CS activities. In England, the transition from ICT courses to a computing curriculum that prioritized CS concepts, instead of computer literacy topics that the ICT teachers typically emphasized before the change, encountered some resistance. Many former ICT teachers were not prepared to implement the new program of study as intended, which leads us to the next key lesson.\n",
      "\n",
      "\n",
      "3. Developing qualified teachers for CS education should be a top priority\n",
      "\n",
      "\n",
      "The case studies highlight the critical need to invest in training adequate numbers of teachers to bring CS education to scale. For example, England took a modest approach to teacher training during the first five years of expanding its CS education K-12 program and discovered that its strategy fell short of its original ambitions. In 2013, the English Department for Education (DfE) funded the BCS to establish and run the Network of Excellence to create learning hubs and train a pool of “master” CS teachers. While over 500 master teachers were trained, the numbers were insufficient to expand CS education at scale. Then, in 2018 the DfE substantially increased its funding to establish the National Center for Computing Education (NCCE) and added 23 new computing hubs throughout England. Hubs offer support to primary and secondary computing teachers in their designated areas, including teaching, resources, and PD (Snowdon, 2019). In just over two years, England has come a long way toward fulfilling its goals of training teachers at scale with over 29,500 teachers engaged in some type of training (Teach Computing, 2020).\n",
      "\n",
      "\n",
      "Several education systems partnered with higher education institutions to integrate CS education in both preservice and in-service teacher education programs. For example, two main institutions in British Columbia, Canada—the University of British Columbia and the University of Northern British Columbia—now offer CS courses in their pre-service teacher education programs. Similarly, in Poland, the Ministry of National Education sponsored teacher training courses in university CS departments. In Arkansas, state universities offer CS certification as part of preservice teacher training while partnering with the Arkansas Department of Education to host in-service professional development.\n",
      "\n",
      "\n",
      "Still other systems partnered with nonprofit organizations to deliver teacher education programs. For instance, New Brunswick, Canada, partnered with the nonprofit organization Brilliant Labs to implement teacher PD programs in CS (Brilliant Labs, n.d.). In Chile, the Ministry of Education partnered with several nongovernmental organizations, including Code.org and Fundación Telefónica, to expand teacher training in CS education. Microsoft Philanthropies launched the Technology Education and Literacy in Schools (TEALS) in the United States and Canada to connect high school teachers to technology industry volunteers. The volunteer experts support instructors to learn CS independently over time and develop sustainable high school CS programs (Microsoft, n.d.).\n",
      "\n",
      "\n",
      "To encourage teachers to participate in these training programs, several systems introduced teacher certification pathways in CS education. For example, in British Columbia, teachers need at least 24 credits of postsecondary coursework in CS education to be qualified to work in public schools. The Arkansas Department of Education incentivizes in-service teachers to attain certification through teaching CS courses and participating in approved PD programs (Code.org, CSTA, ECEP, 2019). In South Korea, where the teaching profession is highly selective and enjoys high social status, teachers receive comprehensive training on high-skill computational thinking elements, such as computer architecture, operating systems, programming, algorithms, networking, and multimedia. Only after receiving the “informatics–computer” teacher’s license may a teacher apply for the informatics teacher recruitment exam (Choi et al., 2015).\n",
      "\n",
      "\n",
      "When faced with shortages of qualified teachers, remote instruction can provide greater access to qualified teachers. For example, a dearth of qualified CS teachers has been and continues to be a challenge for Uruguay. To address this challenge, in 2017, Plan Ceibal began providing remote instruction in computational thinking lessons for public school fifth and sixth graders and integrated fourth-grade students a year later. Students work on thematic projects anchored in a curricular context where instructors integrate tools like Scratch.4 During the school year, a group of students in a class can work on three to four projects during a weekly 45-minute videoconference with a remote instructor, while another group can work on projects for the same duration led by the classroom teacher. In a typical week, the remote instructor introduces an aspect of computational thinking. The in-class teacher then facilitates activities like block-based programming, circuit board examination, or other exercises prescribed by the remote teacher (Cobo & Montaldo, 2018).5 Importantly, Plan Ceibal implements Pensamiento Computacional, providing a remote instructor and videoconferencing devices at the request of schools, rather than imposing the curriculum on all classrooms (García, 2020). With the ongoing COVID-19 pandemic forcing many school systems across the globe to adopt remote instruction, at least temporarily, we speculate that remote learning is now well poised to become more common in expanding CS education in places facing ongoing teacher shortages.\n",
      "\n",
      "\n",
      "4. Exposing students to CS education early helps foster demand, especially among underserved populations\n",
      "\n",
      "\n",
      "Most education systems have underserved populations who lack the opportunity to develop an interest in CS, limiting opportunities later in life. For example, low CS enrollment rates for women at Italian universities reflect the gender gap in CS education. As of 2017, 21.6 percent and 12.3 percent of students completing bachelor’s degrees in information engineering and CS, respectively, were women (Marzolla, 2019). Further, female professors and researchers in these two subjects are also underrepresented. In 2018, only 15 percent and 24 percent of professors and researchers in CS and computer engineering, respectively, were women (Marzolla, 2019). Similar representation gaps at the highest levels of CS training are common globally. Thus, continuing to offer exposure to CS only in post-secondary education will likely perpetuate similar representation gaps.\n",
      "\n",
      "\n",
      "To address this challenge, several education systems have implemented programs to make CS education accessible to girls and other underserved populations in early grades, before secondary school. For instance, to make CS education more gender balanced, the Italian Ministry of Education partnered with civil society organizations to implement programs to spur girls’ interest in CS and encourage them to specialize in the subject later (European Commission, 2009). An Italian employment agency (ironically named Men at Work) launched a project called Girls Code It Better to extend CS learning opportunities to 1,413 middle school girls across 53 schools in 2019 (Girls Code It Better, n.d.). During the academic year, the girls attended extracurricular CS courses before developing their own technologically advanced products and showcasing their work at an event at Bocconi University in Milan (Brogi, 2019). In addition to introducing the participants to CS, the initiative provided the girls with role models and generated awareness on the gender gap in CS education in Italy.\n",
      "\n",
      "\n",
      "In British Columbia, students are exposed to computational thinking concepts as early as primary school, where they learn how to prototype, share, and test ideas. In the early grades of primary education, the British Columbia curriculum emphasizes numeracy using technology and information technology. Students develop numeracy skills by using models and learn information technology skills to apply across subjects. In kindergarten and first grade, curricular objectives include preparing students for presenting ideas using electronic documents. In grades 2 to 3, the curricular goals specify that students should “demonstrate an awareness of ways in which people communicate, including the use of technology,” in English language arts classes, as well as find information using information technology tools. By the time students are in grades 4 and 5, the curriculum expects students to focus more on prototyping and testing new ideas to solve a problem (Gannon & Buteau, 2018).\n",
      "\n",
      "\n",
      "Several systems have also increased participation in CS education by integrating it as a cross-curricular subject. This approach avoids the need to find time during an already-packed school day to teach CS as a standalone subject. For example, in 2015, the Arkansas legislature began requiring elementary and middle school teachers to embed computational thinking concepts in other academic courses. As a result, teachers in the state integrate five main concepts of computational thinking into their lesson plans, including (1) problem-solving, (2) data and information, (3) algorithms and programs, (4) computers and communications, and, importantly, (5) community, global, and ethical impacts (Watson-Fisher, 2019). In the years following this reform, the share of African American students taking CS in high school reached 19.6 percent, a figure that slightly exceeds the percentage of African Americans among all students—a resounding sign of progress in creating student demand for CS education (Computer science on the rise in Arkansas schools, Gov. drafts legislation to make it a requirement for graduation, 2020).\n",
      "\n",
      "\n",
      "After-school programs and summer camps, jointly organized with external partners, have also helped promote demand for CS education through targeted outreach programs to commonly underserved populations. For example, Microsoft Thailand has been holding free coding classes, Hour of Code, in partnership with nonprofit organizations, to encourage children from underprivileged backgrounds to pursue STEM education (Microsoft celebrates Hour of Code to build future ready generations in Asia, 2017). In the past decade, Microsoft has extended opportunities for ICT and digital skills development to more than 800,000 youth from diverse backgrounds—including those with disabilities and residents of remote communities (Thongnab, 2019). Their annual #MakeWhatsNext event for young Thai women showcases STEM careers and the growing demand for those careers (Making coding fun for Thailand’s young, 2018). Also in Thailand, Redemptorist Foundation for People with Disabilities, with over 30 years of experience working with differently abled communities in that country, expanded their services to offer computer trainings and information technology vocational certificate programs for differently abled youth (Mahatai, n.d.).\n",
      "\n",
      "\n",
      "In British Columbia, Canada, the Ministry of Education and other stakeholders have taken steps to give girls, women, and aboriginal students the opportunity to develop an interest in CS education.For example, after-school programs have taken specific steps to increase girls’ participation in CS education. The UBC Department of Computer Science runs GIRLsmarts4tech, a program that focuses on giving 7th- grade girls role models and mentors that encourage them to pursue technology-related interests (GIRLsmarts4tech, n.d.). According to the latest census, in 2016, British Columbia’s First Nations and Indigenous Peoples (FNIP) population—including First Nations, Metis, and Inuits—was 270,585, an increase of 38 percent from 2006. With 42.5 percent of the FNIP population under 25, it is critical for the province to deliver quality education to this young and growing group (Ministry of Advanced Education, Skills and Training, 2018). To this end, part of the British Columbia curriculum for CS education incorporates FNIP world views, perspectives, knowledge, and practices in CS concepts. In addition, the B.C. based ANCESTOR project (AborigiNal Computer Education through STORytelling) has organized courses and workshops to encourage FNIP students to develop computer games or animated stories related to their culture and land (Westor & Binn, 2015).\n",
      "\n",
      "\n",
      "As these examples suggest, private sector and nongovernmental organizations can play an important role in the expansion of CS education, an issue we turn to now.\n",
      "\n",
      "\n",
      "5. Engaging key stakeholders can help address bottlenecks\n",
      "\n",
      "\n",
      "In most reviewed cases, the private sector and nongovernmental organizations played a role in promoting the expansion of CS education. Technology companies not only helped to lobby for expanding CS education, but often provided much-needed infrastructure and subject matter expertise in the design and rollout of CS education. For example, Microsoft Thailand has worked with the Thai government since 1998 in various capacities, including contributing to the development and implementation of coding projects, digital skills initiatives, teacher training programs, and online learning platforms (Thongnab, 2019; Coding Thailand, n.d.). Since 2002, Intel’s Teach Thailand program has trained more than 150,000 teachers. Additionally, Google Coding Teacher workshops train educators on teaching computational thinking through CS Unplugged coding activities (EduTech Thailand, 2019). The workshop is conducted by Edutech (Thailand) Co., Ltd., an educational partner of Google, which adapted the Google curriculum to the Thailand education context. Samsung has been engaged in a smart classroom project that has built futuristic classroom prototypes and provided training for 21st century competencies (OECD/UNESCO, 2016).\n",
      "\n",
      "\n",
      "In England, nongovernmental organizations have played an important role in supporting the government’s expansion of CS education. The DfE has relied on outside organizations for help in executing its CS education responsibilities. The DfE’s NCEE, for instance, is delivered by a consortium including the British Computing Society, STEM Learning, and the Raspberry Pi Foundation—three nonprofit organizations dedicated to advancing the computing industry and CS education in the country (British Computing Society, n.d; STEM Learning, n.d.; Raspberry Pi Foundation, n.d.).\n",
      "\n",
      "\n",
      "Chile’s Ministry of Education developed partnerships with individual NGOs and private companies to engage more students, especially girls. These initiatives offer the opportunity for hands-on learning projects and programming activities that students can perform from their home computers. Some of the same partners also provide online training platforms for teacher PD.\n",
      "\n",
      "\n",
      "Industry advocacy organizations can also play an important role in the expansion of CS education. For example, in Arkansas, the state’s business community has long supported CS education (Nix, 2017). Accelerate Arkansas was established in 2005 as an organization of 70 private and public sector members dedicated to moving Arkansas into a more innovation- and knowledge-based economy (State of Arkansas, 2018). Similarly, in England, a network of organizations called Computing at School established a coalition of industry representatives and teachers. It played a pivotal role in rebranding the ICT education program in 2014 to the computing program that placed a greater emphasis on CS (Royal Society, 2017).\n",
      "\n",
      "\n",
      "To ensure sustainability, one key lesson is that the government should coordinate across multiple stakeholders. The reliance on inputs from external organizations to drive CS education implies that the heavy reliance on NGO-provided training and resources in Chile have been insufficient to motivate more schools and teachers to include CS and computational thinking in classroom learning activities. By contrast, the DfE has effectively coordinated across various nongovernmental organizations to expand CS education. Similarly, Arkansas’s Department of Education is leading an effort to get half of all school districts to form partnerships with universities and business organizations to give students opportunities to participate internships and college-level CS courses while in high school (Talk Business & Politics, 2020). In sum, the experience of decades of educational policies across the education systems reviewed shows that schools require long lasting, coordinated, and multidimensional support to achieve successful implementation of CS in classrooms.\n",
      "\n",
      "\n",
      "6. When taught in an interactive, hands-on way, CS education builds skills for life\n",
      "\n",
      "\n",
      "Several of the cases studied introduced innovative pedagogies using makerspaces (learning spaces with customizable layouts and materials) and project-based learning to develop not only skills specific to CS but also skills that are relevant more broadly for life. For example, Uruguayan CS education features innovative concepts like robotics competitions and makerspaces that allow students to creatively apply their computational thinking lessons and that can spark interest and deepen understanding. In addition, computational thinking has been integrated across subject areas (e.g., in biology, math, and statistics) (Vázquez et al., 2019) and in interdisciplinary projects that immerse students in imaginative challenges that foster creative, challenging, and active learning (Cobo & Montaldo, 2018). For instance, students can use sensors and program circuit boards to measure their own progress in physical education (e.g., measuring how many laps they can run in a given period).\n",
      "\n",
      "\n",
      "Similarly, in New Brunswick, Brilliant Labs provide learning materials to schools so they can offer students CS lessons using makerspaces that encourage students to develop projects, engage with technology, learn, and collaborate. These makerspaces enable students to creatively apply their CS and computational thinking lessons, sparking interest and deepening understanding of CS and computational thinking.\n",
      "\n",
      "\n",
      "Thailand’s curricular reforms also integrated project-based learning into CS education. Thai students in grades 4-6 learn about daily life through computers, including skills such as using logic in problem-solving, searching data and assessing its correctness, and block coding (e.g., Scratch). Then, students in grades 7-9 focus on learning about primary data through objectives that include using programming to solve problems, collecting, analyzing, presenting, and assessing data and information, and textual programming such as Python. Finally, students in grades 10-12 focus on applying advanced computing technology and programming to solve real-world problems, using knowledge from other subjects and data from external sources (Piamsa-nga et al., 2020).\n",
      "\n",
      "\n",
      "After two years of nationwide discussions from 2014 to 2016, the Polish Ministry of National Education announced the creation of a new core curriculum for CS in primary and secondary schools (Syslo, 2020). The new curriculum’s goals included students using technology to identify solutions for problems in every day and professional situations and supporting other disciplines—such as science, the arts, and the social sciences—in innovation (Panskyi, Rowinska, & Biedron, 2019).\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Conclusion\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "CS skills are increasingly necessary to function in today’s technology-advanced world and for the future. They enable individuals to understand how technology works, and how best to harness its potential to improve lives. As these skills take preeminence in the rapidly changing 21st century, CS education promises to significantly enhance student preparedness for the future of work and active citizenship.\n",
      "\n",
      "\n",
      "Our findings suggest six recommendations for governments interested in taking CS education to scale in primary and secondary schools. First, governments should use economic development strategies focused on expanding technology-based jobs to engage all stakeholders and expand CS education in primary and secondary schools. Indeed, such a strategy helps attract and retain investors and foster CS education demand among students. Second, provide access to ICT infrastructure in primary and secondary schools to facilitate the introduction and expansion of CS education. Third, developing qualified teachers for CS should be a top priority. The evidence is clear that a qualified teacher is the most important factor in student learning, and thus preparing the teacher force needed for CS at scale is crucial. Fourth, expose students early to CS education to increase their likelihood of pursuing it. This is especially important for girls and other URM groups historically underrepresented in STEM and CS fields. Fifth, engage key stakeholders (including educators, the private sector, and civil society) to help address bottlenecks in physical and technical capacity. Finally, teach CS in an interactive, hands-on way to build skills for life.\n",
      "\n",
      "\n",
      "Through studying the cases of regional and national governments at various levels of economic development and progress in implementing CS education programs, governments from around the globe can learn how to expand and improve CS education and help students develop a new basic skill necessary for the future of work and active citizenship.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Case studies\n",
      "\n",
      "\n",
      "For a detailed discussion of regional and national education systems from diverse regions and circumstances that have implemented computer science education programs, download the case studies.\n",
      "\n",
      "\n",
      " Download the full report \n",
      "\n",
      "https://www.brookings.edu/search/?s=data%20science\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.brookings.edu/blog/how-we-rise/2021/10/29/reckoning-with-science-medicine-and-scapegoating/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/blog/how-we-rise/2021/10/29/reckoning-with-science-medicine-and-scapegoating/\n",
      "TITLE: Reckoning with science, medicine, and scapegoating\n",
      "Reckoning with science, medicine, and scapegoating\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/research/the-potential-of-the-chips-and-science-act-for-rural-america/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/research/the-potential-of-the-chips-and-science-act-for-rural-america/\n",
      "TITLE: The potential of the CHIPS and Science Act for rural America\n",
      "The potential of the CHIPS and Science Act for rural America\n",
      "BODY:\n",
      "\n",
      "New article found for topic: python\n",
      "URL: https://www.brookings.edu/research/institutionalizing-data-analysis-in-german-federal-governance/\n",
      "TITLE: Institutionalizing Data Analysis in German Federal Governance\n",
      "Institutionalizing Data Analysis in German Federal Governance\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/research/in-central-new-york-a-test-of-the-chips-and-science-acts-promise-for-economic-revitalization/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/research/in-central-new-york-a-test-of-the-chips-and-science-acts-promise-for-economic-revitalization/\n",
      "TITLE: In Central New York, a test of the CHIPS and Science Act’s promise for economic revitalization\n",
      "In Central New York, a test of the CHIPS and Science Act’s promise for economic revitalization\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/news-releases/ben-bernanke-distinguished-senior-fellow-in-residence-receives-nobel-prize-in-economic-sciences/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/news-releases/ben-bernanke-distinguished-senior-fellow-in-residence-receives-nobel-prize-in-economic-sciences/\n",
      "TITLE: Ben Bernanke, distinguished senior fellow in residence, receives Nobel Prize in economic sciences\n",
      "Ben Bernanke, distinguished senior fellow in residence, receives Nobel Prize in economic sciences\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/techstream/how-data-science-can-ease-the-covid-19-pandemic/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/techstream/how-data-science-can-ease-the-covid-19-pandemic/\n",
      "TITLE: How data science can ease the COVID-19 pandemic\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Social distancing and stay-at-home orders in the United States have slowed the infection rate of SARS-CoV-2, the pathogen that causes COVID-19. This has halted the immediate threat to the U.S. healthcare system, but consensus on a long-term plan or solution to the crisis remains unclear.  As the reality settles in that there are no quick fixes and that therapies and vaccines will take several months if not years to invent, validate, and mass produce, this is a good time to consider another question: How can data science and technology help us endure the pandemic while we develop therapies and vaccines?\n",
      "\n",
      "\n",
      "Before policymakers reopen their economies, they must be sure that the resulting new COVID-19 cases will not force local healthcare systems to resort to crisis standards of care. Doing so requires not just prevention and suppression of the virus, but ongoing measurement of virus activity, assessment of the efficacy of suppression measures, and forecasting of near-term demand on local health systems. This demand is highly variable given community demographics, the prevalence of pre-existing conditions, and population density and socioeconomics.\n",
      "\n",
      "\n",
      "Data science can already provide ongoing, accurate estimates of health system demand, which is a requirement in almost all reopening plans. We need to go beyond that to a dynamic approach of data collection, analysis, and forecasting to inform policy decisions in real time and iteratively optimize public health recommendations for re-opening. While most reopening plans propose extensive testing, contact tracing, and monitoring of population mobility, almost none consider setting up such a dynamic feedback loop. Having such feedback could determine what level of virus activity can be tolerated in an area, given regional health system capacity, and adjust population distancing accordingly.\n",
      "\n",
      "\n",
      "We propose that by using existing technology and some nifty data science, it is possible to set up that feedback loop, which would maintain healthcare demand under the threshold of what is available in a region. Just as the maker community stepped up to cover for the failures of the government to provide adequate protective gear to health workers, this is an opportunity for the data and tech community to partner with healthcare experts and provide a measure of public health planning that governments are unable to do. Therefore, the question we invite the data science community to focus on is: How can data science help forecast regional health system resource needs given measurements of virus activity and suppression measures such as population distancing?\n",
      "\n",
      "\n",
      "For the data science effort to work, first and foremost, we need to fix delays in data collection and access introduced by existing reporting processes. Currently, most departments of public health are collecting and reporting metrics that are not helpful, and are reporting them with 48 hour delays, and often with errors. Although there are examples of regional excellence in such reporting, by and large, the recommendations from the health IT community around accurate and fast public health reporting remain ignored. For instance, consider the number of COVID-19 hospitalizations, which is the best indicator of the disease’s burden on the regional health system. At the present time, due to time lags in confirming and reporting cases and a failure to distinguish between current and cumulative hospitalizations, even regions that report hospitalization data often provide only a blurry picture of the burden on the regional health system. Regions should ideally report both suspected and confirmed hospital cases and indicate the date of admission, in addition to the date of report or confirmation.\n",
      "\n",
      "\n",
      "Even with perfect reporting, there are fundamental delays in what such data can tell us. For example, new admissions to a hospital today reflect virus activity as of 9 to 13 days ago (which depends, in turn, on social distancing interventions from up to 17 days prior). Not factoring in such considerations have led to significant over-estimation of hospitalization needs nationwide. We therefore need to measure virus activity via proxy measures that are indicative early in the lifecycle of the virus. We must benchmark these against the number of new and total COVID-19 hospitalizations as well as ideally the number of new infections, assuming it is accurately measured through large scale testing. Available proxy measures include test positivity rates in health systems, case counts, deaths and perhaps seropositivity rates. Ongoing symptom tracking via smartphone apps, daily web or phone surveys, or cough sounds can identify potential hotspots where virus transmission rates are high. Contact tracing, which currently requires significant human effort, can also help tracking of potential cases if it can be scaled using technology under development by major American tech companies. \n",
      "\n",
      "\n",
      "With reliable tracking and benchmarking in place, we can calculate infection prevalence as well as daily growth and transmission rates, which is essential for determining if policies are working. This is a problem not only of data collection but also data analysis. Issues of sensitivity, daily variability, time lags, and confounding need to be studied before such data can be used reliably. For instance, symptom tracking is nonspecific and may have difficulty tracking virus activity at low prevalence. Other emerging data sources such as wastewater and smart thermometer data hold similar promise but will have to grapple with these same issues.\n",
      "\n",
      "\n",
      "We then need to estimate the regional effects of policy interventions such as shelter-in-place orders (via mobility reduction) and contact tracing (via reductions in new cases), first as simple forecasts and eventually maturing to what-if analyses. Several efforts have quantified the impact of mobility on virus transmission and some have suggested “safe” forms of mobility. While there are many potential ways to quantify population mobility — such as via traffic patterns, internet bandwidth usage by address, and location of credit card swipes — the most scalable mechanism to measure mobility appears to be via tracking of smartphones. Groups such as the COVID-19 Mobility Data Network provide such data daily in anonymized, aggregated reports.\n",
      "\n",
      "\n",
      "Once the ability to project from mobility to transmission to health system burden is constructed, we can “close the loop” by predicting how much mobility we can afford given measured virus activity and anticipated health system resources in the next two weeks. Researchers have already attempted to calculate “tolerable transmission” in the form of maximum infection prevalence in a given geography that would not overload health systems. Coupling such tolerable transmission estimates with daily assessments of a valid sample of the population (via testing, via daily surveys, via electronic health record-based surveillance) would allow monitoring of changes in transmission which can alert us to the need to intervene, such as by reducing mobility. As new measures such as contact tracing cut transmission rates, these same monitoring systems can tell us that it is safe to increase mobility further. Continuously analyzing current mobility as well as virus activity and projected health system capacity can allow us to set up “keep the distance” alerts that trade off tolerable transmission against allowed mobility. Doing so will allow us to intelligently balance public health and economic needs in real time.\n",
      "\n",
      "\n",
      "Concretely, then, the crucial “data science” task is to learn the counterfactual function linking last week’s population mobility and today’s transmission rates to project hospital demand two weeks later. Imagine taking past measurements of mobility around April 10 in a region (such as the Santa Clara County’s report from COVID-19 Community Mobility Reports), the April 20 virus transmission rate estimate for the region (such as from http://rt.live), and the April 25 burden on the health system (such as from the Santa Clara County Hospitalization dashboard), to learn a function that uses today’s mobility and transmission rates to anticipate needed hospital resources two weeks later. It is unclear how many days of data of each proxy measurement we need to reliably learn such a function, what mathematical form this function might take, and how we do this correctly with the observational data on hand and avoid the trap of mere function-fitting. However, this is the data science problem that needs to be tackled as a priority. \n",
      "\n",
      "\n",
      "Adopting such technology and data science to keep anticipated healthcare needs under the threshold of availability in a region requires multiple privacy trade-offs, which will require thoughtful legislation so that the solutions invented for enduring the current pandemic do not lead to loss of privacy in perpetuity. However, given the immense economic as well as hidden medical toll of the shutdown, we urgently need to construct an early warning system that tells us to enhance suppression measures if the next COVID-19 outbreak peak might overwhelm our regional healthcare system. It is imperative that we focus our attention on using data science to anticipate, and manage, regional health system resource needs based on local measurements of virus activity and effects of population distancing.\n",
      "\n",
      "\n",
      "Dr. Nigam Shah is an associate professor of Medicine (Biomedical Informatics) at Stanford University and  Associate CIO for data science at Stanford Healthcare.Dr. Jacob Steinhardt is an assistant professor of statistics at University of California, Berkeley.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "https://www.brookings.edu/blog/africa-in-focus/2022/01/26/investment-in-science-and-technology-is-key-to-an-african-economic-boom/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/blog/africa-in-focus/2022/01/26/investment-in-science-and-technology-is-key-to-an-african-economic-boom/\n",
      "TITLE: Investment in science and technology is key to an African economic boom\n",
      "Investment in science and technology is key to an African economic boom\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/research/exploring-the-state-of-computer-science-education-amid-rapid-policy-expansion/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/research/exploring-the-state-of-computer-science-education-amid-rapid-policy-expansion/\n",
      "TITLE: Exploring the state of computer science education amid rapid policy expansion\n",
      "Exploring the state of computer science education amid rapid policy expansion\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/techstream/how-china-harnesses-data-fusion-to-make-sense-of-surveillance-data/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/techstream/how-china-harnesses-data-fusion-to-make-sense-of-surveillance-data/\n",
      "TITLE: How China harnesses data fusion to make sense of surveillance data\n",
      "BODY:\n",
      "\n",
      "Hikvision surveillance cameras are seen in front of a Chinese flag at a shopping area in Shanghai on May 5, 2021. (REUTERS/Aly Song)\n",
      "\n",
      "\n",
      "Across the Chinese government’s surveillance apparatus, its many arms are busy collecting huge volumes of data. Video surveillance footage, WeChat accounts, e-commerce data, medical history, and hotel records: It’s all fair game for the government’s surveillance regime. Yet, taken individually, each of these data streams don’t tell authorities very much. That’s why the Chinese government has embarked on a massive project of data fusion, which merges disparate datasets to produce data-driven analysis. This is how Chinese surveillance systems achieve what authorities call “visualization” (可视化) and “police informatization” (警务信息化). \n",
      "\n",
      "\n",
      "While policymakers around the world have grown increasingly aware of China’s mass surveillance regime—from its most repressive practices in Xinjiang to its exports of surveillance platforms to more than 80 countries—relatively little attention has been paid to how Chinese authorities are making use of the data it collects. As countries and companies consider how to respond to China’s surveillance regime, policymakers need to understand data fusion’s crucial role in monitoring the country’s population in order to craft effective responses.  \n",
      "\n",
      "\n",
      "Data fusion in Chinese surveillance programs \n",
      "\n",
      "\n",
      "As China’s population has embraced online life, the Party-state’s mass surveillance practices have evolved from relying on more manual methods—such as dānwèi (单位) work units, the hùkǒu (户口) residency registration system, and dǎng’àn (档案) secret political files—to using technologies that range from the mundane to the cutting-edge. To achieve the goal of “stability maintenance” (维稳), China’s national surveillance programs utilize varying degrees of data fusion. Data fusion is present, for example, in national defense crisis response platforms (国防动员) developed in the mid-2010s that gather data from multiple “thematic clouds,” including e-commerce, tourism, industry, courts, and law enforcement. Other recent programs that rely on data fusion include Sharp Eyes (雪亮工程), the nation-wide Police Cloud (警务云), and Xinjiang’s Integrated Joint Operations Platform (IJOP, 一体化联合作战平台).  \n",
      "\n",
      "\n",
      "One of the Chinese government’s most prominent data fusion programs is Sharp Eyes, which was launched in 2015 by nine government bodies. The program builds on the infrastructure used by Skynet—a 2005 initiative that focused on surveillance in urban public areas—and extends it into rural areas. Sharp Eyes pulls from a wide variety of data sources. These include surveillance cameras—both privately and government-owned and with and without facial-recognition capabilities—and vehicle and license plate recognition cameras. Public and private video surveillance systems collect facial and other attributes from key locations such as hospitals, schools, entertainment venues, hotels, internet cafes, major road intersections, and storefronts. Sharp Eyes also aims to collect “virtual identities,” such as MAC addresses, phone numbers, and WeChat accounts. \n",
      "\n",
      "\n",
      "Authorities ascertain individuals’ identities by first combining the above information with geographic information source (GIS) data and then sending this data to “societal resource integration platforms,” which exist in Xinjiang and at least four other provinces. According to analysis originally published in the journal China Digital Cable TV, a publication supervised by the Ministry of Education and the Ministry of Science and Technology, these platforms combine facial and vehicle recognition data and match it against private and public video sources. GIS data is superimposed on live video feeds to provide granular location data. Multiple companies can be involved in one platform project. For example, one local Sharp Eyes project in Fujian Province uses products from prominent (and blacklisted) AI companies such as Yitu, Huawei, and Hikvision.  \n",
      "\n",
      "\n",
      "However, Sharp Eyes is not entirely powered by data fusion. In fact, human-centric surveillance is a key design element. In the city of Linyi, where Sharp Eyes was piloted, the local government upgraded citizens’ television cable boxes so they could view surveillance feeds and report crimes by pushing a button on their TV remotes. The Ministry of Justice even provided a patriotic slogan for the effort: “remote control in hand, safety in heart.” (This citizen-centric surveillance strategy originates from the Cultural Revolution, which inspired Sharp Eyes’ name.) As part of Sharp Eyes, mobile apps push video surveillance and public security information to citizens and allow assigned groups of households to report crimes. Command and control centers are staffed by personnel to review footage, take citizens’ reports, and dispatch police accordingly. \n",
      "\n",
      "\n",
      "Another national program that uses data fusion is the Ministry of Public Security’s (MPS) Police Cloud, which has been active since 2015. Provincial police cloud-computing centers fuse data from public and private sources, including ID cards, CCTV footage, medical history, supermarket memberships, IP addresses, social media usernames, delivery records, residential addresses, hotel stays, petition records, and biometrics, according to a 2017 report from Human Rights Watch. In a nod to “visualizing” data, the system aims to uncover connections between individuals that would otherwise be difficult for police officers to detect on their own. It supposedly predicts future actions or threats that might cause social instability, such as protests and acts of terror.  \n",
      "\n",
      "\n",
      "Targeting ‘focus personnel’ \n",
      "\n",
      "\n",
      "China’s data-fusion programs allow its surveillance systems to assemble highly detailed portraits of the country’s citizens, but these systems apply particularly severe scrutiny to “focus personnel,” which includes individuals petitioning the government, those purportedly involved in terrorism, and those “undermining social stability.” China’s Uyghur ethnic minority is among those that fall under the “focus personnel” category, and in the Xinjiang region, the center of Uyghur life, this persecuted minority is subjected to intense surveillance. \n",
      "\n",
      "\n",
      "One key tool in this surveillance regime is the Integrated Joint Operations Platform (IJOP), which monitors Xinjiang residents with unprecedented intrusiveness. The system flags mundane and otherwise legal behavior as warranting further surveillance, imprisonment, or even extralegal internment in Xinjiang’s vast network of concentration camps. The IJOP functions as a data fusion tool by tying an individual’s government-issued ID card to her physical attributes (such as facial features, blood type, and height), as well as tracking where individuals’ phones, ID cards, and vehicles go. The system collects a variety of data from afar: excessive electricity use, the use of WhatsApp and VPNs, driving someone else’s car. The system also relies on highly intrusive methods. Scattered at strategic locations such as malls and mosques around Xinjiang are what are known as “three-dimensional portrait and integrated data doors” (三维人像综合数据门). These “doors” resemble airport metal detectors and possess facial recognition capabilities, ID card verification, and tools to lift a variety of data from mobile phones, such as MAC addresses, IMEI, IMSI, and ESN numbers. The IJOP also ingests data collected by what have been dubbed “anti-terrorism swords,” which are used at police checkpoints to plug into phones and download their contents, according to an Intercept investigation. IJOP sends push notifications to officers, who can pull aside someone walking through a data door. They can then interrogate individuals and detain or imprison them. In this way, the IJOP can restrict individuals’ movements, which are limited based on what threat level the system determines they fall under.  \n",
      "\n",
      "\n",
      "China’s military is another player using data-fusion tools in Xinjiang to build predictive policing systems. In 2019, a professor named Cui Yinglong at the People’s Armed Police Engineering University in the regional capital of Ürümqi developed a “dynamic early warning” system, known as the Tianshan Anti-terrorism Cloud (天山反恐云). It is trained on a database he helped build called the “East Turkistan terrorist activity database,” which collects and fuses data based on terrorist methods, objectives, attack dates, and organization from 1990 to the present. Although little information exists on the inner workings of the app, it purports to “accurately depict” terrorists’ “religious, organizational, and behavioral characteristics.” The cloud is designed to provide soldiers with early warning of terrorist activity and combat decision support, and is apparently in active operational use.  \n",
      "\n",
      "\n",
      "Issues with ‘information islands’  \n",
      "\n",
      "\n",
      "While it is tempting to conclude that China’s surveillance state is effortlessly and automatically tracking and surveilling every person in the country, its monitoring systems are plagued by human inefficiency, unreliable and incomplete basic data, and incompatible datasets and systems. These inefficiencies have resulted in data silos, also known as “government information islands” (政府信息孤岛), a phrase that refers to isolated data pools that are not adequately shared within government bureaucracies. Currently, data is shared horizontally (across departments and regions) and vertically (within the same organizational entity from the local level up). But scholars have found that horizontal integration bodies suffer from a lack of information that is stove-piped in vertical bodies. As a result, authorities are hindered by bureaucratic systems that prevent more comprehensive data access and greater insight.  \n",
      "\n",
      "\n",
      "To better execute data fusion, Chinese authorities are now attempting to tackle longstanding issues with data silos, according to a recent ChinaFile analysis of government procurement notices. In 2019, for example, authorities in Beijing looked to build a “Sharp Eyes Video Sharing and Exchange Platform” that would make available video footage from cameras belonging to a range of different departments on a single platform. By integrating footage from approximately 200,000 cameras—and up to 1 million—the platform would improve access to footage and data re-use. To further improve data fusion, future improvements to Sharp Eyes include implementing a standard data mining approach and overcoming technical difficulties and inconsistent standards.  \n",
      "\n",
      "\n",
      "On the legal front, authorities have also taken several steps to address data silos. While some of this reform preceded the advent of surveillance programs such as Sharp Eyes and the IJOP, scholars Huirong Chen and Sheena Chestnut Greitens note that newer reforms under Xi Jinping coincided with these programs. They include the 2017–18 reorganization of the People’s Armed Police and passing the 2017 National Intelligence Law, which aimed to integrate disparate intelligence and national security authorities. Furthermore, the 13th Five-Year Plan for National Informatization (2016–2020) detailed issues with “information islands,” and called for integrating systems across ministries and departments. In 2017, the Party asked local governments to establish comprehensive information platforms, likely similar to the ones established under Sharp Eyes. \n",
      "\n",
      "\n",
      "Policy implications \n",
      "\n",
      "\n",
      "Responding to the Chinese government’s use of data-fusion systems to power its surveillance systems represents a difficult challenge. Under the Trump Administration, the U.S. Department of Commerce’s Bureau of Industry and Security (BIS) blacklisted multiple Chinese companies for their human rights abuses in Xinjiang, adding them to the so-called “Entity List.” By being added to the Entity List, these firms should have been cut off from U.S. suppliers, but given the opaque nature of supply chains, it is difficult to assess whether sanctioned firms have been able to access U.S.-originating technologies via workarounds. It also difficult to assess whether homegrown R&D initiatives to develop domestic alternatives to U.S. goods have borne fruit. Furthermore, the vast majority of the companies on the Entity List do not have data fusion as their main line of business; rather, they often provide services that feed into fusion architectures. This means U.S. policy is potentially overlooking a key area of Chinese surveillance systems. (Media and nongovernmental organizations have created lists of Chinese and non-Chinese companies that enable surveillance data fusion practices. This database can aid the BIS in making determinations for its Entity List.) \n",
      "\n",
      "\n",
      "On the technical side, governmental organizations, such as the National Science Foundation and the Defense Advanced Research Projects Agency, can fund research to thwart data fusion processes. One emerging line of research seeks to develop privacy-preserving computer vision systems that obscure individuals’ faces. Other lines of research seek to build counter-surveillance technologies, such as specialized clothing, attacking systems through adversarial examples (intentionally destructive inputs to cause model malfunctioning), or data poisoning (modifying training images). Interfering at the stage prior to fusion would not only protect the privacy of those targeted by surveillance systems, but also prevent data fusion processes from accurately functioning.  \n",
      "\n",
      "\n",
      "It is also important to look at the role of U.S. companies. Chinese surveillance systems are heavily reliant on U.S. firms to provide the gear that powers these digital operations. U.S. suppliers such as Intel, NVIDIA, Cisco, Seagate and Western Digital have all been linked to various aspects of Chinese surveillance systems, but the U.S. government has so far been unable to write rules effectively prohibiting the sales of such equipment. In 2020, the State Department released exhaustive guidance for companies’ export considerations, but the document is nonbinding. The difficulty of imposing binding rules against U.S. companies—along with the continued synergy between the commercial sphere and surveillance states’ technical needs—makes it difficult to prevent the export of such technology. As the United States has staked out a clear policy against surveillance-enabled repression, especially in Xinjiang, it is no longer acceptable for companies to deny knowledge of involvement in supporting China’s surveillance state. One of the most viable options to decrease companies’ more problematic exports remains increased public pressure. \n",
      "\n",
      "\n",
      "Increased attention to Chinese data fusion practices—and its supporting companies—would allow U.S. policy to target China’s surveillance state at a core level, rather than only facial and voice recognition elements that feed into fusion architectures. Taking such steps would better protect Uyghurs, “focus personnel,” and the Chinese people writ large from falling under increasingly abusive, unchecked surveillance.  \n",
      "\n",
      "\n",
      "Dahlia Peterson is a research analyst at Georgetown’s Center for Security and Emerging Technology.  \n",
      "\n",
      "\n",
      "Intel provides financial support to the Brookings Institution, a nonprofit organization devoted to rigorous, independent, in-depth public policy research. \n",
      "\n",
      "https://www.brookings.edu/research/superhuman-science-how-artificial-intelligence-may-impact-innovation/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/research/superhuman-science-how-artificial-intelligence-may-impact-innovation/\n",
      "TITLE: Superhuman science: How artificial intelligence may impact innovation\n",
      "Superhuman science: How artificial intelligence may impact innovation\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/blog/future-development/2022/04/28/how-can-better-climate-data-empower-people-to-act/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/blog/future-development/2022/04/28/how-can-better-climate-data-empower-people-to-act/\n",
      "TITLE: Getting people to understand climate data better\n",
      "Getting people to understand climate data better\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/research/what-all-policy-analysts-need-to-know-about-data-science/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/research/what-all-policy-analysts-need-to-know-about-data-science/\n",
      "TITLE: What all policy analysts need to know about data science\n",
      "What all policy analysts need to know about data science\n",
      "BODY:\n",
      "\n",
      "https://www.brookings.edu/techstream/how-satellites-and-ai-can-fix-development-data-problems/\n",
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/techstream/how-satellites-and-ai-can-fix-development-data-problems/\n",
      "TITLE: How satellites and AI can fix development data problems\n",
      "BODY:\n",
      "\n",
      "A satellite image from 2015 depicts a salt-affected water catchment area in drought-struck farmland in Australia, November 26, 2015. (REUTERS/David Gray)\n",
      "\n",
      "\n",
      "Data drives development policy. To determine aid packages and projects, policymakers need good data on everything from population to urban sprawl to economic livelihoods. Yet policymakers creating development policies, whether in response to disasters or with an eye toward the long-term, face a core problem: measuring sustainable development variables.\n",
      "\n",
      "\n",
      "Against the backdrop of an ever more urgent climate crisis, improving efforts to get good data has never been more important. The most recent report from the U.N. Intergovernmental Panel on Climate Change lays bare the scale of this challenge, yet even as scientists warn that time is running out to slow the warming of the planet, persistent disagreement remains about how much wealthy countries should be spending on climate assistance for lower-resourced ones.\n",
      "\n",
      "\n",
      "These questions of how much to spend on aid and where to spend it raise a key issue with development broadly. In the past, poor forecasting and inefficient aid distribution have hindered the effectiveness of development programs, including on climate. If policymakers and researchers cannot get accurate information about a problem, it’s more difficult to forge effective solutions. But new technology for development analysis, driven by a combination of satellite imagery and machine learning, may hold the keys to progress.\n",
      "\n",
      "\n",
      "As it stands, data can be inaccurate, costly to acquire, or hard to get all together. This is especially the case in low-resourced countries. In Africa alone, 34% of countries have gone more than 15 years since their last agricultural survey. Even then, the data that is collected is often incomplete or inaccurate. And yet the need for this high-fidelity data is pressing. When a locust plague struck East Africa in 2020, one of the key challenges in responding to that crisis was simply determining the location of locust swarms. In the absence of effective tools to monitor and respond to the locusts marauding the region, 19 million farmers across East Africa lost their crops, causing widespread food shortages in the region.\n",
      "\n",
      "\n",
      "Our new paper introduces an elegant solution to the problem of measuring sustainable development: machine learning applied to satellite imagery. The explosion in commercial satellites and the public availability of satellite imagery opens up new opportunities to analyze sustainable development-related variables at low cost, high accuracy, and great scale. When we compared satellite imagery from 200 random sample sites across multiple continents, we observed a substantial increase in the number and quality of images captured over time. Sites once imaged a couple times a year are now captured multiple times a week, and these images detail localized activity like infrastructure growth.\n",
      "\n",
      "\n",
      "Satellite imagery is one piece of the puzzle. Another is the growing use, and usefulness, of artificial intelligence (AI)-powered machine learning (ML) models to extract common patterns of information from available data. In the development context, researchers have built models increasingly capable of assessing sustainable development metrics from satellite images. One satellite image of arable land might tell the story of a village’s economic health—its crop yields, its agricultural diversity, and its infrastructure development.\n",
      "\n",
      "\n",
      "In our assessment, ML models leveraging satellite imagery inputs can amplify—and possibly outperform—traditional measurement tools like ground based surveys and censuses, offering a promising path forward. These technologies are unlikely to replace ground-based surveys altogether. But their augmentation of these methods can help address the data problems in sustainable development policy. For example, researchers can use satellite-based estimates of buildings, nighttime lights, and other markers to equip policymakers with more accurate estimates of local population size instead of infrequent, traditional census methods, particularly in lower-resourced countries.\n",
      "\n",
      "\n",
      "Techniques to “train” ML models—to teach them which patterns to extract from available data—are improving. Researchers can now build models even when training data is less readily available or low in quality, as is often the case with sustainable development data. Synthetically created training data—data that is artificially created instead of generated by real-world events, we found, is another route to addressing data shortcomings, especially useful in the development context. In the agricultural setting, for example, crop model simulations trained on synthetic data to predict crop yields have performed as well as or better than approaches that calibrate directly to limited field data.\n",
      "\n",
      "\n",
      "In addition, transfer learning and semi-supervised learning can enable researchers to circumvent issues surrounding data quantity and quality. In the former approach, models leverage large quantities of readily available data to learn a task similar to the task of interest and then “transfer” extracted patterns to sustainable development metrics. In the latter, models extract patterns from unlabelled satellite data (sometimes combined with small amounts of labeled data) without substantial human input. While noisy training data is a persistent problem that distorts model performance and evaluation, we found that models trained on high volumes of noisy data but tested on un-degraded data were stable performers, indicating that ML models can still be robust.\n",
      "\n",
      "\n",
      "ML-driven, satellite-derived assessments of sustainable development variables hold evident promise, but they still face significant challenges. Trust issues loom large. Many ML models are not transparent, and it is often unclear how models arrive at a given outcome—such as predicting that a field’s crop yield will be low based on a satellite image. Policymakers are understandably weary of algorithms that cannot be fully explained. In addition to trust and explainability challenges, ML-driven estimates of sustainable development variables run into issues of scope. While some variables like crop growth can be inferred through ML-based approaches, others like educational attainment cannot be derived from satellite imagery.\n",
      "\n",
      "\n",
      "Looking forward, researchers and practitioners alike can advance the use of satellite-driven assessments by focusing on explainability in models, cultivating public-private partnerships to operationalize model usage, and better understanding how satellite imagery and AI tools can address development data gaps. Policymakers, for their part, can better understand both the potential and the limitations of this emerging technology. After all, research does not occur in a vacuum, and, especially where development is concerned, policymakers have an important role in setting state policies, prioritizing investments, and raising attention to issues.\n",
      "\n",
      "\n",
      "By focusing on strategies, such as, the use of synthetic data, transfer learning, and testing models on a small amount of high-quality data to counter noisy data, researchers and policymakers can leverage the power of machine learning and satellite imagery to change sustainable development for the better.\n",
      "\n",
      "\n",
      "Marshall Burke is an associate professor in the Department of Earth System Science and deputy director of the Center on Food Security and the Environment at Stanford University. Anne Driscoll is a research data analyst at the Center on Food Security and the Environment at Stanford University.David Lobell is a professor in the Department of Earth System Science and the Gloria and Richard Kushel Director of the Center on Food Security and the Environment at Stanford University.Stefano Ermon is an assistant professor in the Department of Computer Science at Stanford University. This post is adapted from the Stanford HAI’s policy brief,“Using Satellite Imagery to Understand and Promote Sustainable Development.”\n",
      "\n",
      "https://www.brookings.edu/blog/techtank/2021/07/22/how-the-national-science-foundation-is-taking-on-fairness-in-ai/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: data%20science\n",
      "URL: https://www.brookings.edu/blog/techtank/2021/07/22/how-the-national-science-foundation-is-taking-on-fairness-in-ai/\n",
      "TITLE: How the National Science Foundation is taking on fairness in AI\n",
      "How the National Science Foundation is taking on fairness in AI\n",
      "BODY:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "class Crawler:\n",
    "    def __init__(self, website):\n",
    "        self.site = website\n",
    "        self.found = {}\n",
    "\n",
    "    def getPage(url):\n",
    "        try:\n",
    "            html = urlopen(url)\n",
    "        except Exception as e:\n",
    "            return None\n",
    "        return BeautifulSoup(html, 'html.parser')\n",
    "\n",
    "    def safeGet(bs, selector):\n",
    "        \"\"\"\n",
    "        Utilty function used to get a content string from a Beautiful Soup\n",
    "        object and a selector. Returns an empty string if no object\n",
    "        is found for the given selector\n",
    "        \"\"\"\n",
    "        selectedElems = bs.select(selector)\n",
    "        if selectedElems is not None and len(selectedElems) > 0:\n",
    "            return '\\n'.join([elem.get_text() for elem in selectedElems])\n",
    "        return ''\n",
    "\n",
    "    def getContent(self, topic, url):\n",
    "        \"\"\"\n",
    "        Extract content from a given page URL\n",
    "        \"\"\"\n",
    "        bs = Crawler.getPage(url)\n",
    "        if bs is not None:\n",
    "            title = Crawler.safeGet(bs, self.site.titleTag)\n",
    "            body = Crawler.safeGet(bs, self.site.bodyTag)\n",
    "            return Content(topic, url, title, body)\n",
    "        return Content(topic, url, '', '')\n",
    "\n",
    "    def search(self, topic):\n",
    "        \"\"\"\n",
    "        Searches a given website for a given topic and records all pages found\n",
    "        \"\"\"\n",
    "        bs = Crawler.getPage(self.site.searchUrl + topic)\n",
    "        searchResults = bs.select(self.site.resultListing)\n",
    "        for result in searchResults:\n",
    "            url = result.select(self.site.resultUrl)[0].attrs['href']\n",
    "            # Check to see whether it's a relative or an absolute URL\n",
    "            url = url if self.site.absoluteUrl else self.site.url + url\n",
    "            if url not in self.found:\n",
    "                self.found[url] = self.getContent(topic, url)\n",
    "            self.found[url].print()\n",
    "\n",
    "\n",
    "\n",
    "siteData = [\n",
    "    ['Reuters', 'http://reuters.com', 'https://www.reuters.com/search/news?blob=', 'div.search-result-indiv',\n",
    "        'h3.search-result-title a', False, 'h1', 'div.ArticleBodyWrapper'],\n",
    "    ['Brookings', 'http://www.brookings.edu', 'https://www.brookings.edu/search/?s=',\n",
    "        'div.article-info', 'h4.title a', True, 'h1', 'div.core-block']\n",
    "]\n",
    "sites = []\n",
    "for name, url, search, rListing, rUrl, absUrl, tt, bt in siteData:\n",
    "    sites.append(Website(name, url, search, rListing, rUrl, absUrl, tt, bt))\n",
    "\n",
    "crawlers = [Crawler(site) for site in sites]\n",
    "topics = ['python', 'data%20science']\n",
    "\n",
    "for topic in topics:\n",
    "    for crawler in crawlers:\n",
    "        crawler.search(topic)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawling Sites through Links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Website:\n",
    "\n",
    "    def __init__(self, name, url, targetPattern, absoluteUrl, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.targetPattern = targetPattern\n",
    "        self.absoluteUrl = absoluteUrl\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag\n",
    "\n",
    "\n",
    "class Content:\n",
    "\n",
    "    def __init__(self, url, title, body):\n",
    "        self.url = url\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "\n",
    "    def print(self):\n",
    "        print(f'URL: {self.url}')\n",
    "        print(f'TITLE: {self.title}')\n",
    "        print(f'BODY:\\n{self.body}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.brookings.edu/blog/fixgov/2023/04/05/what-we-learned-from-the-chicago-mayoral-results/\n",
      "TITLE: What we learned from the Chicago mayoral results\n",
      "What we learned from the Chicago mayoral results\n",
      "BODY:\n",
      "\n",
      "As Chicagoans went to the polls on Tuesday, early signs pointed to a narrow victory for Paul Vallas, the former head of the city’s public school system and noted educational reformer, over Brandon Johnson, a former social studies teacher turned organizer for the Chicago Teachers Union. Vallas led in the pre-election polls by an average of 3 percentage points, a margin that widened to 6 points when undecided voters were asked whether they leaned toward a candidate. A higher share of Vallas’s supporters said that they were certain to cast their ballots, and more of Johnson’s said that they might change their minds about their choice. Vallas enjoyed a strong lead among voters 60 and older, who are the most likely to vote of all age cohorts, while Johnson was doing best among those 30 and younger, who are typically the least likely to participate.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "William A. Galston\n",
      "\n",
      "\t\t\t\t\tEzra K. Zilkha Chair and Senior Fellow - Governance Studies \n",
      "\n",
      " Twitter\n",
      "BillGalston\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The ideological battle lines were clearly drawn. Vallas ran as a moderate, Johnson as an unabashed progressive. Johnson wanted to raise taxes on businesses, visitors of Chicago, and wealthy individuals to fund new social programs, while Vallas advocated fiscal restraint. The centerpiece of Vallas’s campaign was a pledge to crack down on violent crime. By contrast, Johnson expressed early sympathy (some would say support) for the “defund the police” movement that erupted after the murder of George Floyd before moderating his position. Not surprisingly, Vallas enjoyed the fervent backing of Chicago’s police union.\n",
      "When incumbent mayor Rahm Emanuel ran for reelection in 2015, he also faced a progressive candidate, Jesus “Chuy” Garcia, leading some observers to draw parallels between then and now. But these elections differ in two key respects. First: because there was no Black candidate in the 2015 race, the Black vote was up for grabs, and Emanuel won it by a margin of 58 to 42. He also won the white vote by 2 to 1 while Garcia prevailed among Hispanics by a similar margin. This year, Brandon Johnson, a Black candidate with strong community roots, is receiving more than 70 percent support in this key constituency, while Vallas is outpolling Johnson among Hispanics. (Although Vallas is of Greek extraction, his last name — which means “fences” in Spanish — has led some Hispanics to believe that he is one of them.)\n",
      "The second difference between 2015 and today: eight years ago, no single issue dominated the race, and the electorate was almost equally split among the economy, city finances, education, and crime as its chief concern. This year, violent crime dwarfed all other considerations, and the outcome of the race would be seen as a referendum on the candidates’ competing plans for addressing it.\n",
      "It is the centrality of voters’ concerns about crime that gave this local contest national implications. A Vallas victory would have reinforced the tough on crime message that the election of Eric Adams in New York City had sent. If Johnson prevailed, his supporters would be able to argue that only a strong progressive message could bring young people and disaffected minority voters to the polls in large enough numbers to overcome those who wanted to intensify the use of tough, racially tinged methods against street-level criminals — and more broadly, to give progressives a chance to prevail over what they regard as the defenders of the status quo.\n",
      "\n",
      "\n",
      "Related Content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "FixGov\n",
      "Do politicians get rewarded for spending money?\n",
      "\n",
      "Vanessa Williamson\n",
      "Wednesday, March 22, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "FixGov\n",
      "Can government protect women from domestic violence? Not if states do not follow up.\n",
      "\n",
      "Kaitlin N. Sidorsky and Wendy J. Schiller\n",
      "Tuesday, March 21, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "FixGov\n",
      "The gerrymander myth\n",
      "\n",
      "William A. Galston\n",
      "Friday, March 17, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "To the surprise of many veteran observers, this is exactly what happened. With 90,000 absentee ballots still to be counted, Johnson led by a margin of 15,000 votes out of more than 550,000 cast. And because Johnson was receiving nearly 70% of the absentee vote, Vallas already has conceded the race.\n",
      "Although exit polls are not yet available, preliminary results from Chicago’s 50 wards paint a clear picture. Johnson racked up nearly 80% of the Black vote on Chicago’s South Side and ran strongly among white liberals on the Lakefront. Vallas prevailed in the mostly white working-class wards in the Northwest and Southwest sections of the city, but his margins were not large enough to overcome Johnson’s margins elsewhere. With no Hispanic candidate on the ballot, turnout in the Hispanic-majority wards was reportedly anemic. Overall turnout, though, was higher than usual, which the Johnson campaign attributes in part to a surge among younger voters. We do not yet have enough information to confirm this assessment.\n",
      "With a strong boost from the Chicago Teachers Union, which has become a dynamo of progressive policies and organizing in the Windy City, Johnson has an opportunity to advance his progressive agenda and become a trendsetter for other cities. Of course, governance always is more complicated than winning elections. The former will require policy approvals from the City Council and tax increases staunchly opposed by the business community to fund his proposed boost in social spending. How he deals with crime increases, underperforming schools, shaky city finances, and a divided Democrat party will determine how successful he is. Against the backdrop of the presidential campaign and national battles for control of Congress, next year won’t be dull.\n",
      "\n",
      "URL: https://www.brookings.edu/blog/order-from-chaos/2023/03/27/japans-new-security-policies-a-long-road-to-full-implementation/\n",
      "TITLE: Japan’s new security policies: A long road to full implementation\n",
      "Japan’s new security policies: A long road to full implementation\n",
      "BODY:\n",
      "\n",
      "On December 16, 2022, Japan’s government released a new national security strategy, national defense strategy, and defense buildup program. In mid-January, Japanese Prime Minister Fumio Kishida and key Cabinet officials visited Washington to jointly highlight these documents and discuss the next steps for the U.S.-Japan alliance with the Biden administration.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Adam P. Liff\n",
      "\n",
      "\t\t\t\t\tNonresident Senior Fellow - Foreign Policy, Center for East Asia Policy Studies \n",
      "\n",
      " Twitter\n",
      "AdamPLiff\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Jeffrey W. Hornung\n",
      "\n",
      "\t\t\t\t\tSenior Political Scientist - RAND Corporation \n",
      "\n",
      "\t\t\t\t\tAdjunct Professor - Georgetown University \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "At the time, much commentary rightly noted the historic ambition contained within Japan’s new strategies aimed at strengthening deterrence in response to a worsening regional security environment.\n",
      "In all the excitement over these historic announcements, however, much of the discourse has inappropriately treated Japan’s strategies as a done deal, as though their full realization is inevitable. The reality is that an extraordinary alignment of political, economic, fiscal, and other stars will be necessary for Japan’s government to fully implement the Kishida administration’s stated ambitions over the next 5-10 years.\n",
      "To be sure, Japan may ultimately get there. But the road ahead may be bumpy.\n",
      "What Japan’s “three documents” are — and what they are not\n",
      "Japan’s new National Security Strategy (NSS) — the first since 2013 and only the second ever — is Tokyo’s “supreme national security policy document.” It “provides strategic guidance for Japan’s national security policy areas, including diplomacy, defense, economic security, technology, cyber, maritime, space, intelligence, official development assistance (ODA), and energy.” Japan’s new National Defense Strategy (NDS) — the first so-named and the successor to the erstwhile National Defense Program Guidelines (six versions since 1976; last revised in 2018) — is basically a ten-year guideline that is designed to clarify Japan’s defense objectives and the ways and means by which the government intends to achieve them. Its companion document, the Defense Buildup Program, provides “program guidelines” for building and maintaining the critical defense capabilities needed to support the NDS.\n",
      "These documents were approved by Japan’s National Security Council and Cabinet — the members of which are appointed by Japan’s prime minister. As such, they are a clear signal of the current Kishida government’s political and policy intent.\n",
      "However, these documents are not legally-binding commitments, plans, or legislation that have received the imprimatur of Japan’s National Diet, much less been fully resourced.\n",
      "“The most severe and complex security environment since the end of World War II”\n",
      "While Japan’s defense orientation remains fundamentally unchanged in key aspects, these new strategies, together with the outcomes from the January 2023 U.S.-Japan summit and cabinet-level security meetings in Washington, reflect a re-evaluation by Japan’s government of what it can and must do to more effectively enhance deterrence in the face of a rapidly worsening security environment, a changing balance of power in East Asia, and the arrival of a “new era of strategic competition.” They are the latest manifestation of a judgment shared across successive administrations in Japan that Tokyo must adopt a more proactive role in deterrence and regional stability amidst what the documents call Japan’s “most severe and complex security environment since the end of World War II.”\n",
      "The new NSS identifies China and North Korea’s activities as posing, respectively, “an unprecedented and the greatest strategic challenge” and a “grave and imminent threat.” Meanwhile, it notes that “Russia’s aggression against Ukraine has easily breached the very foundation of the rules that shape the international order.” Critically, the NDS judges that Russia’s aggression was possible because “Ukraine’s defense capability … was insufficient” for effective deterrence. All told, the sense of urgency that Japan’s posture must adapt quickly to a changing security environment vastly exceeds that of the 2013 NSS.\n",
      "In response to threats both general and specific, the new strategy calls for “fundamentally reinforcing Japan’s own capabilities.” For example, Japan’s unprecedented call for “counterstrike capabilities” results from a frank recognition that China and North Korea’s ballistic and cruise missile arsenals could overwhelm Japan’s air and missile defense systems. Meanwhile, the call for surging defense spending reflects not only a push for new, expensive capabilities (e.g., counterstrike; unmanned systems) but also an awareness that after years of under-investment in key initiatives (e.g., munitions and parts stockpiles, passive base defenses, cyber and space capabilities), vastly greater resources are needed to strengthen deterrence and resilience.\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "The Crisis of Liberal Internationalism\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tEdited by Yoichi Funabashi and G. John Ikenberry \n",
      "2020\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Dilemmas of a Trading Nation\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Mireya Solís \n",
      "2017\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "P\n",
      "\n",
      "\n",
      "\n",
      "Politicians and Policymaking in Japan\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Gerald L. Curtis \n",
      "2002\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "From here to there: a long (and potentially bumpy) road ahead\n",
      "It is one thing to map out an ambitious vision and list of program priorities, which Japan has already accomplished with the release of last December’s “three documents.” But it is something different to effectively, efficiently, and fully implement it.\n",
      "Show me the money: resourcing\n",
      "As the new documents clearly acknowledge, major defense budget increases are the sine qua non for realizing much of Japan’s new national security vision. Without any doubt, the planned change in the official annual defense budget from 5.4 trillion yen in 2022 ($40 billion in today’s rates) and 8.9 trillion yen in 2027 ($67 billion in today’s rates) — a roughly two-thirds increase — is a historically significant pledge.\n",
      "But the ambitious new spending targets were announced before a concrete plan for how to fully finance them. Generating and sustaining this massive amount of new funding over the next several years — and beyond — may not be easy.\n",
      "Parliamentary debate has so far focused on some combination of sources — including tax increases, debt spending, expenditure cuts, and moving resources around from other budgets. Yet even within Kishida’s own Liberal Democratic Party (LDP), internal fissures have already emerged publicly. Furthermore, a December 2022 poll also showed that tax hikes were opposed by a clear majority. With an approval rating hovering between 30% to 40%, it’s not clear how much political capital Kishida (or his successors) will have to push through potentially unpopular funding measures. Tax increases in Japan have long been politically precarious.\n",
      "There are certainly grounds for optimism in Tokyo. The Lower House just approved the government’s FY2023 budget request, which includes a massive defense budget increase of 26%. But there is no guarantee that achieving these spending levels will be fiscally or politically sustainable in the long run, especially with other recently announced initiatives also requiring massive outlays. Indeed, it is no secret that Japan faces severe demographic, economic, and fiscal headwinds.\n",
      "None of this is to suggest that Japan reaching and sustaining an 8.9 trillion yen defense budget by 2027 is impossible. Rather, the point is only that it would be a mistake to assume it’s inevitable simply because it appears in the December 2022 Defense Buildup Program.\n",
      "Don’t forget the politics\n",
      "Beyond appropriations, legal and other reforms will also be necessary to fully implement pledges contained within the “three documents.” Given important constitutional, political, and other constraints on Japan’s defense posture, the devil may be in the details.\n",
      "Forward momentum is already clear. Some new legislation has already been proposed, including as it concerns historic new reciprocal access agreements with Australia and the United Kingdom. In the coming years, additional legislation and reforms will be needed. For example, to implement the pledge to develop “active cyber defense” or capabilities to obtain human intelligence, the Diet may need to pass legislation related to privacy to assuage concerns about government overreach.\n",
      "Domestic politics will impact whether and how quickly legislation will clear the Diet. Additionally, the precise content will also be shaped by political processes. At a minimum, Kishida’s LDP will need support from Komeito, its more “dovish” junior coalition partner with a clear track record of diluting major national security-related initiatives championed by LDP conservatives.\n",
      "A potpourri of other issues\n",
      "Lastly, even if fully resourced and legislated, Japan’s leaders will need to effectively implement Tokyo’s new national security strategy. First and foremost, they will need to negotiate a lot of political agreements — interagency; intra-coalition; and with the United States and other major international partners.\n",
      "For Japan, an assortment of important tasks remains outstanding if it is to achieve its goals, including: securing qualitatively and quantitatively sufficient manpower; reinvigorating the defense industrial base; establishing new and diverse partnerships with the private sector; developing new doctrines, concepts, and forms of command and control; accelerating technological innovation; and cooperating with other countries in technological innovation and foreign military sales.\n",
      "Adapting the U.S.-Japan alliance to Tokyo’s new ambitions and capabilities could be critical, especially if both countries judge that a rethink of the traditional “shield and spear” division of labor of roles and missions is necessary.\n",
      "And none of these objectives will be pursued in a vacuum. Domestic and international political vicissitudes and unexpected events may force adjustments. The massive earthquake, tsunami, and nuclear disaster that struck Japan in March 2011 and Russia’s ongoing war against Ukraine are only two examples of how unforeseen “shocks” can drastically reshape political and strategic priorities.\n",
      "Final word\n",
      "Pointing out the difficult road ahead is not meant to minimize the significance of the ambitions contained in Japan’s new national security and defense strategies, or to suggest achievement is unlikely. Rather, the intent is simply to highlight that despite the bold steps forward already taken by the Kishida Cabinet, there remain many unknowns about what will come next, and how bumpy the path forward is likely to be. One thing is certain: a lot of hard work — in both Tokyo and Washington — lies ahead.\n",
      "\n",
      "\n",
      "Related Content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Order from Chaos\n",
      "South Korea-Japan rapprochement creates new opportunities in the Indo-Pacific\n",
      "\n",
      "Andrew Yeo\n",
      "Friday, March 17, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Order from Chaos\n",
      "As Kishida meets Biden, what is the state of the US-Japan alliance?\n",
      "\n",
      "Mireya Solís\n",
      "Friday, January 20, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Order from Chaos\n",
      "Will Biden’s Asia trip help the US meet its strategic objectives?\n",
      "\n",
      "Mireya Solís\n",
      "Tuesday, May 24, 2022\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.brookings.edu/research/addressing-the-looming-sovereign-debt-crisis-in-the-developing-world-it-is-time-to-consider-a-brady-plan/\n",
      "TITLE: Addressing the looming sovereign debt crisis in the developing world: It is time to consider a ‘Brady’ plan\n",
      "Addressing the looming sovereign debt crisis in the developing world: It is time to consider a ‘Brady’ plan\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Brahima Sangafowa Coulibaly\n",
      "\n",
      "\t\t\t\t\tVice President and Director - Global Economy and Development \n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development \n",
      "\n",
      " Twitter\n",
      "BSangafowaCoul\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "W\n",
      "\n",
      "\n",
      "\n",
      "Wafa Abedin\n",
      "\n",
      "\t\t\t\t\tResearch and Administrative Assistant to the Vice President and Director - Global Economy and Development \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Among the challenges facing developing countries, none is arguably more crucial than the significantly deteriorated fiscal situation that threatens to erase several years of progress on development agendas. According to some estimates, almost 60 percent of the poorest countries are either in or at high risk of debt distress, nearly doubling since 2015 (Figure 1; World Bank 2022a). Total debt service payments on public and publicly guaranteed (PPG) external debt of the poorest countries rose to over $50 billions in 2021, with repayments now representing 11.3 percent of government revenue in the poorest countries, up from 5.1 percent in 2010 (Figure 2). In most developing countries, the cost of servicing external debt now exceeds expenditures on health, education, and social protection combined (UNICEF, 2021). The current global environment characterized by higher global interest rates and exchange rate depreciations against major currencies is adding to the fiscal challenge by raising the cost of external financing and debt service. Debt service payments for the poorest countries rose 36 percent to over $70 billion last year and are projected to remain elevated through at least 2027 (Figure 3). It is increasingly evident that the Common Framework for Debt Treatments, adopted by the G-20 to help developing countries restructure their debts and address solvency issues and protracted liquidity problems, is facing serious operational challenges. Echoing widely shared concerns about the limitations of the CF, the International Monetary Fund’s Managing Director called for changes to it while the World Bank President urged for the acceleration of its implementation. Likely due to the challenges with the CF, out of the 37 countries at high risk of or in debt distress, only four countries have requested assistance under the framework so far. We need a pre-emptive and wholesale approach to restore fiscal sustainability across the developing world and avert a systemic debt crisis.\n",
      "Figure 1: Low-income countries Debt Sustainability Analysis\n",
      "\n",
      "Source: IMF Annual Report (2022a) and author’s calculations using the IMF Debt Sustainability Analysis for Low-income Countries (2022).\n",
      "Figure 2: External PPG debt servicing cost (2000-2022)\n",
      "\n",
      "Source: Author’s calculations using World Bank International Debt Statistics (2021) and IMF World Economic Outlook database (2022b).\n",
      "Figure 3: Total external PPG debt service coming due\n",
      "\n",
      "Source: Author’s calculations using World Bank International Debt Statistics (2021).\n",
      "In this policy brief, we propose that the G-20 adopt a Brady-like scheme to accelerate the restructuring of external private sector debt to restore debt sustainability in the affected countries with the following design features.\n",
      "First, we recommend the creation of a new special purpose fund—the recovery and sustainability fund (RSF)—to be capitalized by International Financial Institutions (IFIs) and bilateral donors. Second, the funds will be used to secure collateral against new tradable bonds—Recovery and Sustainability bonds (RSBs)— issued by participating indebted countries. The guarantees attached to the RSBs will provide credit enhancement and allow countries to issue the new bonds on terms that are more favorable than those of the current stock of private external debt. As shown in Figure 4, despite holding about 27 percent of the debt, the private sector accounts for 37 percent of the debt service due to the higher cost; the median coupon rate is 7.5 percent. The new bonds would also have longer maturities, ideally 30 years. The beneficiary countries could use the proceeds from the RSBs to retire the outstanding balance on the current private external debt.\n",
      "Figure 4: Share of debt versus share of debt service cost in 2021\n",
      "\n",
      "Source: Author’s calculations using World Bank International Debt Statistics (2021).\n",
      "The lower coupon rates on the RSB along with the longer maturity will lead to sizeable reductions in the debt burden to more sustainable levels. We conduct a simple illustrative simulation, which indicates that this scheme could reduce the debt service payments as a share of government revenues by up to 4 percentage points per year for the average developing country, cutting near half the debt service burden in many cases. We further estimate a total reduction in external debt repayment of up to $100 billion over the next five years. In the remainder of the brief, we document the deterioration in sovereign debt situation, review the challenges with the CF, and outline the case for a Brady-like plan.\n",
      "Download the full policy brief.\n",
      "\n",
      "\n",
      "Related Content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Climate Change\n",
      "Debt-for-adaptation swaps\n",
      "\n",
      "Chetan Hebbale and Johannes Urpelainen\n",
      "Tuesday, March 21, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Global Development\n",
      "Social and development impact bonds by the numbers\n",
      "\n",
      "Emily Gustafsson-Wright and Elyse Painter\n",
      "Thursday, March 9, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Africa in Focus\n",
      "Managing the compounding debt and climate crises\n",
      "\n",
      "Gracelin Baskaran\n",
      "Wednesday, March 1, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "URL: https://www.brookings.edu/research/breaking-the-urban-doom-loop-the-future-of-downtowns-is-shared-prosperity/\n",
      "TITLE: Breaking the ‘urban doom loop’: The future of downtowns is shared prosperity\n",
      "Breaking the ‘urban doom loop’: The future of downtowns is shared prosperity\n",
      "BODY:\n",
      "\n",
      "In 2019, the skylines of many American downtowns were sparkling with new construction. Hudson Yards in New York, the Frost Tower in San Antonio, San Francisco’s Salesforce Tower, the futuristic Amazon Spheres and surrounding towers in Seattle, and thousands of other buildings were all part of the more than 100 million square feet of new office space built in the United States just before the onset of the COVID-19 pandemic.1 Demand for cities seemed stronger than ever.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Tracy Hadden Loh\n",
      "\n",
      "\t\t\t\t\tFellow - Brookings Metro, Anne T. and Robert M. Bass Center for Transformative Placemaking \n",
      "\n",
      " Twitter\n",
      "lohplaces\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Hanna Love\n",
      "\n",
      "\t\t\t\t\tSenior Research Associate - Brookings Metro \n",
      "\n",
      " Twitter\n",
      "hannamlove\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Flash forward four years. By now, COVID-19 has claimed the lives of over 1 million Americans and raised new questions about another potential casualty: Is the pandemic, particularly its impact on the nature of work, taking down the American city? Nowhere is this fear more pronounced than in the nation’s “superstar” cities and their shiny downtowns.\n",
      "This specter of an office real estate apocalypse, “urban doom loop,” transit death spiral, or “ghost towns” is filling some urban observers with existential dread. So too are reports of rising crime and unsheltered homelessness in downtowns. However, this fear is not new, nor is it destiny.\n",
      "But this dread should prompt public and private sector leaders to reassess. In fact, they have an opportunity to seize this moment to chart a new future for American cities—one that reimagines downtowns as prosperous and inclusive places that advance shared prosperity across all neighborhoods.\n",
      "A group of public and private sector leaders in some of the nation’s largest cities—New York, Chicago, Philadelphia, and Seattle—have come together at Brookings Metro to do just this. These leaders were at the forefront of inclusive growth before the pandemic, and are now working with Brookings Metro and each other to identify policy, practice, and governance solutions that can be the seeds for the next generation of shared prosperity in downtowns, cities, and regions.\n",
      "Ghost-town downtowns: An old narrative with a new twist\n",
      "Only a generation ago, white flight and suburban sprawl left many cities as partial ghost towns. Anti-city rhetoric around “blight” and “inner-city crime” was used to justify the most negative aspects of urban renewal and, later, the war on drugs, which had significant consequences for the economic and social fabric of American cities. Suburbanization spread the fruits of growth out over more land area and jurisdictions, while public and private sector abandonment of center cities both concentrated poverty and increased the overall cost of services—weakening city and regional economies and exacerbating racial and economic segregation.\n",
      "The villainization of American city centers began to reverse in the early 2000s, as highly educated young workers and some employers returned, lured by proximity to jobs, amenities, walkability, and a sense of place that downtowns offered. For instance, in 35 of the top 45 U.S. downtowns by job count, the share of the regional population living in the commercial core (Table 1, Columns 5 and 6) grew between 2000 and 2020. And in a select few downtowns, regional job market share grew between 2011 and 2019 (Table 1, Columns 2 and 3), with Boston leading the pack.\n",
      "\n",
      "\n",
      "\n",
      "...\n",
      "\n",
      "\n",
      "NotesCC = commercial core, AC = Activity Center\n",
      "Data sources\r\n",
      "    2019 U.S. Census Bureau Longitudinal Employer-Household Dynamics (LEHD) Origin-Destination Employment Statistics (LODES), available online.\n",
      "\r\n",
      "    2020 U.S. Census\n",
      "\n",
      "\n",
      "\n",
      "The urban “downtown resurgence” of the 21st century provided a new boost for urban cores that had been in decline, but it also created economic cleavages within cities and regions. In cities across geographic and economic spectrums—from Chattanooga, Tenn. to Detroit—subsidized development downtown ushered in newcomers while other (often predominantly Black and Latino or Hispanic) neighborhoods continued to lack basic amenities like grocery stores. At the same time, suburban poverty rates climbed. While not an entirely new phenomenon, on the eve of the COVID-19 pandemic, there was a public consensus in cities that the benefits of downtown growth weren’t extending to many other parts of the city and region.\n",
      "Chicago: “The most obvious challenge that Chicago has struggled with—and has such a reputation for—is the segregation. Literally being able to draw the line from north and south, you see where it changes…You see it at the street level. All of that is just very visible. And it’s happened for so long.”   \n",
      "Over the last three decades, then, city centers went from being perceived as dangerous and declining (a reason for white flight and suburbanization) to increasingly unaffordable and exclusive (a driver of urban inequality) to being “dead” once again (due to the retreat and slow return of office workers, increasing business closures and vacancies, and fear of crime and unsheltered homelessness).\n",
      "This reversal is forcing local and regional leaders to ask: What does the future of downtowns look like? How can downtowns adapt to become less reliant on commuters and office space? What purpose should our downtowns fulfill? And most importantly, how can downtowns be more inclusive, safe, and productive for all residents as they evolve and recover from the pandemic? The timing of these questions—combined with a fire hose of federal relief aimed directly at cities—presents local leaders with a material opportunity to “think big.”\n",
      "New York: “I think we’ve moved the conversation from ‘I’m scared of getting Covid and giving it to my loved ones’ to ‘I’m generally not feeling okay going back to the office because I’m seeing all of this crime, or what we perceive as crime, on the street and I don’t want to go back.’”\n",
      "\n",
      "Methods\n",
      "This brief is part of a mixed-methods research project that seeks to understand the future of downtowns through interviews, spatial data analysis, and direct engagement with local leaders. To understand perceptions of downtown health and recovery, in fall 2022, we spoke with nearly 100 stakeholders in New York,2 Chicago, Seattle, and Philadelphia, including business leaders, major employers, public sector officials, residents, place governance associations, and other nonprofit and civic leaders.3\n",
      " \n",
      "We then juxtaposed our qualitative findings with quantitative spatial analysis of population, employment, travel, and real estate data impacting downtown recovery. To do so, we first needed to develop a definition for “downtown” that is both consistent and comparable across cities, but precise enough to distinguish downtowns from adjacent submarkets. (This is because perceptions of what “downtown” is can vary significantly, and multiple cities have distinct commercial submarkets that have varying dynamics within a single city center.)\n",
      " \n",
      "To define “downtown,” we began by identifying each metro area’s central business district (CBD) as the census block with both the highest job density in the region and the largest absolute number of jobs; this is similar to the approach the International Downtown Association used in prior research. We then identified all adjacent blocks with a live-work quotient (ratio of residents to jobs, shown in Table 1, Column 8) of less than 0.1.4 We defined a metro area’s “commercial core” as the broader area of contiguous census block groups characterized by moderate- to high-density development with more jobs than residents (live-work quotient < 1).5\n",
      " \n",
      "This essay includes an analysis based on these definitions for the 45 largest U.S. downtowns (by job count, as shown in Table 1, Column 1). It is the first in a series of research products designed to inform the conversation—and local decisionmaking—on contemporary issues impacting American downtowns, from housing and adaptive reuse to perceptions and realities of public safety. \n",
      "\n",
      "Downtowns’ challenges didn’t start with the pandemic\n",
      "To help local leaders “think big” about center city recovery, it’s important to understand the pre-pandemic spatial distribution of cities’ jobs, opportunities, and amenities. Prior to the pandemic, the country’s economic geography was already evolving, and there was a need for downtown leaders to understand and adapt to these trends.\n",
      "First, even during the “urban resurgence,” downtowns were not the only game in town. In fact, the knowledge and innovation economy’s demands on place did not favor only the urban core. With so many jobs already suburbanized and with rising demand for new, collaborative, green, and amenity-rich work environments (which some older downtowns could not adapt fast enough to supply), a constellation of activity centers both adjacent to downtowns and their suburbs emerged to meet new demand for density. As many have pointed out, even before the pandemic, office-dominated places like San Francisco’s downtown were not places residents liked to frequent unless they had to.\n",
      "\n",
      "For example, Figure 1 shows the Seattle region’s activity centers (areas where economic, social, and civic assets cluster), which include downtown but also other increasingly desirable neighborhoods such as University District, Bellevue, and Capitol Hill, which have witnessed population and economic growth largely driven by knowledge economy workers.\n",
      "At the same time regions were witnessing the growth of new activity centers, only one-third of U.S. downtowns were gaining job market share in the years between the Great Recession and the pandemic’s arrival. Another one-third were growing only enough to hold steady, while the last third were in decline. Almost all regions were threatened by widening income inequality between neighborhoods.\n",
      "Some cities were responding to their changing economic realities. For example, even prior to the pandemic, Phoenix built a downtown strategy adapted to its relatively low downtown job market share and weak office demand (Table 1) with creative placemaking and by both attracting new anchor institutions and investing in transit connections to other activity centers. Chicago’s pre-pandemic Neighborhood Opportunity Fund aimed to link the wealth generated by downtown development with small businesses in underserved commercial corridors. But others were stuck in an old mindset—overlooking the importance of placemaking, blocking new housing development, limiting the growth of anchor institutions, or failing to invest in transit.\n",
      "While the pandemic did not create the origins of downtowns’ challenges, in its wake, the persistent and likely structural weakening of downtowns’ productivity could become a closed, reinforcing cycle: a doom loop. Some say that given suburbanization, the growth of activity centers, and remote work, downtowns simply no longer matter in the same way they once did (or, worse, that they are smoking craters that people should avoid). Others say downtowns have gotten all the spoils in recent decades, and it’s high time that public dollars focus on investment in neighborhoods. But it’s not that simple.\n",
      "From ‘downtowns versus neighborhoods’ to downtowns supporting neighborhoods\n",
      "Both the “downtowns versus neighborhoods” and “cities versus suburbs” viewpoints present false dichotomies that hold all residents of a region back.\n",
      "First, downtowns are extremely important to their regional economies. As a rule, downtowns are the most intense clusters of jobs in their metro areas—concentrating both higher-wage jobs (such as those in finance, insurance, law, or public administration) and lower-wage jobs (such as those in the arts or hospitality). Drawing on our mapping of regional activity centers, we compared the pre-pandemic job density of downtown activity centers to all other activity centers in their respective regions. For all of the 45 largest U.S. downtowns, downtown activity centers concentrate more jobs than other regional activity centers by a factor ranging from a low of 1.4 in Birmingham, Ala. to a high of 27.5 in Chicago (Table 1, Column 4). With few exceptions, even if 50% of jobs did not return to U.S. downtowns and 100% of jobs returned to every other job center (an extreme and unlikely scenario), downtowns would still be the densest job centers in their relative regions. Their well-being matters to the whole economy.\n",
      "Downtowns also matter to city residents, including workers and small business owners. Jobs in downtowns are generally more physically accessible to city dwellers than those in far-flung suburban areas. Beyond that, downtowns’ draw as regional job centers helps support local businesses. Every day, when suburban commuters arrive at jobs in a city center, the fiscal damage done by suburbanization is partially mitigated by their local spending. To summarize Table 1, the suburban commuter workforce increases the average city’s daytime population—and thus the market for goods and services—by an average of 19% (Table 1, Column 7). In a few notable cases, this boost is 50% or more, as in Washington, D.C. (50%); Boston (56%); and Hartford, Conn. (60%). In cities with a much higher daytime population relative to their resident population, the loss of sales tax and local employment in services and hospitality are likely acute.\n",
      "Finally, downtowns matter to cities’ fiscal health. A typical U.S. downtown such as Chicago produces more than 7 times its tax assessable value relative to land area, as shown in Figure 2. In the rest of the city, this relationship is inverted, with land area producing less than three-quarters of its share of taxable value. The biggest U.S. cities with major office real estate footprints are still anticipating future revenue shortfalls as the “fiscal time bomb” of commercial real estate valuations adjusts to the new market reality and federal aid is used up. The well-being of all parts of these cities depends on maintaining a balanced budget—and an essential part of that is keeping downtowns productive.\n",
      "\n",
      "While the challenges facing downtowns must be addressed to ensure a prosperous future for American cities, investing in strong downtowns does not have to come at the expense of supporting disinvested neighborhoods. Past policies and practices have created a perception (or reality) that downtowns prosper while neighborhoods stagnate. There is an obvious moral and economic mandate to invest in neighborhoods, but downtowns are a critical part of this solution—for example, through the revenue they generate, which can be used to fund neighborhood improvements and create a positive loop where downtowns benefit from healthy neighborhoods as neighbors. This is shared prosperity, but as we consider the future of downtowns and cities, it is not a given.\n",
      "Philadelphia: “Until you fix Kensington, Philly can never be a world-class city. Philly could be a world-class city, but only if it fixes communities like Kensington. No question. You can’t just make Center City shiny.”\n",
      "Actions local leaders can take to advance both downtown and neighborhood recovery\n",
      "In 2021, we wrote that to recover from COVID-19, downtowns would need to adapt by modernizing their office product, diversifying their land use mix, and prioritizing livability (such as quality public spaces, arts, traffic safety, etc.). As public and private sector leaders consider these adaptations (such as in panels like “New” New York), it is critical that all parties resist falling into false dichotomies and forced tradeoffs, and instead work together across silos to create win-win solutions.\n",
      "What kinds of win-win solutions could unlock another generation of value from our most historically productive places? And how might these solutions broaden who benefits from the productivity and prosperity that downtowns produce? In the coming months, Brookings Metro will continue to work with New York, Seattle, Chicago, and Philadelphia to explore and synthesize innovative policy solutions for the future of downtowns across the following domains:\n",
      "\n",
      "Inclusive workforce development and job recovery, including best practices on how to support a diverse workforce pipeline that helps employers find the talent they need and connects residents of all educational backgrounds to good and accessible jobs.\n",
      "Entrepreneurship and small business development, including how to maximize the survival of existing small businesses while cultivating new and diverse small business ownership downtown.\n",
      "Public safety and its impact on downtown recovery, including how to address both the perceptions and realities of crime in cities, with a focus on preventative, place-based solutions and the role of economic development professionals and the business community in promoting safety.\n",
      "Homelessness and its impact on downtown recovery, including understanding the costs and benefits of revising tax structures and adaptive reuse ordinances to build new housing, as well as policy changes to prevent homelessness and displacement and increase the supply of permanent affordable housing.\n",
      "Play and placemaking downtown, including strategies to make downtowns 24-hour destinations through arts, events, and programming, as well as how to measure the inclusivity of such placemaking interventions.\n",
      "Collaborative governance, civic partnerships, and capacity building, including identifying which partnerships are needed to sustain downtown recovery and the key public and private sector actions needed to facilitate supportive capital flows between downtowns and historically disinvested neighborhoods.\n",
      "\n",
      "As Brookings Metro and the four-city cohort delve into policy and practice solutions for each domain over the coming months, we will publish findings and guidance for other city and downtown leaders nationwide to learn from.\n",
      "Conclusion\n",
      "As we experience yet another rhetorical onslaught predicting the decline of cities and their downtowns, it is important to remember that cities have faced their share of crises before. From suburbanization and white flight to the rise of crime in urban centers in the 1990s to the 9/11 terrorist attacks on New York, cities and downtowns—and most especially, the people who activate them— have proved their resilience.\n",
      "The COVID-19 pandemic is no different, except that city leaders are facing a task that is two-fold: 1) ensuring a strong downtown recovery to restore the fiscal and economic well-being of their city and region; and 2) promoting a recovery that intentionally confronts the challenges and inequities that plagued downtowns prior to the pandemic.\n",
      "As the heart of our nation’s economy, downtowns have the unique potential—and imperative—to act boldly and inclusively to rebuild stronger than before. Rather than seeking to return to the pre-pandemic status quo, this is a pivotal moment for downtowns to build equity into revitalization by implementing policy solutions that connect downtown prosperity to more neighborhoods and more people. Only then can cities and regions realize their full potential.\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Hyperlocal\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Jennifer S. Vey and Nate Storring \n",
      "2022\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The authors would like to thank DW Rowlands for her excellent research assistance on this piece. They also extend their sincere gratitude to Fred Cerullo (Grand Central Partnership), David Downey (International Downtown Association), Anne Fadullon (City of Philadelphia), Paul Levy (Center City District), Julie Stein (City of New York), and Jennifer S. Vey for reviewing earlier drafts of this piece. Any errors that remain are solely the authors’.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.brookings.edu/research/the-second-half-of-the-sustainable-development-goal-era-ideas-for-doing-things-differently/\n",
      "TITLE: The ‘Second Half’ of the Sustainable Development Goal era: Ideas for doing things differently\n",
      "The ‘Second Half’ of the Sustainable Development Goal era: Ideas for doing things differently\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "John W. McArthur\n",
      "\n",
      "\t\t\t\t\tDirector - Center for Sustainable Development \n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development \n",
      "\n",
      " Twitter\n",
      "@mcarthur\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "This September, the U.N. will convene a midpoint summit on the Sustainable Development Goals, halfway between their 2015 launch and 2030 deadline. For many leaders gathering in the General Assembly, the mood might be somber. Stark global tensions alongside inadequate SDG progress make for a tough outlook. But a successful summit will need to focus on pragmatics more than sentiment: What has gone well, where could a burst of effort tackle gaps, and—perhaps most importantly—what needs to be done differently? \n",
      "The spirit of doing things differently is central to the purpose of the SDGs. The title of the all-country agreement establishing the goals was nothing shy of “Transforming our world.” It emphasized the “universal, integrated, and interrelated nature” of the goals, which “seek to realize the human rights of all.” Societies were not signing on to cherry-pick goals here and there. A hundred and ninety-three countries committed to transformation across all the goals, everywhere.  \n",
      "But how much has actually changed in the ensuing years? Or more plainly, how many changes have turned out to be for the better?   \n",
      "Fortunately, there are success stories. Compared to 2015, for example, a million fewer children under five are dying every year. India has extended basic sanitation access to more than 165 million additional people. Amid the pandemic, Togo deployed frontier digital technologies to deliver rapid cash assistance to nearly a quarter of its adult population, with priority to informally employed women. Since the SDGs were adopted, more than 200 legal reforms have been enacted across more than 100 countries to address domestic violence, penalize sexual harassment, and prohibit gender discrimination in employment. The United Kingdom’s greenhouse gas emissions dropped by roughly a fifth, despite the country’s remarkable political travails. Between 2015 and early 2023, Canada jumped from protecting around 1 percent to nearly 15 percent of its marine and coastal areas.  \n",
      "A growing range of constituencies are engaging on the SDGs too. More than 60 local government entities have crafted “Voluntary Local Reviews” of their SDG progress since New York City pioneered the first one in 2018. Almost three-quarters of the world’s 250 largest companies by revenue now report against some dimension of the SDGs. Times Higher Education’s “Impact rankings” have prompted huge numbers of universities to benchmark efforts against the SDGs. The Nature family of journals has intentionally expanded their range of outlets to heighten SDG-related scientific debates, most recently including Nature Water, launched earlier this year.  \n",
      "But despite these successes, empirical assessments draw attention to widespread SDG shortfalls. Our own previous work shows the huge number of people’s lives and basic needs at stake around the world, in addition to the planet itself. On many measures of SDG 5 for gender equality, current trends imply waiting decades if not centuries for targets to be achieved. Too few trendlines have changed, and in several areas the world is moving in the wrong direction.  \n",
      "The bottom line is that the world has not been doing enough things differently enough to be on course for SDG success. One 2022 meta-study soberly assessed that the SDGs’ impact on institutions and policies “has been largely discursive,” with limited “institutional impact, from legislative action to changing resource allocation.” Other studies have shown similar gaps in the private sector, where SDG rhetoric still doesn’t translate enough to action.  \n",
      "At the Brookings Center for Sustainable Development (CSD), we confront such findings with both clear-eyed rigor and determination. Our scholars and scholar teams work within their respective domains of expertise to distill challenges into component parts and distinguish between which pieces need reinforcement, which merit targeted innovation, and which need wholesale renewal. We draw inspiration from one another’s commitment to improving understanding and recommending better paths forward. \n",
      "In that spirit, this short compendium captures a cross section of SDG-focused insights and recommendations from CSD-affiliated scholars. Each brief essay describes something with the potential to be done differently during the second half of the SDG era. Across a dozen contributions, topics range from reframing media coverage of the SDGs to measuring and elevating the role of the private sector; from participatory approaches to transforming education systems to new learning paradigms for human rights; from better risk-taking in fragile countries to improving infrastructure and services for care; from fit-for-purpose multilateral development banks to a purpose-driven fund to end extreme poverty; from turbo-charged Canadian SDG approaches to renewed American SDG leadership; from breakthroughs in digital public infrastructure to innovative frontiers in the digitally empowered methods of collective behavior science.  \n",
      "We hope these essays can help spark bigger, broader, and bolder public debates on things to be done better during the second half of the SDG era. The world needs to keep ramping up critical keystones to success—the policies, institutions, technology, data, business innovations, government actions, and multistakeholder consortiums that will help do things differently. Shallow either-or arguments about opportunity versus failure are of little use when many ingredients for SDG success are increasingly in place. The practical challenge lies in fostering the complementary ingredients required to achieve breakthroughs. \n",
      "Over the coming months, CSD scholar teams will continue to push forward on specific challenges at the heart of the “second half” agenda for the SDGs. These vary from renewing multilateral financing for climate and development to reframing fiscal policies for gender equality to reimagining policy for rural America. Through the 17 Rooms initiative, we will continue to innovate around diverse forms of “next step” group actions, both internationally and at the community level.  \n",
      "Across all 17 goals, individual countries and communities deserve to have their own debates on what they’d like to do differently. Where the evidence suggests things haven’t yet been done differently enough, the looming 2030 deadline can help renew motivation to spark change.   \n",
      "The pledge of a world transformed will boil down to people coming together in groups of all scales to drive better actions together. Doing so with urgency can still deliver an inclusive world of sustainable prosperity for all. \n",
      "Download the full essay.\n",
      "\n",
      "URL: https://www.brookings.edu/research/hey-journalists-we-have-a-story-for-you/\n",
      "TITLE: Hey journalists, we have a story for you\n",
      "Hey journalists, we have a story for you\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Matthew Bishop\n",
      "\n",
      "\t\t\t\t\tNonresident Senior Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      " Twitter\n",
      "mattbish\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "“Boring U.N. stuff,” the global journalistic profession seems to have collectively concluded early on about the Sustainable Development Goals (SDGs), before moving swiftly on to more interesting fare—of which, it is true, the past few years have offered a plentiful supply, what with Trump, COVID-19, Ukraine, Harry and Meghan, and other surefire headline grabbers. \n",
      "Yet the media will likely have far more to say about the SDGs as the 2030 end date approaches. Any race against the clock generates tension, drama, and big decisions to report on as the deadline draws near.    \n",
      "More subtly, the COVID-19 pandemic has provided tangible evidence of why taking the sort of broad approach adopted for the SDGs makes sense by highlighting how thoroughly interconnected are issues of personal and planetary health, exclusion, and inequality. Reporting on the interconnections between the various problems the world faces during this era of polycrisis should be far easier for a journalist to pitch to an editor post-pandemic. The smarter newsroom bosses are more likely to recognize that their writers should look beyond their narrow domain expertise to see the bigger picture. Asking a writer “how does your story relate to the SDGs?” could become a useful shorthand for editors wanting to encourage the use of a more panoramic reporting lens. \n",
      "The smarter newsroom bosses are more likely to recognize that their writers should look beyond their narrow domain expertise to see the bigger picture. Asking a writer “how does your story relate to the SDGs?” could become a useful shorthand for editors wanting to encourage the use of a more panoramic reporting lens. \n",
      "Business journalism may lead the rest of the profession in doing this kind of reporting, as the transformation demanded by the SDGs is becoming increasingly material to the companies and investors they write about.    \n",
      "During last year’s 17 Rooms flagship process, hosted by the Center for Sustainable Development at the Brookings Institution and The Rockefeller Foundation, leading business media editors gathered in “Room 12,” a working group focused on actionable next steps for SDG 12 centered around responsible consumption and production. All of the editors said their organizations are now taking the SDGs—and their parallel yardstick in the business world, ESG (environment, social, governance)—far more seriously than they used to, and they expected their interest to keep growing. They also increasingly see a need to upgrade their newsrooms with the expertise needed to cover this story in a sophisticated way. \n",
      "Business pressures from content consumers, advertisers, and conference sponsors are all helping push things in the right direction, but the biggest motivator is that this is no longer “boring U.N. stuff.” There is now a real story to report on, with real CEOs and successful investors wanting to talk about the profits, opportunities, and risks that abound on the road to SDG world. \n",
      "Business editors first started to get it when companies started pledging to become carbon “net-zero.” What could be more natural for an editor than to have reporters look into how exactly these companies proposed to make this transformation, whether their strategy made any sense, and whether they were actually implementing it.   \n",
      "As ESG now moves center stage politically in the U.S. and globally, serious media, especially those with decent business coverage, will likely do what it does best: report, inform, educate, and hold decisionmakers to account.  \n",
      "However superficial the Punch-and-Judy political debate about ESG may seem compared to the seriousness of underlying environmental, social, and economic challenges of global sustainable development, it does stem from serious questions about the extent to which business should, can, and does make genuine efforts to improve the state of the world. Mobilizing the media to answer these questions is just what the SDGs need.  \n",
      "Indeed, a more engaged media will increase general awareness of why the Global Goals are needed, and what will be required to get the job done. It will highlight what is working, and more painfully, highlight failures and a good deal of humbug in private-sector ESG/SDG initiatives. If that latter part hurts, it should also inspire learning and a determination to do better in future. For the SDGs, it will certainly beat being ignored. \n",
      "Download the full essay.\n",
      " \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.brookings.edu/research/caring-about-care-an-sdg-5-priority/\n",
      "TITLE: Caring about Care: An SDG-5 priority\n",
      "Caring about Care: An SDG-5 priority\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Caren Grown\n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Goal 5 is an ambitious and expansive approach to reducing gaps between males and females and enabling women and girls to live their lives to the fullest. It proposes a multidimensional definition of, and comprehensive set of indicators for, tracking gender equality and women’s empowerment, complemented with targets and indicators across other goals. While advances have been made toward many aspects of Goal 5, the U.N. estimates that at the current rate, it will take nearly 300 years to meet all targets.  A high priority for accelerating progress is Target 5.4, which seeks to equalize the time that women and men spend on unpaid care and domestic work, including care for children, the elderly, the sick, and those with disabilities. Available data suggest this target is seriously off track.   \n",
      "Even before COVID-19 exposed glaring gaps in the infrastructure, systems, and services for care, inequality between males and females in unpaid care and domestic work were stark, with women providing the bulk of care worldwide. At the same time, the need for more and better care is urgent.  Many countries will experience an increased demand for care, especially elderly care, over the next 7 years considering demographic trends and climate-related events. Fertility remains high in several countries in sub-Saharan Africa, which will require care for children, while several countries in Asia, the Middle East, and Europe face high growth rates of the elderly population, with many countries expected to turn greyer by the end of the SDGs in 2030. Conflict also increases the need for care, especially for those with physical and emotional disabilities. Unless the supply of publicly supported care for children, the elderly, and those with disabilities increases, and men take up more caring labor, women of working age will face pressure to drop out of the labor force to take on this role.  \n",
      "There are currently no international legal standards for the provision of care. Nonetheless, the global community has adopted the “Five R” framework for care:  recognize, reduce, and redistribute unpaid work, and reward and represent paid caregiving. Comprehensive and coordinated policies, investments, and actions are needed to implement this framework. National governments play a key role in setting policies, including for parental leave and long-term care services1; occupational, safety, and other regulations for public and private care providers; and financing through public expenditure or tax credits.  Governments can also incorporate care as an economic sector in industrial planning and job creation strategies. As they do so, more investment and better regulation to assure the quality of care services will be critical. \n",
      "Since care provision largely takes place at the local level, municipalities also play a key role—delivering or contracting out public services; issuing building codes, licensing requirements, and standards for care facilities; inspecting facilities to ensure standards are met; and providing financial support to families, to name a few functions.2 Bogota is a shining example of a municipality that has adopted a comprehensive care framework, centered around CARE blocks that currently operate in 10 areas of the city, providing a range of quality services for caregivers and recipients.   \n",
      "But there is more to do. For instance, mayors, city councils, and urban planners can incorporate care services and infrastructure into city master plans. Collecting and analyzing data on the demand for care and the supply of care services is a first step. Planners can also link care diagnostics to strategies for housing, transport, land use, energy, water and sanitation, and food security. Meanwhile, cities can begin incorporating care provision into planning for climate risks and adaptation in response to climate shocks.  Employer-supported child care can be particularly impactful in low-income and post-conflict contexts where fiscal space may be constricted and publicly provided services are limited.  The needs of unpaid and informal caregivers should be at the heart of these efforts.  \n",
      "All individuals require care and provide care at some point throughout their lives, many without the support they need.  Continued inaction and lack of investment is no longer an option. By 2030, there is a major opportunity to advance SDG 5.4, and ensure the world is on the way to building quality systems to support caregivers and those who receive it. National- and local-level leaders working together can help pave the way forward.    \n",
      "Download the full essay.\n",
      "\n",
      "\n",
      "URL: https://www.brookings.edu/research/participatory-policymaking-to-transform-education-systems-and-meet-sdg-4/\n",
      "TITLE: Participatory policymaking to transform education systems and meet SDG 4\n",
      "Participatory policymaking to transform education systems and meet SDG 4\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Rebecca Winthrop\n",
      "\n",
      "\t\t\t\t\tDirector - Center for Universal Education \n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development \n",
      "\n",
      " Twitter\n",
      "RebeccaWinthrop\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Still reeling from the COVID-19 pandemic, education systems around the globe are facing a myriad of challenges. The world did not have enough teachers before the pandemic, but today the crisis is even worse. Overwhelmed, exhausted, underpaid, and unappreciated, teachers (like nurses in many places) are leaving the profession.  UNESCO estimates tens of millions more are needed globally—from roughly 9,000 more in the Netherlands to 16.5 million in sub-Saharan Africa. Children’s well-being has also suffered drastically. Child marriage is on the rise in countries in the Global South with UNICEF estimating 10 million more girls at risk of child marriage due to COVID-19. In the Global North, especially, the pandemic has created a mental health crisis with the American Academy of Pediatrics declaring it a U.S. national emergency in October 2021.   \n",
      "Unsurprisingly, the pandemic’s lasting impacts also reach academic learning. Many children, especially those with learning differences and from the most marginalized communities, are being further left behind on mastering essential skills like literacy and numeracy. In low- and middle-income countries, learning poverty (the percentage of 10-year-olds not able to read a simple text) has increased due to the pandemic, rising from 57 percent pre-pandemic to an estimated 70 percent in 2022. In the high-income country context of the United States, equity gaps widened with math proficiency scores for eighth-graders falling to levels not seen since 2000.  \n",
      "Given what educators and systems leaders face, many argue that dialing back our ambition and focusing on closing the learning gaps on essential academic skills, namely literacy and numeracy, should take center stage rather than maintaining our collective focus on developing the lifelong learners envisioned in SDG 4. However, dialing back ambition is not the answer. Finding new and transformative ways to tackle the suite of problems is. The risk of replacing the SDG education goal with a more focused objective on, say, literacy and numeracy is that even though it would be framed as a floor to build on, education systems will orient around it and ultimately treat it as the core objective. It will unwittingly become a ceiling. One has to look no further than the United Nations Millennium Development Goals that focused on access to primary school for boys and girls as the starting point for a quality education, assuming quality teaching and learning would follow. Instead, it became the end goal with millions of children entering new schools and no learning taking place. \n",
      "What is needed is a serious attempt to meet SDG 4 not by tinkering on the margins but by transforming systems so that they address the deep gaps in equity and relevance simultaneously. One approach to doing this is to harness innovation and bring it to the center of systems as we argued in 2018 in “Leapfrogging Inequality: Remaking Education to Help Young People Thrive.” The COVID-19 pandemic showed that education systems can indeed innovate in their core business. Several important pandemic-era innovations have the potential to deeply transform systems. For example, schools meaningfully sharing power with families on how to best collaborate to educate children is one important innovation that would transform systems. As our work in the Brookings Center for Universal Education highlighted, schools with deep relational trust between families and teachers are 10 times more likely to improve students’ academic and well-being outcomes. Maintaining this shift alongside other new practices such as effectively harnessing technology and integrating a focus on student well-being and mental health can help reach more children and provide them with the support and skills they need.       \n",
      "Ultimately, developing and stewarding education system transformation requires a participatory approach that puts those inside the system at the center. Government leaders, teachers, students, and communities need to develop a shared vision of the purpose of their own education systems. Too often,  essential actors feel left on the sidelines with too little space made for their involvement by those with power. Engaging in participatory policymaking approaches is essential to sustain the commitment needed to accompany transformation over time and has been done in contexts ranging from British Columbia, Canada to Bogotá, Colombia. As David Sengeh of the government of Sierra Leone and I wrote together last year, education system transformation hinges on refreshed local assessment of goals—“are they meeting the moment that we are in, are they tackling inequality and building resilience for a changing world, are they fully context aware, are they owned broadly across society”—and then aligning action around shared answers.  With common purpose, education systems can be transformed to dramatically reduce inequalities and produce lifelong learning for all.  \n",
      "Download the full essay.\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.brookings.edu/blog/brown-center-chalkboard/2023/04/05/state-of-the-states-gubernatorial-policy-priorities-in-2023/\n",
      "TITLE: State of the States: Gubernatorial policy priorities in 2023\n",
      "State of the States: Gubernatorial policy priorities in 2023\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Katharine Meyer\n",
      "\n",
      "\t\t\t\t\tFellow - Governance Studies, Brown Center on Education Policy \n",
      "\n",
      " Twitter\n",
      "@katharinemeyer\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Rachel M. Perera\n",
      "\n",
      "\t\t\t\t\tFellow - Governance Studies, Brown Center on Education Policy - The Brookings Institution \n",
      "\n",
      " Twitter\n",
      "@rachelmarisa\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The federal government plays a limited role in education policy—states and local governments are primarily responsible for educating our nation’s youth. The first federal laws about education governance weren’t introduced until 1965 with the Elementary and Secondary Education Act (ESEA) and Higher Education Act (HEA). And still, states are given broad latitude to determine how to best implement these federal laws in their states. Today, the federal government provides only about 10% of funding for K-12 public education and does not directly contribute to college operations (though the U.S. Department of Education administers a large portfolio of financial aid programs to students pursuing postsecondary education). \n",
      "States are often seen as “laboratories of democracy”—individual systems with their own executive, legislative, and judicial branches that may experiment with the development and implementation of policies to inform policy adoption in other states and at the federal level. These state-level policies then tend to “diffuse” throughout the country. One example of the prominent role that states play in agenda setting when it comes to education policy is the Common Core State Standards, K-12 education standards developed by the Council of Chief State School Officers and the National Governors Association, not by the federal government.\n",
      "\n",
      "\n",
      "Related Content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Brown Center Chalkboard\n",
      "The alarming state of the American student in 2022\n",
      "\n",
      "Robin Lake and Travis Pillow\n",
      "Tuesday, November 1, 2022\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Brown Center Chalkboard\n",
      "What if Americans sour on public education? \n",
      "\n",
      "Jon Valant\n",
      "Monday, August 29, 2022\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Brown Center Chalkboard\n",
      "Can public education return to normal after the COVID-19 pandemic?\n",
      "\n",
      "Paul T. Hill and Ashley Jochim\n",
      "Thursday, October 29, 2020\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The present moment reveals important insights into states’ priorities on public education. Most education systems—from early childhood to K-12 to postsecondary—are still reeling from the harmful effects of the pandemic and pandemic-era political conflicts. Many systems are also grappling with the wave of censorship bills and book bans sweeping K-12 schools and colleges across the country. In this context and with so much turnover taking place in state houses across the country—36 states held gubernatorial elections last fall with nine new governors starting their terms in 2023—we aim to understand what these changes mean for education policy. \n",
      "In this series, we focus on gubernatorial education priorities for 2023 to understand what state laboratories are experimenting with. Each year, many of the nation’s governors deliver their “State of the State” reports. Analogous to the President’s State of the Union, these addresses provide governors an opportunity to present their policy priorities for the year and the remainder of their terms. In this series, we first review gubernatorial priorities laid out in governors’ State of the State addresses. We then provide key contextual information about these efforts — what the research says about various proposals and lessons learned from prior attempts to implement similar policies—that we hope can guide the transition from policy priority to policy in practice. \n",
      "To do so, we rely on resources developed by teams at FutureEd, the Education Commission of the States, and the National Governors Association. These teams collated governors’ State of the State videos and transcripts, coded education topics covered, and made these resources publicly available for analysis. We are grateful for their careful work which this series builds on. \n",
      "\n",
      "URL: https://www.brookings.edu/blog/brown-center-chalkboard/\n",
      "TITLE: State of the States: Gubernatorial policy priorities in 2023\n",
      "BODY:\n",
      "\n",
      "URL: https://www.brookings.edu/blog/fixgov/2023/04/05/the-right-to-abortion-will-be-secured-before-the-end-of-the-decade/\n",
      "TITLE: The right to abortion will be secured before the end of the decade\n",
      "The right to abortion will be secured before the end of the decade\n",
      "BODY:\n",
      "\n",
      "When the right to choose an abortion is on the ballot, it wins. And it will keep winning for the rest of the decade until the right to abortion is secured state by state in all but the deepest red states and the U.S. Supreme Court’s decision is rendered moot.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Elaine Kamarck\n",
      "\n",
      "\t\t\t\t\tFounding Director - Center for Effective Public Management \n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Governance Studies \n",
      "\n",
      " Twitter\n",
      "EKamarck\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The latest evidence? On Tuesday, the liberal Milwaukee circuit court judge, Janet Protasiewicz, scored a solid victory over the conservative candidate Daniel Kelly in a race whose outcome would determine the majority on the Wisconsin Supreme Court and pave the way for overturning the 1849 law outlawing abortion.\n",
      "This race is but the latest example of the power this issue has to upend American politics. It happened first in August 2022 when a referendum on the primary ballot in Kansas would have amended the state constitution to make abortion illegal. Turnout was high — more votes were cast (980,000) than in any primary election in Kansas history. In a state Donald Trump won by 15 points, the anti-abortion referendum lost 59% to 41%. The larger more urban counties rejected the referendum by large margins and in the rural counties where it won it won by narrow margins.\n",
      "And now Wisconsin.\n",
      "In the 2020 presidential race, Wisconsin was a battleground state where Biden won a narrow victory (just over 20,000 votes) over Trump. The race for the open Supreme Court seat broke all records for turnout and money spent. Turnout appears to be 1.8 million — over half of the turnout in the 2020 presidential election. Over $45 million was spent according to WisPolitics — “a stunning sum for a statewide, off-year, springtime election.” The campaigning was intense, complete with nearly half of the ads about abortion.\n",
      "As the race went on, Dan Kelly, the conservative candidate with a pro-life record, tried to change the subject and tamp down the abortion issue; realizing as the race progressed (as did others in the 2022 midterms) — that his stance was a loser.\n",
      "As in Kansas, the power of the abortion issue is evident when compared to the presidential vote county by county. In the state’s two most populous counties, Milwaukee and Dane (home to the University of Wisconsin in Madison) Protasiewicz won big, which was to be expected. However, she also exceeded Biden’s vote in those two places — by 3.6% in Milwaukee and 6.3% in Dane. In the state’s next three most populous counties Biden lost to Trump, but in each one Protasiewicz performed better than Biden. In the third largest county in the state, Wausheka, Biden won 38.9% of the vote but Protasiewicz won 42%. In Brown County Biden won 45.6% of the vote but Protasiewicz won 52%. And in Racine County Biden won 47.2% of the vote and Protasiewicz won 49%.\n",
      "\n",
      "\n",
      "Related Content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "FixGov\n",
      "What we learned from the Chicago mayoral results\n",
      "\n",
      "William A. Galston\n",
      "Wednesday, April 5, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "FixGov\n",
      "Do politicians get rewarded for spending money?\n",
      "\n",
      "Vanessa Williamson\n",
      "Wednesday, March 22, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "FixGov\n",
      "Can government protect women from domestic violence? Not if states do not follow up.\n",
      "\n",
      "Kaitlin N. Sidorsky and Wendy J. Schiller\n",
      "Tuesday, March 21, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "As we saw in the 2022 midterms — when the freedom to choose is on the ballot it wins — upending prior voting patterns. The issue today is more real than it was in past races where Democrats always tried to argue its importance. Not surprising. A right withdrawn will always get more attention than a right taken for granted. While we don’t have any exit polls to confirm that abortion was foremost in voters’ minds, the explicit prominence of the issue in Protasiewicz’s campaign and Kelly’s failed attempts to change the topic are evidence that there was one big issue in this race.\n",
      "In the coming years pro-choice advocates will try to put as many pro-choice referenda on the ballot as possible. In 2022, six states had referenda on the ballot and the pro-choice position won in all of them. Wisconsin’s attempt at a referenda was rejected, but the recent Supreme Court race took its place and victory there will likely accomplish the same thing. In 2024 we can expect at least ten states to have citizen-led ballot initiatives enshrining abortion rights in their state constitutions.[1]\n",
      "The strength of the pro-choice position goes well beyond the debate over abortion itself. Deep in the American DNA is the belief that we should have as little government and as much liberty as possible. The law’s intrusion into the complex medical and moral issues surrounding as personal a decision as abortion strikes at the heart of American’s desires to control their own destiny. All indications are that by the end of the decade the Supreme Court’s decision to return abortion rights to states will reinstate abortion across the land.\n",
      "\n",
      "[1] Arizona, Arkansas, Florida, Missouri, Montana, Nebraska, North Dakota, Ohio, Oklahoma and South Dakota.\n",
      "\n",
      "URL: https://www.brookings.edu/blog/fixgov/\n",
      "TITLE: The right to abortion will be secured before the end of the decade\n",
      "\n",
      "\t\t\t\tFixGov\t\n",
      "BODY:\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.brookings.edu/blog/order-from-chaos/2023/04/05/when-might-us-political-support-be-unwelcome-in-taiwan/\n",
      "TITLE: When might US political support be unwelcome in Taiwan?\n",
      "When might US political support be unwelcome in Taiwan?\n",
      "BODY:\n",
      "\n",
      "For a time, it looked as though House Speaker Kevin McCarthy would make a high-profile visit to Taiwan this spring. There was some suggestion that this might lead Beijing to react even more coercively than it did after the previous speaker, Nancy Pelosi, visited in August 2022. Perhaps for that reason, McCarthy will now have a meeting with Taiwanese President Tsai Ing-wen when she transits through Los Angeles, California. Depending on how McCarthy frames his support for Tsai, however, the People’s Republic of China (PRC) might still escalate its military operations around Taiwan to signal its opposition to the alleged “hollowing out” of the U.S. “One China” policy. Depending on the scale of these actions, some Taiwanese voters might again conclude that such symbolic demonstrations of support risk entrapping Taiwan in an escalating U.S.-China rivalry.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Alastair Iain Johnston\n",
      "\n",
      "\t\t\t\t\tGovernor James Albert Noe and Linda Noe Laine Professor of China in World Affairs - Harvard Fairbank Center for Chinese Studies \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Tsai Chia-hung\n",
      "\n",
      "\t\t\t\t\tProfessor - National Chengchi University \n",
      "\n",
      "\t\t\t\t\t2022-23 Visiting Scholar - Harvard Fairbank Center for Chinese Studies \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "G\n",
      "\n",
      "\n",
      "\n",
      "George Yin\n",
      "\n",
      "\t\t\t\t\tDistinguished Research Fellow - National Taiwan University Center for China Studies \n",
      "\n",
      "\t\t\t\t\tResearch Associate - Harvard Fairbank Center for Chinese Studies \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "According to recent surveys we conducted in Taiwan, a majority of respondents believe that Pelosi’s visit was detrimental to Taiwan’s security. At first glance, this seems surprising. In a triangular relationship between a patron state (the United States) and its client (Taiwan) on the one hand, and a shared adversary (China) on the other, one might normally expect the client to welcome visible and credible signals of support.\n",
      "However, even as its security environment appears to be deteriorating, a client might not welcome signals of support from the patron if the client considers those signals to be so provocative that they undermine its security. Typically, it is the patron that worries about entrapment by its client, while the client worries about abandonment by the patron. But our surveys suggest that a considerable portion of Taiwanese voters worry about entrapment by the United States. There are, of course, partisan differences concerning fears of entrapment. Supporters of the Kuomintang (KMT) and independents worry that the convergence of Democratic Progressive Party (DPP) and U.S. preferences toward strategic competition with China makes Taiwan less secure.\n",
      "This concern about entrapment appears to have increased after Pelosi’s visit. We conducted a panel survey in Taiwan with two waves, one in September 2022 and a second in January 2023, to gauge the Taiwanese public’s reaction to Pelosi’s very public demonstration of support for Taiwan. In 2022, we asked respondents, in the wake of Pelosi’s visit and the PRC’s unprecedented military exercises around Taiwan, whether Taiwan faced a serious threat. In January 2023, we followed with a slightly different question about whether Pelosi’s travel had made Taiwan more or less secure.\n",
      "In September 2022, respondents overwhelmingly believed that Pelosi’s trip and the large-scale People’s Liberation Army exercises created a serious threat to Taiwan. Somewhat surprisingly, this response was similar across the political partisan divide in Taiwan. Media reports, in both Taiwan and the West, have suggested that many Taiwanese citizens did not originally seem too bothered by China’s reactions.\n",
      "After the initial shock, the majority of respondents in the January 2023 survey still believed that Pelosi’s visit made Taiwan less secure. But partisan differences were clearer in the answers to this question than they had been in September. A majority of KMT supporters and independents (as well as one-third of DPP supporters) believed that Pelosi’s visit had made Taiwan less secure. By contrast, a majority of DPP supporters felt the opposite was true.\n",
      "Indeed, by January 2023, 52% of our DPP respondents had shifted to the conclusion that Taiwan was more secure in the wake of the visit. In contrast, only 21% of the independent respondents and 11% of the KMT respondents had done so.\n",
      "Such a shift may have resulted from reassuring U.S. policy initiatives or confirmation bias, or both. For example, by January 2023, some DPP supporters may have been responding to the U.S.-Taiwan Initiative on 21st-Century Trade and a $12 billion military aid program. Alternatively, some DPP supporters may have perceived, or wanted to perceive, a change in U.S. policy after our first survey in September 2022 and looked for evidence for their beliefs. Based on answers to other questions we asked in the 2023 survey about the likelihood of possible gestures of U.S. support for Taiwan, those DPP respondents who shifted their perception of Taiwan’s security in a more positive direction between surveys also believed that the United States would diplomatically recognize Taiwan, would send troops to defend Taiwan if attacked by the PRC, and would conclude a trade agreement with Taiwan.\n",
      "A “Folk theory” of Taiwanese Security?\n",
      "In addition to asking respondents about the impact of Pelosi’s visit on Taiwan’s security, we also probed the panel respondents about whether they had a “folk theory” of security — that is, intuitions about the factors influencing Taiwan’s security based on their own biases and lived experiences. We gave respondents five common explanations for cross-strait instability and asked them whether they agreed or disagreed with each.\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Upcoming\n",
      "U.S.-Taiwan Relations\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Ryan Hass, Bonnie Glaser, and Richard Bush \n",
      "2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "US-China Nuclear Relations: The Impact of Strategic Triangles\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tContributions by Robert Einhorn \n",
      "2021\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "China 2049\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tEdited by David Dollar, Yiping Huang, and Yang Yao \n",
      "2020\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "A large portion of respondents appears to have a relatively plausible multi-causal “theory” of how Taiwan’s security is threatened. A large majority agrees that the PRC’s aggressive intentions are a source of instability. A smaller majority believes that increasing levels of support for independence in Taiwan is a source of instability. And only about 55% attribute instability to shifts in U.S. policy toward a “One China, One Taiwan policy.” On the other hand, majorities of respondents tended not to attribute instability to the neglect of defense building in Taiwan or to U.S. strategic ambiguity. Together these results suggest that many voters essentially lean toward the ideas embedded in the traditional U.S. policy of dual deterrence: Taiwan is more secure when the PRC’s aggressive intentions are countered and when the PRC is assured that U.S. policy does not encourage formal independence.\n",
      "Not surprisingly there are partisan differences. DPP supporters are more likely to blame inadequate military spending as a source of instability than KMT supporters and independents. Similarly, while KMT supporters and independents do not think U.S. strategic ambiguity is a source of instability, a small majority of DPP supporters does. While all groups attribute instability to the PRC’s aggressive intentions, DPP supporters overwhelmingly do so. Interestingly, all groups also agree that greater support in Taiwan for independence is a source of instability, though a smaller majority of DPP supporters agree than others. Not surprisingly, a large majority of KMT supporters agree that the hollowing out of the U.S. “One China” policy is a source of instability, but independents and DPP supporters are relatively evenly split.\n",
      "Implications\n",
      "Our surveys have a couple of tentative implications for policy discussions in Washington about how to preserve Taiwan’s security.\n",
      "First, our data suggests that if a substantial constituency in the client state believes such high-profile signals of support are counterproductive, this can weaken the security relationship between patron and client, making coordinated responses to the common adversary more difficult. A McCarthy-Tsai meeting, or a McCarthy visit to Taiwan, may only be prospectively reassuring to a majority of Taiwanese if they believe that the PRC response will be milder than it was after Pelosi’s visit. And if China does react less coercively after McCarthy’s symbolic actions, then it is possible that, retrospectively, at least some KMT and independent voters may conclude such actions are a useful signal of support. On the other hand, even a slightly less threatening PRC response may not be reassuring enough to KMT and independent voters, given their concerns about entrapment. Our surveys suggest that KMT supporters, and to some degree independents, do not identify culturally or politically with the United States nearly as much as DPP respondents do. Many KMT supporters and independents blame growing support for independence as a source of cross-strait instability. So, it may be unlikely that many KMT and independents will interpret a McCarthy-Tsai meeting as a useful symbol of U.S. support in the face of PRC coercion, especially if it is viewed as a partisan statement of support for the DPP government.\n",
      "A second implication of our results concerns the effectiveness of deterrence. Washington’s discourse about how to deter the PRC emphasizes military tools over the synergistic effects of assuring Beijing that the United States is not encouraging or enabling formal Taiwanese independence. Proponents of dual deterrence claim that, since 1972, credible coercion and credible assurance have kept the probability of a PRC takeover of Taiwan relatively low and will likely continue to be more successful compared to alternative U.S. strategies, such as allowing and/or recognizing a de jure independent Taiwan. This may or may not be the case. There is a debate over whether dual deterrence is the best of a bad set of options for reducing the probability of conflict. But our analysis of respondents’ “folk theory” of security suggests that a considerable portion of the Taiwanese population seems to agree with the dual deterrence notion. This may be the conceptual basis for their concern about entrapment by U.S. politicians.\n",
      "\n",
      "\n",
      "Related Content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Order from Chaos\n",
      "Australia’s caution on Taiwan may not last\n",
      "\n",
      "Richard McGregor\n",
      "Wednesday, March 29, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Order from Chaos\n",
      "Taiwan is feeling the pressure from Russian and Chinese autocracy\n",
      "\n",
      "Pavel K. Baev\n",
      "Thursday, March 16, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Order from Chaos\n",
      "The Philippines’ evolving view on Taiwan: From passivity to active involvement\n",
      "\n",
      "Renato Cruz De Castro\n",
      "Thursday, March 9, 2023\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Methodology\n",
      "The 2022 household telephone survey was conducted between September 22 and 29, 2022, by the Election Study Center, National Chengchi University. We randomly drew samples according to the area codes of telephone books. Only respondents who were over 20 years old and registered in Taiwan were eligible for this survey. Data was weighted by gender, age, education, and residence area based on the latest census data. The sample size was 1,127. The margin of error with 95% confidence interval is 2.92%. The 2023 panel study was conducted between January 5 and 9, 2023. We successfully re-interviewed 576 respondents or about 51% of the 2022 sample. The data was also weighted by gender, age, education, and residence area. The main questions we analyze are: “In August this year, U.S. Congress Speaker Nancy Pelosi visited Taiwan, and China immediately held large-scale military exercises around Taiwan. Do you think this is a serious threat to Taiwan’s security?” (September 2022 survey); and “Do you think Pelosi’s visit to Taiwan made Taiwan more or less secure?” (January 2023 survey).\n",
      "\n",
      "URL: https://www.brookings.edu/blog/order-from-chaos/\n",
      "TITLE: When might US political support be unwelcome in Taiwan?\n",
      "\n",
      "\t\t\t\tOrder from Chaos\t\n",
      "BODY:\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.brookings.edu/research/sdg-implementation-for-fragile-countries-needs-more-risk-taking/\n",
      "TITLE: SDG implementation for fragile countries needs more risk-taking\n",
      "SDG implementation for fragile countries needs more risk-taking\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Naheed Sarabi\n",
      "\n",
      "\t\t\t\t\tVisiting Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      " Twitter\n",
      "Sarabinaheed\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "In 2023, concurring economic, social, and environmental crises are disproportionately affecting fragile states, creating a grim outlook for achieving the SDGs by 2030. The Global Peace Index Report for 2022 indicates deteriorating global peacefulness since 2014, with a growing gap between the most peaceful and least peaceful countries. SDG progress has been either stagnating or declining in more than half of the fragile states. Poverty and insecurity are on the rise in conflict-affected and fragile countries, where 20 percent of the global share of those in extreme poverty live; this is expected to rise to more than 50 percent by 2023. In addition, fragile countries are not on track with SDGs related to hunger, good health, and gender.  \n",
      "During the second half of the SDG era, the international community must take more risks than it does in traditional development processes if conflict-affected and fragile countries are to make more SDG gains.  \n",
      "A traditional economic development agenda in developing countries focuses on interdependent issues like growth, governance, poverty reduction, better services, and infrastructure. However, development is reversible and fragile in the occurrence of shocks. Resilience depends largely on the strength of the institutions. The g7+ group of conflict-affected countries advocated strongly, and successfully, for the New Deal principles, established in 2011, to be incorporated into the 2030 Agenda and the SDGs: peace-building and state-building goals (PSGs), engagement to support country-owned and -led pathways out of fragility, and commitments to results.  \n",
      "What does constructive risk-taking look like in fragile countries?  Three steps can help drive better progress: \n",
      "\n",
      "Strengthen emphasis on SDG 16. Peace, reconciliation, and political stability—foundational elements of SDG 16—must be given stronger political weight in both international and country-level SDG discussions. In my own conversations with the g7+ secretariat, they insist that without a stronger commitment to peace, achieving SDG targets will be a challenge. Multi-year development planning needs farsighted and stable political environments that provide a smooth transition from quick response to long-term development efforts. To level the foundations, the international community must put more emphasis on both peace-building and peacemaking efforts. Addressing horizontal inequalities and keeping closer watch on drivers of internal conflict and community-level peace-building are essentials for the way forward.  \n",
      "Build ownership and institutions in a more measurable manner. Donors need to channel more funds through government systems to build trust and ownership in fragile countries. The SDGs represent an enormous undertaking for fragile countries with low spending capacities. Building institutions and capacities takes time. Where the risk levels for corruption and misuse are extremely high, there is a need for joint program implementation between the government and donors. Multi-donor trust funds that channel money and decisionmaking power through government systems can be good models.  \n",
      "Strengthen the basics for data, planning, financing, and expenditure. There should be an expedited effort to coordinate data systems, integrate SDGs into the national planning and budgeting process, and link plans to government and donor expenditure systems and reports. Lack of access to data is a pressing problem in fragile and conflict-affected countries. Governments are constrained by data deprivation due to the flow of funds outside government systems that in turn shape development plans. SDG targets should be translated into realistic multi-year projects through an inclusive approach. Experts in development cooperation need to undertake simple exercises to align SDGs into national data collection, planning, and budgeting cycles.  \n",
      "\n",
      "SDG implementation in fragile countries needs much more partnership and trust between the development partners and host countries. While the international community must harbor more risk in implementing the above actions, host countries must show the commitment to execute and take ownership.  \n",
      "Download the full essay.\n",
      "\n",
      "URL: https://www.brookings.edu/research/a-pact-to-sustain-digital-public-infrastructure-for-the-sdgs/\n",
      "TITLE: A PACT to sustain digital public infrastructure for the SDGs\n",
      "A PACT to sustain digital public infrastructure for the SDGs\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Priya Vora\n",
      "\n",
      "\t\t\t\t\tNonresident Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "George Ingram\n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      " Twitter\n",
      "@GMIngramIV\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Digital technologies can accelerate achievement of the SDGs but only if governments act now to make trusted digital infrastructure universally available. For years digital capabilities have been described as enablers of progress; they are a key reason seven indicators across four SDGs relate to digital capabilities. But here at the halfway mark, meeting the SDGs remains elusive. Digital must again be put forward as an important means to fast-track progress. \n",
      "With 2030 in sight, governments must find pathways to creating foundational digital infrastructure that can be repurposed for a variety of uses—therefore obviating the need to “reinvent the wheel” and massively improve the return on initial investment. \n",
      "There is reason to be excited about foundational infrastructure designed for multiple uses. India is efficiently extending government services to over 93 percent of its population enrolled in the Aadhaar identity platform. That same tool is used for banks to extend financial services to millions of unbanked families. In the midst of war, the government of Ukraine is using its e-government portal to ensure displaced citizens can access proof of identity, health care records, and much more. The same underlying system is the basis for government agencies to share data to better understand the needs of citizens and identify how best to service those needs. However, the picture of digital infrastructure is not always rosy. These powerful systems are accelerators of intent, whether good or bad. Since the consensus on the SDGs was secured, the world has witnessed a range of harms resulting from digital technologies—in the form of exclusion; data misuse and abuse; and monopolistic tendencies that squash competitive markets. \n",
      "The goal therefore is to expand digital infrastructure but ensure the public (state) response is strong enough to maximize participation, agency, choice, and trust (PACT). When digital assets are either publicly owned, publicly regulated, or publicly designed through open protocols–digital infrastructure can be realized for the public good. In other words, digital public infrastructure (DPI) is secured.  \n",
      "While the return on investment for foundational digital infrastructure is high, foreign assistance for DPI remains inadequate. The bulk of donor digital investments are in one-off siloed solutions designed for a specific program or activity, undermining the hope of infrastructural investments that can serve many purposes. Moreover, investment in public sector institutions to regulate and oversee the digital economy is woefully low in most countries.  \n",
      "This must change. Donors need to jointly commit to invest over the long term in DPI solutions—and the necessary institutional safeguards around them—that will serve nations and maximize digital participation, agency, choice, and trust in the quest to achieve the SDGs.     \n",
      "This opportunity is not out of reach. As an example, India spent just over $1 per person to deploy its digital identity system. The figure could be lower for others using open code, which allows low-resource states to leverage the talents of coders around the world who contribute to open-source digital infrastructure solutions. More importantly, because DPI is relevant to most SDG goals, meeting the resource gap may not require new funds. Instead, through a strategic approach to digital public infrastructure and the institutional safeguards, funding intended for siloed technology investments could be redirected to underpin widely available digital infrastructure assets for all. \n",
      "Download the full essay.\n",
      " \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.brookings.edu/research/scaling-private-sector-engagement-in-the-sdgs/\n",
      "TITLE: Scaling private sector engagement in the SDGs\n",
      "Scaling private sector engagement in the SDGs\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Jane Nelson\n",
      "\n",
      "\t\t\t\t\tNonresident Senior Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "George Ingram\n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      " Twitter\n",
      "@GMIngramIV\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Private sector investment and innovation are essential to achieving the Sustainable Development Goals (SDGs). A vanguard of companies is making public commitments and taking action. Yet, business engagement and impact are far from becoming mainstream. A concerted effort is required to scale the quantity, quality, and accountability of private sector activities that could have a measurable impact on supporting the SDGs.  \n",
      "In the 12th U.N. Global Compact-Accenture CEO Study, released in 2023, 98 percent of more than 2,600 chief executives across 18 industries in 128 countries agreed that sustainability is now core to their role. While 87 percent warned that current levels of geopolitical and economic disruption are limiting the delivery of the SDGs, 51 percent believe they could play a critical role to help achieve the goals with increased commitment and action.  \n",
      "In many cases, however, implementation is lagging behind public pledges and ambition. KPMG’s 2022 Survey of Sustainability Reporting found that 74 percent of the world’s largest 250 companies by revenue are reporting on the SDGs. Yet, only 10 percent are reporting on all 17 Goals, and only 6 percent are reporting on their negative as well as positive impacts. Another study by the Global Reporting Initiative and Support the Goals found that 83 percent of 206 companies surveyed in 2021 said they support the SDGs, but only 40 percent set measurable commitments and only 20 percent included evidence to assess their impacts.    \n",
      "What needs to change? Three areas of collective action will be essential. \n",
      "\n",
      " Standardize and require measures of corporate accountability: The time has come to put public disclosure of a company’s material, social, and environmental risks and performance on a par with corporate financial reporting.\n",
      "Among voluntary reporting initiatives, the International Sustainability Standards Board, created in 2021, is on track to issue its first two standards on General Requirements and Climate by mid-2023. Last year, the Global Reporting Initiative updated its guidance for companies to report on the SDGs. In 2022, the Task Force for Climate-related Financial Disclosure reported that more than 3,800 organizations across 99 countries have become supporters of its recommendations, including over 1,500 financial institutions responsible for assets of $217 trillion.\n",
      "But beyond such voluntary efforts, it is time to make this reporting mandatory, independently audited, or assured in accordance with international auditing and assurance standards, and based on comparable, relevant, and reliable metrics. A variety of non-financial reporting requirements and directives are being spearheaded by the European Union, the U.S. Securities and Exchange Commission, and other government entities. Such policy efforts need to find efficient paths to implementation, to scale the experience and lessons of voluntary reporting by some companies to mandatory reporting for all companies.\n",
      "\n",
      "\n",
      " Leverage investment in innovation accelerators and collaborative platforms: Companies should invest more strategically in research, development, and innovation partnerships. Individual company investments in new technologies, products, services, and business models to deliver specific SDGs are necessary but not sufficient. There is untapped potential to leverage different combinations of public, private, and philanthropic finance and undertake joint efforts to accelerate or scale progress in crucial sectors and systems. Recent alliances such as the Breakthrough Energy Network, the First Movers Coalition, the Energy Transition Accelerator, and the Zero Hunger Private Sector Pledge point the way, alongside more established platforms such as GAVI, the Vaccine Alliance. \n",
      "\n",
      "\n",
      " Advocate for enabling policies through business coalitions: Corporations must speak out more ambitiously and transparently on the need for smart public policies and regulations. In the absence of policy reforms and market incentives, business action for the SDGs will never scale beyond the leading companies. At a global level, organizations such as the World Business Council for Sustainable Development and We Mean Business Coalition, alongside some established industry groups, are increasing their public advocacy. More needs to be done at the national level. The Keidanren in Japan, National Business Initiative in South Africa, and Confederation of Danish Industry are three examples of business associations advocating for government leadership on the SDGs.\n",
      "\n",
      "In summary, the first seven years of the 2030 Agenda have seen growing business engagement, but not at the speed and scale that are necessary—or possible. The second half of the SDG era needs to capture and catapult private-sector contributions to the 2030 goals.  \n",
      "Download the full essay. \n",
      "\n",
      "URL: https://www.brookings.edu/research/paradigm-shift-creating-more-just-societies-with-the-sdgs-human-rights-and-innovations-in-higher-education/\n",
      "TITLE: Paradigm shift: Creating more just societies with the SDGs, human rights, and innovations in higher education\n",
      "Paradigm shift: Creating more just societies with the SDGs, human rights, and innovations in higher education\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Sarah E. Mendelson\n",
      "\n",
      "\t\t\t\t\tNonresident Senior Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      "\t\t\t\t\tDistinguished Service Professor of Public Policy and Head of Heinz College in Washington, D.C. - Carnegie Mellon University \n",
      "\n",
      " Twitter\n",
      "SarahMendelson\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "A large share of the human rights community does not yet embrace—let alone know about—the Sustainable Development Goals (SDGs). Ambivalence is driven in part by the voluntary nature of the SDG agenda, as opposed to one shaped by legal requirements. At the same time, however, a downbeat cottage industry has developed around pessimistic themes such as the “end times” of human rights and a failed, last “utopia.” The pessimism stems from what appears to be overreliance on these same legal remedies that in so many cases fail to stem impunity or address inequality.  \n",
      "A new approach requires a paradigm shift in the frameworks that currently dominate human rights.   \n",
      "Innovations in higher education offer a pathway to advance the closely aligned endeavors of creating peaceful, just, and inclusive communities. Universities have a critical role to play in generating a refreshed approach to human rights that includes SDG literacy, helping to grow what I call Cohort 2030.  \n",
      "What would such a paradigm shift look like?  \n",
      "\n",
      "First, the SDG watchwords “Leave No One Behind” (LNOB) should be embedded in human rights education. LNOB is fundamentally a call for human rights to be respected universally; the SDGs apply everywhere to everyone. The SDGs help broaden attention beyond political rights and elevate socioeconomic ones, which during the Cold War, became enmeshed in East-West power struggles and were subsequently downplayed, at least in the United States. The pandemic has laid bare the urgent need to address social justice gaps and socioeconomic inequities in the Global North as well as in the Global South. In fact, these issues are bound up with tackling much of what imperils democracy in the United States today.      \n",
      "Second, human rights education should focus on SDG localization and translation in specific contexts (sometimes referred to as “vernacularization” in the rights literature). The general lack of localization has been diagnosed by human rights scholars as a significant barrier to the realization of rights. When viewed as global and abstract rather than local and experienced by people, the disconnect contributes to both the need to “rescue” the SDGs and the end-times-for-human-rights zeitgeist. Localizing the SDGs, as has occurred in a number of cities around the world, has resulted in innovation and practical applications that directly and positively impact communities. \n",
      "Third, related to both LNOB and localization, human rights and sustainable development research and coursework should increasingly incorporate and generate people-centered data ecosystems, including community data portals that use open-source software and are derived with the input of local community members. During the 2021 flagship 17 Rooms process, co-hosted by the Brookings Center for Sustainable Development and The Rockefeller Foundation, this was a distinct recommendation from rights experts focused on advancing SDG 16. Disaggregated data that reveal local communities’ social justice gaps are necessary (along with policies and funds) to enable more just transitions post-pandemic—which to date, based on research in several North American cities, has not occurred. In the United States, the White House executive order on equity and data that recently went into effect could be a game changer helping generate such ecosystems.  \n",
      "\n",
      "These three steps would link the Universal Declaration of Human Rights, now in its 75th year, with the SDGs, in addition to the numerous subsequent treaties and laws which students still need to learn.  The idea of combining the SDG and human rights agendas has already been highlighted, for example, by the United Nations in its Fourth Phase (2020-2024) of the World Programme for Human Rights, albeit with scant specificity.  It would also be fully consistent with a recent study by the U.S. National Academies of Sciences, Engineering, and Medicine on higher education and the SDGs, which recommends “field building” regarding sustainable development.1 Best of all, it would drive demand for progress not only on SDG 16 (justice) but on the other people-focused SDGs too. The second half of the SDG era needs to create a sustainable future in which rights are realized for all.   \n",
      "Download the full essay.\n",
      "\n",
      "URL: https://www.brookings.edu/research/how-to-make-mdbs-fit-for-purpose-to-support-sdg-implementation/\n",
      "TITLE: How to make MDBs fit for purpose to support SDG implementation\n",
      "How to make MDBs fit for purpose to support SDG implementation\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Amar Bhattacharya\n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Homi Kharas\n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "“Through collective and individual actions, initiatives and additional steps to enhance extensive work already underway in our organizations, the MDBs, and the IMF will help countries, partners, investors, and the global community move from billions to trillions as we take up the challenge of achieving the SDGs.” Upon the adoption of the SDGs, the World Bank recognized that the transformative vision embodied in the SDGs would require a vastly higher level of resources to be successfully implemented. This never happened. MDB net flows averaged  $25 billion in 2013-15 (in constant 2012 dollars) and $41 billion in 2021, an increase driven by the cyclical response to the COVID-19 pandemic, rather than by the structural needs for building the SDG economy. \n",
      "Shareholders are again asking the MDBs to up their game. The Italian G-20 tasked an expert committee to review the capital adequacy frameworks of the banks. U.S. Treasury Secretary Janet Yellen and other shareholders called for an evolution roadmap for the World Bank. The Indian G-20 is setting up an expert group to advise on MDB reform. \n",
      "Three elements of MDB reform would significantly enlarge their contributions to SDG achievement: \n",
      "\n",
      "The MDBs should embrace and drive the new growth story for the 21st century, one based on innovation and investment in green technologies, and on adaptation and resilience to accelerating climate change.  A number of new technologies in electricity production, light road transport, fugitive emissions, building heating, and food and agriculture are reaching a point within the next five years that would make them price-competitive on a mass-market scale. The potential for cheap power can accelerate growth and make access to modern electricity easier for the 775 million people who currently lack it. It avoids the negative local effects of fossil-fuel-sourced pollution on health and education. Yet there are holdouts among MDBs and their clients that view the green transition as potentially holding back the reduction of poverty and social progress. The MDBs should provide the empirical evidence in support of the new growth strategy that is inclusive, sustainable, and resilient, and draws lessons on how to manage the transition.\n",
      "Achieving a green transformation will require investments at scale. Recent granular estimates suggest that emerging markets and developing countries other than China must increase spending on climate investments fourfold by 2030 to $2.4 trillion.  MDBs must play a much more purposive and proactive role in helping countries identify, enable, and foster green investments, moving beyond a project-by-project approach to support system change. Country-led and country-owned coordination mechanisms around priority objectives have to become the norm for scaling up investments and mobilizing the necessary support. For this MDBs have to be able to work much more effectively as a system and with the private sector.  \n",
      "MDBs also have a central role to play in securing the scale and affordability of finance that will be needed.  Private capital can meet the largest share of the $1 trillion in external finance needed by 2030. In the current context of debt overhangs and rising interest rates on capital markets, however, little private capital is flowing to developing countries. MDBs can create a new highway for private finance through partnering with and reducing risks for the private sector. They can use guarantees and insurance products to take risks that the private sector cannot mitigate, such as policy risk, and reduce the cost of capital through blended finance. MDBs must also increase their own direct finance for the large complementary public investments in both the physical and social infrastructure required and where private capital may not be forthcoming. MDBs are unique in that they can scale up financing at affordable terms without relying heavily on contributions from the budgets of major shareholders, although for the poorest shareholders, it will be important to continue to provide them with concessional aid.\n",
      "\n",
      "In taking these steps, MDBs must take to heart the universal character of the SDGs and the spillover benefits that come from coordinated action. They must serve all clients, low-income and middle-income alike. They must pursue sustainable development as an integrated and interwoven package of economic growth, human capital development, gender equality, urbanization, jobs, nature conservation, and biodiversity preservation. Fit for purpose MDBs would provide an accelerant for the second half of the SDG period and for keeping global climate targets in reach. \n",
      "Download the full essay.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.brookings.edu/research/a-purpose-driven-fund-to-end-extreme-poverty-by-2030/\n",
      "TITLE: A purpose-driven fund to end extreme poverty by 2030\n",
      "A purpose-driven fund to end extreme poverty by 2030\n",
      "BODY:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Homi Kharas\n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development, Center for Sustainable Development \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "John W. McArthur\n",
      "\n",
      "\t\t\t\t\tDirector - Center for Sustainable Development \n",
      "\n",
      "\t\t\t\t\tSenior Fellow - Global Economy and Development \n",
      "\n",
      " Twitter\n",
      "@mcarthur\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Ending extreme poverty by 2030 is first among equals within the Sustainable Development Goals. When SDG target 1.1 was formally adopted in 2015, the number of extremely poor people was thought to be around 730 million globally and was falling by roughly 65 million a year. Continuing that trend would have cut poverty rates to zero by 2030. But progress has slowed instead. Recent projections suggest 570 million people might still be poor in 2030, far short of elimination. At the SDG midpoint, rebooting efforts to ending extreme poverty could re-energize the world like nothing else.  \n",
      "First, what went wrong? Much early optimism rested on the World Bank’s prominent ownership of the poverty reduction target. In 2013, the organization adopted the end of extreme poverty as one of its two headline goals, alongside reducing inequality. It promised to operationalize the twin goals through tailored country strategies and transformational forms of engagement. Further, in the 2015 Addis Ababa Action Agenda on Financing and Development, all U.N. member states agreed to a new social compact to end poverty, with fiscally sustainable and nationally appropriate income floors. They vowed to explore funding modalities for such systems.  \n",
      "Unfortunately, these promises lacked follow-through. The shortcomings culminated last October, when the World Bank admitted premature defeat on its own goal, with a statement that “the global goal of ending extreme poverty by 2030 would not be achieved.”  \n",
      "The bumpiness of the path to ending extreme poverty should motivate renewal, not resignation. Fortunately, improvements in technology, policy, and evidence over the past decade make the goal more solvable today than it was in 2015, not less.  \n",
      "As particular inspiration during the strains of COVID-19, widespread country breakthroughs in social protection highlight frontier opportunities for direct income support through cash transfers. Evidence of these and other cash transfer measures is reassuringly positive: A strong reduction in poverty rates and spillover benefits into other SDG areas, mostly education, health, and nutrition. Evidence of the practicalities of implementation has also accumulated. For instance, Togo, a low-income country, was able to build its Novissi digital platform to reach almost 1 million beneficiaries within a year, distributing around $20/month (30 percent of the monthly minimum wage). During the first six months, nearly two-thirds of the beneficiaries were women.  \n",
      "The bumpiness of the path to ending extreme poverty should motivate renewal, not resignation. \n",
      "The digital and mobile technology revolutions have transformed the speed and costs of hyper-targeted transfers to poor people. Transaction costs are now down to single-digit levels as a share of funds for beneficiaries. Dramatic advances in targeting methods, thanks to machine learning and more granular poverty data, have radically improved efficiency and efficacy. \n",
      "For sure, cash transfers are a policy tool, not a panacea. They require a long-term strategy, and a majority of extremely poor people still live in rural areas, where investments in agriculture, infrastructure, and human capital are all key to long-term economic growth. Each country should be empowered with vigorous debate on how to sequence and scale such investments. But cash transfers can serve as a best alternative to a negotiated agreement in these debates. Whenever a better alternative to ending extreme poverty by 2030 cannot be found, cash transfers offer a powerful device for getting the job done—as a ladder out of poverty, not a permanent dependency.  \n",
      "Embedding this logic into a purpose-driven global funding instrument could transform progress on SDG 1. One option would be to encourage the International Development Association (IDA) to orient its operations around the explicit purpose of ending extreme poverty, proactively supporting targeted cash transfer systems in every country that wishes to use them. With an incoming new World Bank president and a request from shareholders to follow through on its existing headline goal, IDA would be ripe for innovation. An alternative would be to build a new purpose-driven multilateral fund to end extreme poverty, with a technical focus on sustained cash transfer systems for extremely poor people. Such a targeted instrument would be consistent with, while much more specific than, calls for a global fund for social protection.  \n",
      "The implied financing needs for such a funding instrument are substantial but realistic. At about $100 billion, the global extreme poverty gap is equivalent to only 0.1 percent of global GDP, its lowest relative size ever, and slightly more than half of total official development assistance. An initial funding modality could start where the needs are greatest, with focused support to the five countries on course to account for more than half the world’s poor in 2030: Nigeria, the Democratic Republic of Congo, Madagascar, Tanzania, and Angola.  \n",
      "The time is ripe for a purpose-driven fund to end extreme poverty. The targeting can be surgical, costs are modest, and implementation is now feasible. Doing so to achieve SDG 1 within the coming seven years would mark a historic global achievement and help renew momentum for the SDGs overall.\n",
      "Download the full essay.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "\n",
    "class Crawler:\n",
    "    def __init__(self, site):\n",
    "        self.site = site\n",
    "        self.visited = {}\n",
    "\n",
    "    def getPage(url):\n",
    "        try:\n",
    "            html = urlopen(url)\n",
    "        except Exception as e:\n",
    "            print(e)\n",
    "            return None\n",
    "        return BeautifulSoup(html, 'html.parser')\n",
    "\n",
    "    def safeGet(bs, selector):\n",
    "        selectedElems = bs.select(selector)\n",
    "        if selectedElems is not None and len(selectedElems) > 0:\n",
    "            return '\\n'.join([elem.get_text() for elem in selectedElems])\n",
    "        return ''\n",
    "\n",
    "    def getContent(self, url):\n",
    "        \"\"\"\n",
    "        Extract content from a given page URL\n",
    "        \"\"\"\n",
    "        bs = Crawler.getPage(url)\n",
    "        if bs is not None:\n",
    "            title = Crawler.safeGet(bs, self.site.titleTag)\n",
    "            body = Crawler.safeGet(bs, self.site.bodyTag)\n",
    "            return Content(url, title, body)\n",
    "        return Content(url, '', '')\n",
    "\n",
    "    def crawl(self):\n",
    "        \"\"\"\n",
    "        Get pages from website home page\n",
    "        \"\"\"\n",
    "        bs = Crawler.getPage(self.site.url)\n",
    "        targetPages = bs.findAll('a', href=re.compile(self.site.targetPattern))\n",
    "        for targetPage in targetPages:\n",
    "            url = targetPage.attrs['href']\n",
    "            url = url if self.site.absoluteUrl else f'{self.site.url}{targetPage}'\n",
    "            if url not in self.visited:\n",
    "                self.visited[url] = self.getContent(url)\n",
    "                self.visited[url].print()\n",
    "\n",
    "\n",
    "brookings = Website('Reuters', 'https://brookings.edu', '\\/(research|blog)\\/', True, 'h1', 'div.post-body')\n",
    "crawler = Crawler(brookings)\n",
    "crawler.crawl()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawling multiple page types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Website:\n",
    "    \"\"\"Common base class for all articles/pages\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Product(Website):\n",
    "    \"\"\"Contains information for scraping a product page\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, productNumber, price):\n",
    "        Website.__init__(self, name, url, TitleTag)\n",
    "        self.productNumberTag = productNumberTag\n",
    "        self.priceTag = priceTag\n",
    "\n",
    "class Article(Website):\n",
    "    \"\"\"Contains information for scraping an article page\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, bodyTag, dateTag):\n",
    "        Website.__init__(self, name, url, titleTag)\n",
    "        self.bodyTag = bodyTag\n",
    "        self.dateTag = dateTag"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}