archived_dataset_merge.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "df99a48d",
   "metadata": {},
   "source": [
    "## Pubmed Abstract/Fulltext Fetching Merging and Cleaning\n",
    "\n",
    "Pubmed does not give fulltext/abstract text in a good format. It gives a CSV with the metadata of searched results. That means you have to use the Entrez fetch and then merge the abstracts and text information with the metadata to have something workable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d973b203",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(       PMID                                              Title  \\\n",
       " 0  28946522  The effect of manual therapy and exercise in p...   \n",
       " 1  28325149  Low Dose Naltrexone in the Treatment of Fibrom...   \n",
       " 2  29514165  High-Dose Erythropoietin for Asphyxia and Ence...   \n",
       " 3  29424396  [Quality of sleep and academic performance in ...   \n",
       " 4  29754671  Rationale and design for the Vascular Outcomes...   \n",
       " \n",
       "                                              Authors  \\\n",
       " 0                 Ulger O, Demirel A, Oz M, Tamer S.   \n",
       " 1  Metyas S, Chen CL, Yeter K, Solyman J, Arkfeld...   \n",
       " 2  Juul SE, Comstock BA, Heagerty PJ, Mayock DE, ...   \n",
       " 3  Bugueño M, Curihual C, Olivares P, Wallace J, ...   \n",
       " 4  Capell WH, Bonaca MP, Nehler MR, Chen E, Kitte...   \n",
       " \n",
       "                                             Citation First Author  \\\n",
       " 0  J Back Musculoskelet Rehabil. 2017 Nov 6;30(6)...      Ulger O   \n",
       " 1  Curr Rheumatol Rev. 2018;14(2):177-180. doi: 1...     Metyas S   \n",
       " 2  Neonatology. 2018;113(4):331-338. doi: 10.1159...      Juul SE   \n",
       " 3  Rev Med Chil. 2017 Sep;145(9):1106-1114. doi: ...    Bugueño M   \n",
       " 4  Am Heart J. 2018 May;199:83-91. doi: 10.1016/j...    Capell WH   \n",
       " \n",
       "                    Journal/Book  Publication Year Create Date       PMCID  \\\n",
       " 0  J Back Musculoskelet Rehabil              2017  2017/09/27         NaN   \n",
       " 1            Curr Rheumatol Rev              2018  2017/03/23         NaN   \n",
       " 2                   Neonatology              2018  2018/03/08  PMC5980685   \n",
       " 3                  Rev Med Chil              2017  2018/02/10         NaN   \n",
       " 4                    Am Heart J              2018  2018/05/15         NaN   \n",
       " \n",
       "       NIHMS ID                                DOI  \\\n",
       " 0          NaN                 10.3233/BMR-169673   \n",
       " 1          NaN  10.2174/1573397113666170321120329   \n",
       " 2  NIHMS937494                  10.1159/000486820   \n",
       " 3          NaN    10.4067/s0034-98872017000901106   \n",
       " 4          NaN          10.1016/j.ahj.2018.01.011   \n",
       " \n",
       "                                           BACKGROUND  \\\n",
       " 0                                               None   \n",
       " 1  ConclusionFibromyalgia is a chronic pain disor...   \n",
       " 2                                               None   \n",
       " 3  Sleeping and studying are the day-to-day activ...   \n",
       " 4                                               None   \n",
       " \n",
       "                                          CONCLUSIONS  \\\n",
       " 0  This study showed that SSE and manual therapy ...   \n",
       " 1  This prospective study lends further support t...   \n",
       " 2                                               None   \n",
       " 3  A bad sleep quality influences academic perfor...   \n",
       " 4                                               None   \n",
       " \n",
       "                                              METHODS  \\\n",
       " 0  A total of one-hundred thirteen patients diagn...   \n",
       " 1  Naltrexone is an opioid receptor antagonist us...   \n",
       " 2                                               None   \n",
       " 3  Students of the first and second year of high ...   \n",
       " 4                                               None   \n",
       " \n",
       "                                            OBJECTIVE  \\\n",
       " 0  To determine the effects of spinal stabilizati...   \n",
       " 1  A significant number of fibromyalgia patients ...   \n",
       " 2                                               None   \n",
       " 3  To determine the quality of sleep and its rela...   \n",
       " 4                                               None   \n",
       " \n",
       "                                              RESULTS  \n",
       " 0  Intragroup analyses both treatments were effec...  \n",
       " 1  Two small prospective pilot studies have previ...  \n",
       " 2                                               None  \n",
       " 3  The interview was answered by 322 first year s...  \n",
       " 4                                               None  ,\n",
       " 10000,\n",
       " 10000,\n",
       " 28946522,\n",
       " 28946522)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# with open(\"clinical_study_abst.txt\") as f:\n",
    "#     text = f.read()\n",
    "    \n",
    "df = pd.read_csv(\"downloads/clin_study_meta.csv\")\n",
    "records = get_pubmed_records([str(pmid) for pmid in df[\"PMID\"]])\n",
    "len([r for r in records[\"PubmedArticle\"] if r is not None])\n",
    "\n",
    "articles = records[\"PubmedArticle\"]\n",
    "\n",
    "g = {\n",
    "    \"PMID\": [str(get(r,\"MedlineCitation.PMID\")) for r in articles],\n",
    "    \"BACKGROUND\": [get_segment(r, \"BACKGROUND\") for r in articles],\n",
    "    \"OBJECTIVE\": [get_segment(r, \"OBJECTIVE\") for r in articles],\n",
    "    \"METHODS\": [get_segment(r, \"METHODS\") for r in articles],\n",
    "    \"RESULTS\": [get_segment(r, \"RESULTS\") for r in articles],\n",
    "    \"CONCLUSIONS\": [get_segment(r, \"CONCLUSIONS\") for r in articles]\n",
    "}\n",
    "\n",
    "\n",
    "df2 = pd.DataFrame(g)\n",
    "df2[\"PMID\"] = df2[\"PMID\"].astype(int)\n",
    "dfx = pd.merge(df, df2, on=\"PMID\")\n",
    "dfx.to_csv(\"enriched_pm_10k.csv\")\n",
    "\n",
    "\n",
    "#dfx.head(), len(df), len(df2), df[:6768][\"PMID\"][0], df2[:6768][\"PMID\"][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac3ca53d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# #export\n",
    "# def get_first(p):\n",
    "#     return get(split_into_sentences(p), 0)\n",
    "# #get(split_into_sentences(t), 0) for t in dfx[\"METHODS\"][:10] if split_into_sentences(t))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e62fbfb",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n",
      "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  import sys\n",
      "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "#reformating data\n",
    "dfy = pd.DataFrame([], columns=[\"PMID\", \"Title\", \"source\", \"DOI\", \"text\"])\n",
    "tags = [\"OBJECTIVE\", \"METHODS\", \"RESULTS\", \"CONCLUSIONS\"]\n",
    "for tag in tags:\n",
    "    df = dfx[dfx[tag] != \"None\"]\n",
    "    df[\"paragraph\"] = df[tag]\n",
    "    df[\"text\"] = dfx[tag].map(get_first)\n",
    "    df[\"source\"] = \"pm_\" + tag\n",
    "    df = df[dfy.columns]\n",
    "    dfy = dfy.append(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c187d2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "dfy = dfy.dropna(subset=[\"text\"])\n",
    "dfy[\"labels\"] = \"0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afdeb9ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "dfy.to_csv(\"downloads/struct_abstracts_negatives_18k.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a408451",
   "metadata": {},
   "source": [
    "### Fetching dataset from Google Drive\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2601edde",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = 'https://drive.google.com/uc?export=download&id='\n",
    "df1 = pd.read_csv(path + \"1t2m6IxieiE0hZ8TcLJe9foc7upngrA_0\")\n",
    "df1[\"source\"] = \" limitation|problem ... with ... l=be that\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab299432",
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = pd.read_csv(path + \"1t-2rU83KdppSrwDnqdQUi6Pvtw13D0bQ\")\n",
    "df2[\"source\"] = \"a ... drawback of\"\n",
    "df3 = pd.read_csv(path + \"1iGdfLJ4MkTO8oOomFJ84JCc7nf2zYBul\")\n",
    "df3[\"source\"] = \"a {negative_adjectives} ...  {problem_nouns}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de8836b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(947, 1455, 18573)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df1), len(df2), len(df3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf8fbf1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence_id</th>\n",
       "      <th>title</th>\n",
       "      <th>article_link</th>\n",
       "      <th>sentence_text</th>\n",
       "      <th>year</th>\n",
       "      <th>mesh_list</th>\n",
       "      <th>journal_title</th>\n",
       "      <th>volume</th>\n",
       "      <th>issue</th>\n",
       "      <th>abstract</th>\n",
       "      <th>source</th>\n",
       "      <th>paragraph_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1263595</td>\n",
       "      <td>Shoe-Integrated, Force Sensor Design for Conti...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/32545528/</td>\n",
       "      <td>One problem with such methods is that they tap...</td>\n",
       "      <td>2020</td>\n",
       "      <td>['Biomechanical Phenomena', 'Body Weight', 'Eq...</td>\n",
       "      <td>Sensors (Basel, Switzerland)</td>\n",
       "      <td>20</td>\n",
       "      <td>12</td>\n",
       "      <td>Traditional pedobarography methods use direct ...</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1821146</td>\n",
       "      <td>The Problem with Gout Is That It's Still Such ...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/27481987/</td>\n",
       "      <td>The Problem with Gout Is That It 's Still Such...</td>\n",
       "      <td>2016</td>\n",
       "      <td>['Allopurinol', 'therapeutic use', 'Arthritis,...</td>\n",
       "      <td>The Journal of rheumatology</td>\n",
       "      <td>43</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1840641</td>\n",
       "      <td>Generation of noble-gas binding sites for crys...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/11752783/</td>\n",
       "      <td>One problem with this approach is that not all...</td>\n",
       "      <td>2002</td>\n",
       "      <td>['Bacteriophage T4', 'enzymology', 'Binding Si...</td>\n",
       "      <td>Acta crystallographica. Section D, Biological ...</td>\n",
       "      <td>58</td>\n",
       "      <td>Pt 1</td>\n",
       "      <td>In recent years, the use of noble-gas complexe...</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2187391</td>\n",
       "      <td>Segmentation of Crohn, Lymphangiectasia, Xanth...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/22874169/</td>\n",
       "      <td>However , a tough problem associated with this...</td>\n",
       "      <td>2012</td>\n",
       "      <td>['Algorithms', 'Artificial Intelligence', 'Cap...</td>\n",
       "      <td>Studies in health technology and informatics</td>\n",
       "      <td>180</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Wireless capsule endoscopy (WCE) is a great br...</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3202138</td>\n",
       "      <td>Vection in virtual reality modulates vestibula...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/31233640/</td>\n",
       "      <td>While significant technological advancements a...</td>\n",
       "      <td>2019</td>\n",
       "      <td>['Female', 'Humans', 'Male', 'Muscle, Skeletal...</td>\n",
       "      <td>REFERENCES</td>\n",
       "      <td>50</td>\n",
       "      <td>10</td>\n",
       "      <td>The popularity of virtual reality (VR) has inc...</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentence_id                                              title  \\\n",
       "0      1263595  Shoe-Integrated, Force Sensor Design for Conti...   \n",
       "1      1821146  The Problem with Gout Is That It's Still Such ...   \n",
       "2      1840641  Generation of noble-gas binding sites for crys...   \n",
       "3      2187391  Segmentation of Crohn, Lymphangiectasia, Xanth...   \n",
       "4      3202138  Vection in virtual reality modulates vestibula...   \n",
       "\n",
       "                                article_link  \\\n",
       "0  https://pubmed.ncbi.nlm.nih.gov/32545528/   \n",
       "1  https://pubmed.ncbi.nlm.nih.gov/27481987/   \n",
       "2  https://pubmed.ncbi.nlm.nih.gov/11752783/   \n",
       "3  https://pubmed.ncbi.nlm.nih.gov/22874169/   \n",
       "4  https://pubmed.ncbi.nlm.nih.gov/31233640/   \n",
       "\n",
       "                                       sentence_text  year  \\\n",
       "0  One problem with such methods is that they tap...  2020   \n",
       "1  The Problem with Gout Is That It 's Still Such...  2016   \n",
       "2  One problem with this approach is that not all...  2002   \n",
       "3  However , a tough problem associated with this...  2012   \n",
       "4  While significant technological advancements a...  2019   \n",
       "\n",
       "                                           mesh_list  \\\n",
       "0  ['Biomechanical Phenomena', 'Body Weight', 'Eq...   \n",
       "1  ['Allopurinol', 'therapeutic use', 'Arthritis,...   \n",
       "2  ['Bacteriophage T4', 'enzymology', 'Binding Si...   \n",
       "3  ['Algorithms', 'Artificial Intelligence', 'Cap...   \n",
       "4  ['Female', 'Humans', 'Male', 'Muscle, Skeletal...   \n",
       "\n",
       "                                       journal_title volume issue  \\\n",
       "0                       Sensors (Basel, Switzerland)     20    12   \n",
       "1                        The Journal of rheumatology     43     8   \n",
       "2  Acta crystallographica. Section D, Biological ...     58  Pt 1   \n",
       "3       Studies in health technology and informatics    180   NaN   \n",
       "4                                         REFERENCES     50    10   \n",
       "\n",
       "                                            abstract  \\\n",
       "0  Traditional pedobarography methods use direct ...   \n",
       "1                                                NaN   \n",
       "2  In recent years, the use of noble-gas complexe...   \n",
       "3  Wireless capsule endoscopy (WCE) is a great br...   \n",
       "4  The popularity of virtual reality (VR) has inc...   \n",
       "\n",
       "                                       source paragraph_text  \n",
       "0   limitation|problem ... with ... l=be that            NaN  \n",
       "1   limitation|problem ... with ... l=be that            NaN  \n",
       "2   limitation|problem ... with ... l=be that            NaN  \n",
       "3   limitation|problem ... with ... l=be that            NaN  \n",
       "4   limitation|problem ... with ... l=be that            NaN  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dff = df1.append(df2).append(df3)\n",
    "dff.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35d134ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "dff[\"PMID\"] = dff[\"article_link\"].map(lambda t: t.split(\"/\")[-2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e36996d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20822, 20975)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dff.drop_duplicates(subset=[\"PMID\"])), len(dff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfdc7153",
   "metadata": {},
   "outputs": [],
   "source": [
    "dff = dff.drop_duplicates(subset=[\"PMID\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7cbf780",
   "metadata": {},
   "outputs": [],
   "source": [
    "dff = dff.rename(columns={'sentence_text': \"text\", \"title\": \"Title\"}, errors=\"raise\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e93d936b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence_id</th>\n",
       "      <th>Title</th>\n",
       "      <th>article_link</th>\n",
       "      <th>text</th>\n",
       "      <th>year</th>\n",
       "      <th>mesh_list</th>\n",
       "      <th>journal_title</th>\n",
       "      <th>volume</th>\n",
       "      <th>issue</th>\n",
       "      <th>abstract</th>\n",
       "      <th>source</th>\n",
       "      <th>paragraph_text</th>\n",
       "      <th>PMID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1263595</td>\n",
       "      <td>Shoe-Integrated, Force Sensor Design for Conti...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/32545528/</td>\n",
       "      <td>One problem with such methods is that they tap...</td>\n",
       "      <td>2020</td>\n",
       "      <td>['Biomechanical Phenomena', 'Body Weight', 'Eq...</td>\n",
       "      <td>Sensors (Basel, Switzerland)</td>\n",
       "      <td>20</td>\n",
       "      <td>12</td>\n",
       "      <td>Traditional pedobarography methods use direct ...</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "      <td>32545528</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1821146</td>\n",
       "      <td>The Problem with Gout Is That It's Still Such ...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/27481987/</td>\n",
       "      <td>The Problem with Gout Is That It 's Still Such...</td>\n",
       "      <td>2016</td>\n",
       "      <td>['Allopurinol', 'therapeutic use', 'Arthritis,...</td>\n",
       "      <td>The Journal of rheumatology</td>\n",
       "      <td>43</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27481987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1840641</td>\n",
       "      <td>Generation of noble-gas binding sites for crys...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/11752783/</td>\n",
       "      <td>One problem with this approach is that not all...</td>\n",
       "      <td>2002</td>\n",
       "      <td>['Bacteriophage T4', 'enzymology', 'Binding Si...</td>\n",
       "      <td>Acta crystallographica. Section D, Biological ...</td>\n",
       "      <td>58</td>\n",
       "      <td>Pt 1</td>\n",
       "      <td>In recent years, the use of noble-gas complexe...</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11752783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2187391</td>\n",
       "      <td>Segmentation of Crohn, Lymphangiectasia, Xanth...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/22874169/</td>\n",
       "      <td>However , a tough problem associated with this...</td>\n",
       "      <td>2012</td>\n",
       "      <td>['Algorithms', 'Artificial Intelligence', 'Cap...</td>\n",
       "      <td>Studies in health technology and informatics</td>\n",
       "      <td>180</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Wireless capsule endoscopy (WCE) is a great br...</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22874169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3202138</td>\n",
       "      <td>Vection in virtual reality modulates vestibula...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/31233640/</td>\n",
       "      <td>While significant technological advancements a...</td>\n",
       "      <td>2019</td>\n",
       "      <td>['Female', 'Humans', 'Male', 'Muscle, Skeletal...</td>\n",
       "      <td>REFERENCES</td>\n",
       "      <td>50</td>\n",
       "      <td>10</td>\n",
       "      <td>The popularity of virtual reality (VR) has inc...</td>\n",
       "      <td>limitation|problem ... with ... l=be that</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31233640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18568</th>\n",
       "      <td>362702792</td>\n",
       "      <td>Falls and injuries in frail and vigorous commu...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/1987256/</td>\n",
       "      <td>These findings suggest that fall-related injur...</td>\n",
       "      <td>1991</td>\n",
       "      <td>['Accidental Falls', 'prevention &amp; control', '...</td>\n",
       "      <td>Journal of the American Geriatrics Society</td>\n",
       "      <td>39</td>\n",
       "      <td>1</td>\n",
       "      <td>Identification of different types of falls and...</td>\n",
       "      <td>a {negative_adjectives} ...  {problem_nouns}</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1987256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18569</th>\n",
       "      <td>362714838</td>\n",
       "      <td>Intermittent catheterization: sterile or clean?</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/1989043/</td>\n",
       "      <td>Bacteriuria -- asymptomatic and symptomatic --...</td>\n",
       "      <td>1991</td>\n",
       "      <td>['Bacteriuria', 'etiology', 'nursing', 'preven...</td>\n",
       "      <td>Rehabilitation nursing : the official journal ...</td>\n",
       "      <td>16</td>\n",
       "      <td>1</td>\n",
       "      <td>Bacteriuria--asymptomatic and symptomatic--alw...</td>\n",
       "      <td>a {negative_adjectives} ...  {problem_nouns}</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18570</th>\n",
       "      <td>362715256</td>\n",
       "      <td>Variceal rebleeding after portosystemic shunti...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/1989102/</td>\n",
       "      <td>Strategies and solutions to a vexing problem .</td>\n",
       "      <td>1991</td>\n",
       "      <td>['Combined Modality Therapy', 'Esophageal and ...</td>\n",
       "      <td>The Surgical clinics of North America</td>\n",
       "      <td>71</td>\n",
       "      <td>1</td>\n",
       "      <td>The purpose of this review was to discuss an a...</td>\n",
       "      <td>a {negative_adjectives} ...  {problem_nouns}</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18571</th>\n",
       "      <td>362724620</td>\n",
       "      <td>Neuroprotective Effect of Natural Products on ...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/26645998/</td>\n",
       "      <td>Peripheral nerve injury ( PNI ) is a serious p...</td>\n",
       "      <td>2016</td>\n",
       "      <td>['Animals', 'Biological Products', 'therapeuti...</td>\n",
       "      <td>Neurochemical research</td>\n",
       "      <td>41</td>\n",
       "      <td>4</td>\n",
       "      <td>Peripheral nerve injury (PNI) is a serious pub...</td>\n",
       "      <td>a {negative_adjectives} ...  {problem_nouns}</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26645998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18572</th>\n",
       "      <td>362750185</td>\n",
       "      <td>Identification of Immunodominant Responses to ...</td>\n",
       "      <td>https://pubmed.ncbi.nlm.nih.gov/26659715/</td>\n",
       "      <td>Malaria , caused by the Plasmodium parasite , ...</td>\n",
       "      <td>2015</td>\n",
       "      <td>['Animals', 'Antigens, Protozoan', 'immunology...</td>\n",
       "      <td>PloS one</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "      <td>Malaria, caused by the Plasmodium parasite, re...</td>\n",
       "      <td>a {negative_adjectives} ...  {problem_nouns}</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26659715</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20822 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       sentence_id                                              Title  \\\n",
       "0          1263595  Shoe-Integrated, Force Sensor Design for Conti...   \n",
       "1          1821146  The Problem with Gout Is That It's Still Such ...   \n",
       "2          1840641  Generation of noble-gas binding sites for crys...   \n",
       "3          2187391  Segmentation of Crohn, Lymphangiectasia, Xanth...   \n",
       "4          3202138  Vection in virtual reality modulates vestibula...   \n",
       "...            ...                                                ...   \n",
       "18568    362702792  Falls and injuries in frail and vigorous commu...   \n",
       "18569    362714838    Intermittent catheterization: sterile or clean?   \n",
       "18570    362715256  Variceal rebleeding after portosystemic shunti...   \n",
       "18571    362724620  Neuroprotective Effect of Natural Products on ...   \n",
       "18572    362750185  Identification of Immunodominant Responses to ...   \n",
       "\n",
       "                                    article_link  \\\n",
       "0      https://pubmed.ncbi.nlm.nih.gov/32545528/   \n",
       "1      https://pubmed.ncbi.nlm.nih.gov/27481987/   \n",
       "2      https://pubmed.ncbi.nlm.nih.gov/11752783/   \n",
       "3      https://pubmed.ncbi.nlm.nih.gov/22874169/   \n",
       "4      https://pubmed.ncbi.nlm.nih.gov/31233640/   \n",
       "...                                          ...   \n",
       "18568   https://pubmed.ncbi.nlm.nih.gov/1987256/   \n",
       "18569   https://pubmed.ncbi.nlm.nih.gov/1989043/   \n",
       "18570   https://pubmed.ncbi.nlm.nih.gov/1989102/   \n",
       "18571  https://pubmed.ncbi.nlm.nih.gov/26645998/   \n",
       "18572  https://pubmed.ncbi.nlm.nih.gov/26659715/   \n",
       "\n",
       "                                                    text  year  \\\n",
       "0      One problem with such methods is that they tap...  2020   \n",
       "1      The Problem with Gout Is That It 's Still Such...  2016   \n",
       "2      One problem with this approach is that not all...  2002   \n",
       "3      However , a tough problem associated with this...  2012   \n",
       "4      While significant technological advancements a...  2019   \n",
       "...                                                  ...   ...   \n",
       "18568  These findings suggest that fall-related injur...  1991   \n",
       "18569  Bacteriuria -- asymptomatic and symptomatic --...  1991   \n",
       "18570     Strategies and solutions to a vexing problem .  1991   \n",
       "18571  Peripheral nerve injury ( PNI ) is a serious p...  2016   \n",
       "18572  Malaria , caused by the Plasmodium parasite , ...  2015   \n",
       "\n",
       "                                               mesh_list  \\\n",
       "0      ['Biomechanical Phenomena', 'Body Weight', 'Eq...   \n",
       "1      ['Allopurinol', 'therapeutic use', 'Arthritis,...   \n",
       "2      ['Bacteriophage T4', 'enzymology', 'Binding Si...   \n",
       "3      ['Algorithms', 'Artificial Intelligence', 'Cap...   \n",
       "4      ['Female', 'Humans', 'Male', 'Muscle, Skeletal...   \n",
       "...                                                  ...   \n",
       "18568  ['Accidental Falls', 'prevention & control', '...   \n",
       "18569  ['Bacteriuria', 'etiology', 'nursing', 'preven...   \n",
       "18570  ['Combined Modality Therapy', 'Esophageal and ...   \n",
       "18571  ['Animals', 'Biological Products', 'therapeuti...   \n",
       "18572  ['Animals', 'Antigens, Protozoan', 'immunology...   \n",
       "\n",
       "                                           journal_title volume issue  \\\n",
       "0                           Sensors (Basel, Switzerland)     20    12   \n",
       "1                            The Journal of rheumatology     43     8   \n",
       "2      Acta crystallographica. Section D, Biological ...     58  Pt 1   \n",
       "3           Studies in health technology and informatics    180   NaN   \n",
       "4                                             REFERENCES     50    10   \n",
       "...                                                  ...    ...   ...   \n",
       "18568         Journal of the American Geriatrics Society     39     1   \n",
       "18569  Rehabilitation nursing : the official journal ...     16     1   \n",
       "18570              The Surgical clinics of North America     71     1   \n",
       "18571                             Neurochemical research     41     4   \n",
       "18572                                           PloS one     10    12   \n",
       "\n",
       "                                                abstract  \\\n",
       "0      Traditional pedobarography methods use direct ...   \n",
       "1                                                    NaN   \n",
       "2      In recent years, the use of noble-gas complexe...   \n",
       "3      Wireless capsule endoscopy (WCE) is a great br...   \n",
       "4      The popularity of virtual reality (VR) has inc...   \n",
       "...                                                  ...   \n",
       "18568  Identification of different types of falls and...   \n",
       "18569  Bacteriuria--asymptomatic and symptomatic--alw...   \n",
       "18570  The purpose of this review was to discuss an a...   \n",
       "18571  Peripheral nerve injury (PNI) is a serious pub...   \n",
       "18572  Malaria, caused by the Plasmodium parasite, re...   \n",
       "\n",
       "                                             source paragraph_text      PMID  \n",
       "0         limitation|problem ... with ... l=be that            NaN  32545528  \n",
       "1         limitation|problem ... with ... l=be that            NaN  27481987  \n",
       "2         limitation|problem ... with ... l=be that            NaN  11752783  \n",
       "3         limitation|problem ... with ... l=be that            NaN  22874169  \n",
       "4         limitation|problem ... with ... l=be that            NaN  31233640  \n",
       "...                                             ...            ...       ...  \n",
       "18568  a {negative_adjectives} ...  {problem_nouns}            NaN   1987256  \n",
       "18569  a {negative_adjectives} ...  {problem_nouns}            NaN   1989043  \n",
       "18570  a {negative_adjectives} ...  {problem_nouns}            NaN   1989102  \n",
       "18571  a {negative_adjectives} ...  {problem_nouns}            NaN  26645998  \n",
       "18572  a {negative_adjectives} ...  {problem_nouns}            NaN  26659715  \n",
       "\n",
       "[20822 rows x 13 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1cd8c74",
   "metadata": {},
   "outputs": [],
   "source": [
    "dff = dff[['Title',\"text\", 'mesh_list', 'journal_title', 'source', 'PMID']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8af93b9a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/ipykernel/__main__.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  if __name__ == '__main__':\n"
     ]
    }
   ],
   "source": [
    "dff[\"labels\"] = \"1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02613125",
   "metadata": {},
   "outputs": [],
   "source": [
    "# df = df.append(dff, ignore_index=True).reset_index()\n",
    "\n",
    "# df[\"PMID\"] = df[\"PMID\"].astype(str)\n",
    "# df[\"labels\"] = df[\"labels\"].astype(str)\n",
    "\n",
    "# df = df.dropna(subset=[\"text\"])\n",
    "# df.to_csv(\"downloads/fat.csv\", index=False)\n",
    "\n",
    "# dfl = pd.read_csv(\"downloads/fat.csv\")\n",
    "# dft = pd.read_csv(\"downloads/struct_abstracts_negatives_18k.csv\")\n",
    "# dfall=dfl.append(dft)\n",
    "# dfall[\"labels\"] = dfall[\"labels\"].astype(str)\n",
    "\n",
    "# dfall = dfall.dropna(subset=[\"text\"])\n",
    "\n",
    "\n",
    "# dfall = dfall[[\"PMID\", \"Title\", \"text\", \"DOI\", \"labels\", \"source\"]]\n",
    "# dfall.to_csv(\"downloads/40k_balanced_pm_acl.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15c4776f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_first(p):\n",
    "    return get(split_into_sentences(p), 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08e991fa",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_amazonei_mxnet_p27",
   "language": "python",
   "name": "conda_amazonei_mxnet_p27"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}