CIC output guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to Parse CIC Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the program CIC is a large json file. This is a guide on parsing the output using python.\n",
    "First, we load the output into a python dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'inconsistencies.json'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-510acc777805>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0minconsistencies\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"inconsistencies.json\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'inconsistencies.json'"
     ]
    }
   ],
   "source": [
    "import json\n",
    "inconsistencies = json.load(open(\"inconsistencies.json\", 'r'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The format of the dictionary is the following:\n",
    "```json\n",
    "{\n",
    "    \"error\" : { \"data_node\" : [  [ group of records (with facets) with one instance id ],\n",
    "                                 [ group of records (with facets) with another instance id ],\n",
    "                                 ... \n",
    "                              ],\n",
    "                ...\n",
    "              },\n",
    "    ...\n",
    "\n",
    "}\n",
    "```\n",
    "so the following call will retrieve the instance id of the first dataset record which has no original record from the data node aims3.llnl.gov:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_instance_id = inconsistencies[\"No original record:\"][\"aims3.llnl.gov\"][0][0][\"instance_id\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason for both index 0s above is to get to the first group, and then to get to the first record. A full list of facets for each record is as follows:  \n",
    "`instance_id`: the instance_id of the record  \n",
    "`number_of_files`: how many files are part of this record  \n",
    "`_timestamp`: the time this record was created  \n",
    "`data_node`: the data node corresponding to the record  \n",
    "`replica`: either true or false, indicates whether record is an original or replica  \n",
    "`institution_id`: the institution corresponding to the record  \n",
    "`latest`: either true or false, indicates whether or not record is latest version of the dataset  \n",
    "`version`: the version of the dataset record  \n",
    "`retracted`: either true or false, indicates whether or not the record has been retracted  \n",
    "`id`: the id of the dataset, different format from instance_id  \n",
    "`activity_drs`: the first (or only) activity id of the dataset  \n",
    "`activity_id`: the list of activity ids of the dataset  \n",
    "`source_id`: the source id of the dataset  \n",
    "`experiment_id`: the experiment id of the dataset  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full list of errors is as follows:  \n",
    "`\"No original record:\"` indicates a batch of replicas with no original record, a sign that the original was deleted improperly  \n",
    "`\"Inconsistent number of files (esgf replica issue):\"` the facet `number_of_files` does not have the same value for at least one record in the dataset, but the timestamp and replica status indicates the record was changed by ESGF  \n",
    "`\"Inconsistent number of files (client issue):\"` same as above, except that the timestamp and replica status indicates that the record was changed by the client  \n",
    "`\"Original record not latest version:\"` the original record in a group was marked not latest  \n",
    "`\"Original record retracted:\"` the original record was marked retracted  \n",
    "`\"Duplicate records:\"` flags all records which are exact duplicates of one another\n",
    "`\"Replica retracted, original not retracted:\"` indicates a group where the replicas were retracted but the original record was not  \n",
    "`\"Failed activity check:\"` record has an invalid activity id according to CMIP6 CMOR table  \n",
    "`\"Failed experiment_id check:\"` record's experiment id does not correspond to its activity id according to CMIP6 CMOR table  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a final example, we will loop through the dictionary and write all the records flagged with a certain data node to a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_NODE = \"aims3.llnl.gov\"\n",
    "fp = open(\"file.txt\", \"w\")\n",
    "for err in inconsistencies.keys():\n",
    "    for node in inconsistencies[err].keys():\n",
    "        if node == DATA_NODE:\n",
    "            for recs in inconsistencies[err][node]:\n",
    "                fp.write(json.dumps(recs))\n",
    "fp.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}