internal_quality_analysis-github.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "### LIBRARIES"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "cc56e56572a75249"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Deal with data\n",
    "import pandas as pd\n",
    "import json\n",
    "from glob import glob\n",
    "\n",
    "# Deal with visualization\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Deal with time\n",
    "import datetime"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-06-12T20:09:03.884001Z",
     "start_time": "2024-06-12T20:09:03.878379Z"
    }
   },
   "id": "a3bcddb987f2383e",
   "execution_count": 2
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Github API\n",
    "\n",
    "A API do GitHub utiliza principalmente o protocolo HTTP, onde as operações são realizadas através de URLs e métodos HTTP, como GET, POST, PUT e DELETE. Os dados são geralmente retornados em formato JSON, tornando-o fácil de processar e interpretar."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "8b4e0d6b878f9fc2"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "data = glob('./analytics-raw-data/GitHub_API-fga-eps-mds-*.json')\n",
    "# print(data)"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "bfc238b8cb27fee1",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Unmarshall json"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "ffa8cabeda8ab48c"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "def unmarshall(json_path: str) -> dict:\n",
    "    with open(json_path) as json_file:\n",
    "        json_obj = json.load(json_file)\n",
    "    return json_obj"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "26a5d6d115efe1a6",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Cálculo dos dados\n",
    "\n",
    "- Carrega dados dos arquivo JSON e armazena-os na variável data.\n",
    "- Processa os dados do JSON para calcular e obter informações relevantes sobre os tempos de feedback de cada execução de workflow, separando-os em listas de feedback_times, authors_feedback_times, e ci_feedback_times.\n",
    "- Calcula o tempo médio de feedback (ci_feedback_time) considerando todas as execuções de workflow.\n",
    "- Calcula o tempo médio de feedback de cada autor e armazena essas informações em authors_avg_feedback_times.\n",
    "- Processa novamente os dados do JSON para contar o número de execuções bem-sucedidas (success_count) e falhas (failure_count) dos workflows."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "7d100c7f5283260c"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "tabela_total = pd.DataFrame()\n",
    "\n",
    "# Processar os dados do JSON e obter as informações relevantes\n",
    "success_count = 0\n",
    "failure_count = 0\n",
    "\n",
    "for json_path in data:\n",
    "    dados = unmarshall(json_path)\n",
    "    # Create a list to store the data for the table\n",
    "    table_data = []\n",
    "    # Processar os dados do JSON e obter as informações relevantes\n",
    "    for run in dados[\"workflow_runs\"]:\n",
    "        updated_at = datetime.datetime.strptime(run[\"updated_at\"], \"%Y-%m-%dT%H:%M:%SZ\")\n",
    "        created_at = datetime.datetime.strptime(run[\"created_at\"], \"%Y-%m-%dT%H:%M:%SZ\")\n",
    "        feedback_time = (updated_at - created_at).total_seconds()\n",
    "        \n",
    "        # informações relevantes\n",
    "        workflow_id = run[\"id\"]\n",
    "        conclusion = run[\"conclusion\"]\n",
    "        author = run[\"actor\"][\"login\"]\n",
    "        \n",
    "        # Adiciona os dados na tabela\n",
    "        table_data.append({\n",
    "            \"Workflow_run ID\": workflow_id,\n",
    "            \"Conclusion\": conclusion,\n",
    "            \"Author\": author,\n",
    "            \"Created at\": created_at,\n",
    "            \"Updated at\": updated_at,\n",
    "            \"Feedback Time\": feedback_time\n",
    "        })\n",
    "        \n",
    "        \n",
    "    # Cria um dataframe com o pandas\n",
    "    df = pd.DataFrame(table_data)\n",
    "\n",
    "    \n",
    "    # table_data_global.append(df)\n",
    "    \n",
    "    # Processar os dados do JSON e obter as informações relevantes\n",
    "    feedback_times = []\n",
    "    authors_feedback_times = {}\n",
    "    ci_feedback_times = []\n",
    "\n",
    "    for run in df.index:\n",
    "        updated_at = df[\"Updated at\"][run]\n",
    "        created_at = df[\"Created at\"][run]\n",
    "        feedback_time = (updated_at - created_at).total_seconds()\n",
    "        feedback_times.append(feedback_time)\n",
    "        \n",
    "    # Calcular o CI FeedBack Time\n",
    "    ci_feedback_time = sum(feedback_times) / len(feedback_times)\n",
    "    df[\"Ci Feedback Time\"] = ci_feedback_time\n",
    "    \n",
    "    # Tabela com todas as linhas\n",
    "    tabela_total = pd.concat([tabela_total, df])\n",
    "    # print('Tabela por dia')\n",
    "    # display(df)\n",
    "\n",
    "    for run in dados[\"workflow_runs\"]:\n",
    "        conclusion = run[\"conclusion\"]\n",
    "        if conclusion == \"success\":\n",
    "            success_count += 1\n",
    "        elif conclusion == \"failure\":\n",
    "            failure_count += 1\n",
    "                \n",
    "print('Tabela com todas as linhas')\n",
    "display(tabela_total)\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "1c0aedb7ec427cd6",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Gerar gráficos\n",
    "\n",
    "- Gráfico de linhas: Mostra o \"CI FeedBack Time\" (tempo de feedback do CI) em relação à \"Data de Atualização\" das execuções de workflow.\n",
    "- Gráfico de barras simples: Exibe o \"Tempo Médio de Feedback do CI\", representando o desempenho médio geral do CI.\n",
    "- Gráfico de pizza: Mostra a \"Quantidade de Workflows com Sucesso e Falha\" em relação ao total de workflows executados, indicando a proporção de workflows que tiveram sucesso e falha."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "caf5e0b5aa86287a"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Gerar gráfico com o CI FeedBack Time por data de atualização\n",
    "\n",
    "temp = tabela_total[['Created at', 'Ci Feedback Time']]\n",
    "dates = temp.groupby(['Created at']).mean().reset_index()  # Calcula a média dos valores por data\n",
    "dates['Created at'] = pd.to_datetime(dates['Created at']) \n",
    "print(dates)\n",
    "\n",
    "\n",
    "# dates = [datetime.datetime.strptime(run[\"updated_at\"], \"%Y-%m-%dT%H:%M:%SZ\") for run in data[\"workflow_runs\"]]\n",
    "plt.figure(figsize=(20, 10))\n",
    "plt.plot(range(len(dates)), dates['Ci Feedback Time'], marker='o', color='b', label='Feedback Time')\n",
    "plt.xlabel(\"Created at\")\n",
    "plt.ylabel(\"CI FeedBack Time\")\n",
    "plt.title(\"CI FeedBack Time por Data de Criação\")\n",
    "plt.xticks(range(len(dates)), dates['Created at'].dt.strftime('%Y-%m-%d'), rotation=45)\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.tight_layout()  # Ajusta a disposição dos elementos para evitar sobreposições\n",
    "plt.show()"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "fc0acb669d16ee73",
   "execution_count": null
  },
  {
   "metadata": {},
   "cell_type": "code",
   "outputs": [],
   "execution_count": null,
   "source": [
    "plt.figure(figsize=(20, 10))\n",
    "bars =  plt.bar(range(len(dates)), dates['Ci Feedback Time'], color='salmon', label='Feedback Time')\n",
    "plt.xlabel(\"Created at\")\n",
    "plt.ylabel(\"CI FeedBack Time\")\n",
    "plt.title(\"CI FeedBack Time por Data de Criação\")\n",
    "plt.xticks(range(len(dates)), dates['Created at'].dt.strftime('%Y-%m-%d'), rotation=45)  # Exibe as datas formatadas no eixo X\n",
    "plt.legend()\n",
    "# Adiciona as labels em cada barra\n",
    "for bar in bars:\n",
    "    height = bar.get_height()\n",
    "    plt.text(bar.get_x() + bar.get_width() / 2, height, str(height), ha='center', va='bottom')\n",
    "\n",
    "plt.tight_layout()  # Ajusta a disposição dos elementos para evitar sobreposições\n",
    "plt.show()\n",
    "\n",
    "# # Gerar gráfico com o tempo médio do CI\n",
    "# plt.figure(figsize=(20, 10))\n",
    "# plt.bar([\"CI\"], [ci_feedback_time], color='salmon')\n",
    "# plt.xlabel(\"CI\")\n",
    "# plt.ylabel(\"Tempo Médio de Feedback\")\n",
    "# plt.title(\"Tempo Médio de Feedback do CI\")\n",
    "# plt.text(0, ci_feedback_time, f'{round(ci_feedback_time, 2)} segundos', ha='center', va='bottom', fontweight='bold')\n",
    "# plt.tight_layout()\n",
    "# plt.show()"
   ],
   "id": "8f74ecd7e0bcd3bb"
  },
  {
   "metadata": {},
   "cell_type": "code",
   "outputs": [],
   "execution_count": null,
   "source": [
    "# Gerar gráfico com a quantidade de workflows com sucesso e falha\n",
    "labels = [\"Success\", \"Failure\"]\n",
    "sizes = [success_count, failure_count]\n",
    "plt.figure(figsize=(20, 10))\n",
    "colors = ['lightgreen', 'crimson']\n",
    "plt.pie(sizes, labels=labels, autopct='%1.1f%%', colors=colors)\n",
    "plt.title(\"Quantidade de Workflows_runs com Sucesso e Falha\")\n",
    "plt.legend(labels, loc='best')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "id": "cebbf6eb1bf5cf89"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}