Merge pull request #101 from BritishGeologicalSurvey/GWBV-endpoint

Add ags_log endpoint

.github/workflows/lint_and_test.yml

@@ -35,12 +35,12 @@ jobs:
     - name: Lint app/ with flake8
       run: |
         # exit-zero treats all errors as warnings. 
-        flake8 app/ --exit-zero 
+        flake8 app/ 
 
     - name: Lint test/ with flake8
       run: |
         # exit-zero treats all errors as warnings. 
-        flake8 test/ --exit-zero  
+        flake8 test/  
 
     - name: Test with pytest
       run: |

Dockerfile

@@ -4,6 +4,5 @@ FROM tiangolo/uvicorn-gunicorn-fastapi:python3.11-slim-2023-02-20
 COPY requirements.txt .
 RUN pip install -r requirements.txt
 
-
 RUN rm -rf /app/*
 COPY ./app /app/app

README.md

@@ -21,11 +21,11 @@ A HTTP API for the [AGS Python library](https://gitlab.com/ags-data-format-wg/ag
 It can:
 
 - Validate AGS files to v4.x of the AGS data format standard
+- Validate AGS files for submission to the [National Geoscience Data Center (NGDC)](http://transfer.bgs.ac.uk/ingestion)
 - Convert between AGS format files and spreadsheet files `.ags` <-> `.xlsx`
+- Download PDF logs of existing files within the National Geoscience Data Centre
 
-Additionally .ags files can be validated for submission to the [National Geoscience Data Center (NGDC)](http://transfer.bgs.ac.uk/ingestion)
-
-It is built on the FastAPI framework, using the official FastAPI docker image as it's base.
+It is built on the FastAPI framework, using the official FastAPI Docker image.
 
 The core Python API provides the functionality to validate and convert AGS geotechnical data. From here, standard Python web frameworks like Uvicorn and Starlette provide the web API/wrapper atop the core Python API.
 
@@ -55,12 +55,11 @@ This will ensure that all references to `self` in responses, and all Swagger and
 
 ### From Source
 
-pyagsapi runs on Python 3.
+pyagsapi runs on Python >= 3.11.
 
-```python
+```bash
 python -m venv pyagsapi
-cd pyagsapi
-. bin/activate
+source pyagsapi/bin/activate
 git clone https://github.com/BritishGeologicalSurvey/pyagsapi.git
 cd pyagsapi
 pip install -r requirements.txt
@@ -77,14 +76,7 @@ Please raise any feature requests, issues or pull requests against this reposito
 
 AGS Validator is written in Python and based on the [FastAPI](https://fastapi.tiangolo.com/) framework. It runs on the [Uvicorn](https://www.uvicorn.org/) ASGI server.
 
-Use the following commands to run the API locally:
-
-```bash
-git clone https://github.com/BritishGeologicalSurvey/pyagsapi
-cd pyagsapi
-pip install -r requirements.txt
-uvicorn app.main:app --reload
-```
+Use the instructions above to run the API locally.
 
 By default, the API is served at [http://localhost:8000](http://localhost:8000).
 

app/errors.py

@@ -7,7 +7,8 @@
 # Define error responses
 error_responses = {
     status.HTTP_404_NOT_FOUND: {"model": ErrorResponse},
-    status.HTTP_422_UNPROCESSABLE_ENTITY: {"model": ErrorResponse}
+    status.HTTP_422_UNPROCESSABLE_ENTITY: {"model": ErrorResponse},
+    status.HTTP_500_INTERNAL_SERVER_ERROR: {"model": ErrorResponse}
 }
 
 

app/main.py

@@ -83,9 +83,20 @@ def custom_openapi():
         return app.openapi_schema
     openapi_schema = get_openapi(
         title="pyagsapi - AGS File Utilities Tools and API",
-        version="4.0.0",
+        version="4.5.1",
         description=("The API performs schema validation, data validation and conversion of your AGS files. "
+                     "It also exports a graphical log from AGS data held by NGDC. "
                      "Schema validation and conversion uses https://gitlab.com/ags-data-format-wg/ags-python-library"),
+        terms_of_service="https://www.bgs.ac.uk/legal-and-policy/terms-of-use/",
+        contact={
+            "name": "BGS Enquiries",
+            "url": "https://www.bgs.ac.uk/about-bgs/contact-us/",
+            "email": "[email protected]",
+        },
+        license_info={
+            "name": "Open Government Licence v3",
+            "url": "https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/",
+        },
         routes=app.routes,
     )
     openapi_schema["info"]["x-logo"] = {

app/routes.py

@@ -1,18 +1,24 @@
 import tempfile
 import shutil
+import requests
 
 from enum import StrEnum
 from pathlib import Path
 from typing import List
 
-from fastapi import APIRouter, BackgroundTasks, File, Form, Request, UploadFile
+from fastapi import APIRouter, BackgroundTasks, File, Form, Query, Request, UploadFile, Response
 from fastapi.responses import FileResponse, StreamingResponse
+from fastapi.exceptions import HTTPException
+
+from requests.exceptions import Timeout, ConnectionError, HTTPError
 
 from app import conversion, validation
 from app.checkers import check_ags, check_bgs
 from app.errors import error_responses, InvalidPayloadError
 from app.schemas import ValidationResponse
 
+BOREHOLE_VIEWER_URL = "https://gwbv.bgs.ac.uk/GWBV/viewborehole?loca_id={bgs_loca_id}"
+
 router = APIRouter()
 
 log_responses = dict(error_responses)
@@ -25,6 +31,11 @@
     "content": {"application/x-zip-compressed": {}},
     "description": "Return a zip containing successfully converted files and log file"}
 
+pdf_responses = dict(error_responses)
+pdf_responses['200'] = {
+    "content": {"application/pdf": {}},
+    "description": "Return a graphical log of AGS data in .PDF format"}
+
 
 # Enum for search logic
 class Format(StrEnum):
@@ -46,6 +57,12 @@ class Checker(StrEnum):
     bgs = "bgs"
 
 
+# Enum for pdf response type logic
+class ResponseType(StrEnum):
+    attachment = "attachment"
+    inline = "inline"
+
+
 checker_functions = {
     Checker.ags: check_ags,
     Checker.bgs: check_bgs,
@@ -89,10 +106,27 @@ class Checker(StrEnum):
                  'This option is ignored when converting to AGS.'),
 )
 
+ags_log_query = Query(
+    ...,
+    title="BGS LOCA ID",
+    description="BGS LOCA ID",
+    example="20190430093402523419",
+)
+
+response_type_query = Query(
+    default=ResponseType.inline,
+    title='PDF Response Type',
+    description='PDF response type: inline or attachment',
+)
+
 
 @router.post("/validate/",
+             tags=["validate"],
              response_model=ValidationResponse,
-             responses=log_responses)
+             responses=log_responses,
+             summary="Validate AGS4 File(s)",
+             description=("Validate an AGS4 file to the AGS File Format v4.x rules and the NGDC data"
+                          " submission requirements. Uses the Offical AGS4 Python Library."))
 async def validate(background_tasks: BackgroundTasks,
                    files: List[UploadFile] = validation_file,
                    std_dictionary: Dictionary = dictionary_form,
@@ -135,8 +169,12 @@ async def validate(background_tasks: BackgroundTasks,
 
 
 @router.post("/convert/",
+             tags=["convert"],
              response_class=StreamingResponse,
-             responses=zip_responses)
+             responses=zip_responses,
+             summary="Convert files between .ags and .xlsx format",
+             description=("Convert files between .ags and .xlsx format. Option to"
+                          " sort worksheets in .xlsx file in alphabetical order."))
 async def convert(background_tasks: BackgroundTasks,
                   files: List[UploadFile] = conversion_file,
                   sort_tables: bool = sort_tables_form,
@@ -179,3 +217,36 @@ def prepare_validation_response(request, data):
         'data': data,
     }
     return ValidationResponse(**response_data, media_type="application/json")
+
+
+@router.get("/ags_log/",
+            tags=["ags_log"],
+            response_class=Response,
+            responses=pdf_responses,
+            summary="Generate Graphical Log",
+            description="Generate a graphical log (.pdf) from AGS data held by the National Geoscience Data Centre.")
+def get_ags_log(bgs_loca_id: int = ags_log_query,
+                response_type: ResponseType = response_type_query):
+    url = BOREHOLE_VIEWER_URL.format(bgs_loca_id=bgs_loca_id)
+
+    try:
+        response = requests.get(url, timeout=10)
+    except (Timeout, ConnectionError):
+        raise HTTPException(status_code=500,
+                            detail="The borehole generator could not be reached.  Please try again later.")
+
+    try:
+        response.raise_for_status()
+    except HTTPError:
+        if response.status_code == 404:
+            raise HTTPException(status_code=404,
+                                detail=f"Failed to retrieve borehole {bgs_loca_id}. "
+                                "It may not exist or may be confidential")
+        else:
+            raise HTTPException(status_code=500,
+                                detail="The borehole generator returned an error.")
+
+    filename = f"{bgs_loca_id}_log.pdf"
+    headers = {'Content-Disposition': f'{response_type.value}; filename="{filename}"'}
+
+    return Response(response.content, headers=headers, media_type='application/pdf')

app/static/js/htmlView.js

@@ -20,7 +20,8 @@ agsHtml.injectResultPopup=function(){
     xhtml=xhtml + '</div>';
     $("#validator").after(xhtml);
     agsHtml.hideResultPopup();
-    $("main input:submit").prop("disabled",true);
+    $("#validator input:submit").prop("disabled",true);
+    $("#converter input:submit").prop("disabled",true);
     return true;
     };
 

app/templates/_base.html

@@ -2,6 +2,9 @@
 <html lang="en">
   <head>
     <title>AGS File Utilities Tool and API</title>
+	<meta name="description" content="This tool and associated API allow AGS schema validation, AGS data validation and conversion of your AGS files. The tool is supporting the development of new national data store of standardised AGS data of the subsurface that can be consulted and accessed to support decision making processes in industry and project planning. " />
+	<meta name="keywords" content="BRITISH,GEOLOGY,EARTH,SCIENCE,GEOLOGICAL,SURVEY,GEOSCIENCE,OPEN, AGS, depositing, geotechnical, data, ac, validator, ags validator" />
+
     <meta charset="UTF-8">
     <meta name="viewport" content="width=device-width, initial-scale=1" />
 

app/templates/landing_page.html

@@ -11,6 +11,7 @@ <h2>Tools</h2>
     <ul>
         <li><a href="#validator">AGS Schema & Data Validator</a></li>
         <li><a href="#converter">File Conversion .ags &#8596; .xlsx</a></li>
+        <li><a href="#ags_log">AGS Log Viewer</a></li>
         <li><a href="#openapi">API Documentation</a></li>
     </ul>
 </section>
@@ -156,6 +157,32 @@ <h2>AGS Converter</h2>
 <br>
 <br>
 <br>
+<section id="ags_log">
+  <h2>AGS Log Viewer</h2>
+    <p>Submit BGS LOCA ID to view log from NGDC held AGS data. Find BGS LOCA ID on <a href="https://mapapps2.bgs.ac.uk/geoindex/home.html?layer=AGSBoreholes">GeoIndex Web Viewer</a> or by querying the <a href="https://ogcapi.bgs.ac.uk/collections/onshoreboreholeindex/items">BGS OGCAPI-Features Server</a></p>
+    <br>
+  <br>
+  <form action="/ags_log/" method="get" id="logForm">
+    <fieldset>
+      <legend>Enter BGS LOCA ID</legend>
+      <input type="text" id="bgs_loca_id" name="bgs_loca_id">
+      <br>
+      <br>
+      <fieldset>
+        <legend>Select Response Type</legend>
+        <input type="radio" id="attachment_response" name="response_type" value="attachment" checked>
+        <label for="attachment_response"> Download Attachment</label><br>
+        <input type="radio" id="inline_response" name="response_type" value="inline" checked>
+        <label for="inline_response"> View in Browser</label><br>
+      </fieldset>
+      <br>
+      <input type="submit" class="submitBtn" value="Get Log" >
+    </fieldset>
+  </form>
+</section>
+<br>
+<br>
+<br>
 <section id="openapi">
       <h2>API Definition</h2>
       <p>

requirements.txt

@@ -9,6 +9,7 @@ Fiona==1.9.1
 Shapely==2.0.1
 pyproj==3.4.1
 geopandas==0.12.2
+requests==2.28.2
 # These libraries are already in FastAPI container but need updated
 fastapi==0.92.0
 uvicorn==0.20.0