pyagsapi dev planning

[volcan01010]

Tasks

• Refactor backend to use AGS4.check_file() rather than the CLI (JSON response)
• Refactor backend to use AGS4.check_file() rather than the CLI (text response)
• Add dictionary selection as query parameter to API
• Refactor backend to handle dictionary selection
• Refactor backend convert function to use library instead of CLI
• Refactor API to handle updated convert
• Add integration tests to other routes and formats
• Add endpoint to do BGS data validation steps
• BGS rule for coordinates in LOCA
• BGS rule for SAMP integrity
• BGS rule for groups and headings
• Flag and reject AGS3 (test/files/real/19684.ags)
• Catch mismatched columns (test/files/real/JohnStPrimarySchool.ags)
• In LOCA - TYPE for LOCA_NATE/NATN/LOCX/LOCY should be 2DP
• Add file summary below metadata:

• 1 data rows in PROJ group.
• 2 data rows in LOCA group.
• 24 groups identified in file
• DICT is present for user defined groups / DICT is not present - there are no user defined groups
  • Check covered by AGS check rule 9
• FILE group is not present but is only required if submission includes non-AGS file

Bonus -

• Make root URL user configurable (via environment variable)
• Fix UnboundLocalError bug in upstream repo
• Make logging messages from bgs.py and ags.py print to console
• Add endpoint to automatically clean files - dataclean
• Report / fix "convert asks for user input" bug in upstream repo - https://gitlab.com/ags-data-format-wg/ags-python-library/-/issues/22 ADD COMMENTS
• Report/fix sys.exit() calls in upstream library
• Submit MR to fix the above
• Add 50 Mb file size limit- Done via k8s NGINX config
• Add to_html function to backend
• Add to_html parameter to API

Questions

• Do we just remove v1 code, given than there are no real users yet?

Details

API specification

/api/v1/?validatedata?

POST: Binary upload of file
Response: {"result": "true"} ?
Errors: 400 - some problem with uploaded file

/api/v1/?clean?

POST: Binary upload of file
Response: {"result": "true"} ?
Errors: 400 - some problem with uploaded file

Backend function specification

def validatedata(filename: Path) -> Dict:
    # return Dictionary / JSON with data errors
    pass

Schema

class Validation(BaseModel):
    filename: str = Field(..., example="example.ags")
    filesize: str = Field(None, example="1 kB")
    checker: str = Field(None, example="python_ags4 v0.3.6")
    dictionary: str = Field(None, example="Standard_dictionary_v4_1.ags")
    time: str = Field(None, example="2021-08-18 09:23:29")
    message: str = Field(None, example="7 error(s) found in file!")
    results: Dict = Field(None, example="")

from:

{'Rule 2a': [{'line': 1,
   'group': '',
   'desc': 'Is not terminated by <CR> and <LF> characters.'}],
 'Rule 3': [{'line': 1,
   'group': '',
   'desc': 'Does not start with a valid data descriptor.'}],
 'Rule 5': [{'line': 1,
   'group': '',
   'desc': 'Contains fields that are not enclosed in double quotes.'}],
 'Rule 13': [{'line': '-', 'group': 'PROJ', 'desc': 'PROJ table not found.'}],
 'Rule 14': [{'line': '-', 'group': 'TRAN', 'desc': 'TRAN table not found.'}],
 'Rule 15': [{'line': '-', 'group': 'UNIT', 'desc': 'UNIT table not found.'}],
 'Rule 17': [{'line': '-', 'group': 'TYPE', 'desc': 'TYPE table not found.'}],
 'Metadata': [{'line': 'File Name', 'group': '', 'desc': 'nonsense.ags'},
  {'line': 'File Size', 'group': '', 'desc': '0 kB'},
  {'line': 'Checker', 'group': '', 'desc': 'python_ags4 v0.3.6'},
  {'line': 'Dictionary', 'group': '', 'desc': 'Standard_dictionary_v4_1.ags'},
  {'line': 'Time (UTC)', 'group': '', 'desc': '2021-08-23 13:19:46'}]}

Discussion

• Default to creating JSON response. For text response, use Jinja2 to render a text version.

[KoalaGeo]

Closing as ongoing tasks moved to #59