SDP v4

Work done one the SAEON Data Portal over the 2023FY (1st April 2022 - 1st April 2023)

UI Updates

• Significant changes to the /home page - new layout, some quick stats, and a nice map background
• Significant changes to the /records page
  • Made records more compact and reorganized information layout
  • Added cute little animation for displaying toolbar
  • Updated default visibility of map and filters for a more professional look and feel
• Updated map to use esri basemap (/records, /record)
• Refactored layout of /login and other flat page (i.e. about, license, etc) to have content that filled the vh and sit correctly on the footer
• Updated content navigation component used on /access, /data-lists, and /usage page (authorized-only pages) to support full screen content
• Updated /access, /data-lists, and /usage pages to render content with a sidebar flush against the right side of screen
• Updated /list/records and /list/records/:id pages to show title and descriptions bespoke to each list. Also these pages now include links back to the originating pages so that users don't get confused if they see the search / record served via the list HTML page instead of the default HTML page
• Updated download notice for thredds/dirisa server - these apps are served via HTTP and not HTTPS, so it's difficult to proxy the download. Instead users are shown the link they must go to manually to trigger the download
• Improved mobile friendliness generally

Search updates

• Improved search matching by anding results instead of oring results
• Fixed Lists feature - a list created from a text search (and other fields visible on the UI gets hidden on list share. This makes it seem like a list is a standalone catalogue.
• Addressed a search bug where facet count and search result count sometimes didn't match. This still occurs in places, but it's because the same term is found in different subjectTypes.
• Updated search box on /render to allow for slightly slower typing (debounce window increased)
• Fixed bug where clicking a subject on /records/:id resulted in mismatched filter state vs filter UI

Reporting capabilities

• Refactored logging to be more focused (removed mousemove, query, click logs. Added appRender, authentication logs)
• Major updates the /usage page in terms of showing a variety of charts and tables, and also giving access to the underlying data via a CSV export
• Researched options for proxying Google analytics via a the server instead of allows CORS requests from the client. Most trackers installed as Browser addons will stop Google Analytics from working. But Google Analytics is not designed to work via proxy and this couldn't be implemented in a useful/sustainable way (it would have required downloading the Google analytics JavaScript, changing it, and then manually keeping it up-to-date). It may be a nice project in the future to compare in-house logs with google logs to see what the effect is of tracker blockers on Google Analytics for general use. The bespoke logging system can capture events that google can't (downloads, authentication, authorization) and logs can be configured for both the server and the client. It's also possible to do 'feature engineering' on the client - i.e. deriving events from underlying logs such as mousemoves, clicks, etc. and saving those in the database.

Deployment

• Simplified deployment of the app to be a single Docker image (this supports a lot of the other items in this milestone)
• Moved the data portal deployment to new Docker Swarm cluster on new VMs running Ubuntu 22

SEO optimizations

• Basic HTML templating allows for individualized keywords, title, and description in the HTML head of the list page (for lists, and records of lists).
• Updated /robots.txt route to generate this page dynamically
• Added a tool to the application to build sitemap files every night, that are then included in the robots.txt file
• Updated the /render route in the application to redirect to /list. This should be updated in all the various places in SAEON WordPress sites where catalogue.saeon.ac.za/render.... urls are used (this may effect SEO)

Code changes

• Significantly refactored a large portion of the code base to be generally cleaner

Integration with ODP

• Added a CLI to the API for running database migrations, generating sitemaps, and running the metadata integration with the ODP
• Updated ODP integration to allow for withdrawn published records without deleting the entire ES index
• Updated ODP integration so that values are 'upserted' (either updated or inserted if not exists). This means that integration runs no longer clear the ES index, so users will never see too few records even while an integration is running
• Integration with ODP now is batched so can handle any amount of records. Although the entire SAEON catalogue is still processed. This is fixable with V2 of the ODP

ODP integration

• Updated the ODP integration to support new work in the ODP v2 setup. This has not been merged yet, and is waiting for the production deployment of the ODP v2

General changes

• Removed IE 11 support (3rd party libraries are dropping support for IE 11 now)
• Updated 3rd party dependencies
• Updated client filters configuration to correctly reflect MIMS requirements (except that changing logos/styling is not supported yet)
• Many other bug fixes, etc.
• Added a button link to submit metadata on all pages (except lists)

Documentation

• Updated documentation to reflect recent work