Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New Date Uploaded for Species Lists #236

Open
rosemaryjoconnor opened this issue Feb 13, 2023 · 11 comments
Open

New Date Uploaded for Species Lists #236

rosemaryjoconnor opened this issue Feb 13, 2023 · 11 comments
Assignees
@charvolant @adam-collins @rosemaryjoconnor
Labels
enhancement
Milestone
v4.0.6

Comments

@rosemaryjoconnor
Copy link

A date indicating last upload date for a species list is required.

Species lists have a lastUpdated field, however this field gets updated whenever any information associated with the list is updated, for example information in List Info. As a result we have no way of knowing the date of the most recent upload of a species list.

This is particularly problematic for Conservation and Sensitive species lists in terms of managing the most recent updates.
@sat01a sat01a added this to the 4.0.6 milestone Jun 5, 2023
@adam-collins
Copy link
Contributor

I believe this should remain as-is. List Info will update the lastUpdated date because it determines how the list is used elsewhere in the ALA. For administrative lists there is often a collectory entry. This page should be kept up to date and should list the source information including the date.

With the fixes that prevent lists reverting to default List Info when new content is uploaded this should be less of an issue.

@rosemaryjoconnor
Copy link
Author

rosemaryjoconnor commented Jun 7, 2023 via email

@peggynewman
Copy link

peggynewman commented Jun 7, 2023 •
edited
Loading

List Info will update the lastUpdated date because it determines how the list is used elsewhere in the ALA. For administrative lists there is often a collectory entry. This page should be kept up to date and should list the source information including the date.

Could you elaborate @adam-collins ?

It looks like we have a few dates (examples from EPBC list)

  • List Info Date Submitted - irrelevant to this discussion (in the example 2015-04-05)
  • List Info Last Update - updates with either a change to List Info or a data refresh (in the example 2023-06-08 - a ListInfo change)
  • List Data Update - when list data was last modified - this is reflected in Last Update, but overwritten if the List Info is changed. Currently there is no way to reliably find this information.
  • Collectory Metadata last updated - only reflects changes to the metadata record (Description field only, not List Info) only visible to users when clicking through to the Collectory from lists. (current example 2023-06-01 14:27:19.0 - my guess is that this is when the Description field was changed).

I think that you're proposing that if we want to retain the date that data was loaded into the list, then we should document it manually in the Description field. Is that right? That will be plain text information. Can we not have a Data Updated field so that it can be more useful?

@adam-collins
Copy link
Contributor

Off topic, I think an audit trail would be more useful.

On topic, I require more details. I think I mistakenly bundled 'metadata updates' into a 'do not track changes' category. I still think the collectory entry should include the information, even if plain text.

I'm not sure the level of detail would be sufficient if adding List Data Update in addition to changing Last Update to Last Metadata Update. It would be triggered by:

  • A new list of species uploaded to replace the existing list.
  • A single row removed. Through the UI.
  • A single row altered to fix the name matching. Through the UI.
  • A single row added. Through the UI.
  • A single value in any of the additional columns is changed. Through the UI.

In this example is it appropriate to update Last Metadata Update when List Data Update is changed?

How would a new names index be recorded? Would we need a Last Name Matched Update?

I note that the collectory page includes the plain text This list was first uploaded by Rosemary O'Connor on the Thu Jun 01 14:27:19 AEST 2023.It contains [totalRecords:2011, successfulItems:2011] taxa. rather than a log of dates when changes were made (only my preference, and probably too late to start now).

Looking again at the collectory there appear to be non-text fields in the Data mobilisation section. Data currency (for List Data Update) and Last checked (for Last Name Matched Update).

In summary, and assuming this issue moves forward:

  1. Rename Last Update to Last Metadata Update.
  2. Add List Data Update to be updated by the events listed above.

@peggynewman
Copy link

Thank you. Off topic - a major feature of the lists redevelopment will be an audit trail or version history of some sort.

For now though, I would be very happy if the 'List Data Update' field reflected any of those 5 changes, whether they be through the UI or not.

However, perhaps you're asking - if there's a new names index and we do a full rematch on lists, should that update the List Data Update. I would say no. Maybe that should be a Last Name Matched Update. That makes sense to me, because there's a difference between those inputs.

There is no change history in the collectory. Those Data currency and Last checked dates are completely incorrect across all data resources. I need to raise an issue on those because they are very embarrassing (ie, we don't "check" or poll anything except perhaps what's we might harvest from an IPT). I don't want to repurpose them for a lists use case. It's important that we have field names that reflect their purpose.

I hope this issue moves forward, it will be useful for our state list stakeholders, so I would like to agree and add:

  1. Rename Last Update to Last Metadata Update Date
  2. Add Last List Data Update
  3. Add Last Name Match Update

Is that possible?

@adam-collins
Copy link
Contributor

Doug had done some previous work adding last uploaded.

Also added last matched.

@adam-collins
Copy link
Contributor

#250

@rosemaryjoconnor
Copy link
Author

rosemaryjoconnor commented Jun 11, 2023 via email

@adam-collins
Copy link
Contributor

Doug's work is integrated into this release. It included the beginnings of the last_uploaded field, Loose Name Search and Name Search Style.

I don't know what the name search is so will need someone to review lists-test for those changes to determine if more work is required.

@rosemaryjoconnor
Copy link
Author

rosemaryjoconnor commented Jun 12, 2023 via email

@adam-collins
Copy link
Contributor

in production

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@charvolant charvolant

@adam-collins adam-collins

@rosemaryjoconnor rosemaryjoconnor

Labels
enhancement
Projects
None yet
Milestone
v4.0.6
Development

No branches or pull requests
5 participants
@peggynewman @charvolant @sat01a @adam-collins @rosemaryjoconnor