Request for new fields to index and expose in search and download

[MortenHofft]

There is quite a lot of ideas for fields to add to the index. So that users can search and download by adding these new filters.
I've collected them all here (at least the ones I could find)

From #400

• earliestEonOrLowestEonothem
• latestEonOrHighestEonothem
• earliestEraOrLowestErathem
• latestEraOrHighestErathem
• earliestPeriodOrLowestSystem
• latestPeriodOrHighestSystem
• earliestEpochOrLowestSeries
• latestEpochOrHighestSeries
• earliestAgeOrLowestStage
• latestAgeOrHighestStage
• lowestBiostratigraphicZone
• highestBiostratigraphicZone
• group
• formation
• member
• bed

From #7

• acceptedKey

#182

• verbatimTaxonKey

From #425

• gbifRegion
• publishedByGbifRegion

From #515

• fieldNumber (only in index as verbatim blob) Request for index on fieldNumber #514
• preparations (currently indexed as keywords) Please add download and filter support for 'preparations' #634
• sex (currently indexed as keywords)
• startDayOfYear (currently indexed as shorts)
• endDayOfYear (currently indexed as shorts)
• higherGeography
• island
• islandGroup
• georeferencedBy
• higherClassification
• previousIdentifications

From #662

• datasetName has since beed aded
• datasetID has since beed aded

From #664

• otherCatalogNumbers

#666 (comment)

• taxonConceptID

From #515 (comment)

• isSequenced

Added 31 oct 2024: From gbif/portal-feedback#5541

• Extended Measurement Or Facts:measurementType
• DNA derived data:DNA_sequence
• GGBN Permit Extension:permitType
• Darwin Core Resource Relationship:relationshipOfResource
• Taxon Description:type

Added 31 oct 2024 from #1099

• dnaSequenceID

[timrobertson100]

Thanks for collating this. We could take the approach of responding to requests as they come in - which is not to be discounted for sure - but perhaps we might just consider what it would take to index everything? Most of the fields will be incredibly sparsely populated, and I'm not sure our original concerns a decade ago of blowing index sizes would hold true today.

[CecSve]

First of all, let me know if this is not the right issue to address this in.

Second, the following dataset has changed the DwC terms recently between versions 1.8 to 1.9, shifting from using 'individualCount' to 'organismQuantity' + 'organismQuantityType': https://www.gbif.org/dataset/91fa1a0d-a208-40aa-8a6e-f2c0beb9b253. When a user downloads the simple version of the dataset, they only get the column 'individualCount' populated with NAs and do not get the updated information stored in 'organismQuantity' + 'organismQuantityType'. Is there a way to secure that data is not 'lost' when publishers begin using new terms for the same thing?

[mdoering]

taxonConceptID would be another one to search for, e.g. avibase ids or taxonid.org identifier:
https://www.gbif.org/occurrence/3457928716

[CecSve]

• sex (currently indexed as keywords)

We are working on a controlled vocabulary gbif/vocabulary#83 but it is not finalized yet.

[MortenHofft]

This issue hasn't been closed so I assume at least some of above is still not implemented. I also just added more fields and a reference to the original issue.

[muttcg]

@MortenHofft
Many of original are closed, as far as I remember some were need discussion. I have a list fields are ready (probably group_ is not relevant anymore, because of SQL downloads):

Fields	Index	Hive	API Field	API Search	Download field	Smal filtered download	Big downloads	Result
earliestEonOrLowestEonothem	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
latestEonOrHighestEonothem	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
earliestEraOrLowestErathem	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
latestEraOrHighestErathem	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
earliestPeriodOrLowestSystem	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
latestPeriodOrHighestSystem	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
earliestEpochOrLowestSeries	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
latestEpochOrHighestSeries	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
earliestAgeOrLowestStage	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
latestAgeOrHighestStage	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
lowestBiostratigraphicZone	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
highestBiostratigraphicZone	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
group	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	hive reserved word group_
formation	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
member	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
bed	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
gbifRegion	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
publishedByGbifRegion	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
fieldNumber	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
preparations	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
sex	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
startDayOfYear	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
endDayOfYear	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
higherGeography	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
island	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
islandGroup	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
georeferencedBy	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
previousIdentifications	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
datasetName	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
datasetID	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
otherCatalogNumbers	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
taxonConceptID	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
isSequenced	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK
associatedSequences	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	OK