Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve publishing formats and throughput on common endpoints #25

Open
jdpye opened this issue Nov 1, 2022 · 2 comments
Open

Improve publishing formats and throughput on common endpoints #25

jdpye opened this issue Nov 1, 2022 · 2 comments
Assignees

Comments

@jdpye
Copy link
Collaborator

jdpye commented Nov 1, 2022

Making general improvements to our publishing strategy into commonly used endpoints (GeoServer, ERDDAP) to mitigate some of the troubles we've experienced in publishing and hosting full-resolution data to those spaces.

Historically we've been able to tip over GeoServers by requesting a few million rows, so for detection data, this might mean an ERDDAP dataset that is lean but informative and can refer to external datasets easily.

@jdpye jdpye self-assigned this Nov 1, 2022
@jdpye
Copy link
Collaborator Author

jdpye commented Nov 17, 2022

Was mentioned by @aubrivliz that exfiltrating data from GeoServer in XML format was the least processing-intensive to GeoServer itself and may allow for some performance over high rowcount querying. We will test this out a bit on our larger table layers and find out whether this solves our bottleneck.

@jdpye
Copy link
Collaborator Author

jdpye commented Jan 19, 2024

Steps ETN took:

updated Geoserver (v 2.?? ?)
optimization.
updated PostgreSQL (14.x now)
indexing of the Geoserver-mapped tables shouldn't have changed but will see.

TODO: OTN to try again and see if issue persists.
OTN to beg ETN for optimization tips for the reverse flow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant