Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update data to use metric units and update the wind field to include scalar variables #280

Open
ian-noaa opened this issue Dec 20, 2023 · 5 comments
Assignees
Labels
Stale task Tasks break a project down into discrete steps

Comments

@ian-noaa
Copy link
Contributor

We'll need to create and run a back-filling script for two related issues. It would make sense to do these at the same time.

  • We need to standardize the database and ingest to consistently utilize metric units throughout.
  • We also need to make sure the wind fields store scalar (speed/direction) & vector (u/v) components in obs & model documents.

Potentially, we could update the Wind field to something like below. That way we could get the Wind object and be able to consistently ask for scalar & vector variables. Whether this is a good idea heavily depends on if it's a good idea in Couchbase.

"Wind": {
  "Vector": {
    "U": 1.234,
    "V": 5.678
  }
  "Scalar": {
    "speed": 1.23,
    "direction": 360
  }
}
@ian-noaa ian-noaa added the task Tasks break a project down into discrete steps label Dec 20, 2023
@randytpierce
Copy link
Contributor

I re-ran the backfill of METAR obs over the weekend to re-calculate the RH using DegF instead of DegC. The obs should all be good now, to my estimation. The backfill is single threaded and it took about 6 hours. That means that if we want to backfill the model data that we have - like to change the units or to calculate WindU and WindV this single threaded technique would take 2(models) * 48 (fcstLens) * 6(hours) => 576 hours which I think is probably a no go. So the model backfill will need to be either multi-threaded or broken up into a bunch of different containers that each process a subset of the data. I lean to the later myself. That would be easy enough. We just need to figure out where to run it. Probably on rancher.

@mollybsmith-noaa
Copy link
Collaborator

After thinking about this over the break, I would really prefer not to store the wind in a different structure than the rest of the variables, as it will make our templating way more difficult.

@ian-noaa
Copy link
Contributor Author

ian-noaa commented Jan 3, 2024

#281 is a possible duplicate of this one.

Copy link

github-actions bot commented Apr 4, 2024

This issue is stale because it has been open 90 days with no activity.

Copy link

This issue is stale because it has been open 90 days with no activity.

@github-actions github-actions bot added the Stale label Jul 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Stale task Tasks break a project down into discrete steps
Projects
None yet
Development

No branches or pull requests

3 participants