Weather forecasts, observations and warnings

[panaaj]

Ref: Previous issue #215 (2016) started to address this topic but with no outcome.

Proposal:
Establish path(s) to contain the data for weather forecasts, observations and warnings/alerts.
Example:
environment.weather.forecasts
environment.weather.observations
environment.weather.warnings

The intent of this issue is to revive the conversation around weather information and it's place within the Signal K model now there are a number of plugins looking to fill this gap.

Rationale:
There are a at least two plugins (possibly more) that provide weather information sources from different services and placing the resultant information in various paths under environment eg: environment.forecast, environment.outside, environment.observations.

The data structure under those paths are varied most likely based on the format of data returned from the weather service source (e.g. OpenWeather, NOAA, etc).
Additionally the use of the path environment.outside (description = "Environmental conditions outside of the vessel's hull","Environmental conditions outside of the vessel's hull") is often used for data coming from sensors on the vessel, and therefore it can be confusing to interchange this with observation data from a weather station at another location.

For a Signal K application to reliably dispay weather information it should not have to know the paths and format of data used by a particular plugin, so the goal here is to define a MVP (minimum viable product) schema for these plugin to align to.

@inspired-technologies and @sbender9 your input will be highly valued.

environment.weather.forecasts

Path containing a series of point forecasts taken at a specific location (e.g. weather station). These could be identifed by their location_id or a timestamp in ISO string format.
Using timestamp as a key will accommodate for multiple point forecasts at varying tie intervals intervals (minutes, hours, etc).

Example with timestamp as key/identifier:

environment: {
   forecasts: {
      <ISOTimeStamp>: {
        outlook: string,  // "sunny", "mostly fine", etc.
        sunrise: timestamp,
        sunset: timestamp,
        temperature: {
           value: number,
           minimum: number,
           maximum: number,
           feelsLike: number,
           dewPoint: number,
        },
        humidity: {
           value: number
        },
        pressure: {
           value: number
        },
        rain: {
           value: number
        },
        wind: {
           speed: number,
           direction: number
        },
        cloudCover: {
           value: number
        },
        visibility: {
           value: number
        },
        position: { latitude: number, longitude: number},
        description: string // location description
      },
      <ISOTimeStamp>: { ... },
      <ISOTimeStamp>: { ... },
      ...
   }

environment.weather.observations

Path containing a series of obersvations at various locations (e.g. weather station). These could be identifed by their location_id or a timestamp in ISO string format.
Using timestamp as a key will accommodate for multiple point forecasts at varying tie intervals intervals (minutes, hours, etc).

Possible structure:

environment: {
   observations: {
      <location_id>: {
        <ISOTimeStamp>: {
           description: string, 
           position: { latitude: number, longitude: number},
           outlook: string,  // "sunny", "mostly fine", etc.
           temperature: {
              value: number,
              minimum: number,
              maximum: number,
              feelsLike: number,
              dewPoint: number,
           },
           humidity: {
              value: number
           },
           pressure: {
              value: number
           },
           rain: {
              value: number
           },
           wind: {
              speed: number,
              direction: number
           },
           cloudCover: {
              value: number
           },
           visibility: {
              value: number
           }
         },
         ...
      },
      ...
   }
   
}

environment.weather.warnings

Warnings could have their own path that contains a series of warnings related to a location or maybe it makes sense to have warnings as a collection within both the observations and forecasts path entries.

Signal K plugins ref:

• signalk-noaa-weather @sbender9
• signalk-openweather-plugin @inspired-technologies

[sbender9]

Warnings should be notifications?

[preeve9534]

"precipitation" (rather than rain)? Then specify type of precipitation as a property.

Maybe also have "obscuration" and a type to indicate the type of obscuration (reduced visibilty)?

[preeve9534]

Straw man for enumeration values:

precipitation: drizzle, rain, snow, hail (maybe also ice-pellets and ice-crystals)...
obscuration: fog, mist, haze, ash, dust, smoke, sand, spray...

obscuration maybe needs a description property with values like shallow, patches, drifting and so on.

How to deal with generic descriptors like "thunderstorm" (thinks .... you get these without precipitation).

[inspired-technologies]

While the idea of having some sort of standard behind weather data is generally beneficial there are also a few concerns to consider:

• Like mentioned by @tkurki in the original issue I'm not convinced that adding hundreds of additional and potentially conflicting data points would be an appropriate approach for SignalK data (usually reflecting sensors just as observed), eg. how would outdated data paths be deleted? Or, alternatively - instead of <ISOTimeStamp> an approach may be to store relative to current time and position <+1h>, <+2h>,, etc. requiring the plugins to constantly update instead of adding new
• In the OpenWeather case we made a decision to only store one specific data set following a to be configured off-set
• Storing actual data (ie. off-set 0) as observation from a weather station is indeed better rather than putting conflicting values to actual sensor data (eg. environment.outside.pressure) but will require a naming convention for location-ids and there may be people out there who still want to take this approach as input for other plug-ins (eg. Barometer-Trend, Barograph).

Overall, rather recommend to use a time-series database like influx to hold longer forecast data series that should auto-expire after getting invalid or obsolete.

Agree with @sbender9 that warnings should only be delivered as SignalK notifications.

OpenWeatherMap delivers ok-ish forecast for land-based or near-coastal positions - however, for proper off-shore weather routing richer APIs may be better suited: looping @tkrofta for squid-sailing-signalk.

[panaaj]

• Or, alternatively - instead of <ISOTimeStamp> an approach may be to store relative to current time and position <+1h>, <+2h>,, etc. requiring the plugins to constantly update instead of adding new

From what I can see weather services provide forecasts as an Array<Forecast_data> for a given source (weather station), each entry representing a time at a period time (supplied as part of the request to the service). The length of the array will reflect the number of time intervals based on the request (or a default number).

If the concern is having a large number of data point keys to manage maybe forecasts should just be an array of values, e.g.

weather: {
   forecasts: [
      { ...forecast data 1...},
      { ...forecast data n...}
   ]
}

and then it is just a matter of managing the array values. The same could apply to observations.

I would expect a user will want the ability to scroll through a series of forecasts within a forward looking time window.

[panaaj]

• Storing actual data (ie. off-set 0) as observation from a weather station is indeed better rather than putting conflicting values to actual sensor data (eg. environment.outside.pressure) but will require a naming convention for location-ids and there may be people out there who still want to take this approach as input for other plug-ins (eg. Barometer-Trend, Barograph)

Maybe to avoid confusion weather data should be treated as a resource and be rooted under api/resources/weather?
In some respects this data set is similar to routes, etc as it is not data associated with the vessel like sensor data, it is external to the vessel.

[panaaj]

Thinking further about this in relation to the pending beta of Course and Resources APIs.... weather forecasts and observations lend themselves nicely to this approach.

Implement API endpoints for environment.weather.forecasts and environment.weather.observations which are accessed via HTTP GET.

HTTP GET .../environment/weather/forecasts
HTTP GET .../environment/weather/observations

A plugin/provider would service requests to these paths and return the data. The data would be managed by the plugin/provider and not placed on the full model and therefore address the concerns related to managing outdated data paths.

As suggested above notifications can be used for warnings e.g. notifications.environment.weather.warnings.

See PR 1381 - resources and course api for an example of how this API can be implemented.

[inspired-technologies]

Thinking further about this in relation to the pending beta of Course and Resources APIs.... weather forecasts and observations lend themselves nicely to this approach.

Implement API endpoints for environment.weather.forecasts and environment.weather.observations which are accessed via

That may be a way forward; however, still think that a snapshot of forecast data for the actual position gives some value to be accessible in the data browser resp. via request, but ...

I would expect a user will want the ability to scroll through a series of forecasts within a forward looking time window

.., for a forecast data series - why doing this through SignalK rather than going to the preferred API directly? With regards to more complex weather enquiries, eg. for an entire area or a defined route I'd have to use very specific visualization software anyways.

[tkrofta]

define a MVP (minimum viable product) schema for these plugin to align to.

Regarding the base schema:

        temperature: {
           value: number,
           minimum: number,
           maximum: number,
           feelsLike: number,
           dewPoint: number,
        },

Next to the traditional outside temps, also water temperature may be predicted:

       temperature: {
           air: number,
           minimum: number,
           maximum: number,
           feelsLike: number,
           dewPoint: number,
           water: number
        },

        wind: {
           speed: number,
           direction: number
        },

In addition, wind shall contain gusts as well

        wind: {
           speed: number,
           gust: number,
           direction: number
        }

        humidity: {
           value: number
        }

Most weather models and sensors nowadays report relativeHumidity rather than absolute, hence, need to be clear on what we mean with humidty.

        rain: {
           value: number
        }

Rain is better defined as precipitation (rate in mm/s and accumulated rain in mm) as well as snow cover, ie.

        precipitation: {
           rate: number,
           accumulated: number,
           snow: number
        }

More advanced modelling APIs may also include storm predictions (Convective available potential energy and a lifted index) as well as sea state (Height in m, Direction in rad, Period in s, Current Speed and Direction) - not consistently available around the globe though:

    storm: {
        energy: number,
        index : number,
    }

     sea: {
        waves: {  
           height: number,
           direction: number,
           period: number }, 
        current: {
           speed: number,
           direction: number }
      }

[panaaj]

., for a forecast data series - why doing this through SignalK rather than going to the preferred API directly?

The definition and use of an extensible common data model is key for interoperability.

The Signal K data model is an abstraction of the commonly used values from various sources (nmea0183, nmea2000, services, directly connected sensors and equipment, etc) into a common model for consumption by client apps and services.
It defines a wide ranging set of attributes and can be extended with additional values where a use case requires it.

So in the context of weather forecasts, they are commonly provided by weather service APIs (i.e. OpenWeather forecasts API) as an array of values that are timestamped, so it would make sense for this to be the basis of the definition in the Signal K schema.

So for example an approach could be:

1. Define an attibute value model

attributeName: {
    value: number | string,
    units: string
}

2. Define an attibute group model

groupName: {
    attributeName: {},
    attributeName: {},
    ...
}

3. Define a weather data model which can contain a combination of attributes and groups and a timestamp indicating the time of day the for which the data is relevent.

{
    timestamp: "2022-04-29T02:16:26.676Z",
    group#1: {
        attribute#g1a: {},
        attribute#g1b: {},
        ...
    },
    group#2: {
        attribute#g2a: {},
        attribute#g2b: {},
        ...
    },
    ...
    attribute#1: {},
    ...
}

4. Forecasts / observations are then arrays of weather data.

[  { weatherdata#1},  { weatherdata#1},  { weatherdata#1}, ... ]

This ensures the weather data model is extensible by allowing common groups and attributes to be defined in the specification and enabling additional values to be added as:

1. Extra attributes in an already defined group
2. New attributes in a custom group
3. Additional attributes in the weather data object

Additionally this caters for values that are found in either forecasts or observations but not in both, yet makes them easily discoverable by a consumer.

For example an observation may contain a feels like temperature which may not be present in a forecast.

Observation:`

{
    timestamp: "2022-04-29T02:16:26.676Z",
    temperature: {
        air: { value: 290, units: 'K"},
        water: { value: 280, units: 'K"},
        feelsLike: { value: 288, units: 'K"},
    },
    ...
}

Forecast:`

{
    timestamp: "2022-04-29T02:16:26.676Z",
    temperature: {
        air: { value: 290, units: 'K"},
        water: { value: 280, units: 'K"},
        minimum: { value: 288, units: 'K"},
        maximum: { value: 288, units: 'K"},
    },
    ...
}

Using an approach like this should simplify specification definition to a "minimum viable set" of common attributes and groups (which can be expanded over time) whilst still catering for the data provideed by a wide range of weather service provider APIs.

[panaaj]

To assist in building out the API schema I have started an OpenAPI document which can be found here:
https://app.swaggerhub.com/apis/panaaj/SignalK-Weather-API/1.0.0