Fix bug operation window

[maxnessbsc]

Error without the modifications is

  * 1: SeriousBug: MovingWindow(1990-01-01 00:00:00 to 1990-02-01 00:00:00) : 1990-01-01 00:00:00 is outside of current period : lower Bound violation
  in  (/gpfs/scratch/bsc32/bsc032519/Bundle_EERIE/ifs-bundle/source/multio/src/multio/action/statistics/OperationWindow.cc:141 gtLowerBound)

[dsarmany]

These changes the lower-bound treatment but leaves the upper-bound treatment unchanged. That would result in boundary values going into both windows statistics. This behaviour is incorrect. For some statistics, like means, it is okay(ish). But for others, like accumulation or min/max, it is wrong.

Even if we 'open' the upper-bound, that is not the behaviour we want for all statistics, even if we may want that for some other statistics.

[maxnessbsc]

Ah ok, I understand. So for monthly means, should we just filter out the first message to avoid the crash?

[dsarmany]

Ah ok, I understand. So for monthly means, should we just filter out the first message to avoid the crash?

For now, I would say yes. Outputting step-zero statistics is a bit misleading anyway. We have been doing that in the past, but it is something arguably we should stop doing. The question, as so often, is backward compatibility and the fact that some of our systems rely on them being output. But I don't think that applies to climate DT.

[MircoValentiniECMWF]

If the problem is with step 0, there is an option in statistics to use step0. If the error is not with step 0, then my guess is that there is something wrong with the data and you need to dump the time history to understand what is going on.

[maxnessbsc]

Adding a filter for step 0 solves the issue :)

[MircoValentiniECMWF]

Can you please paste the lan that caused the issue?

[maxnessbsc]

There might be different bugs. When I run a simulation starting from the 20th of the month:

ABOR1     [PROC=53,THRD=1] :  MULTIO_UTILS_MOD:MULTIO_ERROR_HANDLER - Error: Caught a nested exception on C-C++ API boundary:
Nested FailureAwareException:

  * 1: SeriousBug: MovingWindow(2020-01-01 00:00:00 to 2020-02-01 00:00:00) : 2020-01-20 00:00:00 is outside of current period : lower Bound violation
  in  (/gpfs/scratch/bsc32/bsc032519/Bundle_EERIE/ifs-bundle/source/multio/src/multio/action/statistics/OperationWindow.cc:141 gtLowerBound)

  * 2: FailureAware<Action> with behaviour "propagate" on Client for context: [
Statistics(output frequency = 1, unit = month, operations = average) with Message: Message(version=1, tag=Field, source=Peer(group=null,id=18446744073709551615), destination=Peer(group=null,id=18446744073709551615), metadata={"domain":"g","date":"20200120","time":"0000","expver":"hz9m","class":"d1","type":"fc","stream":"clmn","anoffset":"9","step":"0s","levtype":"pl","param":"131","gribEdition":"2","dataDate":20200120,"dataTime":0,"stepUnits":13,"startStep":0,"endStep":0,"paramId":"131","gridType":"reduced_gg","globalSize":28480,"precision":"single","tablesVersion":"31","setLocalDefinition":"1","grib2LocalSectionNumber":"1","productionStatusOfProcessedData":"12","dataset":"climate-dt","activity":"CMIP6","experiment":"hist","generation":"1","model":"IFS-NEMO","realization":"1","significanceOfReferenceTime":"2","subCentre":"1003","generatingProcessIdentifier":"156","setPackingType":"grid_ccsds","typeOfLevel":"isobaricInhPa"}, payload-size=113920)

Where The period 2020-01-20 00:00:00 should be inside the operation window. The plan is:

name: 'Output pt reduced_gg fields'
enable: '{MULTIO_NEXTGEMS_PLANS_ENABLED}'
actions:
  - type: select
    match:
      - levtype: [pl, pt, pv]
        paramId: [60, 75, 76, 129, 130, 131, 132, 133, 135, 157, 246, 247, 248]
        gribEdition: 2
        gridType: reduced_gg

  - type: metadata-mapping
    mapping: '{MULTIO_RAPS_PLANS_PATH}/General_Plans/mapping-climateDT-monthly-means.yaml'
    overwrite-existing: true
    enforce-match: true

  - type: metadata-mapping
    mapping: '{MULTIO_RAPS_PLANS_PATH}/mapping/mapping-type-of-level-pl.yaml'
    overwrite-existing: false
    enforce-match: false

  - type: statistics
    output-frequency: 1m
    options:
      initial-condition-present: true
      restart: false
      step-frequency: 1
      time-step: 3600
      use-current-time: true
    operations:
      - average       

  - type: encode
    format: grib
    template: '{MULTIO_RAPS_TEMPLATES_PATH}/templates.{MULTIO_RAPS_GRID_TYPE}/reduced_gg_pl_statistical_grib2.tmpl'
    additional-metadata:
      resolution: high

  - type: sink
    sinks:
      - type: fdb5
        enable: '{MULTIO_ENABLE_FDB}'
        config:
          fdb_home: '{MULTIO_NATIVE_GRID_FDB_HOME}'

[maxnessbsc]

Any update on this? I still get the same error with all PL monthly plans, even when the starting date is the 1st of the month.

(@stefaniereuter related to the data-portfolio we have been discussing these days)

[MircoValentiniECMWF]

This message does not make any sense for me, "MovingWindow(2020-01-01 00:00:00 to 2020-02-01 00:00:00) : 2020-01-20 00:00:00". Are you working with a tagged multIO version? If yes which (I need to reproduce the problem)?

[maxnessbsc]

Yes, I'm working with MultIO 2.1.9 and eccodes 2.38.3.