Return units in persistence extension commands and support future persisted states #3736

mherwege · 2023-07-31T14:16:54Z

Closes #3732
Closes #3896

This PR implements unit support in all PersistenceExtension commands, returning the full value including unit for Number item types with a dimension. It therefore changes return types from DecimalType to a State type, whereby the state can include the unit.
This has been tested to work with rules DSL, and it should be possible to extend other scripting languages to support this, making calculations with historic states of items with dimensions more straightforward.
This may require adapting user rules and is therefore a breaking change.

Secondly, this PR makes sure persistence extensions work properly with persisted items in the future:

all Between commands have been adapted to be able to properly work across the now boundary,
a series of Untill commands have been added, equivalent to the Since commands.
nexUpdate and nexState commands have been added.
the historicState command has been deprecated (but still available for backward compatibility) and replaced with persistedState also accepting future dates.
the evolutionRate command has been renamed to evolutionRateSince, evolutionRateUntill and evolutionRateBetween, bringing it in line with the other commands. This change is required to allow one date signatures to differentiate between Since and Untill. evolutionRate is deprecated and kept for backward compatibility, and interpreted as evolutionRateSince.
added removeAllStates Since, Untill and Between commands.

The code now also has full null annotations and checks. In doing so, changes were required to avoid potential null issues. Many of these null issues where due to null being interpreted as now for end dates before, which is not possible when time can be in the future.

The test classes where extended and refactored to also test for all future commands.

Signed-off by: Mark Herwege [email protected]

openhab-bot · 2023-08-15T08:13:02Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/historicstate-returns-uom-but-deltasince-doesnt/148713/2

J-N-K · 2023-09-08T16:17:51Z

I understand what you want to achieve, but why did you chose HistoricItem as return value? The timestamp has no meaning for averages or similar, a simple State would be fine (and could be a DecimalType or a QuantityType (depending on the item's unit).

mherwege · 2023-09-11T07:15:44Z

I understand what you want to achieve, but why did you chose HistoricItem as return value? The timestamp has no meaning for averages or similar, a simple State would be fine (and could be a DecimalType or a QuantityType (depending on the item's unit).

HistoricItem was already used for minimum, maximum and previous, so wanted to stick to that. But I agree the date returned in the HistoricItem is arbitrary. I will adjust.

mherwege · 2023-09-13T09:49:16Z

@J-N-K the methods now return State.

As this is API breaking anyway, I also aligned method names. EvolutionRate whas the only group of methods that did not have a distinction between Since and Between in the name. I believe this is a rarely used method anyway and I think it is cleaner to have them aligned. If you feel differently, I can revert this last commit.

openhab-bot · 2023-10-18T21:07:35Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/blockly-quantity-type-gets-lost-when-using-persistence/150470/4

florian-h05 · 2023-12-04T00:22:01Z

Any chance to get this merged for openHAB 4.1 or is that something for a new major release because it's breaking?

kaikreuzer · 2023-12-05T21:23:18Z

@J-N-K As you had commented before, I hoped that you continue on this PR - any chance to do so?

This may require adapting user rules and is therefore a breaking change.

Breaking user rules is never nice. I'm just trying to get my head around how much this might break. Do you have a typical example where user's will have to adapt their rules? Are these rather exceptional cases or do most rules break by this?
Is there any migration path that we could document as a "quick fix"?
I am not 100% sure whether we should merge this now so shortly before the release. Waiting for a next major release isn't really an option either, that will still take a while...

florian-h05 · 2023-12-06T06:50:32Z

If this gets merged for 4.1, we should try to implement a solution for #3896 for 4.1 as well to avoid having multiple sets of breaking changes.

J-N-K · 2023-12-06T07:00:56Z

I have a rule that calculates the mean temperature over the last three days. It uses averageSince and then adds the unit before posting the update to the item.

I still think this is a good addition, but I have no idea how to make it "less breaking".

mherwege · 2023-12-06T07:10:58Z

I have a rule that calculates the mean temperature over the last three days. It uses averageSince and then adds the unit before posting the update to the item.

Similar for me, and that was the reason to create this PR. It forces you to add the unit again after retrieving the average. That means you need to 'know' the unit in the rule when retrieving to put the right unit back and it goes against the principle of UOM where you don't care.

mherwege · 2023-12-06T07:15:32Z

If this gets merged for 4.1, we should try to implement a solution for #3896 for 4.1 as well to avoid having multiple sets of breaking changes.

@J-N-K @florian-h05 @kaikreuzer If we all agree we want to do this, I could extend this PR to include a solution for #3896.
@florian-h05 Scripting languages may also be affected. Would any change be required in the javascript helper libraries? What about jruby (@jimtng) and habapp (@spacemanspiff2007) ?

florian-h05 · 2023-12-06T07:21:59Z

If we all agree we want to do this, I could extend this PR to include a solution for #3896.

I agree, but I'm no core maintainer ;-)
Thank you for the offer!

Scripting languages may also be affected. Would any change be required in the javascript helper libraries?

Yes, for renamed methods there would be changes required, however those are minor as the library is only wrapping PersistenceExtensions inside a JS class to return JS datatypes.
I would also keep the old ones in the library and log a deprecation warning, so this is less breaking.

For the new functionalities, I have to think about how to integrate them in the library, but I already have an idea.

I can't speak for jRuby, but I guess HABApp is not affected because it is using the REST API to communicate with openHAB and not any Java calls.

kaikreuzer · 2023-12-06T07:28:32Z

If we all agree we want to do this

I agree that I want this change, I am just not sure when (before or after 4.1). I tend towards waiting after the release, so that we can get some feedback by snapshot and milestone users first and maybe come up with a good migration plan for 4.2.

florian-h05 · 2023-12-06T07:36:39Z

Yeah I've already asked myself when this should be implemented.

On one hand, it would be nice to provide the right "tools" together with the new time-series support,
but on the other hand 4.0 had several breaking changes (and so it would be "nice" to have less/no breaking changes in 4.1) and 1.5 weeks till the 4.1 code-freeze is also a bit short to coordinate the changes across all repos and test this.

J-N-K · 2023-12-06T08:04:26Z

Similar for me, and that was the reason to create this PR. It forces you to add the unit again after retrieving the average. That means you need to 'know' the unit in the rule when retrieving to put the right unit back and it goes against the principle of UOM where you don't care.

Full ack. But nevertheless all existing rules using persistence extensions need to be changed.

spacemanspiff2007 · 2023-12-06T08:13:17Z

Similar for me, and that was the reason to create this PR. It forces you to add the unit again after retrieving the average.

But why? I thought when the unit is properly set you can just post a unit with an empty value and the unit from the metadata will be assumed? Imho you should just be able to post the unitless value and everything should work as expected.
The whole point of explicitly providing the unit in the item definition is that there are no ambiguites and the unit can always properly be resolved.

I think HABApp is not affected (but I have to confirm this because currently there are no test cases) because when using the RestAPI the unit is properly assumed when not provided.

mherwege · 2023-12-06T09:19:23Z

But why? I thought when the unit is properly set you can just post a unit with an empty value and the unit from the metadata will be assumed? Imho you should just be able to post the unitless value and everything should work as expected.
The whole point of explicitly providing the unit in the item definition is that there are no ambiguites and the unit can always properly be resolved.

That's true if you immediately post an update to an item that has the same unit in metadata as the unit for the item you retrieve from persistence. Sometimes, these are intermediate steps in calculations. When you retrieve something from persistence, you have to know what unit it is, and you need to consider that in each step in the calculation, convert the value if necessary yourself as it may be in the wrong unit, and then post an update to an item with the value in the unit the item expects. So the rule needs to know in each step what unit it is working in, but it is not explicit. I prefer to have the Quantities with units in the calculations all the way, to avoid all of this. That way the rule does not have to know the unit it retrieves from persistence and the unit of the item it writes to.

mherwege · 2023-12-06T09:21:51Z

I agree that I want this change, I am just not sure when (before or after 4.1). I tend towards waiting after the release, so that we can get some feedback by snapshot and milestone users first and maybe come up with a good migration plan for 4.2.

I will work on extending this with logic for future states with target 4.2 for the whole PR.

spacemanspiff2007 · 2023-12-06T10:15:22Z

That's true if you immediately post an update to an item that has the same unit in metadata as the unit for the item you retrieve from persistence.

That's exactly the example @J-N-K made above and that's why I was confused because this should already work.

J-N-K · 2023-12-06T10:30:30Z

My example was just a rule that needs adjustment, not an example to justify this change :-)

J-N-K · 2023-12-27T19:42:02Z

What's the state here? Are the discussed extensions already included and we can review and merge? The sooner, the better: more time for testing.

mherwege · 2023-12-27T21:00:10Z

What's the state here? Are the discussed extensions already included and we can review and merge? The sooner, the better: more time for testing.

Sorry, not yet. And I won’t be able to work on it for the next few days.

jimtng · 2024-05-03T11:44:08Z

I've just noticed that the "between" variant of the old evolutionRate() was removed in this PR, instead of deprecated. Should we add it back for the sake of compatibility?

mherwege · 2024-05-03T13:30:08Z

I've just noticed that the "between" variant of the old evolutionRate() was removed in this PR, instead of deprecated. Should we add it back for the sake of compatibility?

Yes, makes sense. I missed that one.

jimtng · 2024-05-03T14:30:50Z

Another change I've just noticed: Previously I could call any methods with a null serviceid to indicate that the default persistence service is to be used.

~~Now, due to @NonNullByDefault on the PersistenceExtensions class, and the absence of @Nullable in front of serviceId, this is no longer possible.~~ I think this now fails because of the implementation/code specifically not allowing null serviceid to mean "default persistence service".

This change has caused some errors in existing scripts. Should we restore the previous behaviour?

jimtng · 2024-05-03T14:54:56Z

@mherwege I edited/updated the post above, in case you've only read the original message.

mherwege · 2024-05-03T15:44:12Z

Another change I've just noticed: Previously I could call any methods with a null serviceid to indicate that the default persistence service is to be used.

@jimtng It has never been documented it could be used that way. Also, quickly looking through the old code again, it doesn't look like it worked for all actions, and it even gave wrong result for some (part of the query working and another part returning null and defaulting to a wrong default value, so the combination being wrong).
While it is certainly doable to introduce this, I am wondering if it is worth doing this to fix wrong usage of the actions in the first place.

mherwege · 2024-05-03T17:14:26Z

@jimtng PR is created to allow null serviceId.

jimtng · 2024-05-03T18:07:54Z

@jimtng PR is created to allow null serviceId.

Thanks! I was pondering about your previous response. I agree that it wasn't documented in the javadocs. I've started working on modifications in the jruby helper library to avoid using a null serviceId. However, I think it's a handy thing to support but either way is fine.

… future states (#331) Refs openhab/openhab-core#3736. - Rename `Item::history` to `Item::persistence`. - Rename `HistoricItem` to `PersistedItem` and introduce `PersistedState` for state as string, Quantity & number. - Make variance, deviation, average, sum & delta return the state as `PersistedState` instead of number. - Add `**Until` methods for all existing queries. - Add `nextState` and `nextUpdate` methods. - Rename `historicState` method to `persistedState`. - Remove `latestState` method. - Remove deprecated `evolutionRate` method. - Add `removeAllStatesSince`, `removeAllStatesUntil` & `removeAllStatesBetween` methods. --------- Signed-off-by: Florian Hotze <[email protected]>

See https://github.com/openhab/openhab-js/blob/main/CHANGELOG.md#500. Required to adjust to breaking changes from openhab/openhab-core#3736. Signed-off-by: Florian Hotze <[email protected]>

This adjusts the persistence blocks for the **breaking** changes of `org.openhab.core.persistence.extensions.PersistenceExtensions`. For NashornJS, see openhab/openhab-core#3736 for the changes. For GraalJS, see https://next.openhab.org/addons/automation/jsscripting/#itempersistence for the current API docs and openhab/openhab-js#331 for the changes. --------- Signed-off-by: Florian Hotze <[email protected]>

openhab-bot · 2024-05-17T15:04:11Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/jruby-scripting-official-helper-library/145072/14

njordan77 · 2024-05-25T20:33:47Z

Hello, i changed to M3 and now a rule does throw an error based on this line. Says "unknown variable or line" pointint to the "minus"... Any idea what i have to use instead? Thanks Norbert
var AvailChargePower = (miFro_ACpower.averageSince(now.minusMinutes(2)) - 400)

openhab-bot · 2024-05-25T20:34:20Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/unknown-variable-or-command-in-rule/156191/3

openhab-bot · 2024-05-26T11:07:17Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/openhab-4-2-milestone-discussion/154316/154

openhab-bot · 2024-06-06T18:27:57Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-use-time-series-from-rules-dsl-or-js/154148/23

openhab-bot · 2024-06-09T08:27:58Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-use-time-series-from-rules-dsl-or-js/154148/29

See https://github.com/openhab/openhab-js/blob/main/CHANGELOG.md#500. Required to adjust to breaking changes from openhab/openhab-core#3736. Signed-off-by: Florian Hotze <[email protected]> Signed-off-by: Paul Smedley <[email protected]>

openhab-bot · 2024-09-03T10:23:21Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/persist-state-with-own-timestamp/158160/2

See https://github.com/openhab/openhab-js/blob/main/CHANGELOG.md#500. Required to adjust to breaking changes from openhab/openhab-core#3736. Signed-off-by: Florian Hotze <[email protected]> Signed-off-by: Patrik Gfeller <[email protected]>

See https://github.com/openhab/openhab-js/blob/main/CHANGELOG.md#500. Required to adjust to breaking changes from openhab/openhab-core#3736. Signed-off-by: Florian Hotze <[email protected]>

mherwege requested a review from a team as a code owner July 31, 2023 14:16

wborn added the API breaking label Aug 16, 2023

J-N-K added enhancement An enhancement or new feature of the Core and removed enhancement An enhancement or new feature of the Core labels Sep 8, 2023

mherwege force-pushed the persistence_extensions branch from 1fcb4fe to f06b608 Compare September 13, 2023 09:41

mherwege mentioned this pull request Dec 1, 2023

Extend PersistenceExtensions for working with future states #3896

Closed

mherwege force-pushed the persistence_extensions branch from 10808d8 to 0e82bfc Compare January 10, 2024 16:09

mherwege changed the title ~~Return units in persistence extension commands~~ Return units in persistence extension commands and support future persisted states Jan 10, 2024

mherwege mentioned this pull request May 3, 2024

Persistence extension, re-add deprecated evolutionRate method variants #4212

Merged

mherwege mentioned this pull request May 3, 2024

Persistence extensions: allow null serviceId #4213

Merged

florian-h05 mentioned this pull request May 5, 2024

[jsscripting] Upgrade openhab-js to 5.0.0 openhab/openhab-addons#16719

Merged

florian-h05 mentioned this pull request May 6, 2024

[blockly] Adjust persistence blocks to breaking changes openhab/openhab-webui#2564

Merged

holgerfriedrich mentioned this pull request May 7, 2024

Snapshot 4069: failed to install bundles at startup #4222

Closed

This was referenced May 28, 2024

Persisted group items with base type Number:Dimension not restored correctly in persistence extensions #4254

Closed

Fix persistence extensions for group items and number dimension items with different units #4257

Merged

jlaur mentioned this pull request Oct 26, 2024

Prepare persistence to be ready for QuantityTypes #1954

Open

5 tasks

Return units in persistence extension commands and support future persisted states #3736

Return units in persistence extension commands and support future persisted states #3736

Conversation

mherwege commented Jul 31, 2023 • edited Loading

openhab-bot commented Aug 15, 2023

J-N-K commented Sep 8, 2023

mherwege commented Sep 11, 2023

mherwege commented Sep 13, 2023

openhab-bot commented Oct 18, 2023

florian-h05 commented Dec 4, 2023

kaikreuzer commented Dec 5, 2023

florian-h05 commented Dec 6, 2023

J-N-K commented Dec 6, 2023

mherwege commented Dec 6, 2023

mherwege commented Dec 6, 2023

florian-h05 commented Dec 6, 2023

kaikreuzer commented Dec 6, 2023

florian-h05 commented Dec 6, 2023

J-N-K commented Dec 6, 2023

spacemanspiff2007 commented Dec 6, 2023 • edited Loading

mherwege commented Dec 6, 2023 • edited Loading

mherwege commented Dec 6, 2023

spacemanspiff2007 commented Dec 6, 2023

J-N-K commented Dec 6, 2023

J-N-K commented Dec 27, 2023

mherwege commented Dec 27, 2023

jimtng commented May 3, 2024

mherwege commented May 3, 2024

jimtng commented May 3, 2024 • edited Loading

jimtng commented May 3, 2024

mherwege commented May 3, 2024 • edited Loading

mherwege commented May 3, 2024

jimtng commented May 3, 2024

openhab-bot commented May 17, 2024

njordan77 commented May 25, 2024

openhab-bot commented May 25, 2024

openhab-bot commented May 26, 2024

openhab-bot commented Jun 6, 2024

openhab-bot commented Jun 9, 2024

openhab-bot commented Sep 3, 2024

mherwege commented Jul 31, 2023 •

edited

Loading

spacemanspiff2007 commented Dec 6, 2023 •

edited

Loading

mherwege commented Dec 6, 2023 •

edited

Loading

jimtng commented May 3, 2024 •

edited

Loading

mherwege commented May 3, 2024 •

edited

Loading