Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should sending timestamped data in an observation request response be a supported use case? #1616

Closed
mjviljan opened this issue May 6, 2024 · 14 comments
Labels
question Any question about leshan

Comments

@mjviljan
Copy link

mjviljan commented May 6, 2024

Question

Hi,

I'm trying Leshan out with a water meter installed on an actual water pipe. The meter's manufacturer has specified that the device management server should register as an observer on the object instance resource /10262/0/2 (the object being titled "Interval Data Delivery") which includes several instances of meter readings, all of those timestamped.

Now, when I send an observation request from the server to the device, the device responds by sending a success response and the latest object value — which in this case is an array of several instances. The response is sent in SENML_CBOR format. The server fails to parse the response, giving an error

Unable to decode response payload of request [ObserveRequest [path=/10262/0/2 format=null]] from client […]: Unable to decode node[path:/10262/0/2] : value should not be timestamped

This seems to come from LwM2mNodeSenMLDecoder which has a specific check that there is no timestamp in the resolved record.

The actual request is accepted and when the device starts notifying the server of the resource's changes, the messages are parsed totally fine. Based on my investigation, this seems to happen because the notification messages are decoded using the LwM2MDecoder's method decodeTimestampedData, whereas the payload of the observation request response is using the decode method.

So, I get the notifications fine, but I miss the first values the device is sending, because those are sent already in the observation request response and decoding that fails. This is a problem especially if the connection between the device and the server has been down for some reason for a while, as then the device will send a bigger buffer of stored values in one go in the response, and those are now lost.

I've found these issues which I believe are related, as well as the PR and its comments (esp. about ObserveResponse):

I've tried to investigate this further on my own but I'm unsure about the following:

  • If the PR mentioned above was to include ObserveResponse also, would that fix my case also? (If so, you definitely have my vote on including that also. 😄)
  • Is the device following the standard correctly in the first place when it sends the object instance resource value already in the observation request response? I've tried to re-read this part of the LwM2M spec and it seems the response is usually depicted in the pictures with the value, but it's always just an integer or something, so I'm not quite sure. (I know you're not writing the spec and have trouble getting answers from OMA also, so might not know this either.) If it seems the device shouldn't work like this, I could always try to convince the manufacturer about this also...
@mjviljan mjviljan added the question Any question about leshan label May 6, 2024
@sbernard31
Copy link
Contributor

If the #1610 was to include ObserveResponse also, would that fix my case also?

I didn't get exactly what does your device. So Maybe #1610 will helps but this is not sure.
There is case which will not be supported. (See #1553 (comment))

So to answer more precisely, I should know what contains the first observe response which cause issue in your case.
You will also be able to test the PR when ready (it should be soon).
I will let you know, so you can valid it works before next milestones release which should happen in June. (#1612)

Is the device following the standard correctly in the first place when it sends the object instance resource value already in the observation request response?

If I would know exactly what does your device (which kind of response is sent), I can let your know (from my point of view) if :

  • it follows the specfication
  • it doesn't
  • OR it is in the "grey" zone.

@sbernard31
Copy link
Contributor

@mjviljan #1610 should contains support of timestamp for observe. (with limitation defined above ☝️)

@mjviljan
Copy link
Author

I tested the PR but unfortunately the data I get doesn't work after those changes either.

I'll see if I can share some sensible data from the device here. If not, I'll close this issue soon.

@sbernard31
Copy link
Contributor

I'll see if I can share some sensible data from the device here.

This could be great because :

  1. I can give you my opinion about your device compliancy,
  2. Maybe this doesn't work because you find a bug in Leshan and so it would be good to know that to fix it.

@mjviljan
Copy link
Author

I've taken an example of a SENML_CBOR message that the device sends. As I mentioned in the opening message, parsing this payload fails if the device sends it as a response to an observe request but succeeds if it's sent as a notification of a changed value later.

The raw payload in hex:

89 a4 21 6a 2f 31 30 32 36 32 2f 30 2f 32 00 62 2f 30 22 fb 41 d9 93 5e b1 00 00 00 08 52 83 19 28 1a 00 83 1a 66 4d 27 70 1a 00 01 51 80 81 00 a2 00 62 2f 31 08 58 40 83 19 28 1a 01 83 1a 66 4b dc f8 19 07 08 98 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a2 00 62 2f 32 08 52 83 19 28 1a 02 83 1a 66 4d 27 70 1a 00 01 51 80 81 00 a2 00 62 2f 33 08 54 83 19 28 1b 00 83 1a 66 4d 27 70 1a 00 01 51 80 81 82 00 00 a2 00 62 2f 34 08 55 83 19 28 1c 00 83 1a 66 4c 0e 30 19 38 40 86 00 00 00 00 00 00 a2 00 62 2f 35 08 58 21 83 19 28 1c 01 83 1a 66 4c 0e 30 19 38 40 86 82 00 00 82 00 00 82 00 00 82 00 00 82 00 00 82 00 00 a2 00 62 2f 36 08 56 83 19 28 1e 00 83 1a 66 4d 27 70 1a 00 01 51 80 81 82 18 25 18 61 a2 00 62 2f 37 08 56 83 19 28 1e 01 83 1a 66 4e 78 f0 1a 00 01 51 80 81 1a 7a fd 00 f0 a2 00 62 2f 38 08 41 80

Which is then parsed to CBOR:

[
  {0: "/0", 8: h'8319281A00831A664D27701A000151808100', -2: "/10262/0/2", -3: 1716353732},
  {0: "/1", 8: h'8319281A01831A664BDCF81907089830000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'},
  {0: "/2", 8: h'8319281A02831A664D27701A000151808100'},
  {0: "/3", 8: h'8319281B00831A664D27701A0001518081820000'},
  {0: "/4", 8: h'8319281C00831A664C0E3019384086000000000000'},
  {0: "/5", 8: h'8319281C01831A664C0E3019384086820000820000820000820000820000820000'},
  {0: "/6", 8: h'8319281E00831A664D27701A00015180818218251861'},
  {0: "/7", 8: h'8319281E01831A664E78F01A00015180811A7AFD00F0'},
  {0: "/8", 8: h'80'}
]

And when the parsing succeeds, the actual objects inside are:

[10266, 0, [1716332400, 86400, [0]]]
[10266, 1, [1716247800, 1800, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]]
[10266, 2, [1716332400, 86400, [0]]]
[10267, 0, [1716332400, 86400, [[0, 0]]]]
[10268, 0, [1716260400, 14400, [0, 0, 0, 0, 0, 0]]]
[10268, 1, [1716260400, 14400, [[0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0]]]]
[10270, 0, [1716332400, 86400, [[37, 97]]]]
[10270, 1, [1716418800, 86400, [2063401200]]]
[]

If I debug the device server, LwM2mNodeSenMLDecoder fails the timestamp check for each item.

Does this make any sense, or would you need more info?

@sbernard31
Copy link
Contributor

🤔 at first sight I think this should work ...

I will try to investigate/test that more.

@sbernard31
Copy link
Contributor

I tried to decode the hex value you shared ☝️ with DefaultLwM2mDecoder using SenML_CBOR and I get :

[
  TimestampedLwM2mNode [
      timestamp=2024-05-22T04:55:32Z,
      node=LwM2mMultipleResource [
            id=2,
            values={
                 0=LwM2mResourceInstance [id=0, value=18Bytes, type=OPAQUE],
                1=LwM2mResourceInstance [id=1, value=64Bytes, type=OPAQUE],
                2=LwM2mResourceInstance [id=2, value=18Bytes, type=OPAQUE],
                3=LwM2mResourceInstance [id=3, value=20Bytes, type=OPAQUE],
                4=LwM2mResourceInstance [id=4, value=21Bytes, type=OPAQUE],
                5=LwM2mResourceInstance [id=5, value=33Bytes, type=OPAQUE],
                6=LwM2mResourceInstance [id=6, value=22Bytes, type=OPAQUE],
                7=LwM2mResourceInstance [id=7, value=22Bytes, type=OPAQUE],
                8=LwM2mResourceInstance [id=8, value=1Bytes, type=OPAQUE]},
            type=OPAQUE]
    ]
]

I didn't face issue and works as expected.

I also tried to adapt an integration test and I'm able to decode this payload when I do an ObserveRequest.

So to be clear expected results are :

After testing, it seems to me that code works as expected.
Is there any chance that you didn't test the right version ?

@mjviljan
Copy link
Author

mjviljan commented May 24, 2024

Well, this is embarrassing... Apparently I didn't checkout the PR branch after all before building the test version. 🙄 I thought I did, but it seems I concentrated so much on updating the version numbers in the POM files to make sure I'd get the correct version of Leshan libraries that I forgot to switch branches first. (Edit: Well, I seem to have checked out Jaroslaw's fork, but then used the master branch there...)

I re-tested this now after the PR merge, and it indeed seems to work fine! 🎉

Thank you for the effort in digging into this! 🙇 And sorry for the extra trouble, caused by my mistake!

I believe this will work great when M15 gets released, so I'm closing the issue. Thank you again!

@sbernard31
Copy link
Contributor

No problem 😉
Glad to see that M15 will help you.
Do not hesitate to come back to discuss again about any problem you face with Leshan.

@sbernard31
Copy link
Contributor

@mjviljan there is discussion about #1610 PR.
And we are not sure anymore that this new API is a right way and so we are considering to rollback it 🤔

Discussion are :

Sorry this is maybe not so easy to follow. (but do not hesitate to share your opinion there ☝️)

Anyway, if we revert it and this could impact you because your use case will not work anyway.
Will it be OK for you to just be able to provide a custom encoder to just ignore timestamp validation. (see #1621 (comment)) (and so lost timestamp data ?)

@mjviljan
Copy link
Author

Hi,

Thanks for the ping! I've been following the discussion in Plan for 2.0.0-M15 🎯 but not the others.

I was happy about the change as that would fix my use case nicely, but of course you must think of what works best for Leshan in general and go with that.

I can definitely write a custom decoder for my case. And as I believe the payloads I receive will include the measurement timestamps in the actual data, losing the timestamp from the root node should not be an issue. 🤔

@sbernard31
Copy link
Contributor

I forgot to thank you for your quick feedback.

Just in case I try to clarify my mind about timestamp usage in LWM2M : #1623

For now, there is no decision about what we will do about "time fields" usage issue... 😬

@sbernard31
Copy link
Contributor

(You probably already notice that but finally decide to integrate the timestamp feature, this is now available in 2.0.0-M15)

@mjviljan
Copy link
Author

mjviljan commented Jul 4, 2024

I did, and immediately took that version into use. Works nicely. :-)

Thank you again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Any question about leshan
Projects
None yet
Development

No branches or pull requests

2 participants