How to se threefive to validate the CRC_32 of a SCTE35 message

[fxengelbrecht]

Hi.

How would you go about validating the CRC_32 of a SCTE35 message?
As an example see /DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==
This is a splice_insert message with a segmentation_descriptor.
threefive shows the CRC_32 as: "crc": "0x943a437c" as found in the SCTE35 instance. However the correct CRC_32 is 0x9c216676 if you calculate the CRC_32 based on the content of the message. (This according to the middleman.tv online SCTE35 parser)
I would like to parse a list of SCTE35 and determine which are correct, vs. those with bad CRC_32.
Any guidance would be appreciated.

[futzu]

Let me ask you this, what do you plan on doing when you find a bad crc? How would you handle it and why would you handle it?
What do you perceive as the danger? I'm just curious.

[fxengelbrecht]

I need to inform the upstream service provider that the SCTE35 is malformed. No idea yet on the impact to downstream entities ingesting this content, I just noticed this issue...

As for the validation of the CRC, I thought the following may work.

1. Convert the base64 encode string to hex string
2. Use threefive to decode to json
3. Use threefive to encode the json to hex string.
4. If the last 8 digits in the hex string from step 3 differs from the last 8 digits of the hex string in step 1 then the CRCs are different and the CRC of the source SCTE35 is incorrect.
   Will test this procedure this weekend.

[futzu]

Sorry, I was such a dick yesterday, I assumed you giving me shit and I just wasn't in the mood. I apologize.

Just decode and then encode man, see if the strings match. When threefive encodes, it calculates everything that it can, lengths, types, and crc32. Here's all you have to do.

>>>> from threefive import Cue
>>>> c = Cue('/DAgAAAAAAAAAP/wDwUAAAEDf0//MqD0yAEDAAAAAHXkvig=')
>>>> c.decode()
True
>>>> c.encode()
'/DAgAAAAAAAAAP/wDwUAAAEDf0//MqD0yAEDAAAAAHXkvig='

here's the Cue you had

>>>> cue = Cue('/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==')
>>>> cue.decode()
True
>>>> cue.encode()
'/DA5AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAGQIXQ1VFSQAAAAB/yAAAKTLgAQEwMAAAAACRUPll'

[futzu]

I was worried about false negatives. As I mentioned, threefive re-calculates everything it can when encoding, and also fixes several common errors I've seen over and over. I decided to check the crc32 from the raw bites, much like you thinking with the hex.

a@fu:~$ python3
>>> from threefive import Cue
>>> cue=Cue('/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==')
>>> cue.decode()
True
>>> cue.info_section.crc
'0x943a437c'
>>> from threefive.crc import crc32

>>> hex(crc32(cue.bites[:-4]))
'0x943a437c'
>>> cue.info_section.descriptor_loop_length
23
>>> cue.encode()
'/DA5AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAGQIXQ1VFSQAAAAB/yAAAKTLgAQEwMAAAAACRUPll'
>>> cue.info_section.crc
'0x9150f965'
>>> cue.info_section.descriptor_loop_length
25

One of the more common mistakes that threefive corrects is when people leave off sub_segment_num and'sub_segments_expected from a segmentation descriptor, they are used depending on the value of segmentation_type_id.
sub_segment_num and sub_segments_expected are each 1 byte.
That's why the descriptor_loop _length was 23 before encoding and 25 after, threefive adds them before encoding.

• So to recap,
  
  • middleman.tv is wrong in multiple ways,and I can't explain their results.
  • The crc for the cue from your provider is correct for what they encoded, but they did not include sub_segment_num, and sub_segments_expected in the splice descriptor when they encoded it.

I just saved you a month worth of aggravation trying to figure that one out. You owe me a favor.

try this instead:

Updated!

#!/usr/bin/env python3

from threefive import Cue
from threefive.crc import crc32

cue_list = [
    "/aBadCue",
    "/DAgAAAAAAAAAP/wDwUAAAEDf0//MqD0yAEDAAAAAHXkvig=",
    "/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==",
    "/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==",
]


def vrfy(data):
    stats = {
        "decoded": False,
        "valid_crc": False,
        "complete": False,
    }
    cue = Cue(data)
    try:
        cue.decode()
    except:
        return data, stats
    stats["decoded"] = True
    in_crc = cue.info_section.crc
    stuff = cue.bites[: len(cue.bites) - 4]
    stats["valid_crc"] = (False, True)[hex(crc32(stuff)) == in_crc]
    cue.encode()
    stats["complete"] = (False, True)[cue.info_section.crc == in_crc]
    return data, stats


def do(cue_list):
    """
    do return a dict of cue, valid
    """
    cue_list = set(cue_list)  # remove duplicates
    results = {out[0]: out[1] for out in [vrfy(data) for data in cue_list]}
    return results


if __name__ == "__main__":

    print(do(cue_list))

• output

a@fu:~$ pypy3 vrfy.py 
{'/DAgAAAAAAAAAP/wDwUAAAEDf0//MqD0yAEDAAAAAHXkvig=': {'decoded': True, 'valid_crc': True, 'complete': True}, '/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==': {'decoded': True, 'valid_crc': True, 'complete': False}}

[futzu]

Where did you get that from, middleman?

If it didn't come from me or directly from the specification, it's not relevant and I don't care to hear it.
I had so much trouble trying to verify threefive's results using other SCTE-35 parsers, that I wrote a completely
different parser in Go called cuei.
Trust threefive until you have a reason not to, and then come tell me that reason and I'll fix it.

[futzu]

You're creating your own problems. Had you not been listening to middleman, none of this would have been an issue. This is why I get angry about online parsers, it's always a big waste of time. We wasted hours on this.

[fxengelbrecht]

I prefer using threefive, however, when a vendor tells me "but middleman says..." then I need to verify.

As for the statement "sub_segment_num and sub_segments_expected are only included in the SCTE35 message if segmentation_type_id in [0x34, 0x36, 0x38, 0x3A]. " I got that from the SCTE35 spec. As an example, in the 2022 edition, table 20 on page 59 the pseudo code for the segmentation_descriptor shows:
...
segment_num 8 uimsbf
segments_expected 8 uimsbf
if(segmentation_type_id == ‘0x34’ || segmentation_type_id == ‘0x36’ || segmentation_type_id == ‘0x38’ || segmentation_type_id == ‘0x3A’) {
sub_segment_num 8 uimsbf
sub_segments_expected 8 uimsbf
}

[futzu]

[futzu]

• Here's the Spec
  

• Here's middleman's UPID for that Cue

    {
        segmentation_upid_type: "UserDefined",   <!---- Type 1 has been deprecated, this is old.
        segmentation_upid_length: "0x1",             <!------- if the length is one, 
        Data: "MA==",                                            <!------why is the data 4 bytes long?
        MPU: null,                                                  <! this makes no sense, MPU is a type of UPID
        SegmentationUPIDFormat: "NotSpecified",  <!--- I have no idea what this is supposed to be
        Value: ""                                                        <!--- I have no idea what this is supposed to be
    }

Here's threefive

           "segmentation_upid_type": 1,                                <!---- Type 
            "segmentation_upid_type_name": "Deprecated",   <---- I add the type_name to make it easier
            "segmentation_upid_length": 1,                               <----- Length --  the length is 1 byte
            "segmentation_upid": "0",                                        <---- SegmentationUpid()  -- This is the data, 1 byte

[futzu]

2022 is out of date man. There have been big changes since then.
Tell your client just because it's on the Internet, doesn't mean it's true.
There are multiple errors in middleman's output, that are not a good comparison.

I still for the life of me can't figure out what kind of math middleman is doing,
they are using like the 2020 spec. Comcast's online parser is out of date too.

https://iodisco.com/cgi-bin/scte35decoder uses threefive.

[fxengelbrecht]

Thank you very much. I've downloaded the new SCTE35 and 67 specs and will "read the fancy manual" on both.
And thank you for writing threefive.

[futzu]

I appreciate you saying that.

SCTE-35 doesn't make much sense , it's smart to read the specs.

[futzu]

I don't know if you seen my hls stuff, but there's a bunch of it M3ufu, x9k3, umzz, showcues, and sideways.

[futzu]

showcues is great for debugging hls and scte-35