TS encapsulation of Teletext PES for tsduck

[omikron88]

I made some tests with including teletext packets to tsduck live streams
This worked for me: (written in Python)

The PES packets need to include PTS timestamp. Because tsduck merge plugin controls the insert rate. pts_increment is 1800 for 50Hz time advance, or 3600 for 25Hz time advance. It is in 90kHz units.
Next goes the encapsulation to 188 byte TS packets. pid is the required PID from 0x0100 to 0x1FFE.
It is possible to inject the resulting packets into live tsduck steam with acceptable time error:
-P merge "gener .... "

Of course, there is also need to add the reqired signalisation to PSI tables:
ttx.xml.

edit your language, start page, PID and VBI lines used. And patch the PMT table by
-P pmt --patch-xml ttx.xml

Source files are in next message

Have a nice day
source.zip

[omikron88]

Here are the source files
source.zip

[SonnyWalkman]

Great work, Indentation is out. any chance to add this snippet in a test folder under TS duck? Rob

…

On Sat, 25 Feb 2023 at 07:05, omikron88 ***@***.***> wrote: I made some tests with including teletext packets to tsduck live streams This worked for me: (written in Python) ` def send_ts(): global ts, pid, pes, ts_payload, ts_index, ts_cc, TS_PACKET_SIZE, TS_HEADER_SIZE temp = TS_PACKET_SIZE - TS_HEADER_SIZE if ts_payload >= temp: ts[3] = ts_cc | 0x10 # continuity counter, no scrambling, only payload ts_cc = (ts_cc + 1) % 0x10 # inc.continuity counter ts[TS_HEADER_SIZE:] = pes[ts_index:ts_index + temp] sys.stdout.buffer.write(ts) # out.write(ts) ts_payload -= (TS_PACKET_SIZE - TS_HEADER_SIZE) ts_index += (TS_PACKET_SIZE - TS_HEADER_SIZE) ts[1] = (pid >> 8) & 0x1F # pid not payload unit start ts[2] = (pid & 0xFF) if ts_payload != 0: send_ts() else: ts[3] = ts_cc | 0x30 # continuity counter, no scrambling, adaptation field and payload ts_cc = (ts_cc + 1) % 0x10 # inc.continuity counter ts[TS_PACKET_SIZE - ts_payload:] = pes[ts_index:ts_index + ts_payload] ptr = TS_PACKET_SIZE - ts_payload # point to the first payload byte ts[4] = ptr - TS_HEADER_SIZE -1 ts[5] = 0x00 # no options for temp in range(6, ptr): ts[temp] = 0xFF sys.stdout.buffer.write(ts) # out.write(ts) ts_payload = 0 def init_ts(): global pid, ts_payload, ts_cc, TS_PACKET_SIZE ts = bytearray(b'\xff') * TS_PACKET_SIZE ts_payload = 0 ts_cc = 0 return ts def add_ts(size): global pid, ts_payload, ts_index ts_payload = size ts_index = 0 ts[0] = 0x47 # sync ts[1] = (pid >> 8) & 0x1F # pid ts[2] = (pid & 0xFF) ts[1] |= 0x40 # payload unit start indicator send_ts() def init_pes(pes_size): global packet_index, EBU_UNIT_SIZE pes = bytearray(b'\xff') * pes_size pes[0] = 0x00 pes[1] = 0x00 pes[2] = 0x01 # prefix pes[3] = 0xBD # data temp = pes_size - 6 pes[4] = (temp >> 16) & 0xFF pes[5] = (temp & 0xFF) pes[6] = 0x85 pes[7] = 0x80 # flags pes[8] = 0x24 # header size # 31 0xFF stuffing here pes[45] = 0x10 # ebu teletext packet_index = EBU_UNIT_SIZE return pes def stamp_pts(stamp): global pes pes[9] = ((stamp >> 29) & 0x0F) | 0x21 pes[10] = (stamp >> 22) & 0xFF pes[11] = ((stamp >> 14) & 0xFF) | 0x01 pes[12] = (stamp >> 7) & 0xFF pes[13] = ((stamp << 1) & 0xFF) | 0x01 def add_pes(buffer): global pes, packet_index, pes_size, pts_stamp, pts_increment, EBU_UNIT_SIZE pes[packet_index:] = buffer[0:] packet_index += EBU_UNIT_SIZE if packet_index == pes_size: stamp_pts(pts_stamp) add_ts(pes_size) pts_stamp += pts_increment packet_index = EBU_UNIT_SIZE ` The PES packets need to include PTS timestamp. Because tsduck merge plugin controls the insert rate. pts_increment is 1800 for 50Hz time advance, or 3600 for 25Hz time advance. It is in 90kHz units. Next goes the encapsulation to 188 byte TS packets. pid is the required PID from 0x0100 to 0x1FFE. It is possible to inject the resulting packets into live tsduck steam with acceptable time error: -P merge "gener .... " Of course, there is also need to add the reqired signalisation to PSI tables: ttx.xml: ` ` edit your language, start page, PID and VBI lines used. And patch the PMT table by -P pmt --patch-xml ttx.xml Have a nice day — Reply to this email directly, view it on GitHub <#35>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM4O2S4K3ZEDTUPF5RZSELTWZEICNANCNFSM6AAAAAAVHJMTI4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[SonnyWalkman]

I'm just beginning to play with getting vbit2 to send to TSduck.
Do you have a working TSduck implementation?
Furthermore, can vbit2 pass T42 using UDP into TSduck?

[SonnyWalkman]

I found your bash file.. I got Teletext working with VLC however the PCR's is way high.
ffmpeg is encoding and supplying a 4.5M CBR rate TS stream over multicast socket local on the machine.

• pcrverify: PID 256 (0x0100), PCR jitter: -4,458,984 = 165,147 micro-seconds = 52 packets + 132 bytes + 6 bits (tsp time)

My script below, anything which l could improve?

tsp -v -b 480000 ^
-I ip 224.0.0.1:9999 ^
-P regulate ^
-P merge -b 4700 "python ttxgen.py ttx.t42 2 0 3600 1024" ^
-P pmt --patch-xml TTX.xml ^
-P inject -p 16 NIT.xml=2000 ^
-P inject -p 20 TDT.xml=1000 ^
-P nit --build-service-list-descriptors ^
-P pcrverify ^
-O ip 224.0.0.2:9999

[SonnyWalkman]

The above method kind of worked however, I have heaps of PCR discontinuity errors. Video stopping. lost frames etc.
I've thought about it some more.
I think the better approach is create a completely new TS with its own PCR timing and have that muxed in with the live AV TS?
At least the timing TS requirements are met.

TSduck has the capacity to manufacture a TS from the ground up and just add all what is needed to make the stream compliant. Taken from the TSduck documentation with the TXT stuff added from above.

BITRATE=400000
PCR_PER_SEC=5
PCR_DISTANCE=$(( $BITRATE / $(( $PCR_PER_SEC * 188 * 8 )) ))
PCR_PID=5004
tsp --verbose --bitrate $BITRATE --max-flushed-packets 70
-I null
-P regulate --packet-burst 14
-P filter --every $PCR_DISTANCE --set-label 1
-P craft --only-label 1 --pid $PCR_PID --no-payload --pcr 0
-P continuity --pid $PCR_PID --fix
-P pcradjust --pid $PCR_PID
-P inject PAT.xml --pid 0 --bitrate 15000
-P inject SDT.xml --pid 17 --bitrate 15000
-P merge -b 4700 "python ttxgen.py ttx.t42 2 0 3600 1024"
-P pmt --patch-xml TTX.xml
-P inject -p 16 NIT.xml=2000
-P inject -p 20 TDT.xml=1000
-P nit --build-service-list-descriptors
-P pcrverify
-O ip 224.0.0.3:4500 --packet-burst 7 --enforce-burst

What do you think?
My goal is to : "Keep Teletext ALIVE! on DVB"

[SonnyWalkman]

Hello @omikron88.

Thanks for your contribution.
It's been sometime and I can confirm your Python generator work beautifully with TSduck.

I'm converting your python code to Go code to expand further in generation of both a 4-6 page carousel to cycle a static and a few dynamically generated pages to further provide a live teletext service. I'd like to understand generating the t42 stream to do such? I'll need to update the date and time in the header and developed pages to show system stats such as cpu and gpu, ram usage etc.

I like the idea of a teletext fish tank however working out how I can replicate such in code require some thought.

https://youtu.be/zFkBHuchOnk?si=utewe3efCCaLMgtL

SonnyWalkman

[peterkvt80]

It doesn't sound too difficult to generate the fish tank. You could adapt code from vbit2 as that already generates t42 and has everything needed to generate packets and sequence them correctly.

[SonnyWalkman]

I’m coding in Go there looking at a small device to produce a index with a few dynamic pages like weather, club news and a few engineering pages like test charts and of course fish tank. I have an idea using asciiaquarium however it written in Perl and leverages off a few packages to work. The concept is all there however, structuring fish and other objects in teletext block art is a skill. At 40x25 lines Regards Rob aka SonnyWalkman Sent from Gmail Mobile iPhone 14

…

On Mon, 30 Oct 2023 at 8:11 pm, Peter Kwan ***@***.***> wrote: It doesn't sound too difficult to generate the fish tank. You could adapt code from vbit2 as that already generates t42 and has everything needed to generate packets and sequence them correctly. — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM4O2S2QWOCYKXON5I6FQN3YB5VNXAVCNFSM6AAAAAAVHJMTI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUG43TAOJUGI> . You are receiving this because you commented.Message ID: ***@***.***>

[ZXGuesser]

Following on from a discussion on the facebook teletext group, I've hacked together some code to generate a PTS and encapsulate the PES output in a transport stream in a new branch. It builds a PCR and PTS from the internal field count which tsduck can then resynchronize to another stream.
I've been able to successfully merge this into a live transport stream with tsduck.

[SonnyWalkman]

Wow that was quick. I’ll get if a shot and send you feedback. ZXGuesser, I appreciate your time and motivation in adding PTS time stamping and TS encapsulation.

…

On Sun, 12 Nov 2023 at 7:47 am, ZXGuesser ***@***.***> wrote: Following on from a discussion on the facebook teletext group, I've hacked together some code to generate a PTS and encapsulate the PES output in a transport stream in a new branch: 1c97af2 <1c97af2>. It builds a PCR and PTS from the internal field count which tsduck can then resynchronize to another stream. I've been able to successfully merge this into a live transport stream with tsduck. — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM4O2S4JHIJ4TWFOI3JX5PDYD7P7JAVCNFSM6AAAAAAVHJMTI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWHEYTGNRQGI> . You are receiving this because you commented.Message ID: ***@***.***>

[ZXGuesser]

I got thoroughly nerdsniped on this one, but the chances are I wasn't going to get anything else done today anyway 😂

[ZXGuesser]

The latest code adds a --pid argument and renames the format from PES to ts to reflect that it's now encapsulated.
Before I merge this and do a release, is this new arrangement suitable for your uses @premultiply?

[ZXGuesser]

Made some more small tweaks to correct the TS continuity counters, so tsanalyze doesn't report a zillion continuity errors now.
Merging the teletext stream into an existing transport stream seems to work, provided the target service is resynchronised to the new teletext pid (in tsp -P svresync -p <teletext pid> <service id>). This seems rather a brute force method but there doesn't seem to be any way to re-time the merged pid itself to a reference PCR with tsduck.

I think it's ready to merge but would appreciate more testing, particularly against real DVB hardware, before I release.

[premultiply]

I think Teletext PID without PTS may be the preferred way here.

Or it needs a PCR reference input (e.g. video stream).

[ZXGuesser]

Certainly that was my feeling, that it seems like every teletext decoder is happy ignoring the lack of a PTS, but @SonnyWalkman wanted it for some reason! 😆
IIRC without the PTS tsduck is unable to calculate the bitrate automatically.
I ought to be able to add another config flag to make it optional without too much difficulty. Reading in a reference stream to synchronize to is too complicated in my view and means adding loads more mpeg gubbins to vbit2

[ZXGuesser]

Added another format tsnpts to suppress the PCR and PTS 438b9a5

[SonnyWalkman]

Hello @ZXGuesser and @premultiply,
Thank you for the efforts in adding the code to encapsulate the PES into a DVB TS frame.

@SonnyWalkman wanted it for some reason! 😆

The reasoning behind the PTS and PCR functionality was to acknowledge the DVB standard in providing a PTS to cater for device requiring a PTS even though the device could disregard the PTS altogether.

@ZXGuesser,

Is there a chance to add my fix to stop killing the VBIT process if the remote connection receives zero (TCP disconnect) and negative value on error. Should be handled correctly.

#38

@peterkvt80 may not be able/available to look at the issues with the remote commands returning errors. 'P', 'L' and 'R' commands are broken. If we could resolve these few issues, then there be a few more happy users.

[premultiply]

The reasoning behind the PTS and PCR functionality was to acknowledge the DVB standard in providing a PTS to cater for device requiring a PTS even though the device could disregard the PTS altogether.

I have not found any device with a requirement for PTS on TXT PID yet. They all ignore it completly even if it exists.
Some public broadcasters sent out async PTS on their "HD channels" (simple PID-copy from "SD channel") for years. Nobody cared about it.

[SonnyWalkman]

Just enforcing the specification.

Taken for en_300472v010401 2017

Annex A (informative):
PTS transmission
It appears that the wording "PTS and other optional fields may be present in the PES header" in clause 4.2 on the PES
packet format has led to multiple interpretations.
Though the original intention of the authors was to express that the PTS should always be included in the
transmission - hence the choice of the PES format - it was also recognized that not all decoders might need the accuracy
offered by the PTS. Therefore, clause 5 on the Teletext decoder model, includes the specification of a maximum
retention time of 40 ms for the Bttx buffer of the System Target Decoder's buffer model. By adding this timing
constraint to the transmission channel it was ensured that decoders could work with reasonable accuracy even if the
PTS values in the coded Teletext stream were ignored.
In practice, however, this specification resulted in at least two ways of conveying and subsequently decoding Teletext
data:
• the method where PTS is indeed transmitted and actually used for decoding (and for validation of the 40 ms
timing constraint in the multiplex), and
• the method where not only the PTS is not used for decoding, but is also not transmitted.
As for legacy reasons the transmission of PTS cannot be changed from mandatory to optional (or vice versa depending
on the interpretation of the present document), the recommended practice for new implementations is as follows:
• encoders should support the insertion of Presentation Time Stamps whenever possible, and
• decoders may use the PTS to synchronize the decoding process, but will also need be able to perform the
decoding process before the maximum retention time of 40 ms is elapsed.

[premultiply]

Yep, from todays all-IP practical playout system design it is not easy to mux a teletext PID with a valid PTS as it was in the analog or digital SD days.

IF you (still) have an encoder supporting teletext at all, most of the times this requires a teletext insertion into SDI using SMPTE2031 or OP-47 insertion and transport at the input of each encoding device for each tv service.

But most of the audio+video encoding systems do not have any support for teletext so the only way to go is to add it afterwards by muxing the teletext PID (without PTS) into the the "main" TS with video and audio.

[ZXGuesser]

Released as v2.6.0