Display the transcription in sync with the video

[luangreydive]

I need to display the transcription while the video plays.

Ideally it would sync the transcription to the video, in a similar way Deepgram does in their demos.

If anybody can share a script, that would be great.

Thank you!

[SandraRodgers]

Hi, thanks for your question.

Can you tell me more about what you are building? Is the video coming from a browser page, like a Youtube video? Or is it a live stream? I think the answer to your question depends on your setup - where the video comes from and how you are pulling in the audio to transcribe. Also, could you tell me which Deepgram demo you're referring to?

We'll definitely do our best to help and I really appreciate you taking the time to send us your question.

Sandra

[luangreydive]

I'm attaching a screenshot of the Deempgram demo I was referring to. The video is on a webpage, yes. Our tool records it. Thank you! El vie, 3 mar 2023 a las 11:48, Sandra Rodgers ***@***.***>) escribió:

…

Hi, thanks for your question. Can you tell me more about what you are building? Is the video coming from a browser page, like a Youtube video? Or is it a live stream? I think the answer to your question depends on your setup - where the video comes from and how you are pulling in the audio to transcribe. Also, could you tell me which Deepgram demo you're referring to? We'll definitely do our best to help and I really appreciate you taking the time to send us your question. Sandra — Reply to this email directly, view it on GitHub <#96 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZYVRZZBKRRZ5QUOK544Q3DW2IADLANCNFSM6AAAAAAVNUFP64> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- *Luan Rodriguez* *|* CEO <https://www.greydive.com/?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan> www.greydive.com <https://www.greydive.com/?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan> Lee los últimos artículos de nuestro blog <https://www.greydive.com/blog?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan> Suscríbete a e-commerce lab. <https://www.greydive.com/e-commerce-lab?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan>

[briancbarrow]

Hi @luangreydive,

I didn't see anything attached. Is the video on the webpage a video file, or is this a live streaming video?

[luangreydive]

I sent the attachment through email, here I'm attaching it on Github.

Thanks!

[SandraRodgers]

Hi @luangreydive,

Is this the screenshot of the demo from Deepgram? It would really help if you could send the link the demo. If I could see which demo it is (we have lots of demos so I'm not sure where you got this one), then I could go check the code that was used to see how the demo syncs the subtitles to the video audio.

Here's an idea for how that script to sync the text words to the audio words in the video might work. The Deepgram object response that you get of the transcription includes a property alternatives which gives a word with its start and end time. That can be used to sync up to the time in the video where the speaker is saying the word. You would need to know the start and end of when the word is vocalized in the video, to match it up with the text in the Deepgram transcription. You would take the transcript and the appropriate word timings, track where they are in the audio, and use CSS classes to change the formatting of the words as it moves along.

I'm going to continue to ask around to see if I can find some code example that does that. In the meantime, you've given me a great idea for a tutorial to write.

Sandra

[briancbarrow]

Cool, thanks. Just to make sure I understand, your tool records video. You want to get transcriptions for the video after it is done recording and you have posted it to the webpage, and you want to sync that transcription to the video as it plays.

Is that right?

[luangreydive]

That's right. We already have the transcriptions on the website, we only need to sync them. If there is a script already done that would save us time. Thanks! El lun, 6 mar 2023 a las 19:36, Brian Barrow ***@***.***>) escribió:

…

Cool, thanks. Just to make sure I understand, your tool records video. You want to get transcriptions for the video after it is done recording and you have posted it to the webpage, and you want to sync that transcription to the video as it plays. Is that right? — Reply to this email directly, view it on GitHub <#96 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZYVRZ5KVBKRNGHGHYF457TW2ZRG7ANCNFSM6AAAAAAVNUFP64> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Luan Rodriguez* *|* CEO <https://www.greydive.com/?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan> www.greydive.com <https://www.greydive.com/?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan> Lee los últimos artículos de nuestro blog <https://www.greydive.com/blog?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan> Suscríbete a e-commerce lab. <https://www.greydive.com/e-commerce-lab?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan>

[briancbarrow]

Gotcha. I would recommend using something like ffmpeg to add the subtitles to the video file. This stackoverflow question would be a good place to start looking. https://stackoverflow.com/questions/8672809/use-ffmpeg-to-add-text-subtitles/17584272#17584272

[luangreydive]

That is not exactly what we want :) We want to have a box with the subtitles and have them sync with the video. Thanks! El mié, 8 mar 2023 a las 11:23, Brian Barrow ***@***.***>) escribió:

…

Gotcha. I would recommend using something like ffmpeg to add the subtitles to the video file. This stackoverflow question would be a good place to start looking. https://stackoverflow.com/questions/8672809/use-ffmpeg-to-add-text-subtitles/17584272#17584272 — Reply to this email directly, view it on GitHub <#96 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZYVRZ45EOLDBYC2MZ7SPZLW3CI6ZANCNFSM6AAAAAAVNUFP64> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Luan Rodriguez* *|* CEO <https://www.greydive.com/?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan> www.greydive.com <https://www.greydive.com/?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan> Lee los últimos artículos de nuestro blog <https://www.greydive.com/blog?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan> Suscríbete a e-commerce lab. <https://www.greydive.com/e-commerce-lab?utm_source=firma-mail-luan&utm_medium=firma-mail-luan&utm_campaign=firma-mail-luan&utm_id=firma-mail-luan>

[briancbarrow]

Okay. Are you using one of the SDKs to get the transcriptions? There are helper functions on there that can generate an .srt or .vtt file which you can then use ffmpeg to add them as subtitles on the video file.

Here is the toSRT function in the Node SDK as an example. https://github.com/deepgram/deepgram-node-sdk/blob/ba433b728848da6e360b37658f2f51b7becc26f0/src/types/prerecordedTranscriptionResponse.ts#L47

Does that help? Sorry if I don't understand exactly what you are trying to accomplish. If the ffmpeg solution doesn't work you could write a script that reads the SRT file and displays the appropriate text based on where the video is currently playing. We don't have a script for that right now though.