Real Time transcription of Twilio call: transaction_key":"deprecated"

[Henzelix]

I'm trying to get the transcription of inbound payload from Twilio Voice call. My code is heavily inspired by this blog post https://blog.deepgram.com/deepgram-twilio-streaming/

                        media = json.loads(data)['media']
                        chunk = base64.b64decode(media['payload'])
                        if media['track'] == 'inbound':
                            if current_status=='in-progress' and created_call_sid==callback_call_sid:
                                if deepgram_started==False:
                                    p1 = Process(target=handleDeepgram, args=[ws_array, deepgram_payload])
                                    p1.start()
                                    deepgram_started = True
                                if len(ws_array)>0:
                                    print(f"WS ARRAY: {ws_array}")
                                    if inbound_chunks_started:
                                        if latest_inbound_timestamp + 20 < int(media['timestamp']):
                                            bytes_to_fill = 8 * (int(media['timestamp']) - (latest_inbound_timestamp + 20))
                                            # NOTE: 0xff is silence for mulaw audio
                                            # and there are 8 bytes per ms of data for our format (8 bit, 8000 Hz)
                                            inbuffer.extend(b'\xff' * bytes_to_fill)
                                else:
                                    # make it known that inbound chunks have started arriving
                                    inbound_chunks_started = True
                                    latest_inbound_timestamp = int(media['timestamp'])
                                    # this basically sets the starting point for outbound timestamps
                                    latest_outbound_timestamp = int(media['timestamp']) - 20
                                latest_inbound_timestamp = int(media['timestamp'])
                                # extend the inbound audio buffer with data
                                inbuffer.extend(chunk)



                while len(inbuffer) >= BUFFER_SIZE:
                    print('petla')
                    asinbound = AudioSegment(inbuffer[:BUFFER_SIZE], sample_width=1, frame_rate=8000, channels=1)

                    print("petla wysylanie")

                    deepgram_payload.put(inbuffer)

                    # clearing buffers
                    inbuffer = inbuffer[BUFFER_SIZE:]

In this file I'm opening Websocket connection to Deepgram:

import websocket
from pprint import pprint
from sample_bytes import samplee
from functools import partial

previous_length = 0
ws = None

def handleDeepgram(ws_array, deepgram_payload):
    global previous_length, ws

    extra_headers = {
		'Authorization': 'Token xxxxxxxxx' # API Key I'm using 100% works
	}
    
    websocket.enableTrace(True)
    ws = websocket.WebSocketApp(
        'wss://api.deepgram.com/v1/listen?encoding=mulaw&sample_rate=8000&channels=1&multichannel=false',
        header = extra_headers,
        on_message = on_message,
        on_error = on_error,
        on_close = on_close
    )

    ws_array.append(ws)

    ws.deepgram_payload = deepgram_payload

    # on_open_func = partial(on_open, deepgram_payload)
    ws.on_open = on_open
    ws.run_forever()


def on_message(ws, message):
    print(f"wiadomosc: {message}")

def on_error(ws, error):
    print(f"BLAD: {error}")

def on_close(ws, close_status_code, close_msg):
    # Because on_close was triggered, we know the opcode = 8
    print("on_close args:")
    if close_status_code or close_msg:
        print("close status code: " + str(close_status_code))
        print("close message: " + str(close_msg))

def on_open(ws):
    print('dziala dalej')
    deepgram_payload = ws.deepgram_payload
    while not deepgram_payload.empty():
        payload = deepgram_payload.get()
        sendPayloadToDeepgram(ws, payload)


def sendPayloadToDeepgram(webs, payload):
    try:
        # pprint(f"\TRYING: {payload}")
        webs.send(payload)
        print('przeszlo send')
        pprint(f"\nWYSLANO: {payload}")
    except ConnectionError as e:
        print(f"blad wysylania: {e}")

def closeDeepgram(webs):
    try:
        webs.close()
        print("ZAMKNIETO DEEPGRAM")
    except Exception as e:
        print(f"blad zamykania: {e}")

This is the data that I'm sending to Deepgram: b'\xef\xee\xee\xef\xf0\xf3\xf8\xfd{wspnmlllmnoprtvwwwxyyzxxyzz{|}}}~\xff\xfc\xf8\xf7\xf7\xf7\xf7\xf4\xf2\xf1\xf2\xf3\xf5\xf6\xf6\xf5\xf6\xf9\xfb\xfc\xfd\xfc\xfb\xfc\xfe\x7f~~\xfe\xfe\x7f}{{||{wvxyzzxvvxwuuuurprstuttu{\x7f\xff\xfd\xfc\xfc\xfa\xf7\xf6\xf5\xf7\xfc\xfc\xfb\xfa (it's too long to paste the entire thing)

But this is the response I get from Deepgram: fin=1 opcode=1 data=b'{"transaction_key":"deprecated","request_id":"85a94263-1423-4025-aaee-76f0a1f84201","sha256":"xxxxxxx","created":"2023-05-17T22:13:12.720Z","duration":0.0,"channels":0}'

[nikolawhallon]

This line:

asinbound = AudioSegment(inbuffer[:BUFFER_SIZE], sample_width=1, frame_rate=8000, channels=1)

is creating an AudioSegment object, which is not quite raw audio - the example in the blog post uses AugioSegments to help mix the inbound and outbound channels into a single stereo audio stream, but here you are looking to transcribe just one stream, so this step isn't needed

Can you try removing that line, and then doing this for the send:

deepgram_payload.put(inbuffer[:BUFFER_SIZE])

(or whatever the logical equivalent of the above line is, if I'm not massaging types quite correctly there)

[nikolawhallon]

By the time you get to the deepgram_payload.put part, you will have your raw audio in inbuffer- this is a great opportunity to use Deepgram's streaming test suite to debug the audio by this point - https://github.com/deepgram/streaming-test-suite#3-streaming-other-audio - that section contains info on how you can stream that audio to a test server which will dump the audio into a file for inspection, to make sure the raw audio is not corrupt (one could use a tool like ffmpeg or audacity to convert/inspect the raw 8000 Hz mulaw).

[nikolawhallon]

While the above advice is reasonable, I think it may not be the core issue - I tried out snippets of the code, and the websocket-client library is doing something interesting here.

the line:

webs.send(payload)

will send the payload as a text websocket message. When I use it to send some audio to Deepgram, it prints in my terminal something like:

++Sent raw: b'\x81\xfe\x03\xe8)t\x0f....'
++Sent decoded: fin=1 opcode=1 data=b'RIFFF\xc4\t\x00WAVEfmt \x10\x00\x00\x00....'

it is sending the binary as text (opcode=1) (also one can see that I'm sending a containerized WAV file in this example...)
If I instead do:

webs.send(payload, websocket.ABNF.OPCODE_BINARY)

I get the following output:

++Sent raw: b'\x81\xfe\x03\xe8)t\x0f....'
++Sent decoded: fin=1 opcode=2 data=b'RIFFF\xc4\t\x00WAVEfmt \x10\x00\x00\x00....'

opcode=2! which is binary - Deepgram will stream the binary messages through its pipeline for transcription - one I made this change and used webs.send(payload, websocket.ABNF.OPCODE_BINARY) I was able to stream a file through to Deepgram and get the transcript!

My example isn't the prettiest code, so I won't share the full thing right now, but I'd be happy to flesh out the example with this websocket-client library. And hopefully this change works for you too!

[Henzelix]

Thank you, this totally worked 😍

This is my code:

main.py:

while len(inbuffer) >= BUFFER_SIZE:
                    print('petla')
                    asinbound = AudioSegment(inbuffer[:BUFFER_SIZE], sample_width=1, frame_rate=8000, channels=1)

                    print("petla wysylanie")

                    deepgram_payload.put(bytes(asinbound.raw_data))

                    # clearing buffers
                    inbuffer = inbuffer[BUFFER_SIZE:]

deepgram_websocket.py:

def sendPayloadToDeepgram(webs, payload):
    try:
        # pprint(f"\TRYING: {payload}")
        webs.send(payload, websocket.ABNF.OPCODE_BINARY)
        print('przeszlo send')
        pprint(f"\nWYSLANO: {payload}")
    except ConnectionError as e:
        print(f"blad wysylania: {e}")