OpenFlexure Microscope communication reliability (unresponsive on Hugging Face)

[sgbaird]

I decided to run the script from a terminal instead, and added a print statement to show the receipt of a command. Interestingly, the device is still receiving the command and not throwing errors, but the HF space doesn't receive the message back. If I refresh the space and use the same credentials, then it seems to work OK.

Maybe HF just isn't receiving the message.

cc @kenzo-aspuru-takata

[sgbaird]

@SissiFeng maybe you could take a look at the hugging face spaces code and see where the issue related to messages might be happening?

[SissiFeng]

I suggest adding several code blocks for future debugging:

1. Add logging at key points to track the sending and receiving of commands.
2. Implement a periodic status check mechanism to ensure the connection between the device and the HF space remains active.
3. Implement a retry mechanism for failed commands:
   import tenacity
   @tenacity.retry(stop=tenacity.stop_after_attempt(3), wait=tenacity.wait_fixed(2))
   def send_command_with_retry(command):
   pass

Additionally, I've encountered this before - consider using websockets for real-time bidirectional communication, which can be more stable.

[SissiFeng]

After much testing, it seems a bug on the HiveMQ side. HiveMQ always stops connection with mqtt no matter streamlit or gradio deployment. For example, I’m able to connect with mosquitto and it’s more stable.

I plan to tweak the heartbeat settings and check the TLS certificate configuration to improve the stability of the HiveMQ connection.

[sgbaird]

Sounds good! There could also be some modification needed on the device side. I'm not sure what QoS is set there.

The light mixing demo app has had a fairly high uptime - recently, the issue was mainly on the device side, due to a certificate expiring.

Do you see any major differences in how the MQTT is implemented? https://huggingface.co/spaces/AccelerationConsortium/light-mixing

Could you share a link to your code?

[SissiFeng]

https://github.com/SissiFeng/openflexure-microscope

My MQTT setup follows the structure from the link above: specifying the QoS level and automatically handling TLS settings and so on. Currently, I'm receiving the following message:

Maybe my HiveMQ configuration isn't stable enough. Could you please debug it with another account?

[sgbaird]

Thanks for looking into this! HiveMQ has been reliable in other cases, so I'm not sure it's purely a HiveMQ issue. Maybe we can chat about this on Monday. Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Sissi Feng ***@***.***> Sent: Saturday, October 19, 2024 8:14:39 AM To: AccelerationConsortium/ac-training-lab ***@***.***> Cc: Sterling Baird ***@***.***>; Author ***@***.***> Subject: Re: [AccelerationConsortium/ac-training-lab] OpenFlexure Microscope communication reliability (unresponsive on Hugging Face) (Issue #81) https://github.com/SissiFeng/openflexure-microscope My MQTT setup follows the structure from the link above: specifying the QoS level and automatically handling TLS settings and so on. Currently, I'm receiving the following message: 2024-10-19.08.03.04.png (view on web)<https://github.com/user-attachments/assets/a219d345-32f3-4415-be69-288a2b8c299c> Maybe my HiveMQ configuration isn't stable enough. Could you please debug it with another account? — Reply to this email directly, view it on GitHub<#81 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AK25ABJJAV742M2DJ7GEUEDZ4JEK3AVCNFSM6AAAAABQD5WMZ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRTG44TSOBXHE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[SissiFeng]

https://huggingface.co/spaces/SissiFeng/microscope

[ac-hardware]

SGB:

At least part of the issue is probably the device code that's running on the microscope. Restarting the script got things responsive again on HF.

It's also running on Python 3.7.3. I think @kenzo-aspuru-takata had trouble getting it installed on a different Python version (something more specific to the openflexure OS perhaps), but maybe that could be affecting things.

Tried again, and observed the behavior:
device:

Received command {'command': 'get_pos'}. Executing
Done.

No response on HF.

EDIT: Checked the hf logs and it seems that maybe on the hf side the code is being called many times. I'm seeing about 10 repeats of printouts for each button click, which would obviously cause some issues. Perhaps the separate file structure and having multiple imports is leading to this. Maybe worth condensing to a self-contained app.py implementation to check.

Here's the device code since I've had trouble finding time to get it uploaded.

#maybe implement security for z axis focus and move


import paho.mqtt.client as mqtt
import json
import time
from queue import Queue
import openflexure_microscope_client


import PIL
from PIL import Image
import os
import math
import shutil
import base64
import io
import random

from my_secrets import HOST, USERNAME, PASSWORD, PORT, MICROSCOPE

time.sleep(30)

m = openflexure_microscope_client.MicroscopeClient("localhost")
rangetopx=20000
rangebottomx=-20000
rangetopy = 20000
rangebottomy =-20000

steps_per_full_camera_movemement_x=3600 #estimate this then slightly undershoot if you are having problems with stitching, these values should work though
steps_per_full_camera_movemement_y=2700

client=mqtt.Client()
client.tls_set()
client.username_pw_set(USERNAME, PASSWORD)

commandq = Queue()

def on_message(client, userdata, message):
    received = json.loads(message.payload.decode("utf-8"))
    if commandq.empty():
        commandq.put(received)

client.on_message=on_message  

client.connect(HOST, port=PORT, keepalive=600, bind_address="")

def image_to_base64_with_metadata(image_path):
    # Open the image and get metadata
    with Image.open(image_path) as img:
        # Create a BytesIO buffer
        with io.BytesIO() as buffer:
            # Save the image to buffer with metadata
            img.save(buffer, format="JPEG", exif=img.info.get('exif'))
            # Get the image data in bytes
            image_bytes = buffer.getvalue()
            
            # Encode the image bytes to a Base64 string
            base64_encoded = base64.b64encode(image_bytes).decode('utf-8')
    
    return base64_encoded

def move(x, y, z = False, relative = False):
    p=m.position
    if relative == False and not (x <= rangetopx and x >= rangebottomx and y <= rangetopy and y >= rangebottomy):
        client.publish(microscope+"/return", payload=json.dumps({"command":"move","complete":False,"error":"out of range"}), qos=2, retain=False)
    elif relative == True and not (p["x"]+x <= rangetopx and p["x"]+x >= rangebottomx and p["y"]+y <= rangetopy and p["y"]+y >= rangebottomy):
        client.publish(microscope+"/return", payload=json.dumps({"command":"move","complete":False,"error":"out of range"}), qos=2, retain=False)
    elif relative:
        p = m.position
        p["x"] += x
        p["y"] += y
        if z == False:
            m.move(p)
        else:
            p["z"] += z
            m.move(p)
        client.publish(microscope+"/return", payload=json.dumps({"command":"move","complete":True}), qos=2, retain=False)
    else:
        p = m.position
        l = {"x":x, "y":y, "z":z}
        if z == False:
            l = {"x":x, "y":y, "z":p["z"]}
        m.move(l)
        client.publish(microscope+"/return", payload=json.dumps({"command":"move","complete":True}), qos=2, retain=False)

def focus(amount="fast"):
    if amount == "fast":
        
        m.autofocus()
        
    elif amount == "huge":
       
        m.autofocus(6000)
        
    elif amount == "medium":
        
        m.autofocus(500)
        
    elif amount == "fine":
       
        m.autofocus(100)
        
    elif amount == "all":
        
        m.autofocus(6000)
        m.autofocus()
        m.autofocus(500)
        m.autofocus(100)
        
    else:
       
        m.autofocus(amount)
       
    client.publish(microscope+"/return", payload=json.dumps({"command":"focus","complete":True}), qos=2, retain=False)

def take_image():
    image = m.capture_image()
    buffered = io.BytesIO()
    image.save(buffered, format="PNG")  # Specify the format
    byte_image = base64.b64encode(buffered.getvalue()).decode('utf-8')
    client.publish(microscope+"/return", payload=json.dumps({"command":"take_image","complete":True,"image":byte_image}), qos=2, retain=False)
    
def scan(c1, c2, ov = 1200, foc = 0):
    for i in m.list_capture_ids():
        m.delete_image(i)
    if os.path.isdir('/home/pi/Downloads/scanfolder'):
        shutil.rmtree('/home/pi/Downloads/scanfolder')
    os.mkdir('/home/pi/Downloads/scanfolder')
    p = m.position
    if isinstance(c1, str):
        x1, y1 = map(int, c1.split())
        x2, y2 = map(int, c2.split())
    else:
        x1 = c1[0]
        y1 = c1[1]
        x2 = c2[0]
        y2 = c2[1]
    xd = abs(x1-x2)
    yd = abs(y1-y2)
    essx = steps_per_full_camera_movemement_x-ov
    essy = steps_per_full_camera_movemement_y-ov
    ssx = xd/math.ceil(xd/essx)
    ssy = yd/math.ceil(yd/essy)
    xsc = math.ceil(xd/essx)
    ysc = math.ceil(yd/essy)
    #move to starting position
    if xsc*ysc > 36:
        client.publish(microscope+"/return", payload=json.dumps({"command":"scan","complete":False,"error":"Too many images"}), qos=2, retain=False)
    else:
        p['x']=min(x1,x2)
        p['y']=min(y1,y2)
        m.move(p)
        #scan and download images
        m.scan(params={'grid':[xsc,ysc,1],'stride_size':[ssx,ssy,0],'autofocus_dz':foc,'filename':'SCAN','use_video_port':True,'bayer':True},wait_on_task=True)
        for i in m.list_capture_ids():
            m.download_from_id(i, "/home/pi/Downloads/scanfolder")
            m.delete_image(i)
        image_list=[]
        for filename in os.listdir("/home/pi/Downloads/scanfolder"):
            image_list.append(image_to_base64_with_metadata("/home/pi/Downloads/scanfolder/"+filename))
        client.publish(microscope+"/return", payload=json.dumps({"command":"scan","complete":True,"images":image_list}), qos=2, retain=False)



    


client.loop_start()

client.subscribe(MICROSCOPE+"/command", qos=2)


print("Waiting for commands..")


while True:
    if not commandq.empty():
        command = commandq.get()
        print(f"Received command {command}. Executing")
        if command["command"] == "move":
            try:
                move(x = command["x"], y = command["y"], z = command["z"], relative = command["relative"])
            except KeyError as e:
                print(str(e))
                client.publish(microscope+"/return", payload=json.dumps({"command":"move","complete":False,"error":str(e)}), qos=2, retain=False)
            except Exception as e:
                print(str(e))
                client.publish(microscope+"/return", payload=json.dumps({"command":"move","complete":False,"error":str(e)}), qos=2, retain=False)
        elif command["command"] == "get_pos":
                client.publish(microscope+"/return", payload=json.dumps({"command":"get_pos","complete":True,"pos":m.position}), qos=2, retain=False)
        elif command["command"] == "focus":
            try:
                focus(amount=command["amount"])
            except KeyError as e:
                print(str(e))
                client.publish(microscope+"/return", payload=json.dumps({"command":"focus","complete":False,"error":str(e)}), qos=2, retain=False)
            except Exception as e:
                print(str(e))
                client.publish(microscope+"/return", payload=json.dumps({"command":"focus","complete":False,"error":str(e)}), qos=2, retain=False)
        elif command["command"] == "take_image":
            take_image()
        elif command["command"] == "scan":
            try:
                scan(c1=command["c1"], c2=command["c2"], ov=command["ov"], foc=command["foc"])
            except KeyError as e:
                print(str(e))
                client.publish(microscope+"/return", payload=json.dumps({"command":"scan","complete":False,"error":str(e)}), qos=2, retain=False)
            except Exception as e:
                print(str(e))
                client.publish(microscope+"/return", payload=json.dumps({"command":"scan","complete":False,"error":str(e)}), qos=2, retain=False)
        else:
            print("Command invalid")
            client.publish(microscope+"/return", payload=json.dumps({"command":"INVALID","complete":False}), qos=2, retain=False)
        print("Done.")

[sgbaird]

Maybe both need to be cross referenced against:

https://ac-microcourses.readthedocs.io/en/latest/courses/hello-world/1.4-hardware-software-communication.html

And

https://ac-microcourses.readthedocs.io/en/latest/courses/hello-world/1.4.1-onboard-led-temp.html

While using the public test broker:

HIVEMQ_USERNAME = "sgbaird"
HIVEMQ_PASSWORD = "D.Pq5gYtejYbU#L"
HIVEMQ_HOST = "248cc294c37642359297f75b7b023374.s2.eu.hivemq.cloud"