MS Teams - Bot framework bot is slow

[gykung]

Steps to reproduce

I have a bot, written in Python, bot framework, run on a local VM, connected via ngrok to my bot app for my enterprise. User messages take about 5-7 seconds before it reaches the bot/VM. Does anyone else face a similar issue? What’s the typical latency expected?

One interesting observation: during the crowdstrike incident, the bot responded significantly faster. My enterprise was not affected by crowdstrike.

Sorry for asking this question here, I am not sure where else to ask this.

Expected behavior

Nearly zero lag.

Actual behavior

5-7 seconds lag.

Error details

No response

[Meghana-MSFT]

Hello @gykung - Thanks for raising your query:

The latency experienced by you where user messages take about 5-7 seconds to reach the bot/VM when running on a local VM connected via NgRok is not typically expected. This delay could be due to various factors such as network configuration, VM performance, or ngrok settings.

It's recommended to check and optimize these aspects to reduce latency:
Optimize VM performance: Ensure the VM has enough resources (CPU, memory) to handle requests efficiently.
Create and host a dev tunnel: (Alternative for NgRok and recommended by Microsoft) https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/get-started?tabs=windows
Upgrade ngrok plan: Consider using a paid ngrok plan for better routing and reduced latency.
Test direct hosting: Consider hosting the bot on a cloud provider (e.g., Azure, AWS) instead of a local VM to reduce the latency caused by local network and NgRok/Dev Tunnel.

[Meghana-MSFT]

@gykung - Could you please share the bot id with us at [email protected]. Also, please note that

A host (and a bot is just a webapp host) not receiving a message or it's slow getting there is 999 times out of 1000 a network issue.

Python, like JS, is a single threaded core. So, it's not hard to imagine in a very high load situation where the bot is just backed up. But this is rare since, A) Very high load, B) Mostly applies to compute bound situations. i.e., The bot is running some very complex algorithm during each message.

[gykung]

Hi Meghana, Thank you very much for your reply. The MicrosoftAppId is "515d62a9-34b7-4f90-b213-8b4c7721951f". In case there are analytics on your end that could help us solve the issue, feel free to share with me. I agree it could ultimately be a network issue as it is taking several seconds for the incoming request to reach the Python bot app. I've tried ping but there is no lag. If there are specific Azure admin settings, Azure-related network issues we should trace, feel free to suggest. Thank you. Regards, Guangyi @gykung

…

On Thu, Sep 19, 2024 at 1:26 PM Meghana-MSFT ***@***.***> wrote: @gykung <https://github.com/gykung> - Could you please share the bot id with us at ***@***.*** Also, please note that A host (and a bot is just a webapp host) not receiving a message or it's slow getting there is 999 times out of 1000 a network issue. Python, like JS, is a single threaded core. So, it's not hard to imagine in a very high load situation where the bot is just backed up. But this is rare since, A) Very high load, B) Mostly applies to compute bound situations. i.e., The bot is running some very complex algorithm during each message. — Reply to this email directly, view it on GitHub <#11583 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQWH4JVEWJPUUXBQ4FGJHMDZXJN7PAVCNFSM6AAAAABOE6567GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRQGAYDMMZYGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Meghana-MSFT]

@gykung - We have shared the MicrosoftAppId with engineering team, we will keep you posted on the investigation details. Thank you.

[gykung]

Hi Meghana,Would you or your team have any insights/findings? Thank you.Guangyi @gykung On 20 Sep 2024, at 5:08 PM, Meghana-MSFT ***@***.***> wrote: @gykung - We have shared the MicrosoftAppId with engineering team, we will keep you posted on the investigation details. Thank you. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[gykung]

I presume that the traffic flows this way: User-->teams.microsoft.com-->botframework.com-->ngrok tunnel --> VM hosting the bot, and then in reverse direction to return the response to the user.

I've tried tracert with directly to my ngrok address, with teams.microsoft.com, and botframework.com.

Notably, botframework.com has the most significant lag.

[gykung]

@Meghana-MSFT I've fixed the issue. It was an unexpected issue in the app. Somehow the tracert and network tracing had been giving false clues. Thanks for the time spent on this.

[gykung]

@Meghana-MSFT I've fixed the issue. It was an unexpected issue in the app. Somehow the tracert and network tracing had been giving false clues. Thanks for the time spent on this.

[microsoft-github-policy-service]

Tell us about your experience!

Hi gykung! This issue is closed in our system. We would like your feedback on your experience with our support team and Platform.

Best regards,
Teams Platform