Skip to content
This repository has been archived by the owner on Jul 22, 2024. It is now read-only.

EWS Throttling issues being rolled out worldwide on 365 #293

Open
vlw007 opened this issue Apr 12, 2023 · 25 comments
Open

EWS Throttling issues being rolled out worldwide on 365 #293

vlw007 opened this issue Apr 12, 2023 · 25 comments

Comments

@vlw007
Copy link

vlw007 commented Apr 12, 2023

Starting April 6, we are seeing EWS throttling issues when there is barely any EWS activity on a tenant.

The error looks like this:
[You have exceeded the available concurrent connections for your account. Try again once your other requests have completed.]
ErrorDetails; Key=[Policy], Value=[MaxConcurrency]
ErrorDetails; Key=[MaxConcurrencyLimit], Value=[27]
ErrorDetails; Key=[ErrorMessage], Value=[This operation exceeds the throttling budget for policy part 'MaxConcurrency', policy value '27', Budget type: 'Ews'. Suggested backoff time 0 ms.]

When this step executes, Our SaaS platform has only a single thread (no concurrency) and our backoff time is way larger than zero ms.

We made the request to remove throttling policies for 90 days. The support request suggested that we wait 15 minutes for the policy to be updated

We waited two hours and there does not appear to be any difference. The same error appears. Another application using EWS has also started having problems at almost the exact same time.

Does anyone notice this happening too?
@cmcveen
Copy link

cmcveen commented Apr 12, 2023

Yes, I have noticed this same issue over the past couple of days in a few tenants.

@mariusan
Copy link

mariusan commented Apr 12, 2023 via email •
edited
Loading

@joshmadams
Copy link

I am seeing this error in our tenant as well.

@PhilbyTheGreat
Copy link

We are seeing this in our tenant too.

@PhilbyTheGreat PhilbyTheGreat mentioned this issue Apr 13, 2023
Open
@mariusan
Copy link

Below is the result of a trace. This is in response to the very first call and there are no other uses of the account.

<Trace Tag="EwsResponse" Tid="5" Time="2023-04-13 06:00:12Z" Version="[2.2.1.0](http://2.2.1.0/)">
  <?xml version="1.0" encoding="utf-8"?>
  <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Body>
      <s:Fault>
        <faultcode xmlns:a="http://schemas.microsoft.com/exchange/services/2006/types">a:ErrorExceededConnectionCount</faultcode>
        <faultstring xml:lang="en-US">You have exceeded the available concurrent connections for your account.  Try again once your other requests have completed.</faultstring>
        <detail>
          <e:ResponseCode xmlns:e="http://schemas.microsoft.com/exchange/services/2006/errors">ErrorExceededConnectionCount</e:ResponseCode>
          <e:Message xmlns:e="http://schemas.microsoft.com/exchange/services/2006/errors">You have exceeded the available concurrent connections for your account.  Try again once your other requests have completed.</e:Message>
          <t:MessageXml xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
            <t:Value Name="Policy">MaxConcurrency</t:Value>
            <t:Value Name="MaxConcurrencyLimit">27</t:Value>
            <t:Value Name="ErrorMessage">This operation exceeds the throttling budget for policy part 'MaxConcurrency', policy value '27',  Budget type: 'Ews'.  Suggested backoff time 0 ms.</t:Value>
          </t:MessageXml>
        </detail>
      </s:Fault>
    </s:Body>
  </s:Envelope>
</Trace>

@halliballi123
Copy link

same here, problem started yesterday evening.

@joshmadams
Copy link

This issue started around 7am EST yesterday (4/12/23) for us.

@Yup-maker
Copy link

5 customers also have this problem with Public Folders. (However, I myself can not reproduce it)

This is my Exchange Online Version: 15.20.6277.038 V2018_01_08
This is one of my customers: 15.20.6298.028 V2018_01_08

@halliballi123
Copy link

halliballi123 commented Apr 14, 2023 •
edited
Loading

5 customers also have this problem with Public Folders. (However, I myself can not reproduce it)

This is my Exchange Online Version: 15.20.6277.038 V2018_01_08 This is one of my customers: 15.20.6298.028 V2018_01_08

In our case it's also public folders. We create contacts in public folders.
Version is 15.20.6298.30

@mariusan
Copy link

Might be the same issue:
https://admin.microsoft.com/Adminportal/Home#/servicehealth/:/alerts/EX540990

@vlw007
Copy link
Author

vlw007 commented Apr 14, 2023 via email

@Yup-maker
Copy link

Yes by using ExchangeServerInfo: service.ServerInfo?.ToString()

Are you able to get the Exchange Online Version via an API call ?
…
________________________________ From: halliballi123 @.> Sent: Friday, April 14, 2023 6:11:58 AM To: OfficeDev/ews-managed-api @.> Cc: Vernon L. Weitzman (Cira Apps) @.>; Author @.> Subject: Re: [OfficeDev/ews-managed-api] EWS Throttling issues being rolled out worldwide on 365 (Issue #293) 5 customers also have this problem with Public Folders. (However, I myself can not reproduce it) This is my Exchange Online Version: 15.20.6277.038 V2018_01_08 This is one of my customers: 15.20.6298.028 V2018_01_08 In our case it's also public folders. We create contacts in public folders. — Reply to this email directly, view it on GitHub<#293 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJDIBQRPNUFYCARDXKYCJ63XBDTC5ANCNFSM6AAAAAAW4GA22A. You are receiving this because you authored the thread.Message ID: @.***>

@vlw007
Copy link
Author

vlw007 commented Apr 14, 2023 via email

@ToS0
Copy link

ToS0 commented Apr 14, 2023

Same here - since yesterday

@viebrix
Copy link

viebrix commented Apr 17, 2023

We also had the issue with public folder, but it seems it is right now solved

@Andy1860
Copy link

Same here, seems to be solved.

@viebrix
Copy link

viebrix commented Apr 18, 2023

Greg Taylor from the Exchange Teams asked here why and how people use public folders:
techcommunity - exchange-team-blog
I think this is a great chance to help the team to improve exchange for us.

@halliballi123
Copy link

problem still there

@viebrix
Copy link

viebrix commented Apr 18, 2023

@halliballi123
I'm only user and not experience in this topic. But did you try to request to remove throttling policies for 90 days, as vlw007 is describing in his first post? We also did this step last week. Maybe this makes a difference?

@halliballi123
Copy link

I don't know how and where to put that request

@viebrix
Copy link

viebrix commented Apr 18, 2023

Found this description in the web:
Sign in to Microsoft 365 Admin Center and click on the Help icon in the top corner of the screen.
Search for EWS and select the Increase EWS Throttling Policy article that is suggested.In the Run diagnostics section that is displayed, select Run Tests. If the diagnostic returns a The Exchange Web Services are throttled message, you have the option to change (ease) the policy to allow your migration to complete.
Choose the Duration (Days) that you want to relax the policy for from the drop down menu. You can choose between 20, 60 and 90 days. Place a tick in the checkbox to acknowledge that want to make the specified changes.
Select Update Settings to apply the changes to the policy. Changes can take up to 30 minutes to take effect.
(source of text, support.cloudm.io)

@Andy1860
Copy link

Found this description in the web: Sign in to Microsoft 365 Admin Center and click on the Help icon in the top corner of the screen. Search for EWS and select the Increase EWS Throttling Policy article that is suggested.In the Run diagnostics section that is displayed, select Run Tests. If the diagnostic returns a The Exchange Web Services are throttled message, you have the option to change (ease) the policy to allow your migration to complete. Choose the Duration (Days) that you want to relax the policy for from the drop down menu. You can choose between 20, 60 and 90 days. Place a tick in the checkbox to acknowledge that want to make the specified changes. Select Update Settings to apply the changes to the policy. Changes can take up to 30 minutes to take effect. (source of text, support.cloudm.io)

I already tried this last week when the problems started with no improvement.

@Andy1860
Copy link

I think you just have to wait until it works again. We haven't had any more problems since yesterday afternoon.

Last update in M365 Health Center:

April 17, 2023 7:24 PM
Title: Some users may be unable to access, view, add, or select public folders and calendars in Outlook on the web
User impact: Users may be unable to access, view, add, or select public folders and calendars in Outlook on the web.
Current status: We've determined the status of the fix deployment to currently stand at 80 percent saturation, and we’ll provide an estimated timeline for resolution once available.
Scope of impact: Some users attempting to perform multiple public folder or calendar related actions in Outlook on the web may be impacted.
Start time: Tuesday, April 11, 2023, 1:37 PM (11:37 AM UTC)
Root cause: A recent service update contained a code issue that’s preventing users from accessing, viewing, adding, or selecting public folders and calendars in Outlook on the web.
Next update by: Tuesday, April 18, 2023, 8:00 PM (6:00 PM UTC)

@halliballi123
Copy link

halliballi123 commented Apr 18, 2023 •
edited
Loading

I think you just have to wait until it works again. We haven't had any more problems since yesterday afternoon.

Last update in M365 Health Center:

April 17, 2023 7:24 PM Title: Some users may be unable to access, view, add, or select public folders and calendars in Outlook on the web User impact: Users may be unable to access, view, add, or select public folders and calendars in Outlook on the web. Current status: We've determined the status of the fix deployment to currently stand at 80 percent saturation, and we’ll provide an estimated timeline for resolution once available. Scope of impact: Some users attempting to perform multiple public folder or calendar related actions in Outlook on the web may be impacted. Start time: Tuesday, April 11, 2023, 1:37 PM (11:37 AM UTC) Root cause: A recent service update contained a code issue that’s preventing users from accessing, viewing, adding, or selecting public folders and calendars in Outlook on the web. Next update by: Tuesday, April 18, 2023, 8:00 PM (6:00 PM UTC)

thanks, we tried the fix but still no improvement, so maybe you are right. We have to wait ...

@halliballi123
Copy link

finally it is working again ...

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@mariusan @halliballi123 @ToS0 @viebrix @vlw007 @cmcveen @joshmadams @PhilbyTheGreat @Yup-maker @Andy1860