Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Idempotency tests are too flaky #3339

Closed
andreaTP opened this issue Sep 20, 2023 · 10 comments
Closed

[CI] Idempotency tests are too flaky #3339

andreaTP opened this issue Sep 20, 2023 · 10 comments

Comments

@andreaTP
Copy link
Contributor

I noticed this during the last couple of PRs, I think that this is causing some level of friction:

  • maintainers need to restart multiple times the CI build
  • the PR is always marked as red in the PRs list view
  • we keep receiving notifications about failing jobs
@baywet
Copy link
Member

baywet commented Sep 20, 2023

Thanks for starting the conversation on that.
I've noticed that too. What I didn't get time to investigate is whether this is caused by remaining issues in the code or by some kind of setup issue with the tests themselves.
Do you have any insights to share?

@andreaTP
Copy link
Contributor Author

Do you have any insights to share?

No 😞 , I just noticed it. I was guessing that this turns true (but I cannot prove it):

is caused by remaining issues in the code

The uploaded artifacts from the actions are not that useful as they are bugged (overwriting files instead of keeping the separate folders).

I don't have much time to dedicate to this now unfortunately.

@baywet
Copy link
Member

baywet commented Sep 20, 2023

Fair enough, let's at least get to a place where we have good artifacts to investigate potential issues. Scheduled for 1.8 at the moment. Let's revisit once we make progress on other topics.

@baywet
Copy link
Member

baywet commented Nov 2, 2023

update on this. With the recent improvements we've made, the only place I still see a flaky behaviour is for the twitter API at this point for the last month/6 weeks. (for the stable languages)
I suspect this is caused by a pattern that's still not supported (like different successful responses)
Is that your current understanding of the situation?

@andreaTP
Copy link
Contributor Author

andreaTP commented Nov 3, 2023

Kind of ... looking at the most recent failures:
https://github.com/microsoft/kiota/actions/workflows/idempotency-tests.yml

yes Twitter is the one failing more often, should we remove it from the idempotency tests?

@sebastienlevert
Copy link
Contributor

What makes twitter different from the others? Anything specific in the description that we should be looking at?

@baywet
Copy link
Member

baywet commented Nov 3, 2023

To Seb's point, before we remove it, I think we should try to identify the source. If it's caused by something that "is wrong in the description", I don't have any issue to remove it. If we uncover something though, I'd rather fix the code.

@andreaTP
Copy link
Contributor Author

andreaTP commented Nov 3, 2023

As you prefer, of course fixing the issues is much better!
But we are still excluding quite a bunch of idempotency tests (which are consistently failing).

So, removing the file, is like an "ignore for all" that will, at least, reduce the noise.

@baywet
Copy link
Member

baywet commented Nov 30, 2023

@andreaTP while working on #3814 and with the added improvements of #3813 I think we indirectly addressed this issue.
I'll leave it open so we can monitor the situation but twitter, the problematic API description, is not randomly failing anymore.
I think this is the combination of:

  • resolving a couple of opportunities for collisions
  • resolving a couple of race conditions

@sebastienlevert
Copy link
Contributor

Hey @andreaTP, it's been a week and we haven't see any idempotency issues coming up. We'll close for now, feel free to re-open if they start showing up again!

@github-project-automation github-project-automation bot moved this from Todo to Done in Kiota Dec 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests

3 participants