Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows build: Enable RBE build in Windows CI #11107

Merged
merged 30 commits into from
Jun 27, 2020
Merged

Windows build: Enable RBE build in Windows CI #11107

merged 30 commits into from
Jun 27, 2020

Conversation

sunjayBhatia
Copy link
Member

@sunjayBhatia sunjayBhatia commented May 7, 2020

Commit Message:
Enable RBE build in Windows CI

  • Bumps envoy-build-tools and bazel-toolchains
  • Bumps envoy-build-ubuntu image tag
    • Running Windows CI via docker uses the same mechanism as Linux to find the image tag, the tags are the same for Windows and Linux images
  • Temporarily adds a patch to fix rules_go for Windows RBE issue executing batch scripts, see Windows: Recent bump to rules_go breaks Windows + RBE #11657
  • Tags/fixes some additional tests that fail in CI

Additional Description: N/A
Risk Level: Low, but we may observe differences in performance of Windows CI running remotely, we may need to adjust the worker pool accordingly
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
Fixes #10619

@sunjayBhatia sunjayBhatia changed the title Draft: Demonstrate working RBE+Windows build WIP: Demonstrate working RBE+Windows build May 7, 2020
- image uri/tag point to our temporary testing image, needs to be
updated to the image in the envoyproxy repo
- urls of bazel-toolchains and envoy-build-tools point to our spike
branches, need to be changed to real versions once they are upstreamed
- builds envoy-static and runs tests

Co-authored-by: William A Rowe Jr <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
@stale
Copy link

stale bot commented May 15, 2020

This pull request has been automatically marked as stale because it has not had activity in the last 7 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

@stale stale bot added the stale stalebot believes this issue/PR has not been touched recently label May 15, 2020
@sunjayBhatia
Copy link
Member Author

This PR will be turned into a reviewable PR once envoyproxy/envoy-build-tools#64 is merged

@stale stale bot removed the stale stalebot believes this issue/PR has not been touched recently label May 21, 2020
- Also bump build image tag
- Use same mechanism to find build image tag on Windows and Linux

Signed-off-by: Sunjay Bhatia <[email protected]>
@sunjayBhatia sunjayBhatia changed the title WIP: Demonstrate working RBE+Windows build Windows build: Enable RBE build in Windows CI May 28, 2020
Signed-off-by: Sunjay Bhatia <[email protected]>
@sunjayBhatia
Copy link
Member Author

sunjayBhatia commented May 28, 2020

This may cause format to fail for some PRs since the CI image is bumped to one with llvm 10 (and clang-format 10). If that is undesirable as part of this change, we can submit another to bump that dependency separately.

@stale
Copy link

stale bot commented Jun 4, 2020

This pull request has been automatically marked as stale because it has not had activity in the last 7 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

@stale stale bot added the stale stalebot believes this issue/PR has not been touched recently label Jun 4, 2020
@sunjayBhatia
Copy link
Member Author

just waiting on envoyproxy/envoy-build-tools#70 to help us generate an updated toolchain config

@stale stale bot removed the stale stalebot believes this issue/PR has not been touched recently label Jun 4, 2020
@wrowe wrowe mentioned this pull request Jun 4, 2020
- Brings in newly generated ubuntu, windows docker images for RBE

Signed-off-by: William A Rowe Jr <[email protected]>
@wrowe
Copy link
Contributor

wrowe commented Jun 4, 2020

Unclear whether the released 3.2.0 bazel version will be compatible; the final post-3.1.2 tag which picked up the the windows path fixes was;

        sha256 = "5d48b13686081f5d07dbba1271595d1bb2b5fa1d2f8c049ef15011bd4e6bdd19",
        strip_prefix = "bazel-toolchains-834b89b3f608a9aa4c212e7ee4a05b323b980a85",
        # 2020-05-21
        urls = ["https://github.com/bazelbuild/bazel-toolchains/archive/834b89b3f608a9aa4c212e7ee4a05b323b980a85.tar.gz"],

sunjayBhatia and others added 5 commits June 4, 2020 19:54
Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
Surface additional failing tests for visibility before we get Windows
running in CI.

Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
- yaml-cpp parser should be treating cr+lf as newline per spec

Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
@sunjayBhatia sunjayBhatia marked this pull request as ready for review June 5, 2020 14:24
@sunjayBhatia
Copy link
Member Author

sunjayBhatia commented Jun 16, 2020

Looks like the recent rules_go bump from #11581 is the culprit

Specifically this change: bazelbuild/rules_go#2488

@sunjayBhatia
Copy link
Member Author

Blocked on bazelbuild/rules_go#2542

sunjayBhatia and others added 4 commits June 18, 2020 09:56
can be removed once Bazel/RBE issue is addressed

Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>

Signed-off-by: Sunjay Bhatia <[email protected]>
@sunjayBhatia
Copy link
Member Author

Blocked by PR #11719

wrowe and others added 2 commits June 23, 2020 16:11
Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
@sunjayBhatia
Copy link
Member Author

Blocked on bazelbuild/rules_go#2542

No longer blocked on the rules_go PR, we have a patch in this PR for rules_go and an open issue here to track figuring out the true root cause: #11657

@sunjayBhatia
Copy link
Member Author

cc @nigriMSFT @davinci26 if you weren't aware of this one

This reverts commit acfaa9a.

Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
@wrowe
Copy link
Contributor

wrowe commented Jun 23, 2020

We have rolled up all regression fixes into a single distinct PR. #11719 needs to be applied before this one.

wrowe and others added 2 commits June 25, 2020 09:54
Signed-off-by: Sunjay Bhatia <[email protected]>
Co-authored-by: Sunjay Bhatia <[email protected]>
Signed-off-by: William A Rowe Jr <[email protected]>
Co-authored-by: William A Rowe Jr <[email protected]>
Fix network utility test

Signed-off-by: Sunjay Bhatia <[email protected]>
@sunjayBhatia
Copy link
Member Author

Assuming no more test flakes, this PR should be ready for review @lizan

@sunjayBhatia
Copy link
Member Author

https://dev.azure.com/cncf/envoy/_build/results?buildId=43158&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=168f295b-0553-5364-35f7-923225ecd8b3 represents a Windows build where all tests are compiled and the ones known to pass are run

@sunjayBhatia
Copy link
Member Author

@lizan the TSAN failure seems to also be occurring on master, it should not be related to this change

@dio
Copy link
Member

dio commented Jun 26, 2020

@sunjayBhatia merge master will help I think. Since #11760 is merged.

@sunjayBhatia
Copy link
Member Author

@dio Sure, will do

Can we get a review on this soon so we don't slip behind the newly merged PRs that might cause Windows breakages again?

@lizan lizan merged commit 8614e83 into envoyproxy:master Jun 27, 2020
@wrowe wrowe deleted the windows-rbe branch June 29, 2020 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Windows RBE Tracking Issue
4 participants