[WIP] Add e2e ci for kubernetes with kind. #403

liangyuanpeng · 2023-07-28T05:52:15Z

what problem are you trying to solve? (or if there's no problem, what's the motivation for this change?)

Make sure xline can work with kubernetes, so add e2e ci for kubernetes.

what changes does this pull request make?

add e2e ci for kubernetes.

Now, it's sometimes fail, sometimes success.

successed: https://github.com/liangyuanpeng/Xline/actions/runs/5688707901
failed: https://github.com/liangyuanpeng/Xline/actions/runs/5688679640/job/15418920092

liangyuanpeng · 2023-07-28T06:02:32Z

Now, it's sometimes fail, sometimes success.

I think it have some problem on the Xline side and need to fix.

liangyuanpeng · 2023-07-28T08:18:21Z

/cc @bsbds @Phoenix500526

codecov · 2023-07-29T02:58:17Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.95%. Comparing base (7ec7abb) to head (2fe65aa).
Report is 501 commits behind head on master.

❗ Current head 2fe65aa differs from pull request most recent head 5441d8b. Consider uploading reports for the commit 5441d8b to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #403       +/-   ##
===========================================
+ Coverage   53.38%   78.95%   +25.56%     
===========================================
  Files          96      154       +58     
  Lines       14853    21259     +6406     
  Branches    14853        0    -14853     
===========================================
+ Hits         7930    16786     +8856     
+ Misses       6427     4473     -1954     
+ Partials      496        0      -496

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Phoenix500526 · 2023-07-30T12:18:46Z

Thank you for your contribution. Please add a sign-off in the commit message and modify It to comply with the conventional commit format. For your information: Xline commit message guidelines

liangyuanpeng · 2023-07-31T02:07:11Z

@Phoenix500526

I will rebase it when this pr is ready for review.
For now, it's pending for here:

Now, it's sometimes fail, sometimes success.

I think it have some problem on the Xline side and need to fix.

It's need the Xline maintainer to help check ,Thanks. It's easy to reproduce it, just replace etcd with xline and work with kubernetes.

This PR can help to reproduce it.

liangyuanpeng · 2023-07-31T02:46:15Z

Seems like it's related:

[Feature]: HTTP API of health #387

let me check it.

liangyuanpeng · 2023-08-02T01:43:26Z

It's always successed if i disabled xline debug log.

https://github.com/liangyuanpeng/Xline/actions/workflows/kube.yml

iGxnon · 2023-08-02T08:13:10Z

It's always successed if i disabled xline debug log.

https://github.com/liangyuanpeng/Xline/actions/workflows/kube.yml

How about adding a script to display the debug log of xline within the container? You could consider adding a simple test in the CI that sends a basic request to xline and check the outputs. This might help us in analyzing any potential issues related to xline.

liangyuanpeng · 2023-08-08T08:24:22Z

How about adding a script to display the debug log of xline within the container? You could consider adding a simple test in the CI that sends a basic request to xline and check the outputs. This might help us in analyzing any potential issues related to xline.

Absolutely. I just cut the log here, seems like the xline server have some problem. FYI

grep conflict

2023-08-08T08:01:16.533848Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-657da3b8-0442-495a-b674-74a4390bfc2d"), 486)
2023-08-08T08:01:16.949130Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: SpecExeReady(ProposeId("node1-1a24f9fa-e910-4a97-b7d1-eafad8928c32"), 487)
2023-08-08T08:01:16.949162Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-1a24f9fa-e910-4a97-b7d1-eafad8928c32"), 487)
2023-08-08T08:01:18.964632Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: SpecExeReady(ProposeId("node1-89e0ca86-814f-48f5-ba02-89a411b2f1c3"), 488)
2023-08-08T08:01:18.964658Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-89e0ca86-814f-48f5-ba02-89a411b2f1c3"), 488)
2023-08-08T08:01:20.500827Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: SpecExeReady(ProposeId("node1-10d77a7b-f4f2-4a19-958d-bcdb6fc0e683"), 489)
2023-08-08T08:01:20.500879Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-10d77a7b-f4f2-4a19-958d-bcdb6fc0e683"), 489)
2023-08-08T08:01:20.592684Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: SpecExeReady(ProposeId("node1-6b5111f6-628c-4cb1-b56c-12c59afd41e7"), 490)
2023-08-08T08:01:20.592766Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-6b5111f6-628c-4cb1-b56c-12c59afd41e7"), 490)
2023-08-08T08:01:20.978428Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: SpecExeReady(ProposeId("node1-0d4ea7a2-fb5c-4818-b797-1db8b7774564"), 491)
2023-08-08T08:01:20.978449Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-0d4ea7a2-fb5c-4818-b797-1db8b7774564"), 491)
2023-08-08T08:01:22.992134Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: SpecExeReady(ProposeId("node1-bb7c75ac-e5bc-4d77-87eb-dd213dc86340"), 492)
2023-08-08T08:01:22.992162Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-bb7c75ac-e5bc-4d77-87eb-dd213dc86340"), 492)
2023-08-08T08:01:24.247018Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: SpecExeReady(ProposeId("node1-7ef08fbe-ac08-4fb2-b1e6-7f5135aad687"), 493)
2023-08-08T08:01:24.247057Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-7ef08fbe-ac08-4fb2-b1e6-7f5135aad687"), 493)
2023-08-08T08:01:25.006454Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: SpecExeReady(ProposeId("node1-c3c6aa14-c3e3-41c5-95e1-43181ade0704"), 494)
2023-08-08T08:01:25.006476Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-c3c6aa14-c3e3-41c5-95e1-43181ade0704"), 494)
2023-08-08T08:01:27.019814Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: SpecExeReady(ProposeId("node1-58be4d25-ae1d-4a5c-bc9e-81472bc3baab"), 495)
2023-08-08T08:01:27.019862Z DEBUG curp::server::cmd_worker::conflict_checked_mpmc: new ce event: ASReady(ProposeId("node1-58be4d25-ae1d-4a5c-bc9e-81472bc3baab"), 495)

grep error

ror(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829271Z DEBUG tonic::codec::decode: decoder inner stream error: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829287Z  WARN xline::server::watch_server: Receive WatchRequest error Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829395Z DEBUG tonic::codec::decode: decoder inner stream error: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829415Z  WARN xline::server::watch_server: Receive WatchRequest error Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829498Z DEBUG tonic::codec::decode: decoder inner stream error: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829519Z  WARN xline::server::watch_server: Receive WatchRequest error Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829540Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829557Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829566Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829573Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829580Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829650Z DEBUG tonic::codec::decode: decoder inner stream error: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829668Z  WARN xline::server::watch_server: Receive WatchRequest error Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829688Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829703Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829718Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829730Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829816Z DEBUG tonic::codec::decode: decoder inner stream error: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829837Z  WARN xline::server::watch_server: Receive WatchRequest error Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829858Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829873Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829886Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829898Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829912Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829924Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2023-08-08T08:01:27.829988Z  WARN xline::server::watch_server: Receive WatchRequest error Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.830058Z  WARN xline::server::watch_server: Receive WatchRequest error Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }
2023-08-08T08:01:27.829293Z  WARN xline::server::watch_server: Receive WatchRequest error Status { code: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) })) }

The log generate from v0.5.0

This issue #402 also have some error, but is not same with here.

Maybe we can run ci with no debug log first.

Signed-off-by: Lan Liang <[email protected]>

liangyuanpeng · 2024-03-08T09:20:00Z

Waiting for #678 is merged, and then we can go ahead.

liangyuanpeng · 2024-03-08T10:17:26Z

Releated: #677

Phoenix500526

Please remove trailing spaces and squash these commits.

Phoenix500526 · 2024-03-15T03:38:09Z

.github/workflows/kube.yml

+          du -h --max-depth=1 /xline/kind
+          ls -allh /xline/kind/containers
+      - name: upload kind logs
+        if: always()


I think uploading detailed logs when deployment failed is a better choice.

liangyuanpeng · 2024-03-16T10:18:43Z

Create a clean PR #696 to work for it, closing..

liangyuanpeng changed the title ~~add e2e ci for kubernetes with kind.~~ [WIP] Add e2e ci for kubernetes with kind. Jul 28, 2023

liangyuanpeng force-pushed the ci_e2e_k8s branch from 99542cb to dc59d3d Compare July 28, 2023 05:53

liangyuanpeng force-pushed the ci_e2e_k8s branch from 4ae6b87 to 5695557 Compare July 31, 2023 07:26

liangyuanpeng force-pushed the ci_e2e_k8s branch from f0dd641 to 2ce2ef5 Compare August 17, 2023 02:45

liangyuanpeng marked this pull request as draft August 17, 2023 03:09

liangyuanpeng force-pushed the ci_e2e_k8s branch 2 times, most recently from 72934cb to 0c9b510 Compare August 18, 2023 12:44

add e2e ci for kubernetes with kind

ecdc88d

Signed-off-by: Lan Liang <[email protected]>

liangyuanpeng force-pushed the ci_e2e_k8s branch from 52248b3 to ecdc88d Compare August 18, 2023 13:03

liangyuanpeng added 8 commits August 19, 2023 10:56

add log

ae69f4f

change /tmp/kind to /xline/kind

7ca9eea

change /tmp/kind to /xline/kind

1be81bb

change /tmp/kind to /xline/kind

f020eca

update

84eb2d0

disable cron for wip

653a240

Use xline v0.6.0

eef5b12

Signed-off-by: Lan Liang <[email protected]>

add on pull_request

2fe65aa

liangyuanpeng added 2 commits March 12, 2024 07:34

Update xline image to ghcr.io/liangyuanpeng/xline:ffa824f

e47b0e8

Add more k8s

d4731bc

liangyuanpeng added 3 commits March 12, 2024 07:36

disable xline debug log

60b4181

update xline command

f6bb6ce

set log to debug

5441d8b

liangyuanpeng mentioned this pull request Mar 13, 2024

fix: xline watcher will loss event #678

Merged

Phoenix500526 reviewed Mar 15, 2024

View reviewed changes

liangyuanpeng mentioned this pull request Mar 16, 2024

Add github action to run e2e test with kubernetes cluster. #696

Merged

liangyuanpeng closed this Mar 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add e2e ci for kubernetes with kind. #403

[WIP] Add e2e ci for kubernetes with kind. #403

liangyuanpeng commented Jul 28, 2023 •

edited

Loading

liangyuanpeng commented Jul 28, 2023

liangyuanpeng commented Jul 28, 2023

codecov bot commented Jul 29, 2023 •

edited

Loading

Phoenix500526 commented Jul 30, 2023

liangyuanpeng commented Jul 31, 2023 •

edited

Loading

liangyuanpeng commented Jul 31, 2023

liangyuanpeng commented Aug 2, 2023 •

edited

Loading

iGxnon commented Aug 2, 2023

liangyuanpeng commented Aug 8, 2023 •

edited

Loading

liangyuanpeng commented Mar 8, 2024 •

edited

Loading

liangyuanpeng commented Mar 8, 2024

Phoenix500526 left a comment

Phoenix500526 Mar 15, 2024

liangyuanpeng commented Mar 16, 2024

[WIP] Add e2e ci for kubernetes with kind. #403

[WIP] Add e2e ci for kubernetes with kind. #403

Conversation

liangyuanpeng commented Jul 28, 2023 • edited Loading

what problem are you trying to solve? (or if there's no problem, what's the motivation for this change?)

what changes does this pull request make?

liangyuanpeng commented Jul 28, 2023

liangyuanpeng commented Jul 28, 2023

codecov bot commented Jul 29, 2023 • edited Loading

Codecov Report

Phoenix500526 commented Jul 30, 2023

liangyuanpeng commented Jul 31, 2023 • edited Loading

liangyuanpeng commented Jul 31, 2023

liangyuanpeng commented Aug 2, 2023 • edited Loading

iGxnon commented Aug 2, 2023

liangyuanpeng commented Aug 8, 2023 • edited Loading

liangyuanpeng commented Mar 8, 2024 • edited Loading

liangyuanpeng commented Mar 8, 2024

Phoenix500526 left a comment

Choose a reason for hiding this comment

Phoenix500526 Mar 15, 2024

Choose a reason for hiding this comment

liangyuanpeng commented Mar 16, 2024

liangyuanpeng commented Jul 28, 2023 •

edited

Loading

codecov bot commented Jul 29, 2023 •

edited

Loading

liangyuanpeng commented Jul 31, 2023 •

edited

Loading

liangyuanpeng commented Aug 2, 2023 •

edited

Loading

liangyuanpeng commented Aug 8, 2023 •

edited

Loading

liangyuanpeng commented Mar 8, 2024 •

edited

Loading