-
Notifications
You must be signed in to change notification settings - Fork 408
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
e2e: add e2e test into github action #496
Conversation
@Congrool: GitHub didn't allow me to assign the following users: your_reviewer. Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Welcome @Congrool! It looks like this is your first PR to openyurtio/openyurt 🎉 |
bdea58a
to
95aedfa
Compare
Now give the description of the error. • Failure [5.112 seconds]
[k8s.io] yurttunnel-e2e-test
/home/runner/go/pkg/mod/github.com/kubernetes/kubernetes@v1.18.8/test/e2e/framework/framework.go:698
[k8s.io] [yurttunnel-e2e-test]: pod_operate_test_on_edge
/home/runner/go/pkg/mod/github.com/kubernetes/kubernetes@v1.18.8/test/e2e/framework/framework.go:698
yurttunnel_e2e_test_pod_run_on_edge [It]
/home/runner/work/openyurt/openyurt/test/e2e/yurttunnel/yurttunnel.go:170
fail run exec:test-po-on-edge
Unexpected error:
<*errors.errorString | 0xc0005ec5d0>: {
s: "unable to upgrade connection: unable to read error from server response",
}
unable to upgrade connection: unable to read error from server response
occurred
/home/runner/work/openyurt/openyurt/test/e2e/yurttunnel/yurttunnel.go:203 I tried to use the following command to setup openyurt cluster locally. cd ${YURT_ROOT}
export REGION=cn
sudo -E bash hack/local_up_openyurt.sh And got cluster as follows $ kubectl get pod -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-74ff55c5b-5jzkp 1/1 Running 0 46m 10.244.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system coredns-74ff55c5b-s5xbp 1/1 Running 0 46m 10.244.0.4 openyurt-e2e-test-control-plane <none> <none>
kube-system etcd-openyurt-e2e-test-control-plane 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kindnet-mklwd 1/1 Running 0 45m 172.18.0.2 openyurt-e2e-test-worker <none> <none>
kube-system kindnet-nxw69 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kube-apiserver-openyurt-e2e-test-control-plane 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kube-controller-manager-openyurt-e2e-test-control-plane 1/1 Running 0 41m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kube-proxy-76hgh 1/1 Running 0 45m 172.18.0.2 openyurt-e2e-test-worker <none> <none>
kube-system kube-proxy-l75vd 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system kube-scheduler-openyurt-e2e-test-control-plane 1/1 Running 0 46m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system yurt-controller-manager-7dd9fc956d-sv2g4 1/1 Running 0 42m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
kube-system yurt-hub-openyurt-e2e-test-worker 1/1 Running 0 41m 172.18.0.2 openyurt-e2e-test-worker <none> <none>
kube-system yurt-tunnel-agent-fr6jn 1/1 Running 0 40m 172.18.0.2 openyurt-e2e-test-worker <none> <none>
kube-system yurt-tunnel-server-677bdcbcf6-2gjgd 1/1 Running 0 41m 172.18.0.3 openyurt-e2e-test-control-plane <none> <none>
local-path-storage local-path-provisioner-547f784dff-8w4v8 1/1 Running 0 46m 10.244.0.2 openyurt-e2e-test-control-plane <none> <none> Then I tried to error: unable to upgrade connection: unable to read error from server response I could not logs the agent as well: $ kubectl logs yurt-tunnel-agent-fr6jn -n kube-system
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log yurt-tunnel-agent-fr6jn)) I set the option I0926 04:08:02.313247 1 tracereq.go:99] start handling request GET https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent, from 172.18.0.3:39386 to 172.18.0.2:10250
I0926 04:08:02.313300 1 handler.go:111] request header in localHostProxyMiddleware: map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] with host: 172.18.0.2:10250 and urL: https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent
I0926 04:08:02.313362 1 interceptor.go:78] Sending request to "172.18.0.2:10250".
I0926 04:08:02.313791 1 tunnel.go:36] "Received request for host" method="CONNECT" host="172.18.0.2:10250" userAgent="Go-http-client/1.1"
I0926 04:08:02.314096 1 tunnel.go:71] Set pending(rand=2610529275472644968) to &{0xc0006965a0 0xc00062b200 {} 0x5994a0 true false false false 0 {0 0} <nil> {0xc00047c700 map[] true true} map[] false 0 -1 200 false false [] 0 [83 117 110 44 32 50 54 32 83 101 112 32 50 48 50 49 32 48 52 58 48 56 58 48 50 32 71 77 84] [0 0 0 0 0 0 0 0 0 0] [50 48 48] 0xc0008121c0 0}
I0926 04:08:02.314211 1 desthost_backend_manager.go:49] "Get the backend through the DestHostBackendManager" destHost="172.18.0.2"
I0926 04:08:02.317577 1 server.go:689] "Received DIAL_RSP" random=2610529275472644968 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.318268 1 server.go:269] "Register frontend for agent" frontend=&server.ProxyClientConnection{Mode:"http-connect", Grpc:client.ProxyService_ProxyServer(nil), HTTP:(*net.UnixConn)(0xc00000e298), connected:(chan struct {})(0xc00007b920), connectID:1, agentID:"openyurt-e2e-test-worker", start:time.Time{wall:0xc04c1a6892bb1420, ext:260641359529, loc:(*time.Location)(0x23095c0)}, backend:(*server.backend)(0xc0004568e0)} agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.318607 1 tunnel.go:128] "Starting proxy to host" host="172.18.0.2:10250" agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.318824 1 tunnel.go:163] "Forwarding data on tunnel to agent" bytes=239 totalBytes=239 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.325538 1 server.go:718] "Received data from agent" bytes=2303 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.325697 1 server.go:727] "DATA sent to frontend"
I0926 04:08:02.328240 1 tunnel.go:163] "Forwarding data on tunnel to agent" bytes=1223 totalBytes=1462 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.328252 1 interceptor.go:112] successfully setup TLS connection to "172.18.0.2:10250" with headers:
User-Agent: Go-http-client/1.1
I0926 04:08:02.328376 1 tunnel.go:163] "Forwarding data on tunnel to agent" bytes=186 totalBytes=1648 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.330133 1 server.go:718] "Received data from agent" bytes=161 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.330258 1 server.go:727] "DATA sent to frontend"
I0926 04:08:02.330497 1 interceptor.go:282] interceptor: successfully read the http response from the proxy tunnel for request https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent
I0926 04:08:02.330613 1 interceptor.go:323] interceptor: stop serving request https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent with headers: map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
I0926 04:08:02.330691 1 tracereq.go:103] stop handling request GET https://172.18.0.2:10250/containerLogs/kube-system/yurt-tunnel-agent-4q8cc/yurt-tunnel-agent, request handling lasts 17.399337ms
I0926 04:08:02.330987 1 tunnel.go:163] "Forwarding data on tunnel to agent" bytes=24 totalBytes=1672 agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.331034 1 tunnel.go:141] "EOF from host" host="172.18.0.2:10250" agentID="openyurt-e2e-test-worker" connID=1
I0926 04:08:02.332747 1 tunnel.go:170] "Stopping transfer to host" host="172.18.0.2:10250" agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.332247 1 server.go:718] "Received data from agent" bytes=24 agentID="openyurt-e2e-test-worker" connectionID=1
E0926 04:08:02.332948 1 server.go:725] "send to client stream failure" err="write unix /tmp/interceptor-proxier.sock->@: use of closed network connection"
I0926 04:08:02.332968 1 server.go:732] "Received CLOSE_RSP" connectionID=1
I0926 04:08:02.332987 1 server.go:290] "Remove frontend for agent" agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.333010 1 server.go:747] "Close streaming" agentID="openyurt-e2e-test-worker" connectionID=1
I0926 04:08:02.333708 1 server.go:732] "Received CLOSE_RSP" connectionID=1
E0926 04:08:02.333830 1 server.go:735] "could not get frontent client" err="can't find agentID openyurt-e2e-test-worker in the frontends" for tunnel-agent I0926 04:05:30.567393 1 client.go:224] "Connect to" server="c219f2f5-1d95-4349-af1e-fa0ea7b615c1"
I0926 04:05:30.567499 1 clientset.go:190] "sync added client connecting to proxy server" serverID="c219f2f5-1d95-4349-af1e-fa0ea7b615c1"
I0926 04:05:30.567624 1 client.go:326] "Start serving" serverID="c219f2f5-1d95-4349-af1e-fa0ea7b615c1"
I0926 04:08:02.316217 1 client.go:346] "[tracing] recv packet" type="DIAL_REQ"
I0926 04:08:02.316286 1 client.go:355] received DIAL_REQ
I0926 04:08:02.316871 1 client.go:412] received dial request to tcp:172.18.0.2:10250 with random=2610529275472644968 and connID=1
I0926 04:08:02.321113 1 client.go:346] "[tracing] recv packet" type="DATA"
I0926 04:08:02.321219 1 client.go:416] "received DATA" connectionID=1
I0926 04:08:02.321365 1 client.go:492] "write to remote" connID=1 lastData=239
I0926 04:08:02.325188 1 client.go:461] "received data from remote" bytes=2303 connID=1
I0926 04:08:02.328451 1 client.go:346] "[tracing] recv packet" type="DATA"
I0926 04:08:02.328483 1 client.go:416] "received DATA" connectionID=1
I0926 04:08:02.328568 1 client.go:492] "write to remote" connID=1 lastData=1223
I0926 04:08:02.328620 1 client.go:346] "[tracing] recv packet" type="DATA"
I0926 04:08:02.328647 1 client.go:416] "received DATA" connectionID=1
I0926 04:08:02.328713 1 client.go:492] "write to remote" connID=1 lastData=186
I0926 04:08:02.329756 1 client.go:461] "received data from remote" bytes=161 connID=1
I0926 04:08:02.331452 1 client.go:346] "[tracing] recv packet" type="DATA"
I0926 04:08:02.331485 1 client.go:416] "received DATA" connectionID=1
I0926 04:08:02.331565 1 client.go:492] "write to remote" connID=1 lastData=24
I0926 04:08:02.331670 1 client.go:461] "received data from remote" bytes=24 connID=1
I0926 04:08:02.331716 1 client.go:461] "received data from remote" bytes=0 connID=1
I0926 04:08:02.331731 1 client.go:464] "connection read EOF" connID=1
I0926 04:08:02.331746 1 client.go:382] "close connection" connectionID=1
I0926 04:08:02.333303 1 client.go:346] "[tracing] recv packet" type="CLOSE_REQ"
I0926 04:08:02.333332 1 client.go:427] "received CLOSE_REQ" connectionID=1 for kubectl I0926 12:11:22.501541 1090376 round_trippers.go:425] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.20.2 (linux/amd64) kubernetes/faecb19" 'https://127.0.0.1:45873/api/v1/namespaces/kube-system/pods/yurt-tunnel-agent-4q8cc/log'
I0926 12:11:22.532743 1090376 round_trippers.go:445] GET https://127.0.0.1:45873/api/v1/namespaces/kube-system/pods/yurt-tunnel-agent-4q8cc/log 401 Unauthorized in 31 milliseconds
I0926 12:11:22.532784 1090376 round_trippers.go:451] Response Headers:
I0926 12:11:22.532808 1090376 round_trippers.go:454] Cache-Control: no-cache, private
I0926 12:11:22.532828 1090376 round_trippers.go:454] Content-Type: application/json
I0926 12:11:22.532847 1090376 round_trippers.go:454] Content-Length: 274
I0926 12:11:22.532867 1090376 round_trippers.go:454] Date: Sun, 26 Sep 2021 04:11:22 GMT
I0926 12:11:22.532920 1090376 request.go:1107] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server has asked for the client to provide credentials ( pods/log yurt-tunnel-agent-4q8cc)","reason":"Unauthorized","details":{"name":"yurt-tunnel-agent-4q8cc","kind":"pods/log"},"code":401}
I0926 12:11:22.534648 1090376 helpers.go:216] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server has asked for the client to provide credentials ( pods/log yurt-tunnel-agent-4q8cc)",
"reason": "Unauthorized",
"details": {
"name": "yurt-tunnel-agent-4q8cc",
"kind": "pods/log"
},
"code": 401
}]
F0926 12:11:22.534722 1090376 helpers.go:115] error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log yurt-tunnel-agent-4q8cc)) It seems that yurt-tunnel-server successfully fetched the log of |
Hi, I've revised the code and updated the error log. @rambohe-ch |
@Congrool Thank you for uploading detailed info of yurt-tunnel errors. It's looks like the webhook error that comes from kubelet, because kubelet usually send webhook request(subjectaccessreview) to kube-apiserver when he received pod/logs request. |
@rambohe-ch Thank you for your help. I checked the log of kubelet and yurthub. There's actually something wrong. Sep 26 08:23:28 openyurt-e2e-test-worker kubelet[26717]: E0926 08:23:28.869863 26717 server.go:259] Unable to authenticate the request due to an error: verifying certificate SN=324686761876842496809737091112637902102, SKID=, AKID=35:23:38:84:B4:76:01:63:06:04:1C:B5:32:D3:3B:48:A9:2D:52:57 failed: x509: certificate specifies an incompatible key usage Maybe it's the error that prevent kubelet from sending subjectaccessreview request to APIServer. But I have no idea that how does it occur? |
@Congrool Sorry, maybe i have made a mistake. response status code is 401 for authentication failed, and kubelet use |
@rambohe-ch OK, I retried with root@openyurt-e2e-test-worker:/# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
436e5d29fdb9d b124b0cf14d4a 7 hours ago Running yurt-tunnel-agent 0 53db1f4484d1e
92f9a2c1d71ac 3c29d52ea23d1 8 hours ago Running yurt-hub 0 4bdde362f5edc
227bc01ff75a5 d1ece029a5f6a 8 hours ago Running kube-proxy 0 21749708770fe
cdb2fe84a2bbf 6de166512aa22 8 hours ago Running kindnet-cni 0 552aabc77f71b
root@openyurt-e2e-test-worker:/# crictl logs 92f9a2c1d71ac 2>&1 | grep -i tokenreview
root@openyurt-e2e-test-worker:/# |
@Congrool @SataQiu and @DrmagicE have come across the same error, you can check the issue: #508, and this bug have fixed by @DrmagicE in PR: #509 |
2f49577
to
26aac9f
Compare
@Congrool Please remove WIP from title. |
@rambohe-ch ,Hello, I'm testing on my fork. I'll remove WIP when it passes the test. |
@Congrool ok |
/lgtm |
/lgtm cancel |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Congrool, rambohe-ch The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
e2e: add e2e test into github action
What type of PR is this?
/kind enhancement
What this PR does / why we need it:
test/e2e
to check its correctness.hack/local_up_openyurt.sh
, with which we can quickly setup an openyurt cluster at local host withkind
.Which issue(s) this PR fixes:
Fixes #467
Special notes for your reviewer:
Does this PR introduce a user-facing change?
other Note
Currently, this shell skips yurt-tunnel e2e test at
test/e2e/yurttunnel
, because it always failed at this test. We should find out the reason.