test: EKS e2e test using eksctl #667
Conversation
|
linters are flagging the exec cmds. IMO shelling out commands is not ideal here. I know that AKS has their own SDK that is able to interact with EKS https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/eks so maybe this could b[e worth look into as an alternative? |
rbtr
requested review from
matmerr,
vakalapa and
neaggarwMS
and removed request for
karina-ranadive and
spencermckee
|
Yeah, to do it for real we will want to use aws-sdk, but shelling out to eksctl is fine while we're just trying to say hey, Retina E2E could work on EKS |
|
@whatnick this is great, thanks for putting it together so fast! |
|
The aim overall is to bolster Retina's **ANY CNI** claim with a public
demonstration.
I just wanted to start the ball rolling and from past e2e building
experience I expect this to be slow. Just a POC that this is possible and
also flagging corner cases and other setup necessary
- ECR repo for pulling into cluster OR managing GHCR Credentials in cluster
- AWS-Github OIDC pairing for securely logging into account for GHA,
setting up policies and roles on AWS side.
- Setting up OIDC in EKS during cluster provisioning to hook AWS VPC CNI
and demonstrate using that + retina.
- Possible issues with parallel execution of Azure and AWS tests and
clobbering of the kubeconfig.
The slowness will also give me an opportunity to figure out pulling in
eksctl go code via a `require` + `import` to avoid shelling out.
…On Wed, Aug 28, 2024, 07:37 Evan Baker ***@***.***> wrote:
@whatnick <https://github.com/whatnick> this is great, thanks for putting
it together so fast!
While we review/discuss I do want to set the expectation appropriately
that us getting an AWS account provisioned will likely be the slow/hard
part of this 😓
—
Reply to this email directly, view it on GitHub
<#667 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADX7BGZS7I7VYQNRV56VSTZTT2B5AVCNFSM6AAAAABNF7YW52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJTGY2TSNBQGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
good stuff, thanks for taking a look into this @whatnick |
It has added a lot of requires, but I have updated the PoC to consume eksctl as a package and run the cobra commands. It can be slimmed down to remove fancy things like coloured logging which are not really relevant for this use-case. |
|
More progress by enabling AWS VPC-CNI in Network Policy enforcement mode. kubectl --kubeconfig=/home/whatnick/dev/retina/test/e2e/test.pem get pods -n kube-system
NAME READY STATUS RESTARTS AGE
agnhost-a-0 1/1 Running 0 16s
aws-node-77sfp 2/2 Running 0 2m19s
aws-node-ssjsk 2/2 Running 0 2m15s
aws-node-xs2kv 2/2 Running 0 2m17s
coredns-787cb67946-lrxxh 1/1 Running 0 6m34s
coredns-787cb67946-qr4xx 1/1 Running 0 6m34s
kube-proxy-7h7vk 1/1 Running 0 2m15s
kube-proxy-qxwb7 1/1 Running 0 2m19s
kube-proxy-xcbcf 1/1 Running 0 2m17s
retina-agent-22mcl 1/1 Running 0 34s
retina-agent-cc8b4 1/1 Running 0 34s
retina-agent-hwj5t 1/1 Running 0 34sNetwork policy is enabled kubectl --kubeconfig=/home/whatnick/dev/retina/test/e2e/test.pem get networkpolicy -n kube-system
NAME POD-SELECTOR AGE
deny-all app=agnhost-a 66s
|
whatnick
force-pushed
the
feat/eks-e2e-test
branch
from
223e957 to
db6d7b0
Compare
|
This PR will be closed in 7 days due to inactivity. |
github-actions
bot
added
the
meta/waiting-for-author
|
Will merge to upstream soon. |
whatnick
force-pushed
the
feat/eks-e2e-test
branch
from
4bb5b0f to
4c01faa
Compare
github-actions
bot
removed
the
meta/waiting-for-author
|
Currently disabled Windows tests for AWS, can enable once windows cluster setup via eksctl is tested. |
|
This PR will be closed in 7 days due to inactivity. |
github-actions
bot
added
the
meta/waiting-for-author
|
Pull request closed due to inactivity. |
|
Thanks for re-opening this. Been busy otherwise , will fix conflicts and maintain it over the weekend. |
Description
NOTE : Since this will take a bit of CI and other account provisioning planning to keep this synced to upstream once a week till across the line or I run out of juice.
Add EKS based e2e tests by execing EKSCtl to provision and delete temporary cluster. Currently at POC stage since Account setup etc. are needed to run this in practice in conjuction with secrets and variables associated with this repository.
The AWS integration should be setup via OIDC as shown here : https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
with roles relevant to EKSCtl as shown here :
https://eksctl.io/usage/minimum-iam-policies/
Related Issue
Partially addresses #451
Checklist
git commit -S -s ...). See this documentation on signing commits.Screenshots (if applicable) or Testing Completed
With drop packets metrics Scenario disabled as per #746 , the AWS e2e test suite runs successfully.
go test -run TestE2ERetinaAWS ./test/e2e/ -timeout 40m ok github.com/microsoft/retina/test/e2e 1866.467sFor failing test runs cluster creation and tear down is as below.
Additional Notes
The helm chart install portion of this test fails in practice presumably due to unreachable image registry. May need to push images to corresponding ECR or debug ghcr access.
Opening this PR for feedback and discussions on AWS e2e testing approach. In practice I have successfully deployed retina legacy charts in EKS.
Please refer to the CONTRIBUTING.md file for more information on how to contribute to this project.