Credentials for AWS Agent on EKS Hosted Container

[alexandrud]

Hi,

I’m reopening this discussion. I was hoping this werk will add the feature agent_aws: Support for Assume Role (Werk #10333) | checkmk. I installed the 2.0.0b2 build.

We have a couple of managed AWS EKS clusters. In this kind of setup we can assign roles to the containers running in kubernetes. As such, a container can assume a role and can practically be given the privileges needed by the special agent for AWS.

We cannot use a user because our policy requires all users to have MFA.

I actually tested this and it works. The setup described very shortly is like this:

• role (not user) in IAM with the policies attached
• use IRSA to provide IAM privileges to the container ( Cross account IAM roles for Kubernetes service accounts | Containers (amazon.com))
• check-mk container has an annotation with the IAM role it needs to run the AWS commands

I’ve installed awscli in the container and setup the region (export AWS_DEFAULT_REGION=us-east-1). After this I was able to list all EC2 instances without providing the AWS_ACCESS_KEY_ID or the AWS_SECRET_ACCESS_KEY.

However in the rule you cannot skip the credentials fields.

Would it be possible to add this on your roadmap?

Thank you!

[alexandrud]

A couple of updates:

• I’ve changed the create_session function in agent_aws.py and removed the access key and access secret key. This was a quick and dirty way to not require the keys.
• When using IRSA the following env variables are set on the root account in the container ( Technical overview - Amazon EKS ):

• AWS_DEFAULT_REGION=us-east-1
• AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
• AWS_REGION=us-east-1
• AWS_ROLE_ARN=arn:aws:iam::11111111111111:role/some-role

• Unfortunately these need to be set on the user created after initialization. For testing I’ve set them on that user in .bashrc. A quick way to do this would be from the entry point script, but ultimately the best practices for k8s say the container should not run as root so a side container would be needed for the init part and then run the check-mk container with securityContext.runAsUser: 1000

After these changes I was able to retrieve the data from AWS. This works only on v2.x.x. On v1.6.0 the boto version is older and doesn’t use AWS_WEB_IDENTITY_TOKEN_FILE. ( Credentials — Boto3 Docs 1.16.50 documentation (amazonaws.com))