How to let trivy know to use AWS IAM credentials when pulling from IAM protected ECR repositories #264

mmerrill3 · 2020-12-04T13:40:12Z

mmerrill3
Dec 4, 2020

Hello,
I'm trying to see a way to inject either a docker credentials file, or allow trivy to user IAM credentials when requesting a container from an internal ECR repository.
The pod spec for the trivy scan job is coming from the function getPodSpecForStandaloneMode() in scanner.go. So, it doesn't appear that this is configurable. If I see this right, all docker pull requests will be anonymous.

Is there another way to setup pull images with credentials when the trivy scanner job/pod starts? In particular, local ECR repositories that enforce IAM access rules?

Answered by danielpacak

Dec 7, 2020

@mmerrill3 I have an update on that after playing with AKS and ECR. It seems that it's a matter of configuration. You can try attaching the AmazonEC2ContainerRegistryReadOnly policy to the starboard (for Starboard CLI) or starboard-operator service account.

In the example below the nginx-private refers to the private image 119244888562.dkr.ecr.eu-central-1.amazonaws.com/library/nginx:1.16. After attaching the policy to the service account I was able to generate VulnerabilityReport.

kubectl get deployments.apps -o wide
NAME            READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                               SELECTOR
nginx-private   1/1     1  …

View full answer

mmerrill3 · 2020-12-04T22:41:49Z

mmerrill3
Dec 4, 2020
Author

This is not supported yet, per other discussions.

1 reply

danielpacak Dec 5, 2020

Let me clarify this a bit @mmerrill3

In Starboard CLI v0.6.0 we supported basic authentication mechanism for private registries. However, I introduced regression in v0.7.0 while adding support for Trivy client-server mode. It's already fixed and released in v0.7.1
Starboard Operator does not support basic authentication for private registries, but we are working on it and it will come in one of the next releases. See Basic authentication support in Starboard Operator #261 to see the progress on it.

The basic authentication mechanism and the way we support it in Starboard CLI and Starboard Operator assumes that the registry credentials are specified as image pull secrets attached to a pod template or to the service account that's running a given workload. The other crucial assumption is that the password does not expire or secrets are automatically rotated by the process external to Starboard.

As you may know, the basic authentication passwords for ECR expire and we may need to find a better way to support this particular kind of managed registry.

danielpacak · 2020-12-07T21:56:14Z

danielpacak
Dec 7, 2020

@mmerrill3 I have an update on that after playing with AKS and ECR. It seems that it's a matter of configuration. You can try attaching the AmazonEC2ContainerRegistryReadOnly policy to the starboard (for Starboard CLI) or starboard-operator service account.

In the example below the nginx-private refers to the private image 119244888562.dkr.ecr.eu-central-1.amazonaws.com/library/nginx:1.16. After attaching the policy to the service account I was able to generate VulnerabilityReport.

kubectl get deployments.apps -o wide
NAME            READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                               SELECTOR
nginx-private   1/1     1            1           21h   nginx        119244888562.dkr.ecr.eu-central-1.amazonaws.com/library/nginx:1.16   app=nginx-private

eksctl create iamserviceaccount \
    --name starboard \
    --namespace starboard \
    --cluster starboard-cluster \
    --attach-policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly \
    --approve \
    --override-existing-serviceaccounts

kubectl starboard generate vulnerabilityreports deploy/nginx-private -v 3
I1207 22:37:40.217268   25934 scanner.go:62] Getting Pod template for workload: {Deployment nginx-private default}
I1207 22:37:40.263977   25934 scanner.go:68] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:true}
I1207 22:37:40.307848   25934 runner.go:79] Running task and waiting forever
I1207 22:37:40.308383   25934 runnable_job.go:77] Creating job: starboard/228e6d9c-d882-4f97-85e5-2e6f851b1800
I1207 22:37:40.364849   25934 reflector.go:175] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125
I1207 22:37:40.364865   25934 reflector.go:211] Listing and watching *v1.Job from pkg/mod/k8s.io/client-go@v0.18.6/tools/cache/reflector.go:125
I1207 22:37:51.360474   25934 runnable_job.go:112] Stopping runnable job on task completion with status: Complete
I1207 22:37:51.360493   25934 runner.go:83] Stopping runner on task completion with error: <nil>
I1207 22:37:51.360499   25934 scanner.go:98] Scan job completed: starboard/228e6d9c-d882-4f97-85e5-2e6f851b1800
I1207 22:37:51.405042   25934 scanner.go:176] Getting logs for nginx container in job: starboard/228e6d9c-d882-4f97-85e5-2e6f851b1800
I1207 22:37:51.831822   25934 scanner.go:91] Deleting scan job: starboard/228e6d9c-d882-4f97-85e5-2e6f851b1800
I1207 22:37:51.930688   25934 writer.go:71] Creating VulnerabilityReport "default/deployment-nginx-private-nginx"

/cc @lizrice

5 replies

mmerrill3 Dec 9, 2020
Author

Thanks, I'll try that out. I'll probably come up with my own IAM role for starboard, and let that role access the ECR repos in question.

danielpacak Dec 9, 2020

I think it's going to work with a custom policy the same way it works with built-in AmazonEC2ContainerRegistryReadOnly. Anyway I'd love to see your confirmation @mmerrill3 that it worked in your setup.

mmerrill3 Dec 15, 2020
Author

Not quite there yet.

ok, I setup the policy, and I used kiam to give the starboard operator the role I created. I know its different than what you tried, but I didn't setup my cluster with EKS, rather kOps.

Still looking for a way to add an annotation which would be the IAM role to assume to the trivy job I guess, that would work if it was there.

mmerrill3 Dec 15, 2020
Author

I'm going to try the non-kiam way like you tried, will get back in a bit on the status of that....

edwardpius-watchguard Jan 12, 2021

Is there an equivalent for this AKS for an image pull from an ACR Repo?

mmerrill3 · 2020-12-29T22:55:54Z

mmerrill3
Dec 29, 2020
Author

I set up IRSA, and my repo is allowing the IAM role for starboard operator to download the images. I still get a 401. Was the repo you used open to anybody?

I checked the pod, and I see the projected service account token for IAM. I grabbed the token and decoded it, it appears fine. Here is what got projected:

{
"aud": [
"sts.amazonaws.com"
],
"exp": 1609368117,
"iat": 1609281717,
"iss": "https://s3-us-west-2.amazonaws.com/auth.k8s.ctnror0.dev.mmerrill.net",
"kubernetes.io": {
"namespace": "security",
"pod": {
"name": "starboard-operator-5fbc58864d-cq2f8",
"uid": "baadc00c-5631-442e-a805-220b2a4186d7"
},
"serviceaccount": {
"name": "starboard-operator",
"uid": "719b6bf5-a897-48dd-ad05-3b5b6581c13a"
}
},
"nbf": 1609281717,
"sub": "system:serviceaccount:security:starboard-operator"
}

I see in the code that the service account is copied to the trivy job from the controller (starboard operator).

The same service account is passed to the trivy jobs, and the mutating web hook updates those pods (the new trivy pods) with a projected token as well.

Does trivy use the AWS SDK and exchange the projected token for an STS identity?

0 replies

mmerrill3 · 2020-12-31T16:24:52Z

mmerrill3
Dec 31, 2020
Author

I just tried running trivy by itself, the latest version, within k8s, using the expected service account that the IRSA mutating web hook is looking for. This is not the CLI mode, but rather mimicking what the operator does (creating a pod for each scan). I don't see any use of the projected token from the AWS mutating web hook.

mmerrillmbp:devor0 mmerrill$ kubectl run --serviceaccount k8s-starboard-role --requests cpu=100m,memory=256Mi --limits cpu=100m,memory=256Mi --image aquasec/trivy:0.15.0 -ti --rm --command trivy2 -- sh
If you don't see a command prompt, try pressing enter.
/ # trivy -d image ******.dkr.ecr.us-east-1.amazonaws.com/etcd-proxy-v2:0.0.6
2020-12-31T16:14:56.349Z DEBUG Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2020-12-31T16:14:56.354Z DEBUG cache dir: /root/.cache/trivy
2020-12-31T16:14:56.355Z DEBUG There is no valid metadata file: unable to open a file: open /root/.cache/trivy/db/metadata.json: no such file or directory
2020-12-31T16:14:56.355Z INFO Need to update DB
2020-12-31T16:14:56.355Z INFO Downloading DB...
2020-12-31T16:14:56.356Z DEBUG no metadata file
2020-12-31T16:14:56.766Z DEBUG release name: v1-2020123112
2020-12-31T16:14:56.766Z DEBUG asset name: trivy-light-offline.db.tgz
2020-12-31T16:14:56.766Z DEBUG file name doesn't match
2020-12-31T16:14:56.766Z DEBUG asset name: trivy-light.db.gz
2020-12-31T16:14:56.766Z DEBUG file name doesn't match
2020-12-31T16:14:56.766Z DEBUG asset name: trivy-offline.db.tgz
2020-12-31T16:14:56.766Z DEBUG file name doesn't match
2020-12-31T16:14:56.766Z DEBUG asset name: trivy.db.gz
2020-12-31T16:14:56.777Z DEBUG asset URL: https://github-production-release-asset-2e65be.s3.amazonaws.com/216830441/b37b9b80-4b63-11eb-965d-cf2c8f62a06c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201231%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201231T161439Z&X-Amz-Expires=300&X-Amz-Signature=1c2c5b38c5184b976747a94ff23635d65831634a4403dc65943b90801be53ad2&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=216830441&response-content-disposition=attachment%3B%20filename%3Dtrivy.db.gz&response-content-type=application%2Foctet-stream
19.57 MiB / 19.57 MiB [-------------------------------------------------------------------------------------------] 100.00% 2.66 MiB p/s 7s
2020-12-31T16:15:04.751Z DEBUG Updating database metadata...
2020-12-31T16:15:04.752Z DEBUG DB Schema: 1, Type: 1, UpdatedAt: 2020-12-31 12:25:40.923666345 +0000 UTC, NextUpdate: 2021-01-01 00:25:40.923665945 +0000 UTC, DownloadedAt: 2020-12-31 16:15:04.751857053 +0000 UTC
2020-12-31T16:15:05.304Z FATAL unable to initialize a scanner:
github.com/aquasecurity/trivy/internal/artifact.run
/home/circleci/project/internal/artifact/run.go:73

unable to initialize a docker scanner:
github.com/aquasecurity/trivy/internal/artifact.dockerScanner
/home/circleci/project/internal/artifact/image.go:28
2 errors occurred:
- unable to inspect the image (******.dkr.ecr.us-east-1.amazonaws.com/etcd-proxy-v2:0.0.6): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
- GET https://******.dkr.ecr.us-east-1.amazonaws.com/v2/etcd-proxy-v2/manifests/0.0.6: unsupported status code 401; body: Not Authorized

This shows the mutate occured:

/ # env|grep AWS
AWS_ROLE_ARN=arn:aws:iam::******:role/k8s-starboard-role
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Is this really supported in the starboard operator mode? The docs, and your example, only show the CLI mode.

0 replies

mmerrill3 · 2021-01-01T17:07:49Z

mmerrill3
Jan 1, 2021
Author

I'm still working on this. I'm focused on the use case where the docker socket is not available, and the google container registry library is used by trivy to get the images. This library tries to obtain the manifest first for the ECR image using a GET http command, with basic auth set. The basic auth should be the username "AWS" and the password that comes from the ECR get login token function.

I've verified that the IRSA flow works using docker-java and the latest AWS SDK. So, that shows that the assume role flow works for the roles I'm using, and and OIDC provider setup. I was able to download the image from ECR with a token made by the mutating web hook. So, that part is good.

I'm focused on this Manifest call and why that is giving a 401.

0 replies

mmerrill3 · 2021-01-01T18:45:57Z

mmerrill3
Jan 1, 2021
Author

Well, if there's any issue with the irsa setup, and getting the STS token, the exceptions are ignored, and the anonymous user is used.

This method from the aquasecurity fanal library is used to get the STS token, but any errors are glossed over. It turns out I had an issue with my OIDC provider I setup, and the audience. The audience mismatch exception wasn't being propagated, so the end result was the use of an anonymous authorization for basic auth with the docker repo (ECR).

This is the method in question, maybe adding a log statement of the exception would be helpful to quickly point to issues with the STS workflow.

func GetToken(ctx context.Context, domain string, opt types.DockerOption) (auth authn.Basic) {
	if opt.UserName != "" || opt.Password != "" {
		return authn.Basic{Username: opt.UserName, Password: opt.Password}
	}

	// check registry which particular to get credential
	for _, registry := range registries {
		err := registry.CheckOptions(domain, opt)
		if err != nil {
			continue
		}
		username, password, err := registry.GetCredential(ctx)
		if err != nil {
			// only skip check registry if error occurred
			break
		}
		return authn.Basic{Username: username, Password: password}
	}
	return authn.Basic{}
}

1 reply

mmerrill3 Jan 5, 2021
Author

I added some log statements to my own fanal copy, and built trivy with that fanal, and told starboard-operator (version 0.6.0) to use my trivy image. I saw that I was missing the region environment variable in my trivy job pods. After fixing that in the pod-mutating-webhook for IRSA, I was able to scan local repositories.

mmerrill3 · 2021-01-05T13:42:32Z

mmerrill3
Jan 5, 2021
Author

@danielpacak this is verified and anwered.

1 reply

danielpacak Jan 7, 2021

Great news @mmerrill3 Thank you for following up on this one and I'm glad that it finally worked in your environment! Based on your experience, do you think that there's something worth adding to the existing documentation on ECR integration available at https://aquasecurity.github.io/starboard/integrations/managed-registries/#amazon-elastic-container-registry-ecr ?

giovannirco · 2021-01-15T02:16:15Z

giovannirco
Jan 15, 2021

hey @mmerrill3 it seems after finding out the solution it worked straightforward for you. Thank you for all the information, it helped me a lot enabling IRSA and create the SA in the right way, but now after setting this up the jobs fail with a strange error

Error creating: Internal error occurred: add operation does not apply: doc is missing path: "/spec/volumes/0": missing value

Have you seen this before? The error is so generic that I couldn't find much info on the internet about it. Only a few things related to terraform where they stated that the SA needed automountServiceAccountToken: true or something like that.

The operator pod shows logs like it is creating jobs for all images in all namespaces just like I wanted but something is odd

2 replies

giovannirco Jan 15, 2021

operator pod logs:

{"level":"info","ts":1610674378.5526736,"logger":"controller.pod","msg":"Pushing back scan job","pod":"kube-system/nvidia-device-plugin-daemonset-pcnn4","count":3,"retryAfter":30}
{"level":"info","ts":1610674378.5528862,"logger":"controller.pod","msg":"Checking scan jobs limit","pod":"kube-system/node-local-dns-lzrhn","count":3,"limit":3}
{"level":"info","ts":1610674378.552896,"logger":"controller.pod","msg":"Pushing back scan job","pod":"kube-system/node-local-dns-lzrhn","count":3,"retryAfter":30}
{"level":"info","ts":1610674378.553369,"logger":"controller.pod","msg":"Checking scan jobs limit","pod":"kube-system/kube-proxy-k5cct","count":3,"limit":3}
{"level":"info","ts":1610674378.553381,"logger":"controller.pod","msg":"Pushing back scan job","pod":"kube-system/kube-proxy-k5cct","count":3,"retryAfter":30}
{"level":"info","ts":1610674378.554192,"logger":"controller.pod","msg":"Checking scan jobs limit","pod":"staking/eth-fees-server-5d5fdbbfb9-fmtrc","count":3,"limit":3}
{"level":"info","ts":1610674378.5544195,"logger":"controller.pod","msg":"Checking scan jobs limit","pod":"starboard-operator/starboard-operator-5bcd595686-222wj","count":3,"limit":3}

giovannirco Jan 15, 2021

@danielpacak is there a way I can disable the deletion of the scan job like I can do with --delete-scan-job=false on the cli? I cannot see any logs regarding the Jobs :/

VF-mbrauer · 2022-04-04T20:58:18Z

VF-mbrauer
Apr 4, 2022

You might look on the PR: #1103
Eventually, this is something you can make use of.

Cheers

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to let trivy know to use AWS IAM credentials when pulling from IAM protected ECR repositories #264

{{title}}

Replies: 9 comments 10 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to let trivy know to use AWS IAM credentials when pulling from IAM protected ECR repositories #264

Replies: 9 comments · 10 replies

mmerrill3 Dec 4, 2020 Author

mmerrill3 Dec 9, 2020 Author

mmerrill3 Dec 15, 2020 Author

mmerrill3 Dec 15, 2020 Author

mmerrill3 Dec 29, 2020 Author

mmerrill3 Dec 31, 2020 Author

mmerrill3 Jan 1, 2021 Author

mmerrill3 Jan 1, 2021 Author

mmerrill3 Jan 5, 2021 Author

mmerrill3 Jan 5, 2021 Author

Replies: 9 comments 10 replies

mmerrill3
Dec 4, 2020
Author

mmerrill3 Dec 9, 2020
Author

mmerrill3 Dec 15, 2020
Author

mmerrill3 Dec 15, 2020
Author

mmerrill3
Dec 29, 2020
Author

mmerrill3
Dec 31, 2020
Author

mmerrill3
Jan 1, 2021
Author

mmerrill3
Jan 1, 2021
Author

mmerrill3 Jan 5, 2021
Author

mmerrill3
Jan 5, 2021
Author