Repository navigation
Setup cronjob to collect billing metrics #189
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,5 @@ | ||
| apiVersion: v2 | ||
| name: moc-metrics-collector | ||
| description: Metrics reader token and usage collection jobs for billing | ||
| type: application | ||
| version: 0.2.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| prometheusUrl: https://thanos-querier.example.com | ||
| collectors: | ||
| - clusterName: test-workload0 | ||
| clusterId: 00000000-0000-0000-0000-000000000000 | ||
| bucket: test-workload0-metrics | ||
| s3RemoteKey: test-workload0-metrics-bucket |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| {{- range .Values.collectors }} | ||
| {{- $tokenSecret := .tokenSecret | default (dict "name" "moc-metrics-collector-token" "key" "token") }} | ||
| --- | ||
| apiVersion: batch/v1 | ||
| kind: CronJob | ||
| metadata: | ||
| name: {{ required "collectors[].clusterName is required" .clusterName }}-metrics-collector | ||
| namespace: {{ $.Values.name }} | ||
| spec: | ||
| timeZone: {{ $.Values.timeZone | quote }} | ||
| schedule: {{ $.Values.schedule | quote }} | ||
| concurrencyPolicy: Forbid | ||
| jobTemplate: | ||
| spec: | ||
| template: | ||
| spec: | ||
| containers: | ||
| - name: metrics-collector | ||
| image: {{ $.Values.image }} | ||
| command: ["./collect_metrics.sh"] | ||
| env: | ||
| - name: OPENSHIFT_PROMETHEUS_URL | ||
| value: {{ .prometheusUrl | default $.Values.prometheusUrl | required "collectors[].prometheusUrl or prometheusUrl is required" | quote }} | ||
| - name: PROM_QUERY_INTERVAL_MINUTES | ||
| value: {{ $.Values.queryIntervalMinutes | quote }} | ||
| - name: CLUSTER_TYPE | ||
| value: {{ .clusterType | default "hypershift" | quote }} | ||
| - name: OPENSHIFT_CLUSTER_ID | ||
| value: {{ required "collectors[].clusterId is required" .clusterId | quote }} | ||
| - name: OPENSHIFT_CLUSTER_NAME | ||
| value: {{ .clusterName | quote }} | ||
| - name: S3_METRICS_BUCKET | ||
| value: {{ required "collectors[].bucket is required" .bucket | quote }} | ||
| - name: OPENSHIFT_TOKEN | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: {{ $tokenSecret.name }} | ||
| key: {{ $tokenSecret.key }} | ||
| - name: S3_OUTPUT_ACCESS_KEY_ID | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: {{ .clusterName }}-metrics-bucket | ||
| key: access-key-id | ||
| - name: S3_OUTPUT_SECRET_ACCESS_KEY | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: {{ .clusterName }}-metrics-bucket | ||
| key: secret-access-key | ||
| restartPolicy: OnFailure | ||
| {{- end }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| {{- range .Values.collectors }} | ||
| --- | ||
| apiVersion: external-secrets.io/v1 | ||
| kind: ExternalSecret | ||
| metadata: | ||
| name: {{ required "collectors[].clusterName is required" .clusterName }}-metrics-bucket | ||
| namespace: {{ $.Values.name }} | ||
| annotations: | ||
| argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true | ||
| spec: | ||
| refreshInterval: 1h | ||
| secretStoreRef: | ||
| name: aws-secrets-manager | ||
| kind: ClusterSecretStore | ||
| target: | ||
| name: {{ .clusterName }}-metrics-bucket | ||
| data: | ||
| - secretKey: access-key-id | ||
| remoteRef: | ||
| key: {{ required "collectors[].s3RemoteKey is required" .s3RemoteKey }} | ||
| property: application_key_id | ||
| - secretKey: secret-access-key | ||
| remoteRef: | ||
| key: {{ .s3RemoteKey }} | ||
| property: application_key | ||
| {{- end }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,26 @@ | ||
| name: moc-metrics-collector | ||
|
|
||
| # Daily jobs that pull usage metrics from Prometheus and upload them to S3. | ||
| # Empty by default, so on most clusters this chart only creates the reader | ||
| # service account and its token. Set it per cluster where collection runs: | ||
| # | ||
| # collectors: | ||
| # - clusterName: oac-prod-workload0 | ||
| # clusterId: cac066c0-9e64-4421-928b-93b75c4c4d41 | ||
| # clusterType: standalone # optional, defaults to hypershift | ||
| # prometheusUrl: https://... # optional, defaults to .prometheusUrl | ||
| # bucket: oac-prod-workload0-metrics | ||
| # s3RemoteKey: b2-key/oac-prod-workload0-rw | ||
| # tokenSecret: # optional, defaults to this cluster's | ||
| # name: some-other-reader-token # own reader token | ||
| # key: token | ||
| collectors: [] | ||
|
|
||
| # Prometheus (or Thanos querier) URL for collectors that don't set their | ||
| # own. Deliberately unset: rendering fails if a collector has no URL. | ||
| # prometheusUrl: https://thanos-querier-openshift-monitoring.apps... | ||
|
|
||
| image: ghcr.io/cci-moc/openshift-usage-scripts:v0.5.0 | ||
| schedule: "5 6 * * *" | ||
| timeZone: Etc/UTC | ||
| queryIntervalMinutes: 1 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # Hosted cluster metrics live in this hub's Thanos, so the collectors for | ||
| # them run here. | ||
| prometheusUrl: https://thanos-querier-openshift-monitoring.apps.infra.oac.ocp.massopen.cloud | ||
| collectors: | ||
| - clusterName: oac-prod-workload0 | ||
| clusterId: cac066c0-9e64-4421-928b-93b75c4c4d41 | ||
| clusterType: hypershift | ||
| bucket: oac-prod-workload0-metrics | ||
| s3RemoteKey: b2-key/oac-prod-workload0-rw | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this chart create a service account and token on each cluster? Given that the default value for
collectorsis empty and all templates will be empty ifcollectorsis empty? It seems like this chart won't do anything on any cluster exceptoac-prod-infraThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This applicationset will render this chart on every hub cluster.
And this chart will render this on all managed cluster.
With this setup, every cluster will have the service account and a token available to query metrics from. We currently do not use the token from the managed clusters - we may use it in the future if we have classes.
the default value for
collectorsis empty as you noted, so no cluster by default will get a cronjob. Onlyoac-prod-infrawill get the cronjob as its value files override the empty collectors and that is what we actually want.@QuanMPhm does this clarify things?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@QuanMPhm Btw, I appreciate the questions and a thorough review, thank you!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apologies. This whole time I was too fixated on the git diff, I didn't realized the
moc-metrics-collectorchart contained more than just the cronjob and external secret.