MR pipelines sent through webhooks are rejected in `webhook` way. #889

ZhaoUncle · 2024-09-24T08:46:23Z

issues:

I through webhook to push metrics into exporter,as the project use main branch to ci pipeline, it is ok from exporter logs.
but if I use ci-test branch to test , it is failed. I can't get metrics about ci-test branch ci pipeline

version: gitlab-ci-pipelines-exporter:v0.5.8

logs:

{"gitlab-endpoint":"https://gitlab.test.com","gitlab-rate-limit":"1rps","level":"info","msg":"configured","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull projects from wildcards","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull environments from projects","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull refs from projects","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull metrics","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect projects","on-init":"no","scheduled":"every 14400s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect environments","on-init":"no","scheduled":"every 14400s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect refs","on-init":"no","scheduled":"every 1800s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect metrics","on-init":"no","scheduled":"every 600s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"redis url configured, initializing connection..","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"connected to redis","time":"2024-09-24T08:24:46Z"}
{"host":"127.0.0.1:8082","level":"info","msg":"internal monitoring listener set","path":"","scheme":"tcp","time":"2024-09-24T08:24:46Z"}
{"controller-uuid":"652728c2-3aa6-4f94-94bd-50e95ffdfebe","level":"info","listen-address":":8080","metrics-endpoint-enabled":true,"msg":"http server started","openmetrics-encoding-enabled":false,"pprof-endpoint-enabled":false,"time":"2024-09-24T08:24:46Z","webhook-endpoint-enabled":true}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:16Z"}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:17Z"}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:17Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}

config:

config.yml: |
    log:
      format: json
    server:
      webhook:
        enabled: true
    gitlab:
      url: https://gitlab.test.com
      enable_health_check: true
      enable_tls_verify: true
      maximum_jobs_queue_size: 5000
    pull: 
      environments_from_projects:
        on_init: true
        scheduled: false
      metrics:
        on_init: true
        scheduled: false
      projects_from_wildcards:
        on_init: true
        scheduled: false
      refs_from_projects:
        on_init: true
        scheduled: false
    project_defaults: 
      output_sparse_status_metrics: true
      pull:
        environments:
          enabled: false
          exclude_stopped: true
          regexp: .*
        pipeline:
          jobs:
            enabled: true
            from_child_pipelines:
              enabled: true
            runner_description:
              aggregation_regexp: shared-runners-manager-(\d*)\.gitlab\.com
              enabled: true
          test_reports:
            enabled: true
            test_cases:
              enabled: true
          variables:
            enabled: true
            regexp: .*
        refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
          merge_requests:
            enabled: true
            max_age_seconds: 0
            most_recent: 0
          tags:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
    projects: 
      - name: test/devops/poc/ci-test

The text was updated successfully, but these errors were encountered:

jose-sky · 2024-09-24T16:25:26Z

Same problem. I tried with all the options I can think of (project_defaults, wildcards, projects configuration). I cannot get the branch trunk to work.

project_defaults:
  pull:
    pipeline:
      jobs:
        enabled: true
    refs:
      branches:
        enabled: true
        regexp: .*
        most_recent: 500
        max_age_seconds: 0
      merge_requests:
        enabled: true
        most_recent: 500

I get errors for trunk and MRs and doesn't matter what I change I cannot make it work, I tried many combinations.

Sep 24 16:23:19 gitlab-ci-pipelines-exporter[236447]: time="2024-09-24T16:23:19Z" level=info msg="ref not configured in the exporter, ignoring pipeline webhook" project-name=group-name/apps/core-app ref=refs/merge-requests/4945/merge ref-kind=branch

Sep 24 16:24:17 gitlab-ci-pipelines-exporter[236447]: time="2024-09-24T16:24:17Z" level=info msg="ref not configured in the exporter, ignoring pipeline webhook" project-name=group-name/apps/core-app ref=trunk ref-kind=branch

StefanOid · 2024-09-25T07:49:33Z

hello @mvisonneau ,

issues:

I through webhook to push metrics into exporter,as the project use main branch to ci pipeline, it is ok from exporter logs. but if I use ci-test branch to test , it is failed. I can't get metrics about ci-test branch ci pipeline

version: gitlab-ci-pipelines-exporter:v0.5.8

logs:

{"gitlab-endpoint":"https://gitlab.test.com","gitlab-rate-limit":"1rps","level":"info","msg":"configured","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull projects from wildcards","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull environments from projects","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull refs from projects","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull metrics","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect projects","on-init":"no","scheduled":"every 14400s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect environments","on-init":"no","scheduled":"every 14400s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect refs","on-init":"no","scheduled":"every 1800s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect metrics","on-init":"no","scheduled":"every 600s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"redis url configured, initializing connection..","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"connected to redis","time":"2024-09-24T08:24:46Z"}
{"host":"127.0.0.1:8082","level":"info","msg":"internal monitoring listener set","path":"","scheme":"tcp","time":"2024-09-24T08:24:46Z"}
{"controller-uuid":"652728c2-3aa6-4f94-94bd-50e95ffdfebe","level":"info","listen-address":":8080","metrics-endpoint-enabled":true,"msg":"http server started","openmetrics-encoding-enabled":false,"pprof-endpoint-enabled":false,"time":"2024-09-24T08:24:46Z","webhook-endpoint-enabled":true}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:16Z"}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:17Z"}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:17Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}

config:

config.yml: |
    log:
      format: json
    server:
      webhook:
        enabled: true
    gitlab:
      url: https://gitlab.test.com
      enable_health_check: true
      enable_tls_verify: true
      maximum_jobs_queue_size: 5000
    pull: 
      environments_from_projects:
        on_init: true
        scheduled: false
      metrics:
        on_init: true
        scheduled: false
      projects_from_wildcards:
        on_init: true
        scheduled: false
      refs_from_projects:
        on_init: true
        scheduled: false
    project_defaults: 
      output_sparse_status_metrics: true
      pull:
        environments:
          enabled: false
          exclude_stopped: true
          regexp: .*
        pipeline:
          jobs:
            enabled: true
            from_child_pipelines:
              enabled: true
            runner_description:
              aggregation_regexp: shared-runners-manager-(\d*)\.gitlab\.com
              enabled: true
          test_reports:
            enabled: true
            test_cases:
              enabled: true
          variables:
            enabled: true
            regexp: .*
        refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
          merge_requests:
            enabled: true
            max_age_seconds: 0
            most_recent: 0
          tags:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
    projects: 
      - name: test/devops/poc/ci-test

Looks like your regex is not working... Have you tried looking at that?? Maybe quote the regex expression??
Apologies if it's dumb 😆

ZhaoUncle · 2024-09-25T08:18:28Z

@StefanOid sure, regexp: .*,you can see in projects_defaults settting.
And I test in projects add this part regex rule regexp: ci-test, ci-test is branch name, but it is not ok.

 refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*

StefanOid · 2024-09-25T08:21:43Z

@StefanOid sure, regexp: .*,you can see in projects_defaults settting. And I test in projects add this part regex rule regexp: ci-test, ci-test is branch name, but it is not ok.
 refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*

ci-test is not a valid regex expression. ^ci-test$ would match what you have.
Also, what i meant was, try "dotstar" instead of just .* Or "^ci-test$"

Basically .* in "" (github is removing star symbol for some reason)

ZhaoUncle · 2024-09-25T08:40:07Z

@StefanOid yes, I already add it into regex, but it is failed, same problem

^(?:ci-test)$

StefanOid · 2024-09-25T10:51:15Z

@StefanOid yes, I already add it into regex, but it is failed, same problem
^(?:ci-test)$

can you show me the working configuration you had with main?

ZhaoUncle · 2024-09-25T11:52:30Z

@StefanOid yes, I already add it into regex, but it is failed, same problem
^(?:ci-test)$
can you show me the working configuration you had with main?

don't use main, only ci-test.

StefanOid · 2024-09-25T12:06:13Z

You said it is working for main branch. Is it the same config? Kind Regards, Stefan Birca Tel: 07761916718

…

On Wed, 25 Sep 2024 at 12:52, ZhaoUncle ***@***.***> wrote: @StefanOid <https://github.com/StefanOid> yes, I already add it into regex, but it is failed, same problem ^(?:ci-test)$ can you show me the working configuration you had with main? don't use main, only ci-test. — Reply to this email directly, view it on GitHub <#889 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOIBF6KDKAMLKA6GVK5Y5DZYKPZLAVCNFSM6AAAAABOXYZ3C2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZTHA3DOMRTGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

ZhaoUncle · 2024-09-25T12:13:23Z

You said it is working for main branch. Is it the same config?

yes, only this config, no another

ZhaoUncle · 2024-09-26T06:05:55Z

@StefanOid

config.yml: |
    log:
      level: info
      format: json
    server:
      webhook:
        enabled: true
    gitlab:
      url: https://gitlab.test.com
      enable_health_check: true
      enable_tls_verify: true
      maximum_jobs_queue_size: 5000
    pull: 
      environments_from_projects:
        on_init: false
        scheduled: false
      metrics:
        on_init: false
        scheduled: false
      projects_from_wildcards:
        on_init: false
        scheduled: false
      refs_from_projects:
        on_init: false
        scheduled: false
    project_defaults: 
      output_sparse_status_metrics: true
      pull:
        environments:
          enabled: false
          exclude_stopped: true
          regexp: .*
        pipeline:
          jobs:
            enabled: true
            from_child_pipelines:
              enabled: true
            runner_description:
              aggregation_regexp: shared-runners-manager-(\d*)\.gitlab\.com
              enabled: true
          test_reports:
            enabled: true
            test_cases:
              enabled: true
          variables:
            enabled: true
            regexp: .*
        refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
          merge_requests:
            enabled: true
            max_age_seconds: 0
            most_recent: 0
          tags:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
    projects: 
      - name: test/devops/poc/ci-test
        pull:
          environments:
            enabled: false
            exclude_stopped: true
            regexp: .*
          pipeline:
            jobs:
              enabled: true
              from_child_pipelines:
                enabled: true
              runner_description:
                aggregation_regexp: shared-runners-manager-(\d*)\.gitlab\.com
                enabled: true
            test_reports:
              enabled: true
              test_cases:
                enabled: true
            variables:
              enabled: true
              regexp: .*
          refs:
            branches:
              enabled: true
              exclude_deleted: true
              max_age_seconds: 0
              most_recent: 0
              regexp: ^(?:ci-test)$
            merge_requests:
              enabled: true
              max_age_seconds: 0
              most_recent: 0
            tags:
              enabled: true
              exclude_deleted: true
              max_age_seconds: 0
              most_recent: 0
              regexp: .*

use this config, same problem, main branch is ok, ci-test branch is failed, same err log

ZhaoUncle · 2024-09-27T07:04:24Z

@jose-sky @StefanOid
I del redis all key, and then ci-test branch is ok now.....

irfansofyana mentioned this issue Dec 4, 2024

Minimal configuration to track pipelines & jobs on project branch and MR with webhook #960

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MR pipelines sent through webhooks are rejected in `webhook` way. #889

MR pipelines sent through webhooks are rejected in `webhook` way. #889

ZhaoUncle commented Sep 24, 2024

jose-sky commented Sep 24, 2024

StefanOid commented Sep 25, 2024

ZhaoUncle commented Sep 25, 2024

StefanOid commented Sep 25, 2024 •

edited

Loading

ZhaoUncle commented Sep 25, 2024

StefanOid commented Sep 25, 2024

ZhaoUncle commented Sep 25, 2024

StefanOid commented Sep 25, 2024 via email

ZhaoUncle commented Sep 25, 2024

ZhaoUncle commented Sep 26, 2024 •

edited

Loading

ZhaoUncle commented Sep 27, 2024

MR pipelines sent through webhooks are rejected in webhook way. #889

MR pipelines sent through webhooks are rejected in webhook way. #889

Comments

ZhaoUncle commented Sep 24, 2024

jose-sky commented Sep 24, 2024

StefanOid commented Sep 25, 2024

ZhaoUncle commented Sep 25, 2024

StefanOid commented Sep 25, 2024 • edited Loading

ZhaoUncle commented Sep 25, 2024

StefanOid commented Sep 25, 2024

ZhaoUncle commented Sep 25, 2024

StefanOid commented Sep 25, 2024 via email

ZhaoUncle commented Sep 25, 2024

ZhaoUncle commented Sep 26, 2024 • edited Loading

ZhaoUncle commented Sep 27, 2024

MR pipelines sent through webhooks are rejected in `webhook` way. #889

MR pipelines sent through webhooks are rejected in `webhook` way. #889

StefanOid commented Sep 25, 2024 •

edited

Loading

ZhaoUncle commented Sep 26, 2024 •

edited

Loading