cdc changefeed error，the ticdc_processor_exit_with_error_count value equal 0

[Xuma0512]

What did you do?

1. tidb version ： 6.5.11
2. create changefeed :task-test
3. drop a table on the downstream tidb cluster
4. view the changefeed,changefeed error is found 。
5. view granfana ticdc->changefeed->exit error ，no changefeed found
   

What did you expect to see?

ticdc.ticdc_processor_exit_with_error_count alert is found.

What did you see instead?

none

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

| Release Version: v7.5.0
Edition: Community
Git Commit Hash: 069631e2ecfedc000ffb92c67207bea81380f020
Git Branch: heads/refs/tags/v7.5.0
UTC Build Time: 2023-11-24 08:50:14
GoVersion: go1.21.3
Race Enabled: false
Check Table Before Drop: false
Store: tikv |

Upstream TiKV version (execute tikv-server --version):

6.5.11

TiCDC version (execute cdc version):

6.5.11

[3AceShowHand]

This also on the master branch

[3AceShowHand]

This can be reproduced by the following steps:

1. create one mysql sink changefeed
2. sync some data to the downstream database
3. drop the table on the downstream database

You can observe that the changefeed report error since the table cannot be found, and trigger processor exit logic, at this step, the metrics ticdc_processor_exit_with_error_count is deleted, and then rebuild the processor with new metrics, so that the metrics panel is always 0.

[3AceShowHand]

At the moment, processor level metrics are deleted with the processor exit, we can fix the issue by don't delete such metric, but here comes another question, when to delete the metric ?