Comparison / report on OCR vs CodeRabbit and various issues

[acoliver]

My agents did this a report -- some of it is based on my problems but it shows what issues OCR found vs Code Rabbit (another review tool) and what the difference was between running with StepFun Flash 3.7 vs GLM 5.2 was. (models I have affordable access to by subscription)
Maybe its useful for others as well.

The agent that did it got carried away (I asked it to file bugs for my script) and filed bugs here....some of which already were picked up and became PRs because the OCR developers do not sleep (or rather they sleep when I'm awake and vice versa).

[lizhengfeng101]

Thanks so much for putting this together, @acoliver — and for sharing it publicly. 🙏 Independent, hands-on comparisons like this are exactly the kind of feedback that helps us improve, and the OCR-vs-CodeRabbit breakdown plus the StepFun Flash 3.7 vs GLM 5.2 angle is genuinely useful signal for the community (especially for folks weighing model cost/quality trade-offs).

We really appreciate the bug reports too — no worries at all about your agent "getting carried away." Several of them were spot-on and have already turned into PRs, so it was a net win for the project. 😄 (And yes, the timezone thing is real.)

A couple of things that would help us get the most out of your report:

• If any of the findings point to areas where OCR under-performed (missed issues, false positives, model-specific weaknesses), we'd love to track them — feel free to open issues, or point us at the relevant sections and we'll file them ourselves.
• If you're open to it, we'd be happy to hear what would have made OCR more useful in your workflow.

Thanks again for being such an engaged part of the community — this is the good stuff. 🚀

[acoliver]

So my use case is I'm working to completely automate my software creation. Llxprt-code is a cli based on a fork from gemini-cli optimized for long running generation cycles (it can failover between models or even providers with a compatible context and retry). Jefe is for managing many llxprt code instances and Luther is a workflow system that picks up bugs and submits prs autonomously.

I use coderabbit and some rather pedantic lint/complexity checking to determine if a PR is done yet. Coderabbit seems to do more "big picture" reviews and OCR catches more low level bugs. It would be good if OCR did a bit more big picture stuff.

The other thing is that after a few iterations CR stops finding more it's where it takes like double digit cycles for OCR to stop commenting. It would be good if it stopped finding nits at some point a bit more deterministically.

It may come down to my not tuning it correctly but wording of JSDOC or rust comments isnt a review I value much in largely generated code. It provides a lot of that. I definitely don't want it to do so on the third iteration.

What is missing from the report is I had to adjust my workflow to just arbitrarily stop after 4 iterations because it was in a forever loop. However, it does help me adjust the workflow so the LLM doesnt take OCR so seriously. It might be wording or that OCR tends to rank thing high if they are the highest in the review even if I'd rank them very low.

Anyhow, as the report mentioned I do value the feedback OCR provides as it tends to find thread errors that CR misses. I'm excited how it is turning out! Thanks for the good work!