You should already have 2+ developers looking at most PRs. And these developers should absolutely use AI. The PR author should use AI.
But what you should not do is pipe the AI output directly into the PR and tell the PR author to deal with it. That's adding noise to the PR review process. Everything it says is something the PR author needs to validate as relevant, helpful, etc. A human needs to do that before confronting the author with it.
You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?
In our new world of non-deterministic output (that's why we love LLMs! they say such helpful/agreeable/sometimes wrong stuff!), I think CI won't be sufficient. CI is in the realm of Quality Control; when I build the thing, is it to spec and does it do what I need it to do?
But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time. For example, if a hotel chatbot starts telling guests checkout time is at 11pm instead of 11am, that's a real operational problem and ideally should be caught before it impacts customers.
LLMs introduce new classes of problems/risks that we are just starting to understand and develop the tools to manage.
>But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
The instinct seems good because it's impossible to keep up with all the details if you are running AI full-blast. Absolutely impossible. So testing outputs makes sense.
I have a hard time seeing exactly how we get from here to there. But intuitively I would not be surprised. One of these thing where quality may drop 20% but you can scale 100x.
> In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time.
Is that hotel example real? Curious how exactly you employ this technique—my naive idea was, if talking software development, a sort of 'sanity-check auto-linter agent' catch errors on a regular basis (every 10 seconds, every write, w/e).
Well, it's a real example! No, it's not a true story that I'm aware of, but there are plenty of examples of real chatbots run amok.
I refactored the tool out of my application and it's available now at https://endpointevaluator.com . There's nothing there someone couldn't build themselves, but then you have to spend that time building and maintaining it. I think the ongoing and long term cost of maintaining all this LLM-generated software is under-appreciated. So hopefully there is still space for outsourcing tools that are generic enough to be used by many and provide good value for cost.
If that aspect is critical to your workflow, it seems like you could run the model of your choice off of hugging face, on GPU hardware under your control, so the model won't shift out from under you.
That's one good way to reduce the risk, but I don't think it eliminates it.
Even with the same model and the same input, the output is inconsistent. And what I've observed is that as the size of the input and output grows, the consistency and accuracy of the output seems to decrease. It gets more complicated when you don't control the full input, such as a chatbot with customers.
I think the problem remains even if it can be mitigated by freezing the model and the hardware, which carries the tradeoff of requiring a model you can download and run on your own so you can't use the SOTA models.
Our agents automatically review our PR's - the authors agents automatically see the feedback and make fixes, and automatically merge when everything is green.
A well authored CI review process is significantly better than any human could do. We have the AI review not only the changes but clone and investigate all related repositories that integrate with the code to evaluate interactions, check all open and historical issues that may be related, review production/development logs for the feature, etc.
It will then also go update issues/tasks to reflect the changes after the merge.
Similarly you can have automated agents watching post-deployment to mitigate risk in the "a new feature just went out" time-frame. Just having someone check the related logs in the hours after deployment can catch a lot of issues that otherwise would turn into a fire-drill.
My question as well. Why is this happening in ci/ci? Why aren't contributors performing the automated review and correction step locally?
The only answer I can imagine is that it supports reviewing patches from people who aren't using AI or review by humans but parent doesn't mention this.
Must be a token burner setup. Nice if you really got it to work. Now also add business agents that come up with new features based on actual research. Your automated humanless company is almost there.
They're making money, which is more than most ICs living in cost centers can say.
The human code monkey sometimes thinks too highly of the task of coding. That's never once been the job to be done.
Every engineer is going to become a master of understanding and measuring need, opportunity cost, organizational support and headwinds, political will, etc. These are the true features of the job. The coding part will fall away entirely.
Deliver value. Code used to be the expensive part, but now it's becoming cheap, fungible, and ephemeral.
...of course you're likely just trolling, hard to say with the drivel coming out of Silicon Valley the last few years. In that case: my apologies and keep up the good work :D
This is where we landed too. It works remarkably well. AI guidance is really the crucial thing here. I know a lot of people don't like this, think it's really bad, think it just results in slop... but it doesn't or at least never results in dead end slop. If you find what you get out of AI is weak in some area, then get it to focus on making sure that it does a better job, give it more guardrails / guidance etc till you are happy with the result.
I personally would never dump a claude code response into a PR body an ask someone to address it. I can't believe any developer would find this to be appropriate or fulfilling the duty of reviewing code.
I think it's pretty clear that what you should do instead is go over the result and communicate to the PR author anything you think should be addressed. You don't just say "here are 8 things you might want to address but I cant actually speak to any of them."
To add to this, the fact that Claude so often tries to deviate from defined architecture can be really frustrating if you're working in a mature codebase.
That scenario would be annoying but it's not what we have. The agent joins the PR leaves a few comments and responds (with mixed success) to comments. Personally I'd go back to 2020 and never see an LLM again quite happily but that genie is out of the bottle.
Unfortunately, the results speak for themselves and I could probably find dozens of edge cases which the AI code review caught which would have otherwise been missed, probably until production.
I think there is a subtle difference between you-or-I dumping a chat paste into the PR, versus an automated agent directly commenting in a PR. The former carries a human social implication, the latter does not. The weight of them is different.
If you are talking about AI's opening PR's then maybe you have a point. But a person opening a PR should not be met with unvetted feedback.
To be absolutely clear, AI should be used for PR review. It should be used many times. By the PR author and by all the reviewers. It should not just be piped directly from the agent to the author inside the PR. That causes the author to triage every comment.
That sounds good. But it doesn't work in practice from my experience.
The exact same skill in the hands of one person will have vastly different outcomes in the hands of a different person. The review skill I built myself has been shared with folks. They (say they) use it before they put up a PR. I still catch things with the same skill. That is evolving as I catch the model(s) at BSing. Even if I would re-share it all the time, when I catch it, these other people using the same skill wouldn't catch it.
And that is exactly why building a general purpose review agent (or skill) is harder than me having my own evolving skill. If I have to "properly" release a skill/some automation, I will likely err on the side of not having too many false positives. That is harder than still relying on a human to review the AI reviewer. The same people that were really good actual human reviewers of a PR are probably the ones that are good doing the same thing while AI assisted. And the ones that really needed these other human reviewers and processes to help them be productive are probably going to produce a vibe coded mess if left to their own devices with nothing but some AI.
Have you tried running the same review skill back to back in fresh sessions? My experience has been: 1) build complicated thing with LLM, 2) run code review, 3) it finds like 8 things, 4) fix them, 5) run another code review, 6) it finds another new 8 things, rinse and repeat.
I'd guess people truly are finding things pre-review. It's just that LLM review seems to have a limit to the number of problems it can or will find. While at the same time, LLM-written code can be an almost limitless source of bugs and errors.
It has no sense of when an architecture is doomed and needs fundamental changes. Instead, it will happily continue to chase individual bugs nearly to infinity.
But, maybe your review skill is better than the ones I've tried.
With the way memory systems work, I can see the value in having a different person's AI conduct the review as that AI's 'memory' is going to have a slightly different perspective aligned with the developer piloting it
I'm glad I work in places where there's no such silly pointless rules like how many people need to review a PR.
The PR author asks for feedback if it needs feedback, otherwise it merges it, period.
I don't know why and when the world got convinced that all this bureaucracy is a "best practice", when it's just a practice, that can be good, or a waste of time depending on many factors.
Like do doctors when analyzing your medical records are like "I have a diagnosis, but first let me have two colleagues review it"? No, unless they are at the beginning and it's part of the process or they need one.
And that's way more serious than the crap most HN is writing.
My partner gave birth last month and the hospital had a system where every hour a second midwife would come into the room and look at all the charts and review their colleague’s work with a second, fresh pair of eyes.
Double-checking in medicine is a practice that is usually used only for specific high-risk scenarios.
Maybe yours was, albeit it sounds strange because hourly it doesn't even happen in intensive maternity-unity cares where there's a high risk for the child's health.
In my experience with the Claude Github integration, I found it to be pretty helpful. It’s had a pretty good success rate of catching bugs before they get to master, and for simple ones I can ask it to fix itself.
> you should have 2+ developers looking at most PRs
It’d be nice, but usually not the case in my experience. More eyes is better. AI review should not replace human review, it should supplement it. I find myself spending more time doing end-user testing instead of looking at code vs before.
We had a two human PR requirement until recently we dropped it. It was slowing us down too much now the human developer creating the future is obviously writing it all with AI so they need to check it then depending on the feature and it’s use it requires a PR but it’s not universal and we’ve stepped up our automated test Tan X what it used to be it’s been so far fewer bugs better delivery
If you imagine someone speaking that comment out loud, but speaking as if they were giving a keynote at a Meta or Apple dev con, it becomes much easier to read. The commas and dramatic ellipses just fell into place as I read. Like the matrix, but instead of green kanji raining down, it’s readability-increasing punctuation.
lol
Certainly not always. There's a hedonic adjustment which happens however, where some tasks go very smoothly without much specification and a lot of "you know what I mean" to the LLM, while others then require you to get painfully specific after it badly misinterprets your intent.
Or maybe you can just get too spoiled with it grokking your intent, then become so vague that your vague ideas are actually just bad ideas. Certainly has happened to me.
After almost 4 years with LLMs, if my prompt is too vague that I don’t know how to ask precisely, I use this prompt “I have this idea… {description here} How would ideal prompt look like to make idea realize?” And in second round I polish prompt myself. It usually works.
> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?
I would if the PR was clearly written by AI. I'm fine with the PR author using AI, but only to draft the PR. They should be editing the shit out of it for the final version before submitting it.
AI code reviews are the same as AI pull requests. Do you want that firehose? The suggestions might be good. But do you want to add a deluge of work items to that part of the workflow? Do you want the PR owner to be the one to triage feedback before it gets to them? After they have already done it themselves, perhaps with an even better model?
Human reviews are going away for most PRs these days. If the LLM decides the risk level of the change is low, the PR can be merged without another human in the loop. That's at least the direction many companies are taking. Only require a human judgement when necessary.
We’ve added AI to our auto review process. It does expose when the author is not confident in their solution to push back. Which is interesting. But we do try to target specifically at our patterns and safety.
> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?
of course not, it's disrespectful to the author even if they used an LLM to generate the code. what you should do is actually try to understand what the LLM is saying about the proposed changes, check whether it's talking shit or legit, and if legit, rephrase in your own words why you think a certain thing should be changed
It's almost certainly Gemini but I don't know if it's Flash or Pro.
I would assume though advanced reasoning models would be best, especially since it's out of band. If I were building it maybe I'd run two adversarial models.
We totally use it in CI. We were thoughtful in how we built our review bot and it catches a lot of issues before humans lay eyes on the code. Typically they are lower level code issues that humans would not have noticed. It's not the comprehensive review solution, which is why we still require some level of human attendance to the code, but it's very much worth using. We do set limits on PR size and description quality as well to mitigate the harsher AI slop issues.
Notice you haven't even attempted to address why it's important to have in CI. We already agree AI PR reviews are good. But why should you add this new responsibility for the PR author of triaging feedback?
> Typically they are lower level code issues that humans would not have noticed
The author does not triage the feedback. Review bot assesses priority and agents fix the issues the automated review discovers. Nothing is blocked by the bot, humans can ignore if they think the feedback isn't helpful.
We use AI heavily in development but everyone has their own setup and way of approaching use. AI in PRs provides a consistent review layer beyond what the engineers do themselves, and catches a previously undiscovered issue in about 75% of the PRs.
Haven't had a chance to test astra vs 5.6 in these reviews yet. I've seen mixed feedback on astra generally so have hesitated in the upgrade. Have you tested and found any difference?
You should already have 2+ developers looking at most PRs. And these developers should absolutely use AI. The PR author should use AI.
But what you should not do is pipe the AI output directly into the PR and tell the PR author to deal with it. That's adding noise to the PR review process. Everything it says is something the PR author needs to validate as relevant, helpful, etc. A human needs to do that before confronting the author with it.
You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?