PR Review
Introduction
Code review is a critical aspect of modern software development. In a GitHub workflow, the code review starts when a Pull Request (PR) is created, and concludes when the PR is approved and merged or rejected. The reviewers are typically senior developers or architects. They help ensure that the code submitted to the repository is correct, maintainable, scalable, and secure. That is especially important for open source projects where the many code contributions could be from the community.
However, code reviews in PRs are also often the greatest friction point for software development.
- Senior developers are very busy and very expensive. They have the least amount of time for code reviews.
- Yet, the development process cannot move forward (eg merging the PR) without the code review. Developers are often idle waiting for reviews. For open-source community developers, untimely code reviews discourage further contributions.
- Management often asks the senior developer to report and explain the key changes and risk factors associated with a PR, further delaying the process.
According to a linear b survey of over 700,000 PRs by 26,000 developers, it takes on average more than 4 days to review a PR. Developers waste two days of idle time for every PR they submit. It is an enormous amount of waste of productivity
High-level Flow diagram
How it work
Quality Estimation
The code change quality estimation task is to predict whether a code change is high-quality and ready to be accepted in the review process. This is a binary classification task, i.e., Y = {0, 1}. The input is a code change, i.e., X = {𝐷(𝐶0,𝐶1)}. As stated before, there are usually lots of changes across different source code files in a pull request. It takes reviewers a large amount of time to review all the changes. But it always turns out that most of the changes are minor and don’t need a comment or suggestion. To improve the efficiency of code review, we define and advance to automating the code change quality estimation task. Given the estimates, reviewers can give questionable code changes a higher priority to review and pay more attention to them, saving time and effort. Contributors can also leverage the estimates to improve low-quality code changes before submitting them to reviewers.
Code Review
Code review generation is a sequence generation task. The output is a predicted review comment, i.e., Y = {𝑤1, · · · ,𝑤𝑛} where 𝑤𝑖 is a natural language word and 𝑛 ∈ N is the length of the review comment. The input is still a code change, i.e., X = {𝐷(𝐶0,𝐶1)}, with its context. Researchers use the changed code as input but not the code diff, without taking into account that review comments have to focus on the changed part. It’s not recommended for reviewers to give suggestions to the code context which has not been revised. Considering the naturalness of software, language models can capture the general statistical properties of programs due to programs tend to be repetitive and predictable. In the code review process, there are also common issues in some code changes. For example, reviewers often write “This method should be private” to suggest the contributor add a private decorator to a method. This gives us a chance to learn the general code review patterns and generate comments automatically to lighten the burden of reviewers. In the real-world scenario, they may choose an ideal one for them, free from writing comments manually.
Code Refinement
In the code refinement task, the model takes as input both the original code 𝐶1 written by the contributor and the review comment 𝑅𝑛𝑙 from the reviewer, and targets to generate the revised version code 𝐶2 implementing the requirements mentioned in 𝑅𝑛𝑙 . Different from the code refinement task in other software development scenarios where only source code is taken as input, we design specifically for the code review process to utilize the review comment as guidance for refining. Code refinement is a task designed to assist contributors. When they open pull requests and get review comments as feedback, the model helps to revise the submitted code automatically based on the comments.
use GPTQ-style int4 quantization
