Pull request checks

Turn checks on#

Checks are off for every repository until someone turns them on. Until then, its pull requests get no check. In your Arbor Cloud workspace:

1. Install the Arbor GitHub App on the repository, and add the repository to your workspace.
2. Open Analyses. Under Check every pull request, choose Turn on next to the repository.
3. Open a pull request, or push a commit to one that is open. The “Arbor impact” check appears with the commit’s other checks.

Only a workspace member with write access can turn checks on or off. Analyses then run as that member: each run rechecks their membership, their access to the repository on GitHub and the allowance, as a run they started by hand would. If they lose access, the check says so instead of analyzing, and another member can turn checks on again as themselves.

Only these automatic runs add the check. A run started by hand on the Analyses page saves a private report and adds no check, whether or not checks are on.

Arbor checks a pull request when it is opened, reopened or marked ready for review, and when a commit is pushed to it. Drafts wait until they are marked ready. A pull request that was already open when you turned checks on is checked at its next commit.

What the check shows#

The check is a map for review, not a verdict. Its title states the headline facts, and its summary has these parts:

What it reaches

Entry points whose calls lead to the changed functions, up to eight, each with how many calls away it is. An entry point is a function that looks like a route, handler, job or command by its file and name, and that no production code calls.

What to test

Changed functions with callers outside tests and no test found, up to eight. No test found means the graph shows no test calling the function within four calls, and no test file names it.

Where to start reading

Up to five changed files, ranked by the entry points they reach, the sensitive areas they touch, their functions with no test found and their production callers. Each file shows its owners when the repository’s CODEOWNERS file names them.

Dependencies

Advisories from OSV for package versions the pull request added or updated in Cargo.lock, package-lock.json or npm-shrinkwrap.json. Advisories already present before the pull request aren’t listed. Other lockfiles, such as yarn.lock and pnpm-lock.yaml, aren’t checked.

What Arbor couldn’t see

This run’s own gaps, then the calls Arbor doesn’t follow: reflection, dynamic imports, configuration and HTTP.

Annotations

Up to ten notices, on the lines of changed functions with callers and no test found.

Reaches 1 entry point · 1 called function with no test found

What it reaches
- checkoutHandler in src/routes/checkout.ts, 2 calls away

1 function or type changed, with 2 direct callers outside tests.

What to test
- applyDiscount in src/cart/totals.ts:14: 2 callers, no test found calling or naming it

Where to start reading
1. src/cart/totals.ts: reaches 1 entry point · 1 called function with no test found · 2 production callers · owned by @acme/checkout

Dependencies
No added or updated package version has a known advisory in OSV.

What Arbor couldn't see
- Calls made through reflection, dynamic imports, configuration or HTTP aren't followed, so a test or caller can exist that Arbor did not find.

The full report, linked from the check, has longer lists. Re-run on the check tries again when an analysis didn’t finish. For a commit that already has a complete analysis, it reports the same result.

Team rules in .arbor/checks.yml#

A repository can name the facts that should hold a pull request. Arbor reads .arbor/checks.yml from the pull request’s head commit, so a pull request that edits the file is checked against its own version. Review changes to it as you would code.

# Facts that should hold a pull request for review.
version: 1
fail_when:
  # Billing and sign-in code that other code calls needs a test.
  - untested_change:
      paths: ["src/billing/**", "src/auth/**"]
  # So does anything an entry point reaches.
  - untested_entry_point
  # No new high or critical advisories.
  - new_advisory:
      min_severity: high
  # Payment and session changes get a second look.
  - sensitive_change:
      areas: [payment, session]

untested_change

A changed function with production callers has no test found. Optional paths limits it to matching files.

untested_entry_point

A changed function that an entry point reaches has no test found. Takes no options.

new_advisory

The pull request adds or updates a package version with a known advisory, in Cargo.lock, package-lock.json or npm-shrinkwrap.json. Optional min_severity is any (the default), low, moderate, high or critical. An advisory without a severity matches only any.

sensitive_change

The pull request touches a sensitive area. Optional areas takes auth, crypto, database, file_io, network, secrets, permissions, validation, session or payment. Without areas, any sensitive area matches.

Writing the file

• version: 1 is required. fail_when lists the rules; without it, no rule applies.
• Write each rule as its name alone, or as its name with its options indented beneath it. One list item names one rule, and a rule can appear more than once with different options.
• In paths, use src/billing/** for everything under a folder, *.sql for a file ending, or a path from the repository root. Matching ignores case.
• Rule names, options, severities and areas are lowercase, as shown.

A file Arbor can’t use applies no rule at all, and the check names the problem. That includes a version other than 1, a misspelled rule, option or area, an option the rule doesn’t take, more than 20 rules, a file over 64 KB, a symbolic link and YAML that doesn’t parse.

Rules match facts, never counts or scores, so the check can say which function, file or package held the pull request.

Failing and overriding#

On the Team plan, a matched rule fails the check. Its title names the first rule that matched, as in Team rule matched: A changed function an entry point reaches has no test found. The summary lists every rule as matched or not, with up to five of the functions, files or packages that matched.

A failing check holds a merge only when branch protection or a ruleset on GitHub requires the “Arbor impact” check. Arbor doesn’t change those settings; a repository admin does.

To accept the matched rules for one commit, a workspace member with write access presses Override on the check in GitHub. Arbor records who pressed it and when, and the check turns neutral with a title such as Overridden by octocat: …. Override from anyone else changes nothing. The next commit, or a re-run, is checked again without the override.

On the trial and the Developer plan, rules are still checked and listed, but the check stays neutral. When a rule matches, the check says that rules hold it only on the Team plan. Without a rules file, or with one Arbor can’t use, the check is neutral too.

Limits#

Plan

Rules fail the check only while the workspace is on the Team plan. Checks themselves work on every plan, the trial included.

Allowance

Each pull request uses one from the allowance. It’s held while the first analysis runs and kept once that analysis completes. New commits on it are free in the same billing period, or during the trial. Failed, canceled and unsupported runs don’t use the allowance.

100 runs per pull request

A pull request can have up to 100 runs in one billing period, or in the trial. Every run counts toward that, failed, canceled and unsupported ones included. After that, its new commits aren’t analyzed, and the check says the allowance is used.

Re-runs

One commit can be checked up to 20 times, re-runs included.

Newer commits

A newer commit cancels the automatic run still going for an older one. The canceled run doesn’t use the allowance, though it counts toward the 100 runs, and its check says it was replaced by a newer commit.

When nothing can start

If the allowance is used, the member’s access can’t be confirmed, the workspace is already running as many analyses as its plan allows, or the pull request is larger than Arbor analyzes, the check still completes, neutral, and says which.

Changed functions

The check and its rules use the 40 changed functions with the most entry points and callers.