This article is about a small idea we tried in our open-source project: adding a “risk” mark to Conventional Commits. Just a few symbols help to see fast which commits are safe, which need testing, and which are risky. It started as an experiment, but it really changed how we think about testing and releasing.

I wrote this article following my talk Commit Notations for Quality and Testability at the ProQuality Conference 2025.

Context

This notation was developed by Clare Macrae and me during pair programming sessions on obsidian-tasks — an Obsidian plugin for task management. Obsidian is an Electron-based editor that renders local Markdown files and supports JavaScript plugins.

Our plugin has a query engine with many filters, rendering options, and grouping features, plus a modal window for task creation and editing written in Svelte. As of September 1st, 2025, it had over 2.6M downloads. I joined the project in late 2022, while Clare was already contributing. We develop using TDD (Test-Driven Development) and Pair Programming.

Note that we work in a near-ideal environment: we have no customers, we choose what we commit to, and we use the product ourselves. Because we care about it deeply but only contribute part-time (2–4h per week for me), and since the project is open-source, we cannot afford a separate QA role.

With such constraints, it is crucial to avoid situations where a new release goes out and is immediately followed by a storm of bug reports (this has happened!).

Why share this?

While Clare has been the main driver of this approach in our collaboration, I have also adopted it outside of our shared work. Some of my colleagues asked me about it, so I thought it would be useful to share it with a wider audience. Hopefully, you’ll find it relevant for your environment—or at least inspiring.

I believe our job as software engineers is to work smarter. When I have to manually test the plugin, I don’t feel smart: I need to follow a set of actions exactly, it’s hard to stay focused, I get nervous, and overall the experience is unpleasant. Probably because I am not a QA engineer and lack that mindset.

So we tried using commit notation to identify releases where we could reduce — or even skip — manual testing.

Developing the notation

I am familiar with Conventional Commits, which I enjoy because of its clarity and simplicity. It has only two fields: <type>: <description>. For example: refactor: rename variables. However, it lacks any risk notation. How confident am I that this commit might break something? Is it just a three-line variable rename done with an automated tool? Is there a unit test covering it? Or do I need manual testing or a beta deployment? What if I don’t know the full impact of the change?

Arlo’s Commit Notation (ACN) handles this with a risk field: <risk> <type + visibility> <message>. For example: ^ r rename variables. It’s not very easy to read, but it does provide a risk factor for each commit. Here are the four risk levels from the spec:

Risk Level Code Example Meaning Correctness Guarantees
(Proven) Safe . . r Extract method Addresses all known and unknown risks. Intended Change, Known Invariants, Unknown Invariants
Validated ^ ^ r Extract method Addresses all known risks. Intended Change, Known Invariants
Risky ! ! r Extract method Some known risks remain unverified. Intended Change
(Probably) Broken @ @ r Start extracting method with no name No risk attestation.  

The important thing here is the risk level, we took it from this notation and adapted it into Conventional Commits as: <type>: <risk> <description>

Here, risk is an optional field similar to ACN:

Risk Level Code Meaning
Automated action . Done with IDE tools we can trust, no need to test (e.g. VSCode Abracadabra, JetBrains refactors)
Unit tested - There is a unit test that fails if the code is incorrectly altered
Manually or integration tested ! The change can only be validated with manual actions or deployment
Cannot be tested @ The change cannot be tested in any way
Breaks unit tests ** The change breaks at least one unit test (sometimes needed during development)
Not needed /none/ Risk assessment doesn’t make sense for this commit

Commit analysis

From December 2022 to May 2025, I made mostly during pairing sessions with Clare in my fork of obsidian-tasks. Most of them were later merged:

  • 1539 commits
  • 1530 conventional commits
  • 1259 conventional commits with risk notation
    • 40% automated
    • 47% unit tested
    • 11% manually tested
    • 1% broke unit tests
    • 1% couldn’t be tested

This basically means we need manual testing for only 1 out of 10 commits, reducing manual work by a factor of 10.

Branches

Conventional Commits also describe branch types. But now, with risk assessment per commit, we can also talk about branch risk as a whole.

For example, this branch:

* refactor: - remove comments
|
* feat: - better explanation and error messages from include instructions
|
* test: - check error message for includes
|
* test: - text explanation of nested includes
|
* test: - test nested include instructions
|
* feat: - teach query to parse multi-line include instructions
|
* feat: - give a meaningful error for non-existent includes
|
* test: - test layout instruction
|
* test: - test query source
|
* feat: - accept any single-line include instruction
|
* feat: - accept `include not_done` instruction
|
* test: - add failing test for include instruction
|
* docs: document includes

Looking at all the - markers, its maximum risk is unit tested. This means no manual testing of the changed code is required. However, since it’s a feature branch, exploratory testing is still needed to validate the feature itself, but not the underlying code.

Another example:

* refactor: . rename variable
|
* refactor: - remove unused part of an object
|
* refactor: . convert parameters to object
|
* refactor: - remove unused parameter and code
|
* refactor: ! use local anonymous client
|
* refactor: . move function to dedicated file
|
* refactor: reuse queryAsAnonymousUser()
|
* refactor: ! add resolver parameter
|
* refactor: ! add type assertion and generic parameter
|
* refactor: ! get property with a string value
|
* refactor: . add query parameter
|
* refactor: - add TInput generic parameter
|
* refactor: ! extract queryAsAnonymousUser()
|
* refactor: . rename variable
|
* refactor: - remove unused interface

This is a pure refactoring branch. It still needs manual validation (because of the ! commits) , but the advantage is that we can pinpoint exactly which code needs checking and extract test scenarios accordingly.

Quality gate for CI/CD

Since branch risk can be deduced from commit risk, this notation could also serve as a quality gate for CI/CD. Example rules:

  • Only . commits
    • Release directly
  • At least one - commit
    • Exploratory testing → Release
  • At least one ! commit
    • Manual + integration testing → Release

We didn’t implement this officially, because with just two people it would add unnecessary bureaucracy.

Pros and cons

This commit notation requires discipline in making small, contained commits — something not very common even without such notations. For us, it’s easy because of pair programming and the small team size. But enforcing this in a larger team or company would be a big step. Mistakes can happen: overconfidence (marking risky commits as safe) or excessive caution (marking safe commits as risky, leading to more manual work). With team turnover, proper onboarding would also be needed. It can also increase the total number of commits, which in some cultures might be seen negatively.

Still, the benefits are clear:

  • Less frequent manual testing — sometimes not needed at all
  • Shorter manual testing — by isolating risky commits, only affected features need re-validation
  • Better release planning — merge safer branches first, leave riskier ones later, and even plan releases based on branch risk

I hope this approach sparks some useful thoughts =)