When a test fails, you should be able to begin investigation with nothing more than the test’s name and its failure messages —no need to add more information and rerun the test.

Effective use of unit test frameworks and assertion libraries (JUnit, Truth, pytest, GoogleTest, etc.) serves two important purposes. Firstly, the more precisely we express the invariants we are testing, the more informative and less brittle our tests will be. Secondly, when those invariants don’t hold and the tests fail, the failure info should be immediately actionable. This meshes well with Site Reliability Engineering guidance on alerting.

Consider this example of a C++ unit test of a function returning an absl::Status(an Abseil type that returns either an “OK” status or one of a number of different error codes):

EXPECT_TRUE (LoadMetadata().ok());

EXPECT_OK (LoadMetadata());

Sample failure output

load_metadata_test.cc:42: Failure

Value of: LoadMetadata().ok()

Expected: true

Actual: false

load_metadata_test.cc:42: Failure

Value of: LoadMetadata()

Expected: is OK

Actual: NOT_FOUND: /path/to/metadata.bin

If the test on the left fails, you have to investigate why the test failed; the test on the right immediately gives you all the available detail, in this case because of a more precise GoogleTest matcher.

Here are some other posts on this blog that emphasize making test failures actionable:

Writing Descriptive Test Names - If our tests are narrow and sufficiently descriptive, the test name itself may give us enough information to start debugging.

Keep Tests Focused - If we test multiple scenarios in a single test, it’s hard to identify exactly what went wrong.

Prefer Narrow Assertions in Unit Tests - If we have overly wide assertions (such as depending on every field of a complex output proto), the test may fail for many unimportant reasons. False positives are the opposite of actionable.

Keep Cause and Effect Clear - Refrain from using large global test data structures shared across multiple unit tests, allowing for clear identification of each test’s setup.

Test Failures Should Be Actionable