Page MenuHomePhabricator

Classifiers
Updated 389 Days AgoPublic

This is for Goldilocks 2.0. If you're using Goldilocks 1.0 (part of PawLIB), see the official documentation at https://docs.mousepawmedia.com/pawlib

Classifiers are ways of classifying Tests, based on the Live-In Testing Standard.

Types

Every Test must be of a particular type. The type is passed to the test as a template parameter, usually when inheriting as we create a new Test:

class TestSomeFeature : Test<Type::Behavior>

The LIT standard defines several types, which are formalized in Goldilocks with Type:

Type::Behavior (B)

A Behavior Test tests a single behavior or feature. This is the most common type of test, and is usually the best type for comparative benchmarks.

Type::Consistency (C)

A Consistency Test repeats one scenario a large number of times within known limits, and ensures the output is valid and consistent, usually either matching a target value or being in a range of values.

Type::Developer (D)

A Developer Test is usually similar to a unit test, but which does not fit into one of the other categories.

Type::Edge (E)

An Edge Test tests for edge cases, usually with specific scenarios.

Type::Fuzz (F)

A Fuzz Test creates random or non-deterministic scenarios, usually to simulate user input.

Type::Integration (I)

Integration Tests ensure that connected classes or systems are communicating with one another properly.

Type::Stress (S)

A Stress Test is designed to break more fragile code, usually by inputting large data. Should never be run automatically.

Type::Fatality (X)

A Fatality Test is intended to deliberately crash the program or system while logging, to determine limits. Should never be run automatically. Unlike a Stress Test, these tests are supposed to crash, and are usually written as infinite loops.

Should be used with caution. Goldilocks must display a warning and a complex-input prompt (e.g. requiring the user to type "I KNOW WHAT I AM DOING." before it runs. Should resist being run in a CI/CD. (Perhaps these should also have to be run via a special bombard command, instead of run.
Last Author
jcmcdonald
Last Edited
Jul 16 2021, 5:32 PM