Quickcheck

6.0.0.3

Quickcheck

Mike Sperber
and Ismael Figueroa

This manual provides documentation for a Racket implementation of Quickcheck, a library that tests the specification of a program with randomly generated test inputs.

The original Quickcheck is a library for Haskell.

(require quickcheck)

package: quickcheck

1 Quickstart

1.1 Installation

If you have not installed the package yet, run the following command from the command-line:

raco pkg install quickcheck

Note that this package will require Racket v6.0 or newer to install.

Alternatively, you may use the GUI package manager from DrRacket to install the package.

1.2 Running tests

To use Quickcheck, you first write a specification for a program as a property in ordinary Racket code. For example, the following is a property that expresses the English specification “the string->number function produces numbers when given strings”:

> (define string->number-returns-number
(property ([str arbitrary-string])
(number? (string->number str))))

Given such a property, we can run random tests using quickcheck:

> (quickcheck string->number-returns-number)
Falsifiable, after 0 tests:
str = "\u0002\u0003\u0002"

You may have already guessed that this property will be easily falsified, since most strings do not actually parse as numbers.

Next, let’s write a property that we expect will actually succeed. For example, the following encodes the English specification “given two lists of integers, the result of appending them will have the same length as the sum of the original list lengths”:

> (define append-and-length-agree
    (property ([lst-1 (arbitrary-list arbitrary-integer)]
               [lst-2 (arbitrary-list arbitrary-integer)])
      (= (+ (length lst-1) (length lst-2))
         (length (append lst-1 lst-2)))))

Testing the property reveals that it holds up:

> (quickcheck append-and-length-agree)
OK, passed 100 tests.

2 Running checks

procedure
(quickcheck prop) → void?
prop : testable?

Test the given prop, generating new test cases if prop specifies a generator.

Prints the result of the check, including a counterexample for the property if one was found.

Example:

> (quickcheck (property ((str arbitrary-string))
(string=? str (list->string (string->list str)))))
OK, passed 100 tests.

procedure
(quickcheck-results prop)
→
integer? stamps (or/c boolean? result?)
prop : testable?

Like quickcheck, except it will return three values instead:

an integer representing the number of tests run,
a list of ...,
either #t if the tests were successful, #f if the arguments were exhausted, or an instance of the result structure type on a test failure.

Examples:

> (define-values (ntests stamps ok?)
(quickcheck-results (property ((str arbitrary-string))
(string=? str (list->string (string->list str))))))
> ntests
100
> (first stamps)
'()
> ok?
#t

procedure
(check config prop) → void?
config : config?
prop : testable?

Runs a check like quickcheck using the given config as a configuration.

Prints the result of the check, including a counterexample for the property if one was found.

procedure
(check-results config prop) → result?
config : config?
prop : testable?

Like check, except it will return three result values instead of printing the results. See the quickcheck function for an explanation of the return values.

3 Configuration, results, and utilities

struct
(struct config (max-test max-fail size print-every)
    #:extra-constructor-name make-config)
  max-test : number?
  max-fail : number?
  size : (-> integer? integer?)
  print-every : (-> integer? (listof any/c) any)

A structure type that represents configurations for the test process. An instance of this type must be supplied when calling the check or check-results functions.

The max-test field represents the maximum number of succeeding tests that will be run. The max-fail field represents the maximum number of tests that can be run with no result before the checker terminates.

The size field should be a function of one argument that produces the test size given the current test number. The print-every field should be a function that takes the test number and the generated arguments and is called for its side effect.

struct
(struct result (ok stamp argument-list)
    #:extra-constructor-name make-result)
  ok : (or/c null #t #f)
  stamp : (listof string?)
  argument-list : (listof any/c)

Represents a single result from a test. The ok field is #t on success, #f on failure, and null if there is no test result.

The stamp field represents the labels that were relevant to this test execution. The argument-list is a list of the values generated for checking this test case.

syntax
(property ([id gen/arb-expr] ...) body0 body ...)

gen/arb-expr : (or/c arbitrary? generator?)

Constructs a testable property for functions like quickcheck.

The ids are bound to the result of the given gen/arb-exprs inside the body expressions. The body expressions are used as the bodies of a predicate function that will be run with newly generated values from the specified generators or arbitraries.

Example:

> (property ((str arbitrary-string))
(string=? str (list->string (string->list str))))
#<property>

procedure
(property? val) → boolean?
val : any/c

Returns #t if val is a property for testing, otherwise returns #f.

procedure
(testable? val) → boolean?
val : any/c

Returns #t if val is a value that can be tested with functions like quickcheck. Returns #f otherwise.

Values that can be tested are the following:

boolean values (#t and #f),
instances of the result structure type,
instances of the property structure type,
or instances of the generator structure type.

4 Generators

struct
(struct generator (proc)
#:extra-constructor-name make-generator)
proc : (-> integer? random-generator? any/c)

Represents a source of values for randomly testing a given property with functions like quickcheck.

The proc value should be a function that accepts an integer representing the size of the value, a random number generator, and returns a value for testing.

procedure
(choose-integer lower upper) → generator?
lower : integer?
upper : integer?

Produces a generator that returns integers between lower and upper.

procedure
(choose-real lower upper) → generator?
lower : real?
upper : real?

Produces a generator that returns real numbers between lower and upper.

value
choose-ascii-char : generator?

A generator that returns ASCII characters.

value
choose-ascii-letter : generator?

A generator that returns ASCII letter characters.

value
choose-printable-ascii-char : generator?

A generator that returns ASCII characters that have a printed representation.

procedure
(choose-char lower upper) → generator?
lower : char?
upper : char?

Produces a generator that returns characters with integer values between lower and higher.

procedure
(choose-list elem-gen size) → generator?
elem-gen : generator?
size : integer?

Produces a generator that returns lists of the given size and with elements generated with elem-gen.

procedure
(choose-vector elem-gen size) → generator?
elem-gen : generator?
size : integer?

Produces a generator that returns vectors of the given size and with elements generated with elem-gen.

procedure
(choose-string char-gen size) → generator?
char-gen : generator?
size : integer?

Produces a generator that returns strings of the given size and with elements generated with elem-gen, which must generate characters.

procedure
(choose-symbol char-gen size) → generator?
char-gen : generator?
size : integer?

Produces a generator that returns symbols of the given size and with elements generated with elem-gen, which must generate characters.

procedure
(choose-one-of opts) → generator?
opts : (listof any/c)

Produces a generator that returns one of the values in the list opts.

procedure
(choose-mixed promises) → generator?
promises : (listof (promise/c generator?))

Produces a generator that returns returns values from one of the generator promises, forcing the promise when it does so.

procedure
(choose-with-frequencies freqs) → generator?
freqs : (listof (cons/c integer? generator?))

Produces a generator that returns a value from one of the given generators, weighing each generator by the matching integer in freqs.

procedure
(generator-unit val) → generator?
val : any/c

Produces a generator that always returns the given val.

procedure
(generator-bind gen k) → generator?
gen : generator?
k : (-> any/c generator?)

Produce a generator from the result of calling k on the value generated from the generator gen.

procedure
(generator-sequence gens) → generator?
gens : (listof generator?)

Given the list of generators gens, produce a generator that produces values from them in sequence.

procedure
(sized f) → generator?
f : (-> integer? generator?)

Produces a generator from a function f that constructs a generator given an integer representing a value’s size.

struct
(struct arbitrary (gen trans)
    #:extra-constructor-name make-arbitrary)
  gen : generator?
  trans : (-> any/c generator? generator?)

Represents a source of randomly generated values, except where the values are filtered by the function trans to produce a narrower set of new values.

value
arbitrary-boolean : arbitrary?

Generates a boolean value.

value
arbitrary-char : arbitrary?

Generates a character.

value
arbitrary-ascii-char : arbitrary?

Generates an ASCII character.

value
arbitrary-printable-ascii-char : arbitrary?

Generates a printable ASCII character.

value
arbitrary-integer : arbitrary?

Generates an integer.

value
arbitrary-natural : arbitrary?

Generates a non-negative integer.

value
arbitrary-rational : arbitrary?

Generates a rational number.

value
arbitrary-real : arbitrary?

Generates a real number.

procedure
(arbitrary-mixed pred+promises) → arbitrary?
pred+promises : (listof (cons/c (-> any/c any/c) (promise/c arbitrary?)))

Produces a arbitrary generator given a list matching up predicates to promises that produce arbitrary generators.

procedure
(arbitrary-one-of eql? vals) → arbitrary?
eql? : (any/c any/c -> any/c)
vals : (listof any/c)

Produces an arbitrary generator that generates values from the list of values vals. The values are filtered by the equality function eql?.

procedure
(arbitrary-pair fst rst) → arbitrary?
fst : arbitrary?
rst : arbitrary?

Produces an arbitrary generator that generates pairs of values drawn from the fst and rst generators respectively.

procedure
(arbitrary-list elem) → arbitrary?
elem : arbitrary?

Produces an arbitrary generator that generates lists in which the element values are drawn from elem.

procedure
(arbitrary-vector elem) → arbitrary?
elem : arbitrary?

Produces an arbitrary generator that generates vectors in which the element values are drawn from elem.

procedure
(arbitrary-tuple elem ...) → arbitrary?
elem : arbitrary?

Produces an arbitrary generator that generates constant-length lists in which the element values are drawn from the generators elems in order.

procedure
(arbitrary-record constructor
accessors
elem ...) → arbitrary?
  constructor : procedure?
  accessors : (listof procedure?)
  elem : arbitrary?

Produces an arbitrary generator that generates values of some structure type given a constructor procedure and accessor procedures for that structure type. The values passed to the constructor are drawn from the generators elems.

value
arbitrary-string : arbitrary?

Generates a string.

value
arbitrary-ascii-string : arbitrary?

Generates an ASCII string.

value
arbitrary-printable-ascii-string : arbitrary?

Generates an ASCII string consisting of characters with printable representations.

value
arbitrary-symbol : arbitrary?

Generates a symbol.

procedure
(arbitrary-procedure result arg ...) → arbitrary?
result : arbitrary?
arg : arbitrary?

Generates a procedure that takes arguments drawn from the generators args and which produces a value drawn from the generator result.

5 Operations on properties

syntax
(==> bool-expr prop)

Represents implication for testable properties.

If bool-expr is #t, equivalent to prop. Otherwise, produces a property that returns no result.

procedure
(label str test) → generator?
str : string?
test : testable?

Labels the given test with a str label to help identify what portion of the property is failing.

syntax
(classify really? label-expr testable-expr)

Labels the result of testable-expr with the value of label-expr if really? evaluates to #t. Otherwise just returns the result of testable-expr.

syntax
(trivial really? testable-expr)

Like classify, but always uses the label "trivial".

procedure
(collect lbl test) → generator?
lbl : any/c
test : testable?

Labels the given test with the written form of the value lbl. Similar to the label function.

6 Random number generation

procedure
(make-random-generator s1 s2) → random-generator?
s1 : number?
s2 : number?

Constructs a random number generator, given two seeds s1 and s2.

procedure
(random-generator? val) → boolean?
val : any/c

Returns #t if val is a random number generator constructed by make-random-generator and returns #f in all other cases.

procedure
(random-generator-next rand) →
integer? random-generator?
rand : random-generator?

Produce a random integer and a random number generator.

procedure
(random-generator-split rand)
→
random-generator? random-generator?
rand : random-generator?

Splits the given random number generator and returns two random number generators.

procedure
(random-integer rg low high) → integer?
  rg : random-generator?
  low : integer?
  high : integer?

Returns a random integer between the bounds low and high.

procedure
(random-real rg low high) → real?
  rg : random-generator?
  low : real?
  high : real?

Returns a random real number between the bounds low and high.

1	Quickstart
2	Running checks
3	Configuration, results, and utilities
4	Generators
5	Operations on properties
6	Random number generation