Practical ScalaCheck

Noel Markham (47 Degrees)

Agenda

  • Brief introduction
  • The important classes in ScalaCheck
  • Writing properties
  • Tips and techniques
  • Boilerplate reduction

Introducing ScalaCheck

A brief example

scala> import org.scalacheck.Prop.forAll
import org.scalacheck.Prop.forAll
scala> val prop = forAll { s: String =>
     |   s.length >= 0
     | }
prop: org.scalacheck.Prop = Prop
scala> prop.check
+ OK, passed 100 tests.

Also works out-of-the-box with SBT

Introducing ScalaCheck

What ScalaCheck is:

  • A bridge between types and values
  • What you should be using instead of "stubbed" or sample data
  • A conversation about your implementation and tests
  • Thorough

Introducing ScalaCheck

What ScalaCheck is not:

  • A library for random generation
  • A silver bullet
  • Fast (???)

Introducing ScalaCheck

More examples

forAll { x: Int =>
  Math.abs(x) >= 0
}
[info] ! Falsified after 30 passed tests.
[info] > ARG_0: -2147483648
scala> Integer.MIN_VALUE
res0: Int = -2147483648

scala> Math.abs(Integer.MIN_VALUE)
res1: Int = -2147483648

Introducing ScalaCheck

More examples

forAll { x: Int =>
  Math.abs(x) >= 0
}
[info] ! Falsified after 30 passed tests.
[info] > ARG_0: -2147483648

Constrain generation with ==>

import org.scalacheck.Prop.BooleanOperators
					
forAll { x: Int =>
  x > Integer.MIN_VALUE ==>
  Math.abs(x) >= 0
}
[info] + Math.abs: OK, passed 100 tests.

Introducing ScalaCheck

More examples

def brokenReverse[X](xs: List[X]): List[X] =
  if (xs.length > 4) xs else xs.reverse

forAll { (xs: List[Int]) => xs.length > 0 ==>
  (xs.last == brokenReverse(xs).head)
}
[info] ! Falsified after 3 passed tests.
[info] > ARG_0: List("0", "0", "0", "0", "1")
[info] > ARG_0_ORIGINAL: List("219254809",
                              "-1487516422",
                              "-1988082558",
                              "-2147483648",
                              "-2053120082",
                              "2147483647")

This is shrinking

Introducing ScalaCheck

More examples

forAll { (list1: List[Int], list2: List[Int]) =>
  list1.length < (list1 ::: list2).length
}
[info] ! Falsified after 0 passed tests.
[info] > ARG_0: List()
[info] > ARG_1: List()

Introducing ScalaCheck

More examples

We could use ==> here again

Or we could use a different generator for non-empty lists:

import org.scalacheck.Gen

forAll(Gen.nonEmptyListOf(arbitrary[Int]),
       Gen.nonEmptyListOf(arbitrary[Int])) { (l1, l2) =>
  l1.length < (l1 ::: l2).length    
}
[info] + OK, passed 100 tests.

Important Classes

Gen

  • Stands for Generator
  • Can be used to produce any value for a particular type or a subset of values
  • It is a monad, so we can sequence/chain generators to produce new ones
  • ScalaCheck ships with many generators, and facilities to create more

Gen

Selected API Generators

import org.scalacheck.Gen._
def alphaStr: Gen[String]
def posNum[T](implicit n: Numeric[T]): Gen[T]
def oneOf[T](xs: Seq[T]): Gen[T]
def listOf[T](g: Gen[T]): Gen[List[T]]
def listOfN[T](n: Int, g: Gen[T]): Gen[List[T]]

And one in a different class:

import org.scalacheck.Arbitrary._
def arbitrary[T](implicit a: Arbitrary[T]): Gen[T]

Gen

Sampling Data

scala> import org.scalacheck.Gen._
import org.scalacheck.Gen._

scala> posNum[Int].sample
res0: Option[Int] = Some(80)

scala> posNum[Int].sample
res1: Option[Int] = Some(17)

Gen

Gen is a monad

def cappedString: Gen[String] = for {
  c <- alphaUpperChar
  s <- listOf(alphaLowerChar)
} yield (c :: s).mkString
scala> cappedString.sample
res2: Option[String] = Some(Rmvbrcgtzvdlnssznckgedmyeeoxwiqjvtiby)

The Arbitrary Typeclass

Allows generators to be implicitly summoned

case class Record(s: String)

val genRecord: Gen[Record] = alphaStr.map(Record.apply)

implicit val arbRecord: Arbitrary[Record] = Arbitrary(genRecord)
forAll { r: Record =>
  // test here using generated Record
}

The Arbitrary Typeclass

We can also use the arbitrary[T] method now:

scala> arbitrary[Record].sample
res1: Option[Record] = Some(Record(vpwmFseQRubujRridyQ))

Which Gen instance should we use for the Arbitrary?

  • As a rule, the Gen which produces the full range of values

Designing Properties

Scenario: Yahtzee game

Have a better "hand" of five dice than your opponent

Yahtzee
Straight
Full house
Four of a kind
Three of a kind
def winner(h1: Hand, h2: Hand): Hand = ???

Can you test this without Scalacheck?

Testing Winning Hands

  • Generate two random hands
  • Work out the winning hand
  • Check it wins

Testing Winning Hands

forAll { (h1: Hand, h2: Hand) =>
  val h1Score = {
       if (h1.p1 == h1.p2 &&
           h1.p2 == h1.p3 &&
           h1.p3 == h1.p4 &&
           h1.p4 == h1.p5) Yahtzee
       else if (/* and so on */)				
  }

  val h2Score = // as above

  val winningHand = if(h1Score >= h2Score) h1 else h2

  winner(h1, h2) ?= winningHand
}

Problem: we have reimplemented our application code in Scalacheck

Testing Winning Hands

  • Generate some random dice values
  • Construct known hands
  • Confirm that the correct hand wins
forAll { (y: Die, fhA: Die, fhB: Die) => fhA != fhB ==>
  val yahtzee = Hand(y, y, y, y, y)
  val fullHouse = Hand(fhA, fhA, fhA, fhB, fhB)

  (winningHand(yahtzee, fullHouse) ?= yahtzee) &&
  (winningHand(fullHouse, yahtzee) ?= yahtzee)
}

We still have untested conditions, such as:

  • Different permutations of full house dice positions
  • Comparing all winning hands with each other is quadratic!

Testing Winning Hands

  • Generate a pair of known hands
  • Make sure the one that should win, wins

Step one: create a generator for each winning hand.

val allDice: List[Die] = List(One, Two, Three, Four, Five, Six)

val genYahtzee: Gen[Hand] = oneOf(allDice)
                              .map(d => Hand(d, d, d, d, d))
val genThreeOfAKind: Gen[Hand] = for {
  d1 <- oneOf(allDice)
  d2 <- oneOf(allDice diff List(d1))
  d3 <- oneOf(allDice diff List(d1, d2))
} yield Hand(d1, d1, d1, d2, d3)

Testing Winning Hands

Step two: order the generators to reflect superior hands.

val orderedGenerators: List[Gen[Hand]] = List(
  genYahtzee,
  genStraight,
  genFullHouse,
  genFourOfAKind,
  genThreeOfAKind
)

Testing Winning Hands

Step three: Use ScalaCheck to partition the generators.

forAll(chooseNum[Int](1, orderedGens.length - 1)) { idx =>

  val (winningGens, losingGens) = orderedGens.splitAt(idx)

  // ...

Full disclosure
Due to API restrictions, this is not actually how it is implemented, but the intention is exactly the same. See Github (or talk to me) for details.

Testing Winning Hands

Step four: Pit a hand from one of the better generators against a worse one. The better one should win.

forAll(chooseNum[Int](1, orderedGens.length - 1)) { idx =>

  val (winningGens, losingGens) = orderedGens.splitAt(idx)

  forAll(oneOf(winningGens), oneOf(losingGens))
                          { (winningHand, losingHand) =>

    (winner(winningHand, losingHand) ?= winningHand) &&
    (winner(losingHand, winningHand) ?= winningHand)
  }
}
[info] + Winning hand is chosen correctly: OK, passed 100 tests.

Testing Winning Hands

Bonus step: Verify what exactly was run.

Use the collect function:

forAll(chooseNum[Int](1, orderedGens.length - 1)) { idx =>

  val (winningGens, losingGens) = orderedGens.splitAt(idx)

  forAll(oneOf(winningGens), oneOf(losingGens))
                          { (winningHand, losingHand) =>

    collect(s"${winningHand.score} vs ${losingHand.score}") {
      (winner(winningHand, losingHand) ?= winningHand) &&
      (winner(losingHand, winningHand) ?= winningHand)
    }
  }
}

Testing Winning Hands

Verifying what exactly was tested

[info] > Collected test data:
[info] 20% Yahtzee vs FourOfAKind
[info] 19% Yahtzee vs ThreeOfAKind
[info] 16% Yahtzee vs FullHouse
[info] 13% Yahtzee vs Straight
[info] 10% FullHouse vs ThreeOfAKind
[info] 9% Straight vs ThreeOfAKind
[info] 6% FourOfAKind vs ThreeOfAKind
[info] 6% Straight vs FourOfAKind
[info] 1% Straight vs FullHouse

Tips and Techniques

Exhausting Test Cases

Prefer generators over ==>

forAll { (i1: Int, i2: Int, i3: Int) =>
  (i1 > 0 && i2 > 0 && i3 > 0) ==> {
    passed
  }
}
[info] ! 3 positive integers: Gave up after only 51 passed tests.
                              501 tests were discarded.
forAll(posNum[Int], posNum[Int], posNum[Int]) { (i1, i2, i3) =>
  passed
}
[info] + 3 positive integers: OK, passed 100 tests.

Labelling Generators

Clearer data

forAll(arbitrary[Int], arbitrary[Map[Int, String]]) { (i, m) =>
  m.get(i).isDefined
}
[info] ! Falsified after 0 passed tests.
[info] > ARG_0: 1
[info] > ARG_1: Map()
forAll("Index"           |: arbitrary[Int],
       "Lookup database" |: arbitrary[Map[Int, String]]
      ) { (i, m) =>

  m.get(i).isDefined
}
[info] ! Falsified after 0 passed tests.
[info] > Index: -2147483648
[info] > Lookup database: Map()

Labelling Properties

Better than println

forAllNoShrink { (i: Int, j: Int) =>
  val (max, min) = (i max j, i min j)
  val (maxSq, minSq) = (max * max, min * min)

  minSq <= maxSq
}
[info] ! Falsified after 2 passed tests.
[info] > ARG_0: 1528767008
[info] > ARG_1: 1356090093

Labelling Properties

Better than println

forAllNoShrink { (i: Int, j: Int) =>
  val (max, min) = (i max j, i min j)
  val (maxSq, minSq) = (max * max, min * min)

  s"[min: $min, square: $minSq], [max: $max, square: $maxSq]" |:
    (minSq <= maxSq)
}
[info] ! Falsified after 0 passed tests.
[info] > Labels of failing property:
[info] [min: -2147483648, square: 0],
       [max: -2140727206, square: -1698889820]
[info] > ARG_0: -2140727206
[info] > ARG_1: -2147483648

Creating Successful and
Failed Data

def genPick[A, B](implicit aa: Arbitrary[A],
                           ab: Arbitrary[B]
    ): Gen[(Map[A, B], List[A], List[A])] = for {

  pairs <- arbitrary[Map[A, B]]
  keys = pairs.keySet
  validPicks <- someOf(keys)
  anotherList <- listOf(arbitrary[A])
  invalidPicks = anotherList.filterNot(i => keys.contains(i))
} yield (pairs, validPicks.toList, invalidPicks)
forAll(genPick[Int, String]) { case (mapping, succs, fails) =>
  succs.forall(s => mapping.get(s).isDefined) &&
  fails.forall(f => mapping.get(f).isEmpty)
}
[info] + Using genPick: OK, passed 100 tests.

Integration with CI

More tests on the CI server!

Use the --minSuccessfulTests parameter in SBT:

> testOnly YahtzeeProperties
[info] + Winning hand is chosen correctly: OK, passed 100 tests.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 0 s
> testOnly YahtzeeProperties -- -minSuccessfulTests 1000000
[info] + Winning hand is chosen correctly: OK, passed 1000000 tests.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 17 s

Cogen

Before Scalacheck 1.13.0:

scala> val fa = arbitrary[Int => String].sample.get
fa: Int => String = <function1>

scala> fa(1)
res4: String = 爧뀁ณル㒄똧圞ᄄꉫ薇
scala> fa(2)
res5: String = 爧뀁ณル㒄똧圞ᄄꉫ薇
scala> val fb = arbitrary[Int => String].sample.get
fb: Int => String = <function1>

scala> fb(1)
res6: String = 羔ᱯ끭흓(拣༪㢬鰝ᯛ鎤ⳣ慩
scala> fb(2)
res7: String = 羔ᱯ끭흓(拣༪㢬鰝ᯛ鎤ⳣ慩

For Arbitrary[A => B], you only need Gen[B]

Cogen

But now:

scala> val f = arbitrary[Int => String].sample.get
f: Int => String = <function1>

scala> f(4)
res8: String = ᠣ澲蜕␧

scala> f(5)
res9: String = 荜௥尣흑狠榇ƺ㥀

For Arbitrary[A => B], you need a Cogen[A] and a Gen[B]

Boilerplate Reduction

Removing Arbitraries

Using scalacheck-shapeless

We can use Shapeless's automatic typeclass derivation:

"com.github.alexarchambault"
             %% "scalacheck-shapeless_1.13" % "1.1.0-RC1"

Using scalacheck-shapeless

Full example

> test:console
[info] Starting scala interpreter...
[info]
Welcome to Scala version 2.11.7 (Java HotSpot(TM) 64-Bit Server VM).
Type in expressions to have them evaluated.
Type :help for more information.

scala> case class Coordinates(x: Int, y: Int, description: String)
defined class Coordinates

scala> import org.scalacheck.Shapeless._
import org.scalacheck.Shapeless._

scala> import org.scalacheck.Arbitrary._
import org.scalacheck.Arbitrary._

Using scalacheck-shapeless

Full example

scala> arbitrary[Coordinates].sample
res0: Option[Coordinates] = Some(Coordinates(0,537278256,쪗ܘ暎))

scala> arbitrary[Coordinates].sample
res1: Option[Coordinates] = Some(Coordinates(-1204620568,0,掑὏꺒))

Using scalacheck-shapeless

This can derive any Arbitrary[A] as long as:

  • A is a case class or sealed trait family of case classes
  • An implicit Arbitrary can be summoned for A's constituent parts through automatic derivation or otherwise

But beware: compile times can dramatically increase!

See Dave Gurnell's excellent talk yesterday on Establishing Orbit with Shapeless for more on how it works.

scalacheck-datetime

A library for using sensible dates with ScalaCheck

scala> val now = ZonedDateTime.now
now: java.time.ZonedDateTime = 2016-09-01T10:31:40.801+01:00[Europe/London]

scala> val generator = genDateTimeWithinRange(now, Duration.ofDays(7))
generator: org.scalacheck.Gen[java.time.ZonedDateTime] =
                                      org.scalacheck.Gen$$anon$5@34781a0f

scala> (1 to 3).foreach(_ => println(generator.sample))
Some(2016-09-06T06:48:42.275+01:00[Europe/London])
Some(2016-09-05T03:20:26.826+01:00[Europe/London])
Some(2016-09-06T06:06:35.392+01:00[Europe/London])

See more at https://47deg.github.io/scalacheck-datetime

Thank You

Useful links