Practical ScalaCheck

Noel Markham (47 Degrees)

Agenda

  • Brief introduction
  • The important classes in ScalaCheck
  • Boilerplate reduction
  • Writing properties
  • Tips and techniques

Introducing ScalaCheck

A brief example

scala> import org.scalacheck.Prop.forAll
import org.scalacheck.Prop.forAll

scala> val prop = forAll { s: String =>
     |   s.length >= 0
     | }
prop: org.scalacheck.Prop = Prop

scala> prop.check
+ OK, passed 100 tests.

Also works out-of-the-box with SBT

This presentation is using raw ScalaCheck, there are a few minor differences when using with ScalaTest and other frameworks

Introducing ScalaCheck

What ScalaCheck is:

  • A bridge between types and values
  • What you should be using instead of "stubbed" or sample data
  • A conversation about your implementation and tests
  • Thorough

Introducing ScalaCheck

What ScalaCheck is not:

  • A library for random generation
  • A silver bullet
  • Fast (???)

Introducing ScalaCheck

More examples

forAll { x: Int =>
  Math.abs(x) >= 0
}
[info] ! Falsified after 30 passed tests.
[info] > ARG_0: -2147483648
scala> Integer.MIN_VALUE
res0: Int = -2147483648

scala> Math.abs(Integer.MIN_VALUE)
res1: Int = -2147483648

Introducing ScalaCheck

More examples

forAll { x: Int =>
  Math.abs(x) >= 0
}
[info] ! Falsified after 30 passed tests.
[info] > ARG_0: -2147483648

Constrain generation with ==>

import org.scalacheck.Prop.BooleanOperators
					
forAll { x: Int =>
  x > Integer.MIN_VALUE ==>
  Math.abs(x) >= 0
}
[info] + Math.abs: OK, passed 100 tests.

Introducing ScalaCheck

More examples

def brokenReverse[X](xs: List[X]): List[X] =
  if (xs.length > 4) xs else xs.reverse

forAll { (xs: List[Int]) => xs.length > 0 ==>
  (xs.last == brokenReverse(xs).head)
}
[info] ! Falsified after 3 passed tests.
[info] > ARG_0: List("0", "0", "0", "0", "1")
[info] > ARG_0_ORIGINAL: List("219254809",
                              "-1487516422",
                              "-1988082558",
                              "-2147483648",
                              "-2053120082",
                              "2147483647")

This is shrinking

Introducing ScalaCheck

More examples

forAll { (list1: List[Int], list2: List[Int]) =>
  list1.length < (list1 ::: list2).length
}

[info] ! Falsified after 0 passed tests.
[info] > ARG_0: List()
[info] > ARG_1: List()

Introducing ScalaCheck

More examples

We could use ==> here again

Or we could use a different generator for non-empty lists:

import org.scalacheck.Gen

forAll(Gen.nonEmptyListOf(arbitrary[Int]),
       Gen.nonEmptyListOf(arbitrary[Int])) { (l1, l2) =>
  l1.length < (l1 ::: l2).length    
}
[info] + OK, passed 100 tests.

Important Classes

Gen

  • Stands for Generator
  • Can be used to produce any value for a particular type or a subset of values
  • It is a monad, so we can sequence/chain generators to produce new ones
  • ScalaCheck ships with many generators, and facilities to create more

Gen

Selected API Generators

import org.scalacheck.Gen._
def alphaStr: Gen[String]
def posNum[T](implicit n: Numeric[T]): Gen[T]
def oneOf[T](xs: Seq[T]): Gen[T]
def listOf[T](g: Gen[T]): Gen[List[T]]
def listOfN[T](n: Int, g: Gen[T]): Gen[List[T]]

And one in a different class: 

import org.scalacheck.Arbitrary._
def arbitrary[T](implicit a: Arbitrary[T]): Gen[T]

Gen

Sampling Data

scala> import org.scalacheck.Gen._
import org.scalacheck.Gen._

scala> posNum[Int].sample
res0: Option[Int] = Some(80)

scala> posNum[Int].sample
res1: Option[Int] = Some(17)

Gen

Gen is a monad

def cappedString: Gen[String] = for {
  c <- alphaUpperChar
  s <- listOf(alphaLowerChar)
} yield (c :: s).mkString

scala> cappedString.sample
res2: Option[String] = Some(Rmvbrcgtzvdlnssznckgedmyeeoxwiqjvtiby)

The Arbitrary Typeclass

Allows generators to be implicitly summoned 

case class Record(s: String)

val genRecord: Gen[Record] = alphaStr.map(Record.apply)

implicit val arbRecord: Arbitrary[Record] = Arbitrary(genRecord)
forAll { r: Record =>
  // test here using generated Record
}

The Arbitrary Typeclass

We can also use the arbitrary[T] method now: 

scala> arbitrary[Record].sample
res1: Option[Record] = Some(Record(vpwmFseQRubujRridyQ))

Which Gen instance should we use for the Arbitrary?

  • As a rule, the Gen which produces the full range of values

Boilerplate Reduction

Removing Arbitraries

Using scalacheck-shapeless

We can use Shapeless's automatic typeclass derivation: 

"com.github.alexarchambault"
             %% "scalacheck-shapeless_1.15" % "1.3.0"

Using scalacheck-shapeless

Full example

> test:console
[info] Starting scala interpreter...
[info]
Welcome to sbt 1.5.3 (AdoptOpenJDK Java 12.0.1)
Type in expressions to have them evaluated.
Type :help for more information.

scala> case class Coordinates(x: Int, y: Int, description: String)
defined class Coordinates

scala> import org.scalacheck.Shapeless._
import org.scalacheck.Shapeless._

scala> import org.scalacheck.Arbitrary._
import org.scalacheck.Arbitrary._

Using scalacheck-shapeless

Full example

scala> arbitrary[Coordinates].sample
res0: Option[Coordinates] = Some(Coordinates(0,537278256,쪗ܘ暎))

scala> arbitrary[Coordinates].sample
res1: Option[Coordinates] = Some(Coordinates(-1204620568,0,掑὏꺒))

Using scalacheck-shapeless

This can derive any Arbitrary[A] as long as:

  • A is a case class or sealed trait family of case classes
  • An implicit Arbitrary can be summoned for A's constituent parts through automatic derivation or otherwise

But beware: compile times can dramatically increase!

Designing Properties

Don't generate function inputs

Instead, aim to generate what you need to derive your input and output

Designing Properties

An example: Swapping elements in a list

def swap[A](list: List[A], i1: Int, i2: Int) = ???
forAll { (head: List[String], 
          e1: String, 
          middle: List[String], 
          e2: String, 
          tail: List[String]) =>

  val input = head ::: List(e1) ::: middle ::: List(e2) ::: tail
  val expected = head ::: List(e2) ::: middle ::: List(e1) ::: tail

  val output = swap(input, head.length, head.length + 1 + middle.length)

  expected =? output
}

Designing Properties

Another example: returning data from a service

def service(id: Int): Future[String]
def genPick[A, B](implicit aa: Arbitrary[A],
                           ab: Arbitrary[B]
    ): Gen[(Map[A, B], List[A])] = for {
  pairs <- arbitrary[Map[A, B]]
  keys = pairs.keySet
  validPicks <- someOf(keys)
} yield (pairs, validPicks.toList)
forAll(genPick[Int, String]) { case (mapping, ids) =>
  
  def service(id: Int): Future[String] = mapping.get(id) match {
    case Some(s) => Future.succesful(s)
    case None => Future.failed("Not in map")
  }

  // ...wire up the service here,
  // assert it works as expected with the generated ids...
}

Tips and Techniques

Exhausting Test Cases

Prefer generators over ==>

forAll { (i1: Int, i2: Int, i3: Int) =>
  (i1 > 0 && i2 > 0 && i3 > 0) ==> {
    passed
  }
}
[info] ! 3 positive integers: Gave up after only 51 passed tests.
                              501 tests were discarded.

forAll(posNum[Int], posNum[Int], posNum[Int]) { (i1, i2, i3) =>
  passed
}
[info] + 3 positive integers: OK, passed 100 tests.

Labelling Generators

Clearer data

forAll(arbitrary[Int], arbitrary[Map[Int, String]]) { (i, m) =>
  m.get(i).isDefined
}
[info] ! Falsified after 0 passed tests.
[info] > ARG_0: 1
[info] > ARG_1: Map()
forAll("Index"           |: arbitrary[Int],
       "Lookup database" |: arbitrary[Map[Int, String]]
      ) { (i, m) =>

  m.get(i).isDefined
}
[info] ! Falsified after 0 passed tests.
[info] > Index: -2147483648
[info] > Lookup database: Map()

Labelling Properties

Better than println

forAllNoShrink { (i: Int, j: Int) =>
  val (max, min) = (i max j, i min j)
  val (maxSq, minSq) = (max * max, min * min)

  minSq <= maxSq
}
[info] ! Falsified after 2 passed tests.
[info] > ARG_0: 1528767008
[info] > ARG_1: 1356090093

Labelling Properties

Better than println

forAllNoShrink { (i: Int, j: Int) =>
  val (max, min) = (i max j, i min j)
  val (maxSq, minSq) = (max * max, min * min)

  s"[min: $min, square: $minSq], [max: $max, square: $maxSq]" |:
    (minSq <= maxSq)
}
[info] ! Falsified after 0 passed tests.
[info] > Labels of failing property:
[info] [min: -2147483648, square: 0],
       [max: -2140727206, square: -1698889820]
[info] > ARG_0: -2140727206
[info] > ARG_1: -2147483648

Integration with CI

More tests on the CI server!

Use the --minSuccessfulTests parameter in SBT: 

> testOnly YahtzeeProperties
[info] + Winning hand is chosen correctly: OK, passed 100 tests.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 0 s
> testOnly YahtzeeProperties -- -minSuccessfulTests 1000000
[info] + Winning hand is chosen correctly: OK, passed 1000000 tests.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 17 s

Cogen

Before Scalacheck 1.13.0:

scala> val fa = arbitrary[Int => String].sample.get
fa: Int => String = <function1>

scala> fa(1)
res4: String = 爧뀁ณﾙ㒄똧圞ﾨꉫ薇
scala> fa(2)
res5: String = 爧뀁ณﾙ㒄똧圞ﾨꉫ薇
scala> val fb = arbitrary[Int => String].sample.get
fb: Int => String = <function1>

scala> fb(1)
res6: String = 羔ᱯ끭흓(拣༪㢬鰝ᯛ鎤ⳣ慩
scala> fb(2)
res7: String = 羔ᱯ끭흓(拣༪㢬鰝ᯛ鎤ⳣ慩

For Arbitrary[A => B], you only need Gen[B]

Cogen

But now:

scala> val f = arbitrary[Int => String].sample.get
f: Int => String = <function1>

scala> f(4)
res8: String = ᠣ澲蜕␧

scala> f(5)
res9: String = 荜௥尣흑狠榇ƺ㥀

For Arbitrary[A => B], you need a Cogen[A] and a Gen[B]

scalacheck-toolbox

A library for using sensible data with ScalaCheck

scala> val now = ZonedDateTime.now
now: java.time.ZonedDateTime = 2019-09-01T10:31:40.801+01:00[Europe/London]

scala> val generator = genDateTimeWithinRange(now, Duration.ofDays(7))
generator: org.scalacheck.Gen[java.time.ZonedDateTime] =
                                      org.scalacheck.Gen$$anon$5@34781a0f

scala> (1 to 3).foreach(_ => println(generator.sample))
Some(2019-09-06T06:48:42.275+01:00[Europe/London])
Some(2019-09-05T03:20:26.826+01:00[Europe/London])
Some(2019-09-06T06:06:35.392+01:00[Europe/London])

See more at https://47deg.github.io/scalacheck-toolbox

Thank You

Useful links