Engineering

Doing Property-Based Testing on CRUD, the Thoughtless Way

You don't need to think real hard about your tests to do property-based testing. You can also just learn a few patterns and just apply them.

Howon Lee | September 04, 2019

"Civilization advances by extending the number of important operations which we can perform without thinking about them."

- A. N. Whitehead

Property-based testing is a remarkable modality of testing. It involves using a fuzzer, a random data generator, to not only enter random inputs into a system to make sure it doesn't crash or segfault, but to show that arbitrary properties about the system hold. Because you only define the types of data to create and certain abstract properties, about that data, and a library does the actual generation of examples for you, you get an incredibly high ROI to programmer time spent writing tests. It's not just that the tests are cheaper. Often folks find bugs with property-based testing that they couldn't with other modalities of testing.

However, there is a catch. When you start property-based testing, most people become paralyzed by the blank screen effect. They are frozen by having too many possibilities for test generation. They are also paralyzed because of the sophistication they believe they need to generate tests.

The promise of any new testing method for business use must be that they get more bugs at a reduced cost. Property-based testing is extraordinarily promising for this because it allows us to dramatically reduce an entire category of necessary programmer thought, that of thinking of devious examples in testing. Human thought is the most costly part of the software enterprise.

There are legions of software shops without precise requirements, where the general point of view ends up being shuttling data to and fro. There is many a software firm where the only need is for the general business programmer who doesn’t really get value from knowing what an involutive functor is. There, most of the problem is figuring out what people want, what the unclear requirements are, and figuring out the interoperability of data. There, any new testing method for business use must also come with an advisory on how to get it in on time and under budget. The way to do that is by thinking less.

An aside: unfortunately there are, to a first approximation, only about six really good property-based testing libraries: test.check, FsCheck, Quviq QuickCheck, Hypothesis, Hedgehog, ScalaCheck, and maybe a few others. Of those, the only one written for a language that the ordinary programmer uses is Hypothesis, in Python. So this is really a Python article, although you could just use microservices as dynamic linking over HTTP and use a property-based testing library in a different language.

Most corporate programming is CRUD - create, read, update, deletion - of some data. Most corporate programming inside of CRUD ends up being piping data through different systems, maybe with some serialization and so on. Whether at a huge public tech company or small business, most programming in a corporation ends up being CRUD. There are sophisticated event-based systems which often end up replicating CRUD first off, writing creation, update, read, and tombstone events. We're not talking about those.

import uuid

class Contact:
    data_store = {}
    pkey_id = 0

    def __init__(self, name, email, phone, address, contact_id=None):
        Contact.pkey_id += 1
        if contact_id is not None:
            self.contact_id = contact_id
        else:
            self.contact_id = Contact.pkey_id
        self.set_name(name)
        self.email = email
        self.phone = phone
        self.address = address

    def to_str(self):
        return f"{self.name},{self.email},{self.phone},{self.address}"

    @staticmethod
    def find(c_id):
        return Contact.from_str(Contact.data_store[c_id])

    @staticmethod
    def from_str(str_repr):
        name, email, phone, address = tuple(str_repr.split(","))
        return Contact(name=name, email=email, phone=phone, address=address)

    def set_name(self, name):
        self.name = name
        full_name = self.name.split()
        self.first_name = full_name[0]
        self.last_name = full_name[1]

    def save(self):
        Contact.data_store[self.contact_id] = self.to_str()
        return self

    def delete(self):
        Contact.data_store[self.contact_id] = None
        return self

    def __eq__(self, other):
        if self.name != other.name:
            return False
        if self.email != other.email:
            return False
        if self.phone != other.phone:
            return False
        if self.address != other.address:
            return False
        return True

def add_contact(data, contact_id=None):
    new_contact = Contact(
                        name=data["name"],
                        email=data["email"],
                        phone=data["phone"],
                        address=data["address"])
    new_contact.save()
    return new_contact.contact_id

def read_contact(contact_id):
    return Contact.find(contact_id)

def update_contact(contact_id, data):
    curr_contact = Contact.find(contact_id)
    curr_contact.name = data["name"]
    curr_contact.email = data["email"]
    curr_contact.phone = data["phone"]
    curr_contact.address = data["address"]
    curr_contact.save()
    return curr_contact

def delete_contact(contact_id):
    curr_contact = Contact.find(contact_id)
    return curr_contact.delete()

Three Big Mainstays

I think about three big mainstays for doing property-based tests in general, not necessarily in a CRUD situation, without thinking too much. Consider this function, which strips whitespace from email, and gets rid of the + postfix that Gmail and Outlook let you add from email

1. Just call the function

import hypothesis
import hypothesis.strategies
@hypothesis.given(hypothesis.strategies.emails()):
def test_simple_call(email):
    canon_email(email)

This usually catches only syntax errors, does input validation and makes sure that paths reasonably reached by arbitrary input don't explode messily. In this case this just finds that the escape sequence \s is invalid.

2. Just calling the function twice (idempotence)

@hypothesis.given(hypothesis.strategies.emails())
def test_idempotence(email):
    assert canon_email(email) == canon_email(canon_email(email))

This makes sure that a function which doesn't change state or is idempotent (meaning that result of two calls f(f(x)) are the same as one call f(x)), is valid with respect to that property.

This finds:

AssertionError: assert '@a.com' == None where '@a.com' = canon_email('+@a.com')
and None = canon_email('@a.com')  where '@a.com' = canon_email('+@a.com')

Which is just saying that this will not be happy if there’s no actual email username portion, just the + addendum syntax that Outlook and Gmail use. Adding that case after the regex substitution, it now passes.

parts[0] = re.sub(r'[\.|_]', '', parts[0].split('+')[0])
if parts[0] == '':
        return None

Would you have thought of that immediately?

3. Inverse (when you can write an inverse)

This makes sure that two functions which are supposed to be inverses of each other are actually inverses of each other.

Usually in property-testing documentation people put this up as great for serialization-deserialization pairs.

If you're willing to write a fair bit of relatively thoughtless production code in order to help out testing, I find this to be a great workhorse. This is because you can make very many functions invertible, if you put enough outputs in and are willing to put more parameters in the inverse functions. Like so, for a text whitespace stripping function:

import os
import os.path
import string
import hypothesis as hy
import hypothesis.strategies as hy_st

def s3_path_to_parts(s3_path):
    # removes s3://
    path = s3_path.lstrip("s3://")
    parts = path.split('/')
    bucket = parts[0]
    key = '/'.join(parts[1:])
    fname = parts[-1]
    return bucket, key, fname


def parts_to_s3_path(bucket, key, fname):
    return 's3://{}'.format(os.path.join(bucket, key))


def valid_aws_chars():
    return string.ascii_letters + string.digits + "!-_.*'()"


@hy_st.composite
def s3_path(draw, elements=hy_st.text(valid_aws_chars(), min_size=3)):
    texts = draw(hy_st.lists(elements, min_size=2))
    return "s3://" + "/".join(texts)


@hy.given(s3_path())
def test_inverse(s3_path):
    bucket, key, fname = s3_path_to_parts(s3_path)
    assert parts_to_s3_path(bucket, key, fname) == s3_path

Sometimes this is too much trouble, but sometimes it isn’t too much trouble. What this finds is a pretty boneheaded but difficult to spot bug:

AssertionError: assert 's3://aa/aaa' == 's3://3aa/aaa'

Meaning, lstrip will take any set of characters, it won’t do a matching pattern. So you need to replace with an actual regex expression.

So what do these workhorses mean for our CRUD functionality? It means that we can just bang out 7 decent property-based tests without thinking about it.

(1) Create should be invertible. Inverted by deletion.

(2) Update should be invertible. Inverted by an update the other way.

(3) Read should be idempotent.

(4-7) All 4 CRUD operations should have sane behavior with inputs.

import hypothesis as hy
import hypothesis.strategies as hy_st
import contact
import pytest
import re
import os
import string
from typing import Optional, Tuple


@hy_st.composite
def contact_data(draw):
    data = {}
    data["name"] = draw(hy_st.text())
    data["email"] = draw(hy_st.emails())
    data["phone"] = draw(hy_st.text())
    data["address"] = draw(hy_st.text())
    return data


@hy.given(contact_data())
def test_create_simple_call(data):
    contact.add_contact(data)


@hy.given(contact_data())
def test_create_simple_call(data):
    c_id = contact.add_contact(data)
    contact.read_contact(c_id)


@hy.given(contact_data(), contact_data())
def test_update_simple_call(data1, data2):
    c_id = contact.add_contact(data1)
    contact.update_contact(c_id, data2)


@hy.given(contact_data())
def test_delete_simple_call(data):
    c_id = contact.add_contact(data)
    c_id = contact.delete_contact(c_id)


@hy.given(contact_data())
def test_inverse_create_delete(data):
    c_id = contact.add_contact(data)
    contact0 = contact.read_contact(c_id)
    contact1 = contact.delete_contact(c_id)
    assert contact0 == contact1


@hy.given(contact_data(), contact_data())
def test_inverse_update(data1, data2):
    hy.assume(data1 != data2)
    c_id = contact.add_contact(data1)
    contact0 = contact.read_contact(c_id)
    contact1 = contact.update_contact(c_id, data2)
    contact2 = contact.update_contact(c_id, data1)
    assert contact2 == contact0 
    assert contact1 != contact0


@hy.given(contact_data())
def test_idempotence_read(data):
    c_id = contact.add_contact(data)
    res1 = contact.read_contact(c_id)
    res2 = contact.read_contact(c_id)
    assert res1 == res2

In the process of running these tests, you find:

  1. You can’t assume that name.split() returns 2 components.
  2. You have to escape the ad hoc serialization format.

These are worthwhile additions to the spec, not just good tests.

So once you lay down a lot of pretty thoughtless but incredibly cost-effective tests, it may or may not be worth thinking about those tests some more. But you can always have at least these tests, without thinking too much, if you're doing CRUD.

I do not think that property-based tests are some pinnacle of the software writer's craft. If you want a special class of programmer with a special kind of point of view and a special attitude towards testing to write property-based tests, you can advocate for thinking real hard about the tests and picking out tests well-adapted to the problem, and the problem posed in a mathematically sophisticated way. But you don’t need it to just get stuff down.

Thanks to F. Hebert and my coworkers for reading this post.



About the Author:

Find out how DataGrail can work for your business

We obviously take privacy very seriously, your email address is only required so we can email you details about your demo.

Get The Weekly Grail in your inbox every week!

Thanks! Check your inbox to verify your email.