How to implement parameterized testing in Python?

Time:2020-4-24

Before that, I turned to a series of articles on unit testing framework, which introduced unittest, nose / nose2 and pytest, the three most popular Python testing frameworks.

In this paper, we want to continue to talk about the topic of testing for a very common test scenario, namely parametric testing, and try to connect these test frameworks in series, make a horizontal comparison and deepen understanding.

1. What is parametric testing?

For ordinary tests, a test method only needs to run once, while for a test method, parameterized tests may need to pass in a series of parameters and then conduct multiple tests.

For example, if we want to test the login function of a system, we may need to pass in different user names and passwords respectively for testing: using the user name containing illegal characters, using the unregistered user name, using the super long user name, using the wrong password, using reasonable data, etc.

A parametric test is a data-driven test that tests different parameters on the same method to cover the results of all possible expected branches. Its test data can be separated from the test behavior, put into files, databases or external media, and then read by the test program.

2. How to realize parametric testing?

Generally speaking, a test method is the smallest test unit, and its function should be atomized and unitary as much as possible.

First, let’s look at two ways to realize parameterized test: one is to write a test method and traverse all test parameters in it; the other is to write the logic of traversal parameters outside the test method, and then call the test method in turn.

These two ideas can achieve the purpose of testing. In a simple business, there is no problem. However, in fact, they all have only one test unit, which is not optimistic when counting the number of test cases or generating test reports. Scalability is also a problem.

So, how does the existing testing framework solve this problem?

They all use decorators. The main ideas are:Use the original test method (such as test ()), to generate multiple new test methods (such as test1 (), test2 ()…) , and assign parameters to them in turn.

Because test frameworks usually count a test unit as a “test”, this idea of “more than one life” has great advantages over the previous two ideas when they count test results.

3. How to use parametric testing?

In Python standard libraryunittestIt does not support parameterized testing. To solve this problem, two libraries have been developed: one isddt, one isparameterized

DDT is the acronym for “Data-Driven tests”. Typical usage:

import unittest
from ddt import ddt,data,unpack

@ddt
class MyTest(unittest.TestCase):
    @data((3, 1), (-1, 0), (1.2, 1.0))
    @unpack
    def test_values(self, first, second):
        self.assertTrue(first > second)

unittest.main(verbosity=2)

The results of the operation are as follows:

test_values_1__3__1_ (__main__.MyTest) ... ok
test_values_2___1__0_ (__main__.MyTest) ... FAIL
test_values_3__1_2__1_0_ (__main__.MyTest) ... ok

==================================================
FAIL: test_values_2___1__0_ (__main__.MyTest)
--------------------------------------------------
Traceback (most recent call last):
  File "C:\Python36\lib\site-packages\ddt.py", line 145, in wrapper
    return func(self, *args, **kwargs)
  File "C:/Users/pythoncat/PycharmProjects/study/testparam.py", line 9, in test_values
    self.assertTrue(first > second)
AssertionError: False is not true

----------------------------------------------
Ran 3 tests in 0.001s

FAILED (failures=1)

The results show that there are three tests, and show the running state and the information of assertion failure in detail.

It should be noted that each of the three tests has a name, which also carries the information of its parameters. However, the original test UU values method is missing and has been split into three.

In the above example, the DDT library uses three decorators (@ DDT, @ data, @ unpack), which is really ugly. Let’s look at a relatively better parameterized Library:

import unittest
from parameterized import parameterized

class MyTest(unittest.TestCase):
    @parameterized.expand([(3,1), (-1,0), (1.5,1.0)])
    def test_values(self, first, second):
        self.assertTrue(first > second)

unittest.main(verbosity=2) 

The test results are as follows:

test_values_0 (__main__.MyTest) ... ok
test_values_1 (__main__.MyTest) ... FAIL
test_values_2 (__main__.MyTest) ... ok

=========================================
FAIL: test_values_1 (__main__.MyTest)
-----------------------------------------
Traceback (most recent call last):
  File "C:\Python36\lib\site-packages\parameterized\parameterized.py", line 518, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "C:/Users/pythoncat/PycharmProjects/study/testparam.py", line 7, in test_values
    self.assertTrue(first > second)
AssertionError: False is not true

----------------------------------------
Ran 3 tests in 0.000s

FAILED (failures=1)

This library only uses a decorator, @ parameterized.expand, which is much cleaner in writing.

Also remind me that the original test method has disappeared, instead of three new test methods, but the naming rules of the new method are different from the example of DDT.

After introducing unittest, let’s see what’s deadnoseAnd the newnose2。 The nose framework is a unit test with plugins. The above usage is the same.

In addition, nose2 also provides its own parameterized implementation:

import unittest
from nose2.tools import params

@params(1, 2, 3)
def test_nums(num):
    assert num < 4

class Test(unittest.TestCase):
    @params((1, 2), (2, 3), (4, 5))
    def test_less_than(self, a, b):
    assert a < b

Finally, let’s look at the pytest framework, which implements parameterized testing as follows:

import pytest

@pytest.mark.parametrize("first,second", [(3,1), (-1,0), (1.5,1.0)])
def test_values(first, second):
    assert(first > second)

The test results are as follows:

==================== test session starts ====================
platform win32 -- Python 3.6.1, pytest-5.3.1, py-1.8.0, pluggy-0.13.1
rootdir: C:\Users\pythoncat\PycharmProjects\study collected 3 items

testparam.py .F
testparam.py:3 (test_values[-1-0])
first = -1, second = 0

    @pytest.mark.parametrize("first,second", [(3,1), (-1,0), (1.5,1.0)])
    def test_values(first, second):
>       assert(first > second)
E       assert -1 > 0

testparam.py:6: AssertionError
.                                                         [100%]

========================= FAILURES ==========================
_________________________ test_values[-1-0] _________________________

first = -1, second = 0

    @pytest.mark.parametrize("first,second", [(3,1), (-1,0), (1.5,1.0)])
    def test_values(first, second):
>       assert(first > second)
E       assert -1 > 0

testparam.py:6: AssertionError
===================== 1 failed, 2 passed in 0.08s =====================
Process finished with exit code 0

We still need to remind everyone that pytest has changed from one to three, but we can’t see the information of new named methods. Does this mean that it doesn’t produce new test methods? Or just hiding the information about the new method?

4. Final summary

The concept and implementation of parameterized testing and its application in three mainstream Python testing frameworks are introduced above. I only used the simplest example, in order to quickly popularize science.

But the topic is not over yet. For the several parameterized libraries we mentioned, regardless of the differences in writing methods, what are the differences between them at the specific code level?

Specifically, how do they turn a method into multiple methods and bind each method with corresponding parameters? In the process of implementation, what difficult problems need to be solved?

When analyzing some source code, I found this topic quite interesting, so I am going to write another article. So that’s all. Thank you for reading.

How to implement parameterized testing in Python?

The official account.Python cat】, series of high-quality articles, including meow star philosophy cat series, python advanced series, good book recommendation series, technical writing, high-quality English recommendation and translation, etc., welcome to pay attention.

Recommended Today

The way of nonlinear optimization

Mathematical knowledge 1、 Nonlinear functionLinear function is another name of a function of first degree, then nonlinear function means that the function image is not a function of a straight line.Nonlinear functions include exponential function, power function, logarithmic function, polynomial function and so on. 2、 Taylor expansion1. Taylor formula:Taylor’s formula is to add a_ The […]