11. Pytest — Test parameterization

Time:2019-11-27

Catalog

  • 1. @pytest.mark.parametrizesign
    • 1.1. empty_parameter_set_markoption
    • 1.2. Multiple tag combinations
    • 1.3. Marking test module
  • 2. pytest_generate_testsHook Method

Previous index: https://www.cnblogs.com/luizyao/p/11771740.html

In practical work, test cases may need to support multiple scenarios. We can abstract the parts strongly related to the scenarios into parameters, and drive the execution of the use cases by assigning parameters;

Parameterized behavior is expressed at different levels:

  • fixtureParameterization of: reference 4, fixtures: explicit, modular and extensible–fixtureParameterization of;

  • Parameterization of test cases: Using@pytest.mark.parametrizeMultiple parameters orfixtureCombination;

In addition, we can alsopytest_generate_testsThis hook method defines the parameterization scheme;

1. @pytest.mark.parametrizesign

@pytest.mark.parametrizeThe fundamental role ofcollectIn the process of test case, through theSpecified parametersTo add the value ofCall (execute)

First, let’s take a look at its definition in the source code:

# _pytest/python.py

def parametrize(self, argnames, argvalues, indirect=False, ids=None, scope=None):

Focus on the following parameters:

  • argnames: a comma separated string, or a list / tuple, indicating the specified parameter name;

    aboutargnamesIn fact, we have some limitations:

    • Can only be a subset of the tagged object’s input parameters:

      @pytest.mark.parametrize('input, expected', [(1, 2)])
      def test_sample(input):
          assert input + 1 == 1

      test_sampleNo statement inexpectedParameter, if we force declaration in the tag, we get the following error:

      In test_sample: function uses no argument 'expected'
    • The marked object cannot be included in the parameter. Parameters with default values are defined:

      @pytest.mark.parametrize('input, expected', [(1, 2)])
      def test_sample(input, expected=2):
          assert input + 1 == expected

      althoughtest_sampleStatementexpectedParameter, but it is also given a default value. If we force declaration in the tag, we will get the following error:

      In test_sample: function already takes an argument 'expected' with a default value
    • Will overwrite the one with the same namefixture

      @pytest.fixture()
      def expected():
          return 1
      
      
      @pytest.mark.parametrize('input, expected', [(1, 2)])
      def test_sample(input, expected):
          assert input + 1 == expected

      test_sampleIn the tagexpected(2)Overwrite with the same namefixture expected(1), so this use case can be tested successfully;

      Here you can refer to: 4. Fixtures: explicit, modular and extensible — overriding in use case parametersfixture

  • argvaluesOne:Iteratable objectShow thatargnamesThe assignment of parameters is as follows:

    • IfargnamesContains multiple parameters, thenargvaluesThe iteration return element of must be measurable (that is, supportedlen()Method), and the length andargnamesThe number of declared parameters is equal, so it can be a tuple / list / set, etc., indicating the actual parameters of all input parameters:

      @pytest.mark.parametrize('input, expected', [(1, 2), [2, 3], set([3, 4])])
      def test_sample(input, expected):
          assert input + 1 == expected

      Note: considering the de duplication of sets, we do not recommend using it;

    • IfargnamesContains only one parameter, thenargvaluesThe iteration return elements of can be specific values:

      @pytest.mark.parametrize('input', [1, 2, 3])
      def test_sample(input):
          assert input + 1
    • If you also notice what we mentioned earlier,argvaluesIs an iterative object, so we can implement more complex scenes; for example: fromexcelRead arguments in file:

      def read_excel():
          #Read the device information from the database or excel file, which is simplified as a list
          for dev in ['dev1', 'dev2', 'dev3']:
              yield dev
      
      
      @pytest.mark.parametrize('dev', read_excel())
      def test_sample(dev):
          assert dev

      There are many ways to implement this scenario. You can also directlyfixtureIntermediate loadingexcelBut their performance in the test report will be different;

    • As you may remember, in the last tutorial (10, skip, and xfail tags — combinedpytest.paramMethod), we usepytest.parambyargvaluesParameter assignment:

      @pytest.mark.parametrize(
          ('n', 'expected'),
          [(2, 1),
          pytest.param(2, 1, marks=pytest.mark.xfail(), id='XPASS')])
      def test_params(n, expected):
          assert 2 / n == expected

      Now let’s analyze this behavior in detail:

      whetherargvaluesWhether measurable objects (lists, tuples, etc.) or specific values are passed in, we will encapsulate them into one in the source codeParameterSetObject, which is aNamed tupleIncludevalues, marks, idThree elements:

      >>> from _pytest.mark.structures import ParameterSet as PS
      >>> PS._make([(1, 2), [], None])
      ParameterSet(values=(1, 2), marks=[], id=None)

      If you pass aParameterSetWhat happens to objects? Let’s go to the source code to find out:

      # _pytest/mark/structures.py
      
      class ParameterSet(namedtuple("ParameterSet", "values, marks, id")):
      
          ...
      
          @classmethod
          def extract_from(cls, parameterset, force_tuple=False):
              """
              :param parameterset:
                  a legacy style parameterset that may or may not be a tuple,
                  and may or may not be wrapped into a mess of mark objects
      
              :param force_tuple:
                  enforce tuple wrapping so single argument tuple values
                  don't get decomposed and break tests
              """
      
              if isinstance(parameterset, cls):
                  return parameterset
              if force_tuple:
                  return cls.param(parameterset)
              else:
                  return cls(parameterset, marks=[], id=None)

      You can see that if you pass aParameterSetObject, then it returns itself(return parameterset), so the two expressions in the following example are equivalent:

      # src/chapter-11/test_sample.py
      
      import pytest
      
      from _pytest.mark.structures import ParameterSet
      
      
      @pytest.mark.parametrize(
          'input, expected',
          [(1, 2), ParameterSet(values=(1, 2), marks=[], id=None)])
      def test_sample(input, expected):
          assert input + 1 == expected

      Here, you may have guessed,pytest.paramTo encapsulate aParameterSetObject; then let’s go to the source code to verify it!

      # _pytest/mark/__init__.py
      
      def param(*values, **kw):
          """Specify a parameter in `pytest.mark.parametrize`_ calls or
          :ref:`parametrized fixtures `.
      
          .. code-block:: python
      
              @pytest.mark.parametrize("test_input,expected", [
                  ("3+5", 8),
                  pytest.param("6*9", 42, marks=pytest.mark.xfail),
              ])
              def test_eval(test_input, expected):
                  assert eval(test_input) == expected
      
          :param values: variable args of the values of the parameter set, in order.
          :keyword marks: a single mark or a list of marks to be applied to this parameter set.
          :keyword str id: the id to attribute to this parameter set.
          """
          return ParameterSet.param(*values, **kw)

      As we expected, now you should know better how to giveargvaluesPass it on;

  • indirectargnamesA subset of or a Boolean value of; passes the arguments of the specified parameterrequest.paramRedirect to the same name as the parameterfixtureIn order to meet the more complex scene;

    Please refer to the following examples for specific usage:

    # src/chapter-11/test_indirect.py
    
    import pytest
    
    
    @pytest.fixture()
    def max(request):
        return request.param - 1
    
    
    @pytest.fixture()
    def min(request):
        return request.param + 1
    
    
    #Default indirect is false
    @pytest.mark.parametrize('min, max', [(1, 2), (3, 4)])
    def test_indirect(min, max):
        assert min <= max
    
    
    #The parameter corresponding to min max is redirected to the fixture with the same name
    @pytest.mark.parametrize('min, max', [(1, 2), (3, 4)], indirect=True)
    def test_indirect_indirect(min, max):
        assert min >= max
    
    
    #Redirect only the arguments corresponding to Max to fixture
    @pytest.mark.parametrize('min, max', [(1, 2), (3, 4)], indirect=['max'])
    def test_indirect_part_indirect(min, max):
        assert min == max
  • ids: an executable object for generating testsID, or a list / tuple indicating the test of all new casesID

    • If list / tuple is used to directly indicate the testID, then its length should be equal toargvaluesLength:

      @pytest.mark.parametrize('input, expected', [(1, 2), (3, 4)],
                        ids=['first', 'second'])
      def test_ids_with_ids(input, expected):
          pass

      Collected testsIDAs follows:

      collected 2 items
    • If the same test is specifiedIDpytestThe index is automatically added later:

      @pytest.mark.parametrize('input, expected', [(1, 2), (3, 4)],
                        ids=['num', 'num'])
      def test_ids_with_ids(input, expected):
          pass

      Collected testsIDAs follows:

      collected 2 items
    • If in the specified testIDNon use inASCIIThe default value is byte sequence:

      @pytest.mark.parametrize('input, expected', [(1, 2), (3, 4)],
                        IDS = ['num ','Chinese'])
      def test_ids_with_ids(input, expected):
          pass

      Collected testsIDAs follows:

      collected 2 items

      We can see what we expect to showChinese, which actually shows\u4e2d\u6587

      What if we want to get the desired display? Go to the source code to find out:

      # _pytest/python.py
      
      def _ascii_escaped_by_config(val, config):
          if config is None:
              escape_option = False
          else:
              escape_option = config.getini(
                  "disable_test_id_escaping_and_forfeit_all_rights_to_community_support"
              )
          return val if escape_option else ascii_escaped(val)

      We can do this bypytest.iniIntermediate enablingdisable_test_id_escaping_and_forfeit_all_rights_to_community_supportOption to avoid this:

      [pytest]
      disable_test_id_escaping_and_forfeit_all_rights_to_community_support = True

      Tests collected againIDAs follows:

    • If the test is generated through an executableID

      def idfn(val):
          #Add 1 to each val
          return val + 1
      
      
      @pytest.mark.parametrize('input, expected', [(1, 2), (3, 4)], ids=idfn)
      def test_ids_with_ids(input, expected):
          pass

      Collected testsIDAs follows:

      collected 2 items

      As we can see from the above example, for a specificargvaluesparameter(1, 2)In other words, it’s broken down into1and2Respectively passed toidfn, and pass the return value through the-Symbols are linked together as a testIDReturn, not(1, 2)Introduced as a whole;

      Let’s see how it is implemented in the source code:

      # _pytest/python.py
      
      def _idvalset(idx, parameterset, argnames, idfn, ids, item, config):
          if parameterset.id is not None:
              return parameterset.id
          if ids is None or (idx >= len(ids) or ids[idx] is None):
              this_id = [
                  _idval(val, argname, idx, idfn, item=item, config=config)
                  for val, argname in zip(parameterset.values, argnames)
              ]
              return "-".join(this_id)
          else:
              return _ascii_escaped_by_config(ids[idx], config)

      As we guess, first passzip(parameterset.values, argnames)takeargnamesandargvaluesThe processed return value is then passed"-".join(this_id)Connect;

      In addition, if we are careful enough, we can see from the above source code that the assumption has passedpytest.paramDesignatedidProperty, it will be overwrittenidsTest corresponding toIDLet’s confirm:

      @pytest.mark.parametrize(
          'input, expected',
          [(1, 2), pytest.param(3, 4, id='id_via_pytest_param')],
          ids=['first', 'second'])
      def test_ids_with_ids(input, expected):
          pass

      Collected testsIDAs follows:

      collected 2 items

      testIDyesid_via_pytest_paramRather thansecond

    So muchidsWhat’s the use of?

    I think its main function is to further refine test cases, distinguish different test scenarios, and provide a new method for targeted test execution;

    For example, for the following test cases, you can use the-k 'Window and not Non'Options, execute only andWindowsRelated scenarios:

    # src/chapter-11/test_ids.py
    
    import pytest
    
    
    @pytest.mark.parametrize('input, expected', [
        pytest.param(1, 2, id='Windows'),
        pytest.param(3, 4, id='Windows'),
        pytest.param(5, 6, id='Non-Windows')
    ])
    def test_ids_with_ids(input, expected):
        pass
  • scopeStatementargnamesThe scope of the parameter in theargvaluesThe test cases are divided, which affects the collection order of test cases;

    • If we explicitly indicate the scope parameter; for example, declare the parameter scope as a module level:

      # src/chapter-11/test_scope.py
      
      import pytest
      
      
      @pytest.mark.parametrize('test_input, expected', [(1, 2), (3, 4)], scope='module')
      def test_scope1(test_input, expected):
          pass
      
      
      @pytest.mark.parametrize('test_input, expected', [(1, 2), (3, 4)], scope='module')
      def test_scope2(test_input, expected):
          pass

      The collected test cases are as follows:

      collected 4 items

      The following is the default collection order, and we can see the obvious difference:

      collected 4 items
    • scopeIf not specified (orscope=NoneWhenindirectBe equal toTrueOr include allargnamesParameter, the scope is allfixtureThe minimum scope of the scope; otherwise, it will always befunction

      # src/chapter-11/test_scope.py
      
      @pytest.fixture(scope='module')
      def test_input(request):
          pass
      
      
      @pytest.fixture(scope='module')
      def expected(request):
          pass
      
      
      @pytest.mark.parametrize('test_input, expected', [(1, 2), (3, 4)],
                              indirect=True)
      def test_scope1(test_input, expected):
          pass
      
      
      @pytest.mark.parametrize('test_input, expected', [(1, 2), (3, 4)],
                              indirect=True)
      def test_scope2(test_input, expected):
          pass

      test_inputandexpectedThe scope ofmodule, so the scope of the parameter is alsomodule, the collection order of use cases is the same as the previous section:

      collected 4 items

1.1. empty_parameter_set_markoption

By default, if@pytest.mark.parametrizeOfargnamesIf the parameter in does not receive any arguments, the result of the use case will be set toSKIPPED

For example, whenpythonVersion less than3.8Returns an empty list (currentPythonVersion is3.7.3):

# src/chapter-11/test_empty.py

import pytest
import sys


def read_value():
    if sys.version_info >= (3, 8):
        return [1, 2, 3]
    else:
        return []


@pytest.mark.parametrize('test_input', read_value())
def test_empty(test_input):
    assert test_input

We can do this bypytest.iniSet inempty_parameter_set_markOption to change this behavior. The possible values are:

  • skipDefault value
  • xfail: skip execution and mark the use case asXFAILEquivalent toxfail(run=False)
  • fail_at_collect: escalate oneCollectErrorAbnormal;

1.2. Multiple tag combinations

If more than one use case is marked@pytest.mark.parametrizeMark as follows:

# src/chapter-11/test_multi.py

@pytest.mark.parametrize('test_input', [1, 2, 3])
@pytest.mark.parametrize('test_output, expected', [(1, 2), (3, 4)])
def test_multi(test_input, test_output, expected):
    pass

The actual collected use cases are all their possible combinations:

collected 6 items

1.3. Marking test module

We can pass thepytestmarkAssign values, parameterize a test module:

# src/chapter-11/test_module.py

import pytest

pytestmark = pytest.mark.parametrize('test_input, expected', [(1, 2), (3, 4)])


def test_module(test_input, expected):
    assert test_input + 1 == expected

2. pytest_generate_testsHook Method

pytest_generate_testsMethod is called during test case collection and receives ametafuncObject, through which we can access the context of the test request, and more importantly, we can use themetafunc.parametrizeMethod to customize the parameterized behavior;

Let’s first look at how this method is used in the source code:

# _pytest/python.py

def pytest_generate_tests(metafunc):
    # those alternative spellings are common - raise a specific error to alert
    # the user
    alt_spellings = ["parameterize", "parametrise", "parameterise"]
    for mark_name in alt_spellings:
        if metafunc.definition.get_closest_marker(mark_name):
            msg = "{0} has '{1}' mark, spelling should be 'parametrize'"
            fail(msg.format(metafunc.function.__name__, mark_name), pytrace=False)
    for marker in metafunc.definition.iter_markers(name="parametrize"):
        metafunc.parametrize(*marker.args, **marker.kwargs)

First of all, it checkedparametrizeIf you accidentally write it["parameterize", "parametrise", "parameterise"]One of them,pytestAn exception is returned and the correct word is prompted; then, loop through allparametrizeAnd call themetafunc.parametrizeMethod;

Now, let’s define a parameterization scheme of our own:

In the following use case, we examine the givenstringinputWhether it’s only composed of letters, but we didn’t type itparametrizeTag, sostringinputConsidered to be afixture

# src/chapter-11/test_strings.py

def test_valid_string(stringinput):
    assert stringinput.isalpha()

Now, we are looking forward tostringinputAs a normal parameter, and assign from the command line:

First, we define a command line option:

# src/chapter-11/conftest.py

def pytest_addoption(parser):
    parser.addoption(
        "--stringinput",
        action="append",
        default=[],
        help="list of stringinputs to pass to test functions",
    )

Then, we passedpytest_generate_testsMethod willstringinputBehavior byfixtrueChange toparametrize

# src/chapter-11/conftest.py

def pytest_generate_tests(metafunc):
    if "stringinput" in metafunc.fixturenames:
        metafunc.parametrize("stringinput", metafunc.config.getoption("stringinput"))

Finally, we can pass--stringinputCommand line options forstringinputParameters are assigned:

λ pipenv run pytest -q --stringinput='hello' --stringinput='world' src/chapter-11/test_strings.py
..                                                                     [100%] 
2 passed in 0.02s

If we don’t--stringinputOption, equivalent toparametrizeOfargnamesIf the parameter in does not receive any arguments, the result of the test case will be set toSKIPPED

λ pipenv run pytest -q src/chapter-11/test_strings.py
s                                                                  [100%] 
1 skipped in 0.02s

Be careful:

Whether or notmetafunc.parametrizeMethod or@pytest.mark.parametrizeTags, their parameters(argnames)It cannot be repeated, otherwise an error will occur:ValueError: duplicate 'stringinput'