Skip to content

Label-based test selection for allure-pytest #725

@delatrie

Description

@delatrie

Currently, allure-pytest has a bunch of filters to run tests selectively:

  • --allure-epics
  • --allure-features
  • --allure-severities
  • --allure-stories
  • --allure-ids

We need to replace them with one general filtering mechanism based on labels.
Once this is done, the old filtering parameters could be deprecated in favor of the new one.

Motivation

Label-based test selection is more flexible in general. Since epics, features, stories, severities and ids are all labels, those filtering scenarios will be automatically supported.
In addition, other scenarios, like #633 will be supported automatically.
Additionally, custom labels (like ones, created from system env vars, see #721) would also be usable for test selection.
The syntax will become a little bit bulkier but it's always possible to wrap the invocation with a particular set of filters into a command runner like poetry or hatch.

Other information

New parameter may look like this:
--allure-label LABEL=VALUE
or
--allure-label-filter LABEL=VALUE (longer, but more expressive)

Conjunction of filters

It should be possible to specify the parameter multiple times to replace the following usage

pytest --alluredir=allure-results --allure-epics=my-epic --allure-severities=blocker

with the new syntax:

pytest --alluredir=allure-results --allure-label epic=my-epic --allure-label severity=blocker

This usage only selects tests matching both filters at the same time.

Disjunction of filters

It should be also possible to provide an alternative to calls like this:

pytest --alluredir=allure-results --allure-severities=blocker,critical

where tests with either severity=blocker or severity=critical labels are selected.

Since all the current filters use the comma symbol to separate allowed values, we can maintain this behavior. Additionally, we should provide for possibility of a value itself containing a comma by implementing an escaping mechanism (currently this is not supported; but labels like feature, or story may potentially contain commas).

Example 1: selecting tests with severity either blocker or critical:
    pytest --alluredir=allure-results --allure-label severity=blocker,critical
Example 2: selecting tests for the features Red, green or Plus\minus:
    pytest --alluredir=allure-results --allure-label 'feature=Red\, green,Plus\\minus'

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions