Parallel Execution
- 🚀 Speed up test runs by splitting scenarios across multiple nodes
- ⚙️ Configure parallel execution with
--slicer-total
and--slicer-index
- ☁️ Set up parallel runs in CI with GitHub Actions, GitLab CI, or Drone
Introduction
Need to run your test suite faster? Vedro makes it easy to execute tests in parallel across multiple nodes or containers.
Each node runs a different “slice” of the test suite. Together, the nodes split the work and reduce overall test time — perfect for CI pipelines.
Running in Parallel
Use two command-line options to configure parallel execution:
--slicer-total
: total number of nodes (slices)--slicer-index
: index of the current node (from0
toslicer-total - 1
)
Example: 2 nodes
Node 1
$ vedro run --slicer-total=2 --slicer-index=0
Node 2
$ vedro run --slicer-total=2 --slicer-index=1
CI/CD Integration
Most CI systems support parallel execution through job matrices or native parallel support. Here’s how to use slicing with the most common platforms.
GitHub Actions
Use a matrix strategy to run Vedro in parallel:
# .github/workflows/test.yml
name: Test
on: push
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
node-total: [2]
node-index: [0, 1]
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.12'
- name: Install Dependencies
run: pip install -r requirements.txt
- name: Run Tests
run: vedro run --slicer-total=${{ matrix.node-total }} --slicer-index=${{ matrix.node-index }}
See GitHub Actions docs for more.
GitLab CI
GitLab supports parallel jobs natively using the parallel
keyword:
# .gitlab-ci.yml
test:
image: python:3.12
parallel: 2
before_script:
- pip install -r requirements.txt
script:
- vedro run --slicer-total=$CI_NODE_TOTAL --slicer-index=$((CI_NODE_INDEX-1))
GitLab uses 1-based indexing (CI_NODE_INDEX
starts from 1), so subtract 1 to match Vedro’s 0-based indexing.
See GitLab CI docs for more.
Drone CI
Drone supports parallel pipelines using multiple steps
. Here’s how to split a run across two nodes:
# .drone.yml
kind: pipeline
type: docker
name: default
steps:
- name: fork1
image: python:3.12
commands:
- pip install -r requirements.txt
- vedro run --slicer-total=2 --slicer-index=0
- name: fork2
image: python:3.12
commands:
- pip install -r requirements.txt
- vedro run --slicer-total=2 --slicer-index=1
- name: join
image: python:3.12
depends_on: [ fork1, fork2 ]
To test locally:
$ drone exec