Measuring Coverage
Code coverage is a crucial metric in software testing, providing insights into which parts of the codebase are exercised by tests and identifying untested sections. Measuring coverage ensures that critical logic is thoroughly tested, reducing the risk of undetected bugs. In the Python ecosystem, the de facto standard for measuring test coverage is coverage.py
.
Setting Up Coverage Measurementβ
1. Install coverage.py
β
First, install coverage.py
using pip
:
$ pip install coverage
This tool will track executed code during test runs and generate coverage reports.
2. Run Tests with Coverage Trackingβ
To measure coverage while running Vedro tests, execute the following command:
$ coverage run -m vedro run scenarios/
Hereβs what happens:
coverage run
starts tracking code execution.-m vedro run scenarios/
instructs Vedro to execute all scenarios in thescenarios/
directory.
3. Generate and View Coverage Reportsβ
Once tests complete, generate a text-based coverage report:
$ coverage report
For a more detailed and interactive HTML report:
$ coverage html
Then, open htmlcov/index.html
in a web browser to explore coverage results visually.
Uploading Coverage Data to Codecovβ
Codecov automates coverage reporting, integrates with CI/CD pipelines, and provides analytics on test coverage trends.
GitHub Actions Integrationβ
To automate test execution and coverage reporting in GitHub Actions, create .github/workflows/coverage.yml
:
name: Run Tests and Upload Coverage
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install Dependencies
run: |
pip install -r requirements.txt
pip install coverage
- name: Run Tests with Coverage
run: |
coverage run -m vedro run
coverage xml -o coverage.xml
coverage report
- name: Upload Coverage Report
uses: codecov/codecov-action@v4
with:
files: ./coverage.xml
token: ${{ secrets.CODECOV_TOKEN }}
For additional details, refer to the official GitHub Actions documentation.
Best Practices for Coverage Analysisβ
1. Aim for Meaningful Coverageβ
Strive for high coverage but avoid focusing solely on percentage. Instead, ensure critical paths, edge cases, and business logic are well tested.
2. Exclude Non-Essential Codeβ
Exclude test utilities, third-party libraries, and boilerplate code from coverage reports. For more details on configuration options, refer to the official documentation.
3. Enforce Coverage Thresholdsβ
To prevent regressions, enforce minimum coverage percentages in CI pipelines:
$ coverage report --fail-under=80
This ensures that test coverage remains above 80% before merging changes.
For Codecov users, configuration options can be customized using a codecov.yml
file. Refer to the Codecov documentation for details.