Test rules in CI

Add veriproof policy test to the pipeline that changes your rules. It replays the tests you already saved in VeriProof and fails the build when a change would decide a test differently:

the positive, negative, and boundary tests recorded with each rule’s readiness check;
the saved rule test datasets of recorded sessions, for each rule the change touches.

The run reads rules and recorded sessions only. It never deploys, activates, or changes a rule. VeriProof doesn’t call your agent’s endpoint.

Before you start

Save rule tests. Check a rule’s readiness in the portal with its positive, negative, and boundary tests, and save a rule test dataset from recorded sessions you care about.
Create a service principal for the pipeline with the Developer or Governance Engineer role, limited to the groups it tests.
Store VERIPROOF_CLIENT_ID and VERIPROOF_CLIENT_SECRET as pipeline secrets, and your portal address as a variable.
Make the approved CLI wheel available to the pipeline, for example as a pipeline artifact downloaded from Tools and SDKs.

What to test

You change rules by	Run
Editing a definition document in the repository	`veriproof policy test --application ID --definition FILE --fail-on-change`
Saving a draft in the portal	`veriproof policy test --application ID --candidate DRAFT_ID --fail-on-change`
Nothing yet; you want a nightly check	`veriproof policy test --group SLUG --fail-on-change`

The command exits with 0 when nothing changed and 5 when an outcome changed or couldn’t be evaluated. Other codes mean the run didn’t happen; see exit codes.

GitHub Actions


name: Rule regression tests
 
on:
  pull_request:
    paths:
      - "veriproof/**.json"
 
jobs:
  rule-tests:
    runs-on: ubuntu-latest
    env:
      VERIPROOF_CLIENT_ID: ${{ secrets.VERIPROOF_CLIENT_ID }}
      VERIPROOF_CLIENT_SECRET: ${{ secrets.VERIPROOF_CLIENT_SECRET }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install the VeriProof CLI
        run: python -m pip install ./tools/veriproof_cli-1.0.1-py3-none-any.whl
      - name: Configure the deployment
        run: veriproof profile add production "${{ vars.VERIPROOF_PORTAL_URL }}"
      - name: Test the refund assistant's rules
        run: >
          veriproof policy test
          --application "${{ vars.REFUND_ASSISTANT_ID }}"
          --definition veriproof/refund-assistant.json
          --junit results/veriproof-rules.xml
          --fail-on-change
      - name: Publish results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: veriproof-rule-tests
          path: results/veriproof-rules.xml

Azure DevOps


trigger: none
pr:
  paths:
    include:
      - veriproof/*
 
pool:
  vmImage: ubuntu-latest
 
variables:
  - group: veriproof-pipeline   # VERIPROOF_PORTAL_URL, REFUND_ASSISTANT_ID, and the secrets below
 
steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: "3.12"
 
  - script: python -m pip install ./tools/veriproof_cli-1.0.1-py3-none-any.whl
    displayName: Install the VeriProof CLI
 
  - script: veriproof profile add production "$(VERIPROOF_PORTAL_URL)"
    displayName: Configure the deployment
 
  - script: >
      veriproof policy test
      --application "$(REFUND_ASSISTANT_ID)"
      --definition veriproof/refund-assistant.json
      --junit $(Build.ArtifactStagingDirectory)/veriproof-rules.xml
      --fail-on-change
    displayName: Test the refund assistant's rules
    env:
      VERIPROOF_CLIENT_ID: $(VERIPROOF_CLIENT_ID)
      VERIPROOF_CLIENT_SECRET: $(VERIPROOF_CLIENT_SECRET)
 
  - task: PublishTestResults@2
    condition: always()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: $(Build.ArtifactStagingDirectory)/veriproof-rules.xml
      testRunTitle: VeriProof rule tests

Read the results

The JUnit file has a Rule tests suite with one case per saved test, and a Saved datasets suite with one case per dataset and changed rule. A failed case names the expected and actual outcome, such as expected deny, got allow, and the recorded session when there is one.

A changed outcome isn’t always a mistake. If the new decision is the one you want, update the rule’s tests when you check its readiness in the portal, and the next run expects the new outcome.

Each run is bounded in rules, test cases, and replayed sessions. When a run stops early, its notes say what it covered, and the portal shows the same notes with the latest run.