Compare commits

...

11 Commits

Author SHA1 Message Date
Volodymyr Zotov
316f605680 Update docs 2025-10-02 13:18:05 -05:00
Volodymyr Zotov
7135f439a5 Update check-external-pr check when e2e test success 2025-10-02 13:14:04 -05:00
Volodymyr Zotov
625297c1fb Add doc explaining how to test forked PR 2025-10-02 12:41:01 -05:00
Volodymyr Zotov
e967f7ce1e Revert job name 2025-10-02 12:40:35 -05:00
Volodymyr Zotov
5199e9f350 Revert "Update status check to 'success' only if e2e tests are passed"
This reverts commit 878b38deac.
2025-10-02 12:37:04 -05:00
Volodymyr Zotov
878b38deac Update status check to 'success' only if e2e tests are passed 2025-10-02 12:36:27 -05:00
Volodymyr Zotov
1a40b4fb95 Pass jobm name as env 2025-10-02 12:35:06 -05:00
Volodymyr Zotov
4b5d3c5f1a Use propper job name e2e-tests 2025-10-02 12:34:10 -05:00
Volodymyr Zotov
8d4908fb72 Updated comment to mention sha= portion to dispatch the event 2025-10-02 11:50:15 -05:00
Volodymyr Zotov
6cf52a7bfc Use process.env.job instead of hardcoding job name 2025-10-02 11:29:31 -05:00
Volodymyr Zotov
7ef9ec6de7 Update e2e job names to avoid confustion and fix updating check status 2025-10-02 08:35:37 -05:00
4 changed files with 113 additions and 3 deletions

View File

@@ -1,4 +1,4 @@
name: E2E Tests
name: E2E Tests [reusable]
on:
workflow_call:

View File

@@ -1,4 +1,4 @@
# Write comments "/ok-to-test <hash>" on a pull request. This will emit a repository_dispatch event.
# Write comments "/ok-to-test sha=<hash>" on a pull request. This will emit a repository_dispatch event.
name: Ok To Test
on:

View File

@@ -36,6 +36,7 @@ jobs:
env:
ref: ${{ github.event.client_payload.pull_request.head.sha }}
conclusion: ${{ needs.e2e-tests.result }}
job: check-external-pr
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
@@ -44,7 +45,7 @@ jobs:
ref: process.env.ref
});
const check = checks.check_runs.filter(c => c.name === 'e2e-test');
const check = checks.check_runs.filter(c => c.name === process.env.job);
const { data: result } = await github.rest.checks.update({
...context.repo,

109
docs/fork-pr-testing.md Normal file
View File

@@ -0,0 +1,109 @@
# Fork PR Testing Guide
This document explains how to test external pull requests using the dispatch action workflow system.
## Overview
The testing system consists of three main workflows:
1. **E2E Tests [triggered by maintainer]** (`test-e2e.yml`) - Runs automatically for internal PRs, requires approval for external PRs
2. **Ok To Test** (`ok-to-test.yml`) - Processes slash commands to trigger fork testing
3. **E2E tests [fork]** (`test-e2e-fork.yml`) - Triggered manually via slash commands for external PRs
## How It Works
### 1. Initial PR State
When a PR is created or updated:
- The `test-e2e.yml` workflow automatically runs
- For **external PRs**: The `check-external-pr` job detects it's from a fork and **fails intentionally**
- For **internal PRs**: The workflow proceeds normally with e2e tests
- External PRs show ❌ for the check-external-pr job with a message asking for maintainer approval
### 2. Manual Approval Required
**Important**: External PRs require manual approval before workflows can run.
**Steps for maintainers:**
1. Go to the PR page
2. Click **"Approve workflow to run"** button
3. The workflow will then execute
**Note**: The `test-e2e.yml` workflow will still fail for external PRs even after approval, as it's designed to prevent automatic execution. Need to use the slash command instead.
### 3. Testing External PRs
Once the initial checks have run (and failed), maintainers can test the PR using slash commands:
#### Step-by-Step Process:
1. **Navigate to the PR**
- Go to the pull request you want to test
2. **Add a comment with slash command**
```
/ok-to-test sha=<commit-sha>
```
Replace `<commit-sha>` with the actual commit SHA from the PR.
**Note**: Use the short SHA.
3. **Dispatch Action Triggers**
- The `ok-to-test.yml` workflow processes the slash command
- It triggers a `repository_dispatch` event with type `ok-to-test-command`
- The `test-e2e-fork.yml` workflow starts
4. **Workflow Execution**
The fork workflow runs two jobs:
**a) `e2e-tests` job** (conditional)
- Only runs if:
- Event is `repository_dispatch`
- SHA parameter is not empty
- PR head SHA contains the provided SHA
- Calls the reusable `e2e-tests.yml` workflow
- Runs the actual E2E tests if conditions are met
**b) `update-check-status` job** (conditional)
- Runs after `e2e-tests` completes
- Updates the existing check for job named "e2e-tests"
- Sets the conclusion based on `e2e-tests` result:
- ✅ **Success** if tests pass
- ❌ **Failure** if tests fail
## Troubleshooting
### Workflow Not Running
- **Check**: Ensure you've approved the workflow to run
- **Action**: Click "Approve workflow to run" in the PR checks tab
### SHA Not Found
- **Check**: Verify the SHA exists in the PR commits
- **Action**: Use `git log --oneline` to find valid commit SHAs
### Dispatch Not Triggering
- **Check**: Ensure the slash command format is correct
- **Action**: Use exact format: `/ok-to-test sha=<sha>`
### Check Runs Not Updating
- **Note**: The fork workflow updates the existing "E2E Tests [reusable]" check run
- **Behavior**: The same check run is updated with new results rather than creating a new one
## Security Notes
- Only users with **write** permissions can trigger dispatch commands
- The fork workflow runs in the main repository context, allowing it to update check runs
- Manual approval is required for external PR workflows
- External PRs are automatically detected and prevented from running tests automatically
## Workflow Files
- **`.github/workflows/ok-to-test.yml`** - Ok To Test - Slash command processor
- **`.github/workflows/test-e2e.yml`** - E2E Tests [triggered by maintainer] - Initial PR checks
- **`.github/workflows/test-e2e-fork.yml`** - E2E tests [fork] - Dispatch action handler
- **`.github/workflows/e2e-tests.yml`** - E2E Tests [reusable] - Reusable workflow for actual testing
## Testing Checklist
- [ ] PR created with failing check-external-pr check (for external PRs)
- [ ] Workflow approved to run (if required)
- [ ] Slash command posted with valid SHA
- [ ] Dispatch action triggered successfully
- [ ] `check-external-pr` check run updated with correct conclusion