Update fork-pr-testing doc

This commit is contained in:
Volodymyr Zotov
2025-10-28 11:14:56 -05:00
parent 56288bc0af
commit 2a931cd782

View File

@@ -1,109 +1,32 @@
# Fork PR Testing Guide
This document explains how to test external pull requests using the dispatch action workflow system.
This document explains how testing works for external pull requests from forks.
## Overview
The testing system consists of three main workflows:
The testing system consists of two main workflows:
1. **E2E Tests** (`test-e2e.yml`) - Runs automatically for internal PRs, requires approval for external PRs
2. **Ok To Test** (`ok-to-test.yml`) - Processes slash commands to trigger fork testing
3. **E2E tests [fork]** (`test-e2e-fork.yml`) - Triggered manually via slash commands for external PRs
1. **E2E Tests** (`test-e2e.yml`) - Runs automatically for internal PRs, need manual trigger on external PRs.
2. **Ok To Test** (`ok-to-test.yml`) - Dispatches `repository_dispatch` event when maintainer put's `/ok-to-test sha=<commit hash>` comment in the forked PR thread.
## How It Works
### 1. Initial PR State
When a PR is created or updated:
- The `test-e2e.yml` workflow automatically runs
- For **external PRs**: The `check-external-pr` job detects it's from a fork and **fails intentionally**
- For **internal PRs**: The workflow proceeds normally with e2e tests
- External PRs show ❌ for the check-external-pr job with a message asking for maintainer approval
### 1. PR is created by maintainer:
### 2. Manual Approval Required
**Important**: External PRs require manual approval before workflows can run.
For the PR created by maintainer `E2E Test` workflow starts automatically. The PR check will reflect the status of the job.
**Steps for maintainers:**
1. Go to the PR page
2. Click **"Approve workflow to run"** button. For contributors who have made more than one contribution, workflows will start automatically after approval.
3. The workflow will then execute
### 2. PR is created by external contributor:
**Note**: The `test-e2e.yml` workflow will still fail for external PRs even after approval, as it's designed to prevent automatic execution. Need to use the slash command instead.
For the PR created by external contributor `E2E Test` workflow **won't** start automatically.
Maintainer should make a sanity check of the changes and run it manually by:
1. Putting a comment `/ok-to-test sha=<latest commit hash>` in the PR thread.
2. `E2E Test` workflow starts.
3. After `E2E Test` workflow finishes, the commit with a link and workflow status will appear in the thread.
4. Maintainer can merge PR or request the changes based on the `E2E Test` results.
### 3. Testing External PRs
Once the initial checks have run (and failed), maintainers can test the PR using slash commands:
#### Step-by-Step Process:
## Notes
1. **Navigate to the PR**
- Go to the pull request you want to test
2. **Add a comment with slash command**
```
/ok-to-test sha=<commit-sha>
```
Replace `<commit-sha>` with the latest commit SHA from the PR.
**Note**: Use the short SHA.
3. **Dispatch Action Triggers**
- The `ok-to-test.yml` workflow processes the slash command
- It triggers a `repository_dispatch` event with type `ok-to-test-command`
- The `test-e2e-fork.yml` workflow starts
4. **Workflow Execution**
The fork workflow runs two jobs:
**a) `run-e2e-tests` job** (conditional)
- Only runs if:
- Event is `repository_dispatch`
- SHA parameter is not empty
- PR head SHA contains the provided SHA
- Calls the reusable `run-e2e-tests.yml` workflow
- Runs the actual E2E tests if conditions are met
**b) `update-check-status` job** (conditional)
- Runs after `e2e-tests` completes
- Updates the existing check for job named "run-e2e-tests" from 'test-e2e.yml' workflow
- Sets the conclusion based on `run-e2e-tests` result:
- ✅ **Success** if tests pass
- ❌ **Failure** if tests fail
## Troubleshooting
### Workflow Not Running
- **Check**: Ensure you've approved the workflow to run
- **Action**: Click "Approve workflow to run" in the PR checks tab
### SHA Not Found
- **Check**: Verify the SHA exists in the PR commits
- **Action**: Use `git log --oneline` to find valid commit SHAs
### Dispatch Not Triggering
- **Check**: Ensure the slash command format is correct
- **Action**: Use exact format: `/ok-to-test sha=<sha>`
### Check Runs Not Updating
- **Note**: The fork workflow updates the existing "E2E Tests [reusable]" check run
- **Behavior**: The same check run is updated with new results rather than creating a new one
## Security Notes
- Only users with **write** permissions can trigger dispatch commands
- The fork workflow runs in the main repository context, allowing it to update check runs
- Manual approval is required for external PR workflows
- External PRs are automatically detected and prevented from running tests automatically
## Workflow Files
- **`.github/workflows/ok-to-test.yml`** - Ok To Test - Slash command processor
- **`.github/workflows/test-e2e.yml`** - E2E Tests - Initial PR checks
- **`.github/workflows/test-e2e-fork.yml`** - E2E tests [fork] - Dispatch action handler
- **`.github/workflows/e2e-tests.yml`** - E2E Tests [reusable] - Reusable workflow for actual testing
## Testing Checklist
- [ ] PR created with failing check-external-pr check (for external PRs)
- [ ] Workflow approved to run (if required)
- [ ] Slash command posted with valid SHA
- [ ] Dispatch action triggered successfully
- [ ] `check-external-pr` check run updated with correct conclusion
- Only users with **write** permissions can trigger the `/ok-to-test` command.
- External PRs are automatically detected and prevented from running e2e tests automatically.
- Running e2e test on the external PR is optional. Maintainer can merge PR without running it. Maintainer decides whether it's needed to run an E2E test.