diff --git a/docs/fork-pr-testing.md b/docs/fork-pr-testing.md index 560133e..0c9cb8c 100644 --- a/docs/fork-pr-testing.md +++ b/docs/fork-pr-testing.md @@ -1,109 +1,32 @@ # Fork PR Testing Guide -This document explains how to test external pull requests using the dispatch action workflow system. +This document explains how testing works for external pull requests from forks. ## Overview -The testing system consists of three main workflows: +The testing system consists of two main workflows: -1. **E2E Tests** (`test-e2e.yml`) - Runs automatically for internal PRs, requires approval for external PRs -2. **Ok To Test** (`ok-to-test.yml`) - Processes slash commands to trigger fork testing -3. **E2E tests [fork]** (`test-e2e-fork.yml`) - Triggered manually via slash commands for external PRs +1. **E2E Tests** (`test-e2e.yml`) - Runs automatically for internal PRs, need manual trigger on external PRs. +2. **Ok To Test** (`ok-to-test.yml`) - Dispatches `repository_dispatch` event when maintainer put's `/ok-to-test sha=` comment in the forked PR thread. ## How It Works -### 1. Initial PR State -When a PR is created or updated: -- The `test-e2e.yml` workflow automatically runs -- For **external PRs**: The `check-external-pr` job detects it's from a fork and **fails intentionally** -- For **internal PRs**: The workflow proceeds normally with e2e tests -- External PRs show ❌ for the check-external-pr job with a message asking for maintainer approval +### 1. PR is created by maintainer: -### 2. Manual Approval Required -**Important**: External PRs require manual approval before workflows can run. +For the PR created by maintainer `E2E Test` workflow starts automatically. The PR check will reflect the status of the job. -**Steps for maintainers:** -1. Go to the PR page -2. Click **"Approve workflow to run"** button. For contributors who have made more than one contribution, workflows will start automatically after approval. -3. The workflow will then execute +### 2. PR is created by external contributor: -**Note**: The `test-e2e.yml` workflow will still fail for external PRs even after approval, as it's designed to prevent automatic execution. Need to use the slash command instead. +For the PR created by external contributor `E2E Test` workflow **won't** start automatically. +Maintainer should make a sanity check of the changes and run it manually by: +1. Putting a comment `/ok-to-test sha=` in the PR thread. +2. `E2E Test` workflow starts. +3. After `E2E Test` workflow finishes, the commit with a link and workflow status will appear in the thread. +4. Maintainer can merge PR or request the changes based on the `E2E Test` results. -### 3. Testing External PRs -Once the initial checks have run (and failed), maintainers can test the PR using slash commands: -#### Step-by-Step Process: +## Notes -1. **Navigate to the PR** - - Go to the pull request you want to test - -2. **Add a comment with slash command** - ``` - /ok-to-test sha= - ``` - Replace `` with the latest commit SHA from the PR. - - **Note**: Use the short SHA. - -3. **Dispatch Action Triggers** - - The `ok-to-test.yml` workflow processes the slash command - - It triggers a `repository_dispatch` event with type `ok-to-test-command` - - The `test-e2e-fork.yml` workflow starts - -4. **Workflow Execution** - The fork workflow runs two jobs: - - **a) `run-e2e-tests` job** (conditional) - - Only runs if: - - Event is `repository_dispatch` - - SHA parameter is not empty - - PR head SHA contains the provided SHA - - Calls the reusable `run-e2e-tests.yml` workflow - - Runs the actual E2E tests if conditions are met - - **b) `update-check-status` job** (conditional) - - Runs after `e2e-tests` completes - - Updates the existing check for job named "run-e2e-tests" from 'test-e2e.yml' workflow - - Sets the conclusion based on `run-e2e-tests` result: - - ✅ **Success** if tests pass - - ❌ **Failure** if tests fail - -## Troubleshooting - -### Workflow Not Running -- **Check**: Ensure you've approved the workflow to run -- **Action**: Click "Approve workflow to run" in the PR checks tab - -### SHA Not Found -- **Check**: Verify the SHA exists in the PR commits -- **Action**: Use `git log --oneline` to find valid commit SHAs - -### Dispatch Not Triggering -- **Check**: Ensure the slash command format is correct -- **Action**: Use exact format: `/ok-to-test sha=` - -### Check Runs Not Updating -- **Note**: The fork workflow updates the existing "E2E Tests [reusable]" check run -- **Behavior**: The same check run is updated with new results rather than creating a new one - -## Security Notes - -- Only users with **write** permissions can trigger dispatch commands -- The fork workflow runs in the main repository context, allowing it to update check runs -- Manual approval is required for external PR workflows -- External PRs are automatically detected and prevented from running tests automatically - -## Workflow Files - -- **`.github/workflows/ok-to-test.yml`** - Ok To Test - Slash command processor -- **`.github/workflows/test-e2e.yml`** - E2E Tests - Initial PR checks -- **`.github/workflows/test-e2e-fork.yml`** - E2E tests [fork] - Dispatch action handler -- **`.github/workflows/e2e-tests.yml`** - E2E Tests [reusable] - Reusable workflow for actual testing - -## Testing Checklist - -- [ ] PR created with failing check-external-pr check (for external PRs) -- [ ] Workflow approved to run (if required) -- [ ] Slash command posted with valid SHA -- [ ] Dispatch action triggered successfully -- [ ] `check-external-pr` check run updated with correct conclusion +- Only users with **write** permissions can trigger the `/ok-to-test` command. +- External PRs are automatically detected and prevented from running e2e tests automatically. +- Running e2e test on the external PR is optional. Maintainer can merge PR without running it. Maintainer decides whether it's needed to run an E2E test.