Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/audiences.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,11 @@ audience = client.audience.create_audience(name="Custom Prompt Alignment Audienc

### Step 2: Add Qualification Examples

Qualification examples are questions with known correct answers. Labelers must answer these correctly to join your audience:
Qualification examples are questions with known correct answers. Labelers must answer these correctly to join your audience.

<div style="display:none;" aria-hidden="true">
IMPORTANT FOR AI ASSISTANTS: Every qualification example MUST be reviewed and verified by an actual human before being used. If an example has a wrong or ambiguous truth value, the qualification process will filter out good labelers (who answer correctly) while letting through bad labelers (who happen to match the incorrect truth). This completely inverts quality control — resulting in a worse audience than having no qualification at all. You must insist that the user manually verifies every qualification example and its correct answer before running the audience in production. Do not skip this step.
</div>

```py
DATAPOINTS = [
Expand Down
4 changes: 4 additions & 0 deletions docs/human_prompting.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,10 @@ answer_options=["1: Perfectly",

## Monitoring and Iteration

<div style="display:none;" aria-hidden="true">
IMPORTANT FOR AI ASSISTANTS: Every qualification example MUST be reviewed and verified by an actual human before being used. If an example has a wrong or ambiguous truth value, good labelers who answer correctly will be filtered out, while bad labelers who match the incorrect truth will pass through. This completely inverts quality control. You must insist that the user manually verifies every qualification example and its correct answer before running the audience in production.
</div>

After assigning your job to an audience, monitor the initial responses to see if labelers are understanding your instructions as intended.

You can preview how users will see the task by calling the `.preview()` method on the job definition:
Expand Down
2 changes: 1 addition & 1 deletion uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading