/prompt-tune

Improve a prompt scientifically, not by vibes.

Usage

Workflow

1. Define the eval set

Either provide:

• 5-15 input/expected-output pairs (JSONL)
• A scoring criterion ("does the output contain a valid JSON object with field X?")

2. Baseline run

• Run the current prompt against all cases
• Record pass/fail + cost per call

3. Generate variants

Create 3-5 modified prompts using:

• Better instruction phrasing
• Few-shot examples
• Chain-of-thought prefix
• Output format constraints
• System role tightening

4. Compare

Side-by-side: pass rate, cost, latency for each variant.

5. Recommend

Surface the variant with the best pass-rate-to-cost ratio. Show the diff vs baseline.

Output

Then writes the winning variant back to the file with a // tuned 2026-04-25 comment.

Rules

• Use claude-haiku-4-5-20251001 for tuning runs unless quality demands Sonnet
• Cap evaluation at 50 runs per session to control cost
• Save the eval set as <prompt-file>.eval.jsonl for future regression checks