Analyze A/B test results with statistical significance, confidence intervals, practical significance, and recommended actions.
Copy & paste the prompt below into your preferred LLM. Unless a specific AI model is mentioned, you can use whichever you prefer.
You are a data scientist specializing in experimentation and A/B testing. Help me analyze my test results. My test: - What was tested: [DESCRIBE THE A/B TEST — what changed between control and variant] - Hypothesis: [WHAT YOU EXPECTED TO HAPPEN] - Primary metric: [CONVERSION RATE / REVENUE / CLICK-THROUGH / ENGAGEMENT / OTHER] - Control results: [SAMPLE SIZE AND METRIC VALUE] - Variant results: [SAMPLE SIZE AND METRIC VALUE] - Test duration: [HOW LONG DID IT RUN?] - Traffic split: [50/50 / OTHER] - Secondary metrics: [ANY OTHER METRICS TRACKED?] - Segment data: [ANY BREAKDOWNS BY DEVICE, GEO, NEW VS. RETURNING, etc.?] Deliver: 1. **Statistical Analysis:** - Relative lift (% change) - Absolute difference - Statistical significance (p-value) - Confidence interval (95%) - Statistical power assessment - Was the sample size sufficient? 2. **Practical Significance:** Is the observed difference meaningful for the business? Estimated annual impact in dollars/users 3. **Validity Checks:** - Sample Ratio Mismatch (was traffic split as intended?) - Novelty/Primacy effects - Simpson's Paradox check (does the result reverse in subgroups?) - Was the test run long enough (at least one full business cycle)? 4. **Segment Analysis:** If segment data provided — did the result differ for subgroups? 5. **Recommendation:** Ship variant / keep control / run a follow-up test. Clear rationale 6. **Next Test Suggestions:** Based on these results, what should you test next? 7. **Documentation:** Summary for the experimentation log with test ID, dates, results, and decision Underpowered tests and premature conclusions waste more time than they save. Let the math decide.
This prompt works across ChatGPT, Claude, and Gemini because it uses universal prompting principles - analytical framing and role assignment and sequential task breakdown - rather than model-specific tricks that break when you switch platforms. Expect actionable analytical insights with methodology documentation and visualization recommendations. The constraints in this prompt prevent the model from falling back on vague, unhelpful responses.
These data analysis tips will help you get stronger results when using "A/B Test Results Analyzer" and similar prompts in this category.
"A/B Test Results Analyzer" is particularly useful in these situations. If any of these scenarios sound familiar, this prompt will save you significant time.
When you use "A/B Test Results Analyzer" with ChatGPT, Claude, or Gemini, here is what to expect in the AI output.
Adapt "A/B Test Results Analyzer" to your specific situation by modifying these key areas. The more context you add, the better the results.