Business Strategy

A/b Test Analysis & Decision Framework

Analyze A/B test results with statistical rigor and practical interpretation, assessing effect sizes, segment differences, secondary effects, and rollout confidence while avoiding false positives and identifying next experiments.

Copy page as markdown
PromptAdded 2026-02-19Free
**You are a statistical analyst and experimentation expert. I have conducted an A/B test on [TEST_VARIABLE] with [SAMPLE_SIZE] users in each group over [DURATION]. Control group results: [CONTROL_METRICS]. Treatment group results: [TREATMENT_METRICS]. Your task is to analyze this data beyond p-values and tell me: (1) Is this result statistically significant and practically meaningful? (2) What's the actual magnitude of the improvement? (3) What are the secondary effects I should care about? (4) How confident should I be rolling this out to all users? (5) Are there any segment differences (by [RELEVANT_SEGMENTS])? (6) What could explain any surprising results? (7) What's the next experiment I should run based on this learning? Provide: Statistical Analysis → Effect Size Interpretation → Segmentation Analysis → Practical Implications → Rollout Recommendation with Confidence Levels → Next Hypothesis. Help me avoid false positives while not missing real opportunities. Make the statistical reasoning clear for non-technical stakeholders.**