Skip to main content
A/B testing allows you to run multiple integrations simultaneously and distribute requests between them based on configurable weights. This enables you to compare AI models, test new backends, and implement gradual rollouts.
A/B testing active

A/B testing with multiple active integrations

​
How It Works

When your bot receives a message:
  1. Check Active Integrations: Bot finds all integrations with weight > 0
  2. Calculate Distribution: Total weight determines probability for each
  3. Select Integration: Randomly select based on weights
  4. Send Request: Forward message to selected integration
  5. Track Performance: Log which integration was used

​
Setting Up A/B Testing

1

Create Multiple Integrations

Create 2 or more integrations for your bot. For example:
  • Integration A: “GPT-4” (OpenAI)
  • Integration B: “Claude 3.5 Sonnet” (Anthropic)
2

Assign Weights

Set weights for each integration:
  • GPT-4: Weight 50
  • Claude: Weight 50
This creates a 50/50 split.
3

Enable Integrations

Ensure both integrations are active (not disabled).
4

Send Messages

Messages will automatically distribute according to weights.

​
Weight Distribution

Weights determine the probability of each integration being selected:

​
Equal Distribution

​
Unequal Distribution

​
Gradual Rollout

Start with a small percentage and increase over time: Week 1:
Week 2:
Week 3:
Week 4:

​
Use Cases

​
Model Comparison

Compare different AI models on the same traffic:

​
Feature Testing

Test new features or prompts:
Safely test changes on a small percentage of traffic.

​
Fallback Strategy

Use weights with fallback integrations:
The backup integration only runs when primary fails.

​
Best Practices

Start Small

Begin with 5-10% traffic to new integrations

Define Success

Know what you’re measuring before starting

Run Long Enough

Collect enough data for statistical significance

One Variable at a Time

Test one change at a time for clear results

​
Statistical Significance

Don’t draw conclusions too early: Wait until each integration has served enough requests to see patterns.

​
Avoid Common Pitfalls

Don’t:
  • Change weights daily (let tests run)
  • Test too many variables at once
  • Ignore statistical significance
  • Compare apples to oranges (different use cases)
Do:
  • Test one change at a time
  • Keep detailed notes
  • Use consistent metrics
  • Document learnings

​
Configuration Examples

​
Canary Deployment

Gradually roll out a new model:

​
Multi-Variant Testing

Test three options:
Each gets roughly 1/3 of traffic.

​
Champion vs. Challenger

Keep a proven option dominant:
The champion serves most traffic while you evaluate the challenger.

​
Advanced Techniques

​
User-Based Testing

Use custom headers to route specific users:
Requires custom logic in your integration selection.

​
Geographic Testing

Route by user location (if available):

​
Ending an A/B Test

When your test concludes:
1

Analyze Results

Review all collected metrics and determine the winner.
2

Choose Winner

Decide which integration to use going forward.
3

Update Weights

Set winner to Weight 100, others to Weight 0 (or delete them).
4

Document Findings

Record what you learned for future reference.
Keep losing integrations configured but disabled (Weight 0) so you can easily re-test if needed.

​
Troubleshooting

​
Uneven Distribution

If traffic doesn’t match weights:
  • Low Traffic: Need more requests for distribution to even out
  • Caching: Check if responses are cached
  • Time of Day: Traffic patterns may affect distribution

​
One Integration Always Fails

If one integration has high error rate:
  • Check timeout settings
  • Verify API credentials
  • Test integration manually
  • Review error logs

​
Next Steps

Webhook Setup

Configure integration endpoints

Custom Headers

Add routing logic with headers