How To Run An A/B Test In Framer

I used to spend ages wondering whether a button should be blue or green, but Framer's new A/B testing function has totally gotten rid of the guesswork for me.

You can now test two versions of a page and let your actual visitors decide which is best for maximising your return. The results I've had from this tool have been eye-opening, humbling, and sometimes completely baffling.

This post will take you through the complete setup of Framer A/B testing, share real examples from tests I've run, and show you how to avoid the mistakes that'll cost you good data.

What Makes Framer's A/B Testing Great

Most A/B testing tools require external scripts, separate dashboards, and a lot of hoping nothing breaks when you go live, but with Framer, the process looks like this: 

Right-click your page → "New A/B Test" → make your changes → publish. 

That's it.

Framer's testing doesn't slow down your site or cause that annoying content flicker you get with other tools, either. Your visitors will have no idea, and regardless of which variant they land on, you’ll still get real-time results.

No switching between tools, no waiting for data to sync, no extra steps. Everything happens where you're already working.

The one limitation worth knowing upfront is that Framer uses a cookie-free system that resets daily, so you can only track conversions within the same day someone first visits your test. 

For most sites this won’t be a problem, but if you have a longer sales cycle, you might need to keep it in mind.

Before You Start Testing

Just to get an uncomfortable truth out there, A/B testing is only going to work if you have decent traffic.

You’ll need enough visitors to generate statistically reliable data, which usually means at least a few hundred visitors per variant over the course of your test. If your page gets less than 1,000 visitors per month, A/B testing probably isn't worth your time yet.

What you need before starting:

• A page with decent traffic (ideally 1,000+ monthly visitors)

• A clear conversion goal (button click, form submission, page view)

• An idea about what might improve that conversion

Pages that work well for testing:

• Landing pages with clear call-to-actions (CTAs)

• Pricing pages

• Homepage hero sections

• Sign-up or contact forms

Skip testing on:

• Low-traffic pages

• Pages without conversion goals

• About pages or other informational content

The other requirement to get started is being on one of Framer's Business plans (Launch or Scale) with the Advanced Analytics add-on. 

You’ll get a 30-day free trial, then it's event-based pricing with 25,000 events free each month. (An "event" is a page view, link click, or form submission. Most smaller sites stay within the free tier.)

Step-by-Step: Setting Up Your First Test

Alright, let's go through the actual process. 

Step 1: Pick Your Test Page 

Choose a page that gets decent traffic and has a clear conversion goal. For example, I picked my main page because it gets about 2,000 visitors per month and has an obvious call-to-action (to click through to my templates).

Don't overcomplicate it. Pick something straightforward where you can measure success easily.

Step 2: Create Your Variant 

Right-click on your page in the Framer editor and select "New A/B Test." Framer will automatically make a duplicate of your page – this is your variant.

It’s good practice to name your variants something descriptive, so you can remember what you were testing when you look at results later.

Step 3: Make One Meaningful Change 

Don’t change everything at once. Pick one element to test. 

In my test, I only changed the hero headline. Same design, same layout, same everything else. Just the headline.

Step 4: Set Up Conversion Tracking 

Click "Configure" at the top of your variant page. You'll need to add a tracking ID to whatever element you want to measure as a conversion.

For my test, I added a tracking ID to the ‘browse templates’ button. Framer then knows to count every time someone clicks that button as a conversion event.

You can track multiple events, but keep it simple for your first test. Pick the one action that matters most to your business.

Step 5: Launch and Wait 

Hit "Start" and publish your site. Framer will split your traffic 50/50 between the original and variant.

This is the hardest part – waiting. Speaking from experience here, checking results every few hours is completely pointless and will drive you crazy. Give it some time.

The whole setup process took me about 3 minutes, and the wait for meaningful results took 18 days. We’ll get into how to know when it’s time to end your testing a little later on. 

Understanding Framer A/B Test Results

Framer will show you three key metrics:

Conversion Rate is your main number – the percentage of visitors who completed your goal action. If 100 people see your page and 5 click your button, that's a 5% conversion rate.

Lift tells you the percentage improvement (or decline) compared to your original. A +15% lift means your variant is performing 15% better than the control.

Probability of Being Best is the magic number. This shows how confident Framer is that your variant is actually winning. You want this to hit 90% before making any decisions. Anything lower is basically a coin flip.

You'll also see some bands around the result lines – that's the credibility interval. When the bands get narrower and stop overlapping between variants, that's when you know you've got solid data.

My rules for calling winners:

• 90%+ probability of being best

• At least 200 conversions per variant

• Test has run for minimum 1 week to account for day-of-week variations

When to End Your A/B Test

This is probably the most important decision you'll make during the entire process. Here’s how you’ll know it’s time to end testing.

When you have a clear winner: If one variant hits 90% probability of being best and you've got decent sample sizes, you can confidently stop the test. Click "Stop Test" in Framer, and you'll get the option to replace your control version with the winning variant.

Make sure to publish your site again after making the switch – this ensures all future visitors see the winning version.

When results are inconclusive: Sometimes, you'll run a test for weeks and never get a clear winner. This happens more often than you'd think, especially if you're testing similar variants or don't have tons of traffic.

You have two options: keep running the test longer (if you have patience and traffic), or stop it and try testing something completely different. 

There's no shame in inconclusive results, though. All results are great results – in this case they tell you that whatever you tested probably doesn't make a difference to your users.

When something looks obviously wrong: If your results seem completely crazy – like a 500% improvement from changing a font size – something's probably broken. Check your tracking setup, make sure both variants are getting traffic, and verify that your conversion events are firing correctly.

The post-test cleanup

Once you've implemented your winning variant, take a few minutes to document what you learned. What did you think would happen? What actually happened? What will you test next?

You can delete the losing variants to keep your project clean, but save screenshots of your results. You'll want to reference them later when planning future tests.

Key Takeaways

• Start simple and test one thing at a time: Your first test should be simple – a headline change, different CTA text, or button placement. 

• Traffic is everything: Don't bother testing if you're not getting at least a few hundred visitors per variant. 

• Patience beats intuition: Wait for that 90% confidence level before calling winners. I know it's exciting to see early results, but jumping to conclusions will lead you astray more often than not.

• Focus on what actually drives business results: Test your headlines, value propositions, and calls-to-action before you worry about button colors or font choices. 

• The daily reset limitation is real: Keep Framer's cookie-free approach in mind when choosing what to test. It works great for same-day conversions but isn't ideal for longer sales cycles.

• Document everything: Keep notes on what you tested, why you tested it, and what you learned. 

Not sure where to start with your Framer site design? Take my free 60-second template quiz to get matched with the perfect template for your business, plus get 30% off when you're ready to launch.