A/B testing

Using data to optimize digital marketing communications has become paramount. I designed an A/B testing mechanism that enhanced Nutshell’s existing suite of marketing tools.

An SVG illustration of 2 envelopes with letters in them: the left has a blue letter, the right has a purple letter, a gold star, and sparkles, representing an A/B test where the right design is the winner.

Timeline:

Nov ‘23 - Jan ‘24

Teammates:

Tools:

The challenge

Nutshell had recently become an emergent force in the digital marketing industry, and we recognized the critical need for an advanced A/B testing feature. We envisioned a tool that would empower customers to refine their engagement strategies while integrating seamlessly with the existing marketing toolkit.

Competitor analysis

To carve out a niche of our own, I embarked on a comprehensive analysis of different competitors' offerings. This deep dive into the A/B testing functionalities of HubSpot, MailChimp, and Campaign Monitor helped provide a clear blueprint of industry standards, common pitfalls, and potential areas for innovation.

Screenshots of comparable solutions from three competitors, HubSpot, Mailchimp, and Campaign Monitor, are organized in Figma. Each column showcases the respective interface designs and features for a side-by-side comparison.

Product screenshots and images pulled from help articles to convey different competitors' UI and UX approaches.

We focused on identifying key differentiators that could position Nutshell as the go-to solution for A/B testing, ensuring a blend of user-friendly design and advanced capabilities.

Early brainstorming & ideation

I worked especially closely with our engineering team in the early phases of this project to come up with initial designs that blended creativity and technical insight. These brainstorming sessions helped shape the feature’s conceptual framework, prioritize simplicity in the user interface, and set the stage for a seamless integration with Nutshell’s existing tools.

A flowchart detailing the information architecture brainstorming for integrating A/B testing into the current message-sending infrastructure.

Information architecture diagrams finalized with engineering to determine how the current message-sending infrastructure could expand to support A/B testing.

Final designs

The final A/B testing feature provided Nutshell customers with a comprehensive toolkit for creating, managing, and analyzing A/B tests.

The A/B testing interface shows two email versions (A and B) with different designs and messages. Users can select criteria to determine the winner (open rate or click rate), set the duration before declaring a winner, choose the percentage of recipients for the test, and decide what to do in the event of a tie.

The A/B testing tab to configure settings for an experiment comparing content, subject lines, or preview text.

After an initial launch to a handful of beta testers, I continued to work closely with Nutshell’s engineers to refine the feature based on feedback and other performance and usage metrics. This iterative process led to improvements such as a clearer progress bar, more flexible testing options, and better analytics.

Three UI elements displaying the progress of an A/B test. Each shows a notification bar indicating 'Version A is leading in your A/B test' with Version A having a 19% open rate compared to Version B's 12% open rate. The progress bar shows the test's sending progress, with a tooltip indicating '20 hours remaining, 300 messages delivered.' The bottom screenshot expands to show additional details for Versions A and B.

In response to user feedback highlighting uncertainty between initiating an A/B test and receiving the results, we swiftly implemented a progress bar to clearly display the test's sending progress.

Impact & lessons learned

Developing an A/B testing feature for Nutshell strengthened its standing in the digital marketing industry. This process emphasized the significance of user-centered design, underpinned by thorough competitive analysis.

The success of this feature underscored the critical role of responsive product evolution in staying competitive in a fast-paced industry.