Experiments - Feature Reference

This document describes the advanced features and implementation considerations for experiments in Google Analytics.

Overview

The Google Analytics Content Experiments Framework enables you to test almost any change or variation to a website or app to measure its impact on a specific objective; for example, increasing goal completions or decreasing bounce rates. This allows you to identify changes worth implementing based on the direct impact they have on the performance of your property.

Note: This document describes advanced features and implementation considerations. If you're looking for general information about Content Experiments read Benefits of Experiments in the Help Center.

Understanding Experiments

To help you understand how Google Analytics Experiments work, the following sections discuss concepts and considerations for:

• Configuration — creating and configuring experiments.
• Collection and Processing — choosing variations and sending experiment data to Google Analytics.
• Reporting — experiment dimensions and metrics that are available for reporting.

Note: This guide is intended to be library and environment agnostic. Developers should refer to the available developer guides for their environment and library to learn more about implementation details.

Configuration

To run a Google Analytics experiment there are a few conceptual things you need to consider such as what to test and the experiment variations you plan to show users. Next you need to decide if experiment optimization decisions will be managed by Google Analytics or by an external service, and based on that decision determine the experiment objective you want to optimize. Finally, once you create and run the experiment, you will have an experiment ID and a list of variations with Ids that can be used to serve the experiment to users.

The following list of concepts need to be addressed before creating a new experiment:

• Defining the Variations
• Setting the Serving Framework
• Defining the Objective Goals/Metrics
• Running the Experiment

Defining the Variations

Defining variations is dependent on what you want to test. It could be a single element on a website page, or an entire page, the text size of an offer on a kiosk screen, or the result set of a database query. It is up to you how to define a variation. The only requirement when creating an experiment in Google Analytics is that you provide a name for each variation.

Variation Id

Google Analytics represents variations as a list, using a 0-based index, where the first element in the list is the "Original". The list of variations for an experiment is always in the same order and cannot be modified once an experiment is running. The index of a variation is used as the Variation Id.

Note: It is possible to create experiments with variations that do not have valid URLs. This is normal for experiments where the user is not being redirected when shown a variation. In these cases you can safely ignore warnings in the Google Analytics web interface related to experiment validation failures.

Setting the Serving Framework

Optimization decisions for an experiment can be handled by Google Analytics or externally by you or a service. You can configure this option by setting the servingFramework field when you create an experiment using the Management API. Setting the serving framework also affects the value of the JavaScript snippet property of an experiment.

The serving framework and use case for each option are:

• REDIRECT (default) – You intend to rely on Google Analytics to perform client-side redirects to show users the chosen variation. This is the same serving framework used when experiments are created using the Google Analytics web interface. So the only difference in this case is that you are creating the experiment via the Management API.
• API – You will choose variations and handle displaying variations to users but will still rely on Google Analytics to handle experiment optimizations. For example, this should be used if you want to eliminate client-side redirects but still rely on Google Analytics to run experiments. This is typically for when the Management API is used to choose variations server-side in cases where dynamic content is being served to users.
• EXTERNAL – You will choose variations, handle experiment optimization, and only report the chosen variation to Google Analytics. For example, this should be used by 3rd-party optimization platforms that want to integrate with Google Analytics for reporting purposes. In this case, the Google Analytics statistical engine will not run.

For additional details on the servingFramework property see the Experiments API Reference

Defining Objective Goals/Metrics

If the servingFramework of an experiment is set to EXTERNAL then the experiment objective and decisions should be handled externally by you or another service. In this case you are not required to define an objective when creating an experiment.

If the servingFramework is set to API or REDIRECT then the responsibility of handling experiment decisions is managed by Google Analytics. For this case your are required to define an experiment objective to optimize. This objective is what drives the experiment towards a final outcome and is why it's important that Google Analytics tracking is properly implemented on the property for experiments to work. If Google Analytics isn't implemented then it will be impossible to determine when an objective has been reached. The objective can be to minimize or maximize a goal or a predefined metric such as time on site, or page views.

Goal

If you're using a Goal as an experiment objective, note that the goal must exist before it can be used. Also, goals in the experiments web interface are shown with a name and number that follows the pattern Goal Name (Goal (n) Completions), where (n) is the goal number (from 1-20). However, when managing experiments using the Management API, goals are represented with the metric ga:goal(n)Completions. For example, to use Goal 2 Completions with the API, set the experiment objective as ga:goal2Completions.

Metric

For a list of metrics that can be used as experiment objectives, see the Experiments Reference

Running the Experiment

Experiments can be created and modified through the web interface as described in Run an Experiment (Help Center) or programmatically using the Management API.

Once an experiment is properly configured and running, you will have an experiment Id and a list of variations, each with its own variation Id. Using this information it is now possible to begin serving an experiment and sending data to Google Analytics.

The next step is to start sending data to Google Analytics using a collection API/SDK.

Collection and Processing

As users interact with a property and are exposed to a running experiment, the decision about which variation to show a user depends on whether or not they have been previously exposed to the experiment. If they are returning then show the variation previously chosen, otherwise if they are new to the experiment choose and show a variation. Once it's been determined what variation to show the user, the experiment Id and variation Id are sent to Google Analytics along with a hit such as a pageview, event, transaction, etc.

Finally, for collection and processing to work properly for experiments, Google Analytics has to be implemented on the property so that the experiment objective is getting measured. This will ensure that if a user is exposed to an experiment and at some point reaches the experiment objective it will be attributed to the correct experiment ID and variation shown to the user.

The following list of concepts need to be addressed to measure experiments in Google Analytics:

• Choosing a Variation for New Users
• Choosing a Variation for Returning Users
• Sending Experiment Data to Google Analytics
• Experiment Conversions and Outcomes

Choosing a Variation for New Users

When a new user is exposed to an experiment running on your property you need to choose a variation (if any) to show the user. The responsibility of this decision is based on which servingFramework is configured for the experiment.

• API or REDIRECT – In this case, Google Analytics makes the decision and uses a multi-armed bandit approach to evaluate and manage experiments.
• EXTERNAL – In this case, the Google Analytics statistical engine is disabled and it is your responsibility to evaluate and manage experiment decisions. In other words, any approach can be used to choose variations for users.

In either case, the final outcome of this step is that a variation will be chosen for the user and you will have the variation Id. The next step is to send the experiment and variation ID to Google Analytics.

Choosing a Variation for Returning Users

When users interact with your property and are exposed to an experiment for the first time, a choice is made to decide what to show the user. However, once that variation choice is made for the user it should remain the same for subsequent exposures to the same experiment. For this reason it is necessary to anonymously store experiment details for a user in a secure but accessible location. When the user returns to the property you can retrieve the ID of the variation previously chosen for the user and take the appropriate action.

Since you have the variation Id for the user, the next step is to send the experiment and variation ID to Google Analytics.

Sending Experiment Data to Google Analytics

Experiment data needs to be sent to Google Analytics every time a user is exposed to an experiment and is shown a variation.

For example, if a website has an experiment running on a single page of the website, experiment data needs to be sent to Google Analytics each time a user is exposed to that single page, even of they are a returning user. For pageviews on other parts of the site where no experiments are running, it is not necessary to send experiment data to Google Analytics. Keep in mind that returning users should be shown the same variation each session and that in all cases you need to ensure that Google Analytics tracking is implemented properly to measure when objectives are reached by users.

Once an experiment variation is chosen the following experiment data needs to be sent to Google Analytics:

• Experiment ID — the ID of the experiment the user has been exposed to.
• Variation ID — the index of the chosen variation shown to the user.

Note: If you are managing experiments and have set the servingFramework field of an experiment to EXTERNAL, it is likely you have your own internal ID for experiment variations. In this case, to send experiment data to Google Analytics, you will need to map the internal variation ID you're using to the corresponding Google Analytics variation Id of the experiment.

The experiment ID and variation ID are sent to Google Analytics as parameters attached to other hits, like pageviews, events, or ecommerce transactions. For details on how to send experiment data along with hits during collection, review the experiments developer guide for the collection API/SDK you're using.

The processing of experiment data continues until a winning variation is declared by Google Analytics, or the experiment is manually stopped or expires with no clear winner.

Experiment Conversions and Outcomes

The process of determining the outcome of an experiment depends on the servingFramework configured for the experiment.

• EXTERNAL – For this case an experiment objective is not required when creating an experiment in Google Analytics since this should be handled by you or an external service. The outcome of the experiment is not managed by Google Analytics so it is up to the external service to make decisions on the outcome of the experiment.

• API and REDIRECT – In this case Google Analytics manages the experiment so it will determine the performance of variations and determine the outcome of an experiment. It requires that the experiment have a defined objective. In addition, conversion events and metrics related to the objective need to be measured. For example, if the experiment objective is an event based goal then you need to make sure event tracking is properly implemented on the property so that this goal can actually be achieved. This also applies to objective metrics. For example, a transaction objective requires ecommerce tracking to be implemented.

  When a goal/objective is reached or completed by a user, Google Analytics will automatically determine, during processing, which active experiments the user has been exposed to and if a match is found, the conversion will be attributed to the experiment and the specific variation shown to the user.

  If experiments reports are not being populated with data you should confirm that Google Analytics tracking is properly implemented on your property and that goals and conversions are being measured as expected.

Reporting

After experiment data is collected and processed, it becomes available in the Experiments report in the web interface, or as experiments dimensions and metrics in the Core Reporting API.

Experiments APIs and Libraries

There are several different APIs and libraries available to help you perform the various operations that are required as part of an advanced experiments implementation:

Management API

The Management API can be used to create/insert new experiments, update existing experiments, retrieve a list of experiments, get the details of a single experiment, and delete experiments. The insert, update and delete operations require the authorized user to be an admin of the property. For more information see the Management API Reference, and Developer Guide .

Content Experiments JavaScript API

The Content Experiments JavaScript API can be used for website implementations to set the chosen variation for a user and experiment and to get a previously stored variation. It can also be used to query Google Analytics to choose a variation for a user if you require client-side handling of users and experiments. For more details see the Content Experiments JavaScript API method reference.

As part of the collection process for experiments, you will also need to ensure that Google Analytics measurement is properly implemented on your property.