This document describes the advanced features and implementation considerations for experiments in Google Analytics.
Overview
The Google Analytics Content Experiments Framework enables you to test almost any change or variation to a website or app to measure its impact on a specific objective; for example, increasing goal completions or decreasing bounce rates. This allows you to identify changes worth implementing based on the direct impact they have on the performance of your property.
Understanding Experiments
To help you understand how Google Analytics Experiments work, the following sections discuss concepts and considerations for:
- Configuration — creating and configuring experiments.
- Collection and Processing — choosing variations and sending experiment data to Google Analytics.
- Reporting — experiment dimensions and metrics that are available for reporting.
Configuration
To run a Google Analytics experiment there are a few conceptual things you need to consider such as what to test and the experiment variations you plan to show users. Next you need to decide if experiment optimization decisions will be managed by Google Analytics or by an external service, and based on that decision determine the experiment objective you want to optimize. Finally, once you create and run the experiment, you will have an experiment ID and a list of variations with Ids that can be used to serve the experiment to users.
The following list of concepts need to be addressed before creating a new experiment:
- Defining the Variations
- Setting the Serving Framework
- Defining the Objective Goals/Metrics
- Running the Experiment
Defining the Variations
Defining variations is dependent on what you want to test. It could be a
single element on a website page, or an entire page, the text size of an offer
on a kiosk screen, or the result set of a database query. It is up to you how
to define a variation. The only requirement when creating an experiment in
Google Analytics is that you provide a name
for each variation.
Variation Id
Google Analytics represents variations as a list,
using a 0-based index, where the first element in the list is the "Original".
The list of variations for an experiment is always in the same order and
cannot be modified once an experiment is running. The index of
a variation is used as the Variation Id
.
Setting the Serving Framework
Optimization decisions for an experiment can be handled by Google Analytics or
externally by you or a service. You can configure this option by setting the
servingFramework
field when you create an experiment using the
Management API.
Setting the serving framework also affects the value of the
JavaScript snippet property of an experiment.
The serving framework and use case for each option are:
REDIRECT
(default) – You intend to rely on Google Analytics to perform client-side redirects to show users the chosen variation. This is the same serving framework used when experiments are created using the Google Analytics web interface. So the only difference in this case is that you are creating the experiment via the Management API.API
– You will choose variations and handle displaying variations to users but will still rely on Google Analytics to handle experiment optimizations. For example, this should be used if you want to eliminate client-side redirects but still rely on Google Analytics to run experiments. This is typically for when the Management API is used to choose variations server-side in cases where dynamic content is being served to users.EXTERNAL
– You will choose variations, handle experiment optimization, and only report the chosen variation to Google Analytics. For example, this should be used by 3rd-party optimization platforms that want to integrate with Google Analytics for reporting purposes. In this case, the Google Analytics statistical engine will not run.
For additional details on the servingFramework
property see the
Experiments API Reference
Defining Objective Goals/Metrics
If the servingFramework
of an
experiment is set to EXTERNAL
then the experiment objective and
decisions should be handled externally by you or another service. In this case
you are not required to define an objective when creating an experiment.
If the servingFramework
is
set to API
or REDIRECT
then the responsibility of
handling experiment decisions is managed by Google Analytics. For this case
your are required to define an experiment objective to optimize.
This objective is what drives the experiment towards a final outcome and is
why it's important that Google Analytics tracking is properly implemented
on the property for experiments to work. If Google Analytics isn't implemented
then it will be impossible to determine when an objective has been reached.
The objective can be to minimize or maximize a goal or a predefined metric
such as time on site, or page views.
Goal
If you're using a
Goal
as an experiment objective, note that the goal must exist before it can be
used. Also, goals in the
experiments web interface
are shown with a name and number that follows the pattern Goal Name
(Goal (n) Completions), where (n)
is the goal number
(from 1-20). However, when managing experiments using the
Management
API, goals are represented with the metric
ga:goal(n)Completions
. For example, to use
Goal 2 Completions
with the API, set the
experiment objective as ga:goal2Completions
.
Metric
For a list of metrics that can be used as experiment objectives, see the Experiments Reference
Running the Experiment
Experiments can be created and modified through the web interface as described in Run an Experiment (Help Center) or programmatically using the Management API.
Once an experiment is properly configured and running, you will have an experiment Id and a list of variations, each with its own variation Id. Using this information it is now possible to begin serving an experiment and sending data to Google Analytics.
The next step is to start sending data to Google Analytics using a collection API/SDK.
Collection and Processing
As users interact with a property and are exposed to a running experiment,
the decision about which variation to show a user depends on whether or not
they have been previously exposed to the experiment. If they are returning
then show the variation previously chosen, otherwise if they are new to the
experiment choose and show a variation. Once it's been determined what
variation to show the user, the experiment Id
and
variation Id
are sent to Google Analytics along with a hit such
as a pageview, event, transaction, etc.
Finally, for collection and processing to work properly for experiments, Google Analytics has to be implemented on the property so that the experiment objective is getting measured. This will ensure that if a user is exposed to an experiment and at some point reaches the experiment objective it will be attributed to the correct experiment ID and variation shown to the user.
The following list of concepts need to be addressed to measure experiments in Google Analytics:
- Choosing a Variation for New Users
- Choosing a Variation for Returning Users
- Sending Experiment Data to Google Analytics
- Experiment Conversions and Outcomes
Choosing a Variation for New Users
When a new user is exposed to an experiment running on your property you
need to choose a variation (if any) to show the user. The responsibility of
this decision is based on which
servingFramework
is
configured for the experiment.
API
orREDIRECT
– In this case, Google Analytics makes the decision and uses a multi-armed bandit approach to evaluate and manage experiments.EXTERNAL
– In this case, the Google Analytics statistical engine is disabled and it is your responsibility to evaluate and manage experiment decisions. In other words, any approach can be used to choose variations for users.
In either case, the final outcome of this step is that a variation will
be chosen for the user and you will have the variation Id
. The
next step is to send the experiment and variation ID to Google Analytics.
Choosing a Variation for Returning Users
When users interact with your property and are exposed to an experiment for the first time, a choice is made to decide what to show the user. However, once that variation choice is made for the user it should remain the same for subsequent exposures to the same experiment. For this reason it is necessary to anonymously store experiment details for a user in a secure but accessible location. When the user returns to the property you can retrieve the ID of the variation previously chosen for the user and take the appropriate action.
Since you have the variation Id
for the user, the
next step is to send the experiment and variation ID to Google Analytics.
Sending Experiment Data to Google Analytics
For example, if a website has an experiment running on a single page of the website, experiment data needs to be sent to Google Analytics each time a user is exposed to that single page, even of they are a returning user. For pageviews on other parts of the site where no experiments are running, it is not necessary to send experiment data to Google Analytics. Keep in mind that returning users should be shown the same variation each session and that in all cases you need to ensure that Google Analytics tracking is implemented properly to measure when objectives are reached by users.
Once an experiment variation is chosen the following experiment data needs to be sent to Google Analytics:
- Experiment ID — the ID of the experiment the user has been exposed to.
- Variation ID — the index of the chosen variation shown to the user.
The experiment ID and variation ID are sent to Google Analytics as parameters attached to other hits, like pageviews, events, or ecommerce transactions. For details on how to send experiment data along with hits during collection, review the experiments developer guide for the collection API/SDK you're using.
The processing of experiment data continues until a winning variation is declared by Google Analytics, or the experiment is manually stopped or expires with no clear winner.
Experiment Conversions and Outcomes
The process of determining the outcome of an experiment depends on the
servingFramework
configured for
the experiment.
EXTERNAL
– For this case an experiment objective is not required when creating an experiment in Google Analytics since this should be handled by you or an external service. The outcome of the experiment is not managed by Google Analytics so it is up to the external service to make decisions on the outcome of the experiment.API
andREDIRECT
– In this case Google Analytics manages the experiment so it will determine the performance of variations and determine the outcome of an experiment. It requires that the experiment have a defined objective. In addition, conversion events and metrics related to the objective need to be measured. For example, if the experiment objective is an event based goal then you need to make sure event tracking is properly implemented on the property so that this goal can actually be achieved. This also applies to objective metrics. For example, a transaction objective requires ecommerce tracking to be implemented.When a goal/objective is reached or completed by a user, Google Analytics will automatically determine, during processing, which active experiments the user has been exposed to and if a match is found, the conversion will be attributed to the experiment and the specific variation shown to the user.
If experiments reports are not being populated with data you should confirm that Google Analytics tracking is properly implemented on your property and that goals and conversions are being measured as expected.
Reporting
After experiment data is collected and processed, it becomes available in the Experiments report in the web interface, or as experiments dimensions and metrics in the Core Reporting API.
Experiments APIs and Libraries
There are several different APIs and libraries available to help you perform the various operations that are required as part of an advanced experiments implementation:
Management API
The Management API can be used to create/insert new experiments, update existing experiments, retrieve a list of experiments, get the details of a single experiment, and delete experiments. The insert, update and delete operations require the authorized user to be an admin of the property. For more information see the Management API Reference, and Developer Guide .
Content Experiments JavaScript API
The Content Experiments JavaScript API can be used for website implementations to set the chosen variation for a user and experiment and to get a previously stored variation. It can also be used to query Google Analytics to choose a variation for a user if you require client-side handling of users and experiments. For more details see the Content Experiments JavaScript API method reference.
As part of the collection process for experiments, you will also need to ensure that Google Analytics measurement is properly implemented on your property.