How to Design Autonomous Optimization Projects

Olcan Sercinoglu

Sep 09, 2018

Introduction

In a previous post we proposed a new Growth process (illustrated below) based on a system of Autonomous Optimization (AO) projects. In this post, we describe how to design these AO projects to maximize growth.

growth-autonomous.png#asset:679

The goal in each AO project is to quickly achieve and maintain the optimal metrics for that project. Unlike traditional experiments that are meant to be stopped at some point, AO projects are meant to be upgraded periodically to achieve even better metrics and drive further growth.

ao-overview.png#asset:695

In each design cycle, we can create new AO projects to optimize new metrics, or upgrade existing projects to optimize existing metrics better. Either way, we want to design projects that can achieve better metrics faster.

Design For Speed

A well-designed AO project should be able to achieve its optimal metrics in at most 2-4 weeks. Fast optimization is critical because it enables: 

Contrary to popular belief, data rate is not the most important factor for optimization speed, because it can take dramatically less data (and time) to optimize well-designed projects, as we describe below.

Metrics and Sessions

We measure and optimize metrics per session, where session is defined by the metric. For example, to optimize page views per visit on our website, we need each session to correspond to a visit, typically defined as a sequence of page view events that happen within a short time (e.g. 30 minutes). 

A session can be any meaningful sequence of events and (attached) properties. We say that events are observed or decided by the AO system. Each decision splits the session into two parts: the context used to make the decision, and the outcome used to define metrics that can be optimized by the decision. 

Below is an illustration of an example session from an AO project designed to optimize checkout revenue per visit, where the session is a visit and the metric is the amount paid at checkout (if any) during that visit.

ao-session.png#asset:694

Defining Metrics from Outcome Events

Metrics are simply numeric values that we measure for each session and then average across sessions. Most common metrics measure the existence (0/1) or count (0, 1, 2, …) of an event (e.g. number of page views), or the value of a numeric property attached to an event (e.g. total price at checkout). Below is an example metric definition in an Amp.ai project:

ao-metric.png#asset:693

Faster Metrics Are Better

Metrics that are determined faster/earlier enable faster optimization, since the session becomes usable for learning as soon as the metric is determined. This can be achieved by limiting the session length or the measurement period. 

Metrics that take more than a week can be measured for monitoring purposes but should not be designated as optimization metrics. Instead, we should define other metrics that are faster and still likely to drive improvements in the key metric. 

For example, if we are interested in checkout revenue at 30 days, but the user is most likely to complete checkout within 7 days, then we can simply optimize 7-day revenue and can expect this to drive an improvement in 30-day revenue also.

Higher-Signal Metrics Are Better

Metrics that vary more across sessions enable faster optimization by providing more signal (high or low metric) for learning the impact of our variants. 

For example, a common issue in “conversion funnels” is that the most desirable event is also the least frequently occurring event at the bottom of the funnel, and the metric based on it (e.g. checkout completion or revenue) tends to be 0 in most sessions. The metric having the exact same value (e.g. 0) in a large fraction of sessions reduces the discernible signal for learning from those sessions. 

Thankfully, we can often optimize for other events or metrics along the funnel that provide more signal and are still likely to drive improvements in the key metric.

Prioritizing Metrics: Cooperative vs Competitive

AO projects often have many metrics that we care about and would like to monitor or optimize. Amp.ai projects allow multiple optimization metrics as long as they are prioritized such that a single key metric has the highest priority. 

Optimization metrics can be cooperative or competitive. Cooperative metrics are commonly found in conversion funnels, where the metrics along the funnel are usually positively correlated. In this case metric priorities should generally increase along the funnel and the final metric at the end of the funnel should have the highest priority. 

Competitive metrics happen when optimizing one key metric (e.g. email response rate) eventually negatively impacts another key metric (e.g. email unsubscribe rate). In this case we need to establish a trade-off among the metrics. We can do this by iteratively adjusting priorities based on optimization results.

Variants and Segments

Unlike traditional experiments, the variants in AO are not restricted to be applied on predefined segments. Instead, each variant is applied on the best segment that can be discerned from data (more on this in subsequent sections).

Intuitively, best variants for AO (in order of preference) are:

  1. High-impact on big segment (e.g. +30% on 50% of sessions).
  2. High-impact on not-too-small segment (e.g. +30% on 10% of sessions). 
  3. Low-impact on big segment (e.g. +3% on 50% of sessions).

The reason we prefer high-impact variants is that the AO system can then still achieve big overall impact by utilizing multiple variants, as long as they address segments that are not-too-small (and not-too-overlapping, more on this below). Using variants in this manner is commonly referred to as personalization or contextualization.

Thankfully, we are not expected to know (nor could we know) these segments, since the best segments can really only be determined based on data. However, we should still consider segments as part of the intuitive hypotheses that we use in designing our variants (according to the criteria above). We can then refine our hypotheses based on the best variants/segments that are actually found in those projects, enabling us to design even better variants in future upgrade cycles.

How Many Variants? 

The number of variants should be naturally determined by the criteria above and our intuitive hypotheses around segments. We recommend the following additional considerations:

With these additional considerations, we expect 3-10 variants in early projects and 10-30 in later projects as we develop our intuition around segments.

Decision Points and Context

Unlike randomized experiments, variants in AO are intelligent decisions made at one or more decision points. Amp.ai projects allow multiple decision points, and multiple variables can be decided at each decision point.

For example, in the checkout optimization example illustrated below, many variables are decided at a single decision point. Every allowed combination of these variables is a variant available to the AO system.

ao-session.png#asset:694

We generally recommend that decisions be made as late as possible, ideally at the exact point that the decision is needed. This allows more context for the decision and helps avoid unnecessary/unused decisions (more on this below).

More context is generally good because variant-specific segments in AO can only be discerned through the context available at decision points. Thus, we should always review the events/properties that are observed prior to each decision point and consider introducing additional context that may enable more of our key segments to be discerned from the data.

Filtering Out Zero-Impact Sessions

A session is zero-impact if it does not contain a decision that can potentially impact the metric. Zero-impact sessions can slow down optimization by reducing the signal available for learning. 

Zero-impact sessions can happen when decisions are made before they are needed. For example, if we decide some content (e.g. the footer on our website) before it is visible to the user, then the decision has zero impact unless the content becomes visible to the user (e.g. by scrolling down to reveal the footer).

Amp.ai projects allow zero-impact sessions to be filtered out based on the existence of decision events and optionally other events that indicate potential impact (e.g. the user scrolling down).

Data Rate (Sessions per Day)

We measure the data rate for an AO project in sessions per day. Faster data enables faster optimization since more data (and signal) is available for learning. Faster data cannot save a poorly-designed AO project with low-signal metrics or low-impact variants. However, even the best-designed AO projects will struggle to optimize in 2-4 weeks with less than 1,000 sessions per day.

Baseline Sessions vs Optimized ("Amped") Sessions

In AO projects it is important to reserve a certain percentage of sessions where optimization is disabled and all decisions take predefined values that correspond to some known baseline variant. These baseline sessions can then be used to measure the relative performance of the optimized sessions. 

Amp.ai projects start with 100% in the baseline. In order to start optimization, we approve a decision point and then allocate some percentage of baseline sessions to be optimized or amped.

ao-amped.png#asset:692

Conclusion

In conclusion, well-designed AO projects should:

Upgrading projects is then a matter of improving on the above criteria, as well as (optionally) removing variants or context events/properties that are not utilized sufficiently by the AO system to be worth maintaining.

In future posts we will describe how to implement AO projects using Amp.ai and dig deep into specific use cases across the customer funnel and software stack.