Conductrics Blog

Conductrics takes privacy seriously. Conductrics AB Testing has been developed in conformance with Privacy by Design principles in order to give our clients the privacy options they need. This is important for: 1) maintaining customer trust by safeguarding their privacy; and 2) in order to comply with privacy regulations that require systems to be designed with default behaviors that provide privacy protections. 

For example, Article 25 of the GDPR requires “Data protection by design and by default” and explicitly proposes the use of pseudonymization as a means to comply with the overarching data minimization principle. Technology should be designed in such a way that by default the scope of data collected is limited to “only personal data which are necessary for each specific purpose of the processing are processed.”  In other words, if a task can be performed without the use of an additional bit of data, then, by default, that extra bit of data should not be collected or stored. 

With Conductrics, clients are able to run AB Tests without the need to collect or store AB Test data that is linked directly to individuals, or associated with any other information about them that is not explicitly required to perform the test.  Out of the box, Conductrics collects AND stores data using data minimization and aggregation methods.

This is unlike other experimentation systems that follow what can be called an identity-centric strategy. This approach links all events, experimental treatments, sales etc., back to an individual, usually via some sort of visitor Id. 

For clarity, lets look at an example. In the table below there are three experiments running; Exp1, Exp2, and Exp3. Each experiment is joined back to a customer such that there is a table (or logical relationship) that links each experiment and the assigned treatment back to a visitor. 

While this level of detail can be useful for various ad hoc statistical analysis, it also means that the experimentation system has less privacy control since all the information about a person is linked back to each experiment. 

One can use pseudonymization, where the explicitly identifiable information (the Name, Email, and often Cust ID) can be removed from the table. However, what many fail to realize is that even if we remove those fields but keep all of the other associations at the same individual level, the remaining data often still contains more information than required and might be at a higher risk of being able to leak personal information about individuals. 

Happily, it turns out that for most standard experiments there is no need to collect or store data at the individual level to run the required statistical procedures for AB Tests – or even for certain versions of regression analysis useful for things like AB Test interaction checks. Instead of collecting, linking, and storing experiment data per individual, Conductrics collects and stores experiment data that we both de-identify AND aggregate AS IT IS COLLECTED. This makes it possible to both collect and store the experimental data in an aggregate form such that the experimentation platform NEVER needs to know identifying individual level data. 

Privacy by Design: Data Minimization and Aggregation

In the context of privacy engineering, network centricity (“the extent to which personal information remains local to the client”) is considered a key quality attribute in the Privacy by Design process. This can be illustrated by contrasting so-called “global privacy” with “local privacy”. 

In the global privacy approach identifiable, non minimized data is collected and stored, but only by a trusted curator. For example, the US Gov’t collects census data about US citizens. The US Census Bureau acts as the trusted curator, with access to all of the personal information collected by the census. However, before the Census Bureau releases any data publicly, it applies de-identification and data minimization procedures to help ensure privacy. 

Local privacy, by contrast, is a stronger form of privacy protection because data is de-identified and minimized as it is collected and stored. This removes the need for a trusted curator. Conductrics is designed to allow for a local privacy approach by default, such that Conductrics only collects and stores data in its minimized form.

In the process of doing so, Conductrics is also respecting the key privacy engineering objective of dissociability – a key pillar of the NIST Privacy Framework. 

How does this data collection work?

Rather than link all of the experiments back to a common user in a customer table / relational model above, Conductrics stores each experiment in its own separate data table. For simple AB Tests each of these tables in turn only need to record three primary values for each treatment in the experiment: 1) The count of enrolments (the number of users per treatment); 2) The total value of the conversions or goals for each treatment; and 3) the total value of the square of each conversion value. Let’s take a look at a few examples to make this clear. At the start of the experiment the AB Test’s data table is empty.

AB Test 1

As users are assigned to treatments, Conductrics receives telemetry from each user’s client (browser) with only the information needed to update the data table above. For example, say a customer on the client website gets assigned to the ‘B’ treatment in the AB Test. A message is sent to Conductrics (optionally, the messages can be first passed to the organization’s own servers via Conductrics Privacy Server for even more privacy control).

This message looks something like ‘ABTest1:B’ and the AB Test’s data table in Conductrics is updated as follows:

If the user then purchases the product attached to the test, the following message is sent ‘ABTest1:B:10’. Conductrics interprets this as a message to update the Sales field for Treatment B by $10 and the Sales^2 field by 100 (since 10^2=100). Notice that no visitor identifier is needed to make this update (Note: The actual system is a bit more complex, ensuring that the Sales^2 records properly account for multiple conversion events per user, but this is the general idea.)

As new users enter the test and purchase, the AB Test’s data table accumulates and stores the data in its minimized form. So by the end of our experiment we might have a data table that looks like this:

A nice property of this is that not only has the data been collected and stored in a minimized form for increased privacy, but it is also in a form that is much more efficient to calculate statistical tests. This is because most of the required summation calculations that are required for things like t-tests and covariance tables have already been done. 

Interestingly we are not limited to just simple AB Tests. We can extend this approach to include additional segment data. For example if we wanted to analyze the AB Test data based on segments – say built on customer value levels; Low (L) ,Medium (M),and High (H), the data would be stored as follows.

There is of course no need to limit the number of segments to just one. We can have any number of combinations of segments. Each combination is known as an equivalence class. As we increase the segment detail of each equivalence classes we also increase the number (based on the cardinality of the segment combinations).

So as the number of classes increase, there is an associated increase the possibility of identifying or leaking information about persons entered into a test. To balance this, Conductrics collection of user traits/segments follows from ideas of K-Anonymization, a de-identification technique based on the idea that the collected user data should have groups of at least K other users who look exactly the same. Depending on each experiment’s context and sensitivity, Conductrics can either store each user segment/trait group in its own separate data table, effectively keeping the segment data de-linked, or store the data at the segment combination level.

This gives our clients the ability to decide how strict they need to be with how much the data is aggregated. In the near future Conductrics will be adding reporting tools that will allow clients to query each experiment to help manage and ensure that data across all of their experiments is stored at the appropriate aggregation level.  

What about the Stats? 

While statistics can be very complicated, calculations behind most of the common approaches for AB Testing ( e.g  Welch t-tests. z-tests, Chi-Square, F-Tests, etc.) under the hood are really mostly just a bunch of summing and squaring of numbers. 

For example, the mean (average) value for each treatment is just the sum of Sales divided by the number of enrollments for each treatment. By dividing the sales column by the count column for each treatment we can calculate the average conversion value per treatment. 

The calculations for the variance and standard errors are a bit more complicated but they use a version of the following general form for the variance which uses the aggregated sum of squares data field (where each value is squared first then those squared values are all added up):

It turns out that this approach can be generalized to the multivariate case, and using similar logic, we can also run regressions required for MVTs and interaction checks. Technical readers might be interested to notice that the inner product elements of the Gram Matrix for ANOVA/OLS with categorical covariates are just conditional sums that we can pluck from the aggregated data table with minimal additional calculations.

Intentional Data Collection and Use

Of course there are use cases where it is appropriate and necessary to collect more specific, and identifiable information.  However, it is often the case that we can answer the very same questions without ever needing to.  The main point is rather than collecting, linking, and storing all of the extra information by default just in case it might be useful, we should be intentional for each use case about what is actually needed, and proceed from there.

Lastly, we should note that disassociation methods and data minimization by default are not substitutes for proper privacy policies – perhaps one could argue that alternative approaches, such as differential privacy might be, but generally the types of methods discussed here are not). Rather, privacy engineering is a tool that sits alongside required privacy and legal compliance policies and contracts.

If you would like to learn more about Conductrics please reach out here. 

The Fusion of Experimentation with Market Research

Customer centric 

For over a decade Conductrics has been providing innovative software for online Experimentation and Optimization. Innovations include the industry’s first API driven A/B Testing and multiarmed bandit software, as well as the first to introduce transparency in our machine learning predictive targeting with human-interpretable contextual bandits capabilities.

Working alongside our client partners, it became clear that even though tools for experimentation and optimization are extremely powerful, up until now they have been inherently limited.   Limited because no matter the type of A/B Test or ML algorithm, experimentation has lacked visibility into a key, and necessary, source of information – the voice of the customer. 

We are excited to announce that we have created Conductrics Market Research a first of its kind solution that integrates Optimization with Market research. Current approaches require separate tools and treat experimentation and research as separate solutions. We believe the true power of customer experience optimization can be only unlocked when experimentation and research are treated as one, like a film and its soundtrack. Full understanding and meaning can only happen when both are perfectly synced with one another.  

Top Level Benefits of combining research with experimentation and optimization include:

1. Learn directly from the customer about what they care about and their unmet needs in order to feed your experimentation program and to drive drive the fruitful development of new product and new customer experiences.
2. Now you can learn not only ‘What’ work, but ‘Why’ it works, by adding direct customer feedback alongside customer behavior in your A/B Test metrics. 
3. Discover if new customer experiences and product features improve customer attitudes and long term loyalty (e.g. NPS) as well as improve sales and conversions.
4. Tailor the customer’s  journey in real time and show them that you are actually listening to them, by delivering real-time customized online experiences based on their specific survey feedback.
5. Strengthen your culture of experimentation by improving internal messaging by combining direct customer quotes and your tests’ impact on NPS alongside your current statistical reporting. 

New Platform Capabilities

• Easy Set up and Launch of Native Surveys directly from Conductrics
• Connect any A/B Test to any Survey: 
  • Numeric and Yes/No survey responses can be used directly as goals for A/B Test.
  • Collect and connect qualitative/open text customer responses back to each A/B Test treatment experience. 
• Append in-session user data (e.g logged in, loyalty status, A/B test treatments etc.) to survey response data for enhanced survey reporting.
• Create in-session user traits directly from survey responses and attach targeting rules to them to customize online user experience in real-time or to run targeted A/B tests.
• Attach any standard behavioral goals, such as sales, to surveys to auto-generate A/B Tests to determine if offering the survey adversely affects sales or conversions.  

Conductrics Market Research

As we roll out the new integrated platform, customers will gain access to Conductrics Market Research and see it as a new top-level capability along side A/B Testing, Predictive Targeting, and Rule-Based Targeting.

For over a decade, Conductrics has been innovating and delivering industry-first capabilities. We are very excited to be the first to combine direct voice of consumer data alongside traditional experimentation data to provide you with the integrated capabilities needed to create better user experiences, drive long-term loyalty and increase direct revenue.

Get a first look at some of the unique capabilities that are available to your experimentation and research teams that are rolling out at Conductrics. 

To learn more contact us here.

A First Look

In this post we will take our first look at some of the new features the Conductrics Market Research release. We will follow up over the coming weeks and months with more peeks and details of the new capabilities. 

As we roll out the new integrated platform, customers will gain access to Conductrics Market Research and see it as a new top-level capability along side A/B Testing, Predictive Targeting, and Rule-Based Targeting.

When creating a new survey, along with the standard survey creation options, there are three primary additional features that are unique to Conductrics. 

1 Survey Responses as A/B Test Goals

When creating a numeric survey question you will be given the option to ‘Track as a Goal/Conversion’ event. If this option is selected then the value of this question can be used as a goal in any of your A/B Tests – just like any other behavioral measure.   

For example, say we have a three question survey and we would like to capture ‘How Likely To Recommend’ as a survey response, but we would also like to see how an upcoming A/B Test might affect the likelihood to recommend. By clicking the option to track the Recommend question as a goal, all new A/B Tests will be eligible to use this survey response as a conversion goal.

2 Auto Create NPS Goal

While any metric survey question can be used as an A/B Testing goal, often what the C-Suite really cares about is changes to NPS. By also selecting the optional ‘Enable for Net Promoter Score (NPS)’ as shown in the image above, Conductrics will auto create 4 NPS related goals:

1. NPS score (% Promoters – % Detractors)*100
2. Number Promoters
3. Number Detractors
4. Number of Passives. 

What is so powerful about this is that you can now see NPS scores, and their associated confidence intervals and testing statistics, in your A/B Testing reports along with your sales and conversion goals.

We believe this is the only survey solution, even including dedicated survey tools, that provides confidence intervals around NPS and the ability to use NPS in A/B Tests.

3 Real-Time Customer Experience Customization 

As part of the Survey creation workflow there is also the option of assigning survey responses as user traits for filtering and enriching reporting for A/B Tests, or to creating real-time customized in-session customer experiences.

For example, if we wanted to customize the customer experience based on how a user responded to the “What is the Purpose of Your Visit?” question, we just select the ‘Retain Response as Visitor Trait’ in the question setup and the value of ‘Purpose of Visit’ will automatically be sent into an in-session Conductrics visitor trait ‘Purpose’

This makes this information immediately available to all reporting, A/B Testing, and Predictive Targeting modules within Conductrics.

To customize the in-session experience we can use Conductrics Rules-Based Targeting module. Once the above survey is saved it will auto-populate Conductrics User Targeting Conditions builder.  Below we see that the rules builder auto generated a user trait called ‘Purpose’ that has the four associated survey response values as options.  These can be used either directly or in any logical combination with any other set of in-session visitor traits.

To keep it simple, we set up a collection of rules that will trigger the relevant in-session user experience based on just the user’s responses to the ‘Purpose of Visit’ question. In the follow screen shot, we show our variations, or user experiences, on the left, and on the right we have our targeting rules that will trigger and deliver the experiences based on the user’s response to the ‘Purpose of Visit’ question. These user experiences can be content from Conductrics web creation tools, backend content like feature flags, or even different server side algorithms.

For example if a customer submits a survey and answers the ‘What is the Purpose of Your Visit?’ question with ‘To Purchase Now’’, then they will immediately be eligible for the targeting rule that will deliver ‘Product Offers’ content. 

Survey Response Data

Of course, Conductrics Research also provides survey reporting to aide human decision making. Conductrics provides simple to use filtering and summary features to help you more easily understand what your customers are hoping to communicate with you.

Along with the tabular view, there is also a card view, so that you can ‘flip through’ individual survey responses and see all an individual customer’s responses enriched with all of their associated in-session data that has been made available to the Conductrics platform.

Optionally, you can download the in-session enriched survey data for use in your favorite statistical software package.

Are Surveys Affecting Conversions?

An important question that arises with in-session market research is if offering a survey in a sensitive area of the site might adversely affect customer purchase or conversion. Conductrics provides the answer. Simply assign a behavioral conversion goal, like sales, or sign-up, to the survey. Conductrics will automatically run a special A/B Test alongside the survey to test the impact that offering a survey has on conversion by comparing all eligible users who were offered a survey vs those who were not offered.

Now your market research teams can learn where and when it is okay to place customer surveys by determining if conversions actually are affected and if so, if it is by a large enough amount to offset the value of direct customer feedback.

For over a decade, Conductrics has been innovating and delivering industry-first capabilities. We are very excited to be the first to combine direct voice of consumer data alongside traditional experimentation data to provide you with the integrated capabilities needed to create better user experiences, drive long-term loyalty and increase direct revenue. This is just a first quick look at some of the unique capabilities that are available to your experimentation and research teams that are rolling out at Conductrics. 

To learn more contact us here.

Seemingly simple ideas underpinning AB Testing are confusing. Rather than getting into the weeds around the definitions of p-values and significance, perhaps AB Testing might be easier to understand if we reframe it as a simple ruling out procedure.

Ruling Out What?

There are two things we are trying to rule out when we run AB Tests

1. Confounding
2. Sampling Variability/Error

Confounding is the Problem Random Selection is a Solution

What is Confounding?

Confounding is when unobserved factors that can affect our results are mixed in with the treatment that we wish to test. A classic example of potential confounding is the effects of education on future earnings. While people who have more years of education tend to have higher earnings, a question Economists like to ask is if extra education drives earnings or if natural ability, which is unobserved, causes how may years of education and amount of earnings people receive. Here is a picture of this:

We want to be able to test if there is a direct causal relationship between education and earnings, but what this simple DAG (Direct Acyclic Graph) shows is that education and earnings might be jointly determined by ability – which we can’t directly observe. So we won’t know if it is education that is driving earnings or if earnings and education are just an outcome of ability.

The general picture of confounding looks like this:

What we want is a way to break the connection between the potential confounder and the treatment.

Randomization to the Rescue
Amazingly, if we are able to randomize which subjects are assigned to each treatment we can break, or block, the effect of unobserved confounders and we can make causal statements about the treatment on the outcome of interest.

Why? Since the assignment is done based on random draw, the user, and hence any potential confounder is no longer mixed in with the treatment assignment. You might say that the confounder no longer gets to choose its treatment. For example, if we were able to randomly assign people to education, then high and low ability students each would be just as equally likely to be in the low and high education groups, and their effect on earnings would balance out, on average, leaving just the direct effect of education on earnings. Random assignment lets us rule out potential confounders, allowing us to focus just on the causal relationship between treatment and outcomes*.

So are we done? Not quite. We still have to deal with uncertainty that is introduced whenever we try to learn from sample observations.

Sampling Variation and Uncertainty

Analytics is about making statements about the larger world via induction – the process of observing samples from the environment, then applying the tools of statistical inference to draw general conclusions. One aspect of this that often goes underappreciated is that there is always some inherent uncertainty due to sample variation.  Since we never observe the world in its entirety, but only finite, random samples, our view of it will vary based on the particular sample we use. This is reason for the tools of statistical inference – to account for this variation when we try to draw out conclusions.

A central idea behind induction/statistical inference is that we are only able to make statements about the truth within some bound, or range,  and that bound only holds in probability.

For example, the true value is represented as the little blue dot But this is hidden from us.

Instead what we are able to learn is something more like a smear.

The smear tells us that the true value of the thing we are interested in will lie somewhere between x and x’ with some P probability. So there is some P probability, perhaps 0.05, that our smear won’t cover the true value.

This means that there are actually two inter related sources of uncertainty:

1) the width, or precision, of the smear (more formally called a bound)

2) the probability that the true value will lie within the smear rather than outside of its upper and lower range.

Given a fixed sample (and a given estimator), we can reduce the width of the smear (make it tighter, more precise), only by reducing the probability that the truth will lie within it  – and vice versa, we can increase the probability that the truth will lie in the smear only by increasing (make it looser, less precise) its width. This is a more general concept that the confidence interval is an example of – we say the treatment effect is likely within some interval (bound) with a given probability (say 0.95). We will always be limited in this way. Yes we can decrease the width, and increase the probability that it holds by increasing our sample size, but it is always with diminishing returns [in the order of O(1/sqrt(n)].

AB Tests and P-values To Rule Out Sampling Variations

Assuming we have collected the samples appropriately, and certain assumptions hold, by removing potential confounders there will now be just two potential sources of variation between our A and B interventions:

1) the inherent sampling variation that is always part of sample based inference that we discussed earlier; and
2) a causal effect – the effect on the world that we hypothesize exists when doing B vs A.

AB tests are a simple, formal process to rule out, in probability, the sampling variability.  Through the process of elimination if we rule out the sampling variation as the main source of the observed effect (with some P probability), then we might conclude the observed difference is due to a causal effect. The  P-value ( the probability of seeing the observed difference, or greater, just due to random sampling) – relates to the probability that we will tolerate in order to rule out that the sampling variation is a likely source for the observed difference. 

For example, in the first case we might not be willing to rule out sampling variability, since our smears overlap with one another – indicating that the true value of each might well be covered by either smear.

However in this case, where our smears are mostly distinct from one another, we have little evidence that the sampling variability is enough to lead to such a difference between our results and hence we might conclude the difference is due to a causal effect.

So we look to rule out in order to conclude**

To summarize, causal statements, via AB Tests/RCTs, randomize treatment selections to generate random samples from each treatment in order to block confounding so that we can safely use the tools from statistical inference to make causal statements .

* RCTs are not the only way to deal with confounding. When studying the effect of education on earnings, un able to run RCTs, Economists used the method of instrumental variables to try to deal with confounding in observational data.

**technically ‘reject the null’ – think of Tennis if ‘null’ trips you up – it’s like zero. We ask ‘Is there evidence, after we account for the likely difference due to sampling’ to reject that the difference we see, e.g the observed difference in conversion rate between B and A, is likely due to just sampling variations.

***If you want to learn about other ways of dealing with confounding beyond RCTs a good introduction is Causal Inference: The Mixtape – by Scott Cunningham.

As AB testing becomes more commonplace, companies are moving beyond thinking about how to best run experiments to how to best set up and run experimentation programs. Unless the required time, effort, and expertise is invested into designing and running the AB Testing program, experimentation is unlikely to be useful.

Interestingly, some of the best guidance for getting the most out of experimentation can be found in a paper published almost 45 years ago by George Box. If that name rings a bell, it is because Box is attributed with coining the phrase “All models are wrong, but some are useful”. In fact, from the very same paper that this phrase comes from we can discover some guiding principles for running an a successful experimentation program.

In 1976 Box published “Science and Statistics” in the Journal of the American Statistical Association. In it he discusses what he considers to be the key elements to successfully applying the scientific method. Why might this be useful for us? Because in a very real sense, experimentation and AB Testing programs are the way we implement the scientific method to business decisions. They are how companies DO science. So learning about how to best employ the scientific method directly translates to how we should best set up and run our experimentation programs.

Box argues that the scientific method is made up, in part, of the following:
1) Motivated Iteration
2) Flexibility
3) Parsimony
4) Selective Worry

According to Box, the attributes of the scientific method can best thought of as “motivated iteration in which, in succession, practice confronts theory, and theory, practice.” He goes on to say that, “Rapid progress requires sufficient flexibility to profit from such confrontations, and the ability to devise parsimonious but effective models [and] to worry selectively …”.

Let’s look at what he means in a little more detail and how it applies to experimentation programs. 

Learning and Motivated Iteration

Box argues that learning occurs through the iteration between theory and practice. Experimentation programs formalize the process for continuous learning about marketing messaging, customer journeys, product improvements, or any other number of ideas/theories. 

Box: “[L]earning is achieved, not by mere theoretical speculation on the one hand, nor by the undirected accumulation of practical facts on the other, but rather by a motivated iteration between theory and practice. Matters of fact can lead to a tentative theory. Deductions from this tentative theory may be found to be discrepant with certain known or specially acquired facts. These discrepancies can then induce a modified, or in some cases a different, theory. Deductions made from the modified theory now may or may not be in conflict with fact, and so on.”

As part of the scientific method, experimentation of ideas naturally requires BOTH a theory about how things work AND the ability to collect facts/evidence that may or may not support that theory. By theory, in our case, we could mean an understanding of what motivates your customer, why they are your customer and not someone else’s, and what you might do to ensure that they stay that way. 

Many times marketers purchase technology and tools in an effort to better understand their customers. However, without a formulated experimentation program, they are missing out on one half of the equation. The main takeaway is that just having AB Testing and other analytics tools are not going to be sufficient for learning. It is vital for YOU to also have robust theories about customer behavior, what they care about, and what is likely to motivate them. The theory is the foundation and drives everything else. It is then through the iterative process of guided experimentation, that then feeds back on the theory and so on, that we establish a robust and useful system for continuous learning. 

Flexibility

Box;  “On this view efficient scientific iteration evidently requires unhampered feedback. In any feedback loop it is … the discrepancy between what tentative theory suggests should be so and what practice says is so that can produce learning. The good scientist must have the flexibility and courage to seek out, recognize, and exploit such errors … . In particular, using Bacon’s analogy, he must not be like Pygmalion and fall in love with his model.”

Notice the words that Box uses here: “unhampered” and “courage”. Just as inflexible thinkers are unable to consider alternative ways of thinking, and hence never learn, so it is with inflexible experimentation programs. Just having a process for iterative learning is not enough. It must also be flexible. By flexible Box doesn’t only mean it must be efficient in terms of throughput. It must also allow for ideas and experiments to flow unhampered, where neither influential stakeholders nor the data science team holds too dearly to any pet theory. People must not be afraid of creating experiments that seek to contradict existing beliefs, nor should they fear reporting any results that do.  

Parsimony

Box: ”Since all models are wrong the scientist cannot obtain a “correct” one by excessive elaboration. On the contrary, following William of Occam [we] should seek an economical description of natural phenomena. Just as the ability to devise simple but evocative models is the signature of the great scientist so over elaboration and overparameterization is often the mark of mediocrity.”

This is where the  “All Models are Wrong” saying comes from! I take this to mean that rather than spend effort seeking the impossible, we should instead seek what is most useful and actionable –  “how useful is this model or theory in helping to make effective decisions?”

In addition, we should try to keep analysis and experimental methods as simple as required for the problem.  Often companies can get distracted, or worse, seduced by a new technology or method that adds complexity without advancing the cause.  This is not to say that more complexity is always bad, but whatever the solution is, it should be the simplest one that can do the job. That said, the ‘job’ may be really for signaling/optics rather than to solve a specific task. For example, to differentiate a product or service as more ‘advanced’ than the competition, regardless if it actually improves outcomes. It is not for me to say if those are good enough reasons for making something more complex, but I do suggest being honest about it and going forward forthrightly and with eyes wide open. 

Worry Selectively

Box: “Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.”

This is my favorite line from Box. Being “alert to what is importantly wrong” is perhaps the most fundamental and yet underappreciated analytic skill. It is so vital not just in building an experimentation program but for any analytics project to be able to step back and ask “while this isn’t exactly correct, will it matter to the outcome, and if so by how much?” Performing this type of sensitivity analysis, even if informally in your own mind’s eye, is an absolutely critical part of good analysis. You don’t have to be an economist to think, and decide at the margin.

Of course if something is a mouse or a tiger will depend on the situation and context. That said, in general, at least to me, the biggest tiger in AB Testing is fixating on solutions or tools before having defined the problem properly. Companies can easily fall into the trap of buying, or worse, building a new testing tool or technology without having thought about: 1) exactly what they are trying to achieve; 2) the edge cases and situations where the new solution may not perform well; and 3) how the solution will operate within the larger organizational framework. 

As for the mice, they are legion.  They have nests in all the corners of any business, whenever spotted causing people to rush from one approach to another in the hopes of not being caught out.  Here are a few of the ‘mice’ that have scampered around AB Testing:

• One Tail vs Two Tails (eek! A two tailed mouse – sounds horrible)
• Bayes vs Frequentist AB Testing
• Fixed vs Sequential designs
• Full Factorial Designs vs Taguchi designs

There is a pattern here. All of these mice tend to be features or methods that were introduced by vendors or agencies as new and improved, frequently over-selling their importance, and implying that some existing approach is ‘wrong’.  It isn’t that there aren’t often principled reasons for preferring one approach over the other. In fact, often, all of them can be useful (except for maybe Taguchi MVT – I’m not sure that was ever really useful for online testing) depending on the problem. It is just that none of them, or others, will be what makes or breaks a program’s usefulness.

The real value in an experimentation program are the people involved, and the process and culture surrounding it – not some particular method or software.  Don’t get me wrong, selecting software and the statistical methods that are most appropriate for your company matters, a lot, but it isn’t sufficient.  I think what Box says about the value of the statistician should be top of mind for any company looking to run experimentation at scale:
 “… the statistician’s job did not begin when all the work was over-it began long before it started. …[The Statistician’s] responsibility to the scientific team was that of the architect with the crucial job of ensuring that the investigational structure of a brand new experiment was sound and economical.” 

So too for companies looking to include experimentation into their workflow. It is the experimenter’s responsibility to ensure that the experiment is both sound and economical and it is the larger team’s responsibility to provide an environment and process, in part by following Box, that will enable their success.  

If you are looking to upgrade your AB Testing software and improve your experimentation program please contact us here to learn more.

What’s the impact on your Conductrics implementation?

As you are likely aware, many browsers have begun to restrict the ability to “track” visitor behavior, in an effort to protect visitor privacy. The focus is especially on third-party scripts that could track users as they move between totally unrelated sites on the Internet.

Apple’s efforts in this regard are particularly well-known, with the introduction of their Intelligent Tracking Prevention (ITP) policies.

• ITP has gone through several revisions, each placing additional restrictions on how visitor information can be stored and for how long.
• While ITP is most well-known for affecting Safari users on iPhone and iPad, it also affects other browsers on those devices, such as Firefox for iOS / iPadOS. Safari users on the Mac are also affected.

While Conductrics has never engaged in any sort of visitor-tracking outside of your own agents/tests, ITP and other similar restrictions are now a fact of life. There will be some impact on what you can do with Conductrics (or other similar service).

When Using Client-Side Conductrics in Web Pages

When you use Conductrics Express or our Local JavaScript API in your pages, your Conductrics tests/agents do their work locally in the browser. They also store their “visitor state” information locally in the browser.

This visitor state information includes the variation assignments per agent (such as whether the visitor has been selected for the “A” or “B” variation for each of your tests), and some other information such as which goals/conversions have been reached and whether any visitor traits have been set.

ITP says that this visitor state information will be cleared between a user’s visits to your pages, if more than 7 days have passed since their last visit. For visit types, the visitor state will be cleared after one day (the one-day rule is triggered if visitors arrive at your site via link decoration, common in social media campaigns).

What does this all mean?

• If a visitor goes 7 days (*) or more between visits, your client-side Conductrics implementation will see the visitor as a “new” visitor.
• So, after 7 days (*), the visitor might get a different variation than on their previous visit(s). They would also be counted again in the Conductrics reporting.
• If the visitor hits a goal/conversion event, but it’s been 7 days (*) since they were exposed to a variation for a given test/agent, the goal/conversion will not be counted.

How should we change our testing or optimization practice?

• For A/B or MVT testing, try to focus on tests that would reasonably be expected to lead to a conversion event within 7 days (*).
• Generally speaking, focus on tests or optimizations where it wouldn’t be “absolutely terrible” if the visitor were to be re-exposed to a test, possibly getting a different variation.
• You could consider using rules/conditions within your Conductrics agents such that Safari browsers are excluded from your tests. However, that will likely reduce the number of visitors that you do expose (particularly on mobile), probably requiring you to run the test for longer, and also possibly “skewing” your results since you’d be likely excluding most visitors on Apple devices.
• You could consider looking at ITP vs non-ITP browsers (or iOS / iPadOS vs non) when evaluating the results of a test, to see if there are any important, noticeable differences between the two groups for your actual visitors.
• Conversely, Conductrics could be configured to treat all visitors as if they were subject to ITP’s “7 day rule” (or one day), even on browsers that don’t currently impose such restrictions by default (thus leveling the field between ITP and non-ITP browsers). Contact Conductrics to discuss.

(*) The “7 day rule” might actually be one day, depending on whether the visitor lands on your site via “link decoration” (that is, via a URL that has identifiers included in query parameters or similar). See this article from the WebKit team for details.

Frequently Asked Questions

These frequently asked questions are about client-side Conductrics usage.

Q: How is the Visitor State information stored?

A: By default, client-side Conductrics stores its visitor state information in the browser’s “Local Storage” area. Alternatively, it can be configured to instead store the information as a cookie. The main reason to choose the cookie option is to allow the visitor state to be shared between subdomains. ITP is in effect in equal measure regardless of whether your Conductrics implementation is set to use Local Storage or Cookies.

Q: What does Conductrics do to work around ITP?

A: We don’t try to defeat or “work around” ITP or similar policies. The WebKit team has made it very clear that ITP is a work in progress and will continue to address any loopholes. Rather than try to implement workarounds that would likely be short-lived, we think our customers will be better served if we focus on helping you make sense of your test results in an ITP world.

Q: Which browsers are affected by ITP?

A: Technically, ITP is implemented in WebKit browsers, which includes Safari on mobile and desktop. That said, other browsers such as Firefox have similar policies, so often the term “ITP” is used colloquially to refer to any browser restrictions on cookies and other browser-based storage and tracking.

When Using the Conductrics REST API

All of the above is about using client-side Conductrics. If you use the Conductrics REST-style API, the “visitor state” information is retained on the Conductrics side, using an identifier that you pass in when calling the API.

Because the identifier is “on your side” conceptually, whether your REST API tests are affected by ITP will depend on how you store the identifier that you provide to Conductrics.

• If the identifier is associated with a “logged in” or “authenticated” session, it is probably stored in some way on your side such that it can live for a long time, and thus would not be affected by ITP.
• If the identifier is being stored as a first-party cookie by your client-side code, it is also subject to ITP or similar policies, so your Conductrics tests will be affected to more or less the same degree as discussed above. However, if the identifier is being stored as “server cookie” only (with the HttpOnly and Secure flags on the cookie itself), then it is probably not affected by ITP.
• For native mobile apps (or other devices or systems such as kiosks, set-tops, or IVRs), you probably have an identifier that is stored “natively”, without having anything to do with web pages, so your implementation would probably not be affected by ITP.

 Questions?

As always, feel free to contact Conductrics regarding ITP, visitor data, or any other questions you may have. We’re here to help!

December 1, 2020 – Austin, Texas – 

Conductrics, a digital experimentation and artificial intelligence (AI) SaaS company, announces its partnership with Search Discovery, a premier data transformation company. Search Discovery will now offer Conductrics optimization technology along with industry-leading optimization consulting support.

Together, the two companies will offer clients a superior integrated solution that satisfies a market need: Clients across industries are searching for both technological solutions and strategic guidance to help drive their internal innovation and growth. This partnership will make it simple for clients to work smarter and faster, use better experimentation techniques, and leverage both Conductrics’ and Search Discovery’s core competencies to build best-in-class optimization and experimentation programs.

Conductrics offers robust and flexible experimentation software that supports the specific requirements of Marketing, Product, and IT departments. Teams are able to seamlessly manage and quickly deploy their experiments using Conductrics’ integrated communication and implementation tools. 

Search Discovery provides strategic consulting for clients to manage and run successful experimentation and personalization programs at scale.

“Aside from the natural business fit, our two teams work well together. The expert team at Search Discovery has an impressive track record of helping A-list clients build and grow world-class optimization programs,” comments Matt Gershoff, Conductrics’ co-founder and CEO. “This new partnership will enable us to provide clients with the optimal combination of technology and experimentation expertise.”

“The Conductrics platform has optimal flexibility, transparency, and the power needed to help us support our clients’ data-driven decision-making across every digital experience—even in today’s increasingly complex privacy environment,” says Kelly Wortham, Search Discovery’s Senior Optimization Director. “The Conductrics team’s ability to quickly customize the platform with our clients’ rapidly changing requirements makes this partnership even more exciting for Search Discovery.”

About Conductrics

In 2010, Conductrics released one of the industry’s first REST APIs for delivering AB Testing, multi-arm bandits, and predictive targeting to empower both Marketing and IT professionals. With Conductrics, marketers, product managers, and consumer experience stakeholders can quickly and easily optimize the customer journey, while IT departments benefit from the platform’s simplicity, ease of use, and integration with existing technology stacks. Visit Conductrics at www.conductrics.com. 

About Search Discovery

Search Discovery is a data transformation company that helps organizations use their data with purpose to drive measurable business impact. Their services and solutions help global organizations at every stage of data transformation, including strategy, implementation, optimization, and organizational change management. Search Discovery delivers efficient operations, deeper insights, and improved-decision making across marketing, sales, finance, operations, and human resources. Visit searchdiscovery.com.

Nate Weiss, Conductrics CTO, demonstrates key new features and functionalities of the latest software release.  This latest release focuses on streamlining and improving the user experience for faster and easier execution of optimization programs.

The new features that are demonstrated include: 

• A revamped and simplified user experience
• Streamlined A/B Testing/Optimization workflows
• Improved Program Management tools
• Upgraded API and mobile developer libraries

 SEPTEMBER 24, 2020 – AUSTIN, TX – Digital experimentation and Artificial Intelligence (AI) SaaS company Conductrics, today announced its latest major release of its experience optimization platform built expressly for marketers, developers, and IT professionals. The updated platform, which shares the company’s name, is a cloud-based A/B testing and decision optimization engine. This latest release focuses on streamlining and improving the user experience for faster and easier execution of optimization programs.   

These upgrades make it easier for clients to scale optimization programs across different organizational and functional teams in order to deliver ideal digital experiences for their customers. The new features include: 

• A revamped and simplified user experience (UX), 
• Streamlined A/B Testing and Optimization workflows, 
• Improved Program Management tools,  
• Upgraded API and mobile developer libraries.

“Since our start in 2010, our goal has been to make it faster and easier for developers and marketers to work together in order to confidently discover and deliver the best customer experiences,” says Matt Gershoff, Conductrics’ co-founder and CEO. “As other technologies morph and become increasingly more complex, we remain focused on developing accessible, leading-edge optimization and experimentation technology.” 

The new release will be available in mid-October.  Current clients will have the option to use the legacy platform or the new platform – no action is needed on their part. A webinar will be held on October 13^th to demonstrate the new features and benefits – a link to register is on the company website.

About Conductrics

In 2010, Conductrics released one of the industry’s first REST APIs for delivering AB Testing, multi-arm bandits and predictive targeting to empower both Marketing and IT professionals. With Conductrics, marketers, product managers, and consumer experience stakeholders can quickly and easily optimize the customer journey, while IT departments benefit from the platform’s simplicity, ease of use, and integration with existing technology stacks. Visit Conductrics at www.conductrics.com. 

For more information, contact press@conductrics.com

Conductrics has a history of providing innovative approaches to experimentation, testing and optimization.  In 2010, we introduced one of the industry’s first REST API for experimentation which also supported multi-arm bandits and predictive targeting.  Continuing our goal to provide innovative solutions, we’ve developed Express Macros and Templates to safely and easily build tests at scale.

To illustrate, let’s say you frequently run headline testing on a news page or change certain areas of a landing page over and over. In such situations, you don’t want test authors to have full page-editing capabilities – you just want them to have access to specific sections of the page or the site. Additionally, because variations of the same basic test will be conducted over and over, it is imperative that the test setup is simple, easy to repeat and scaleable.

Conductrics Templates and Express Macros

Conductrics Templates and Express Macros are an answer to this. They are an easy way to create reusable forms that allow your team to set up and run multiple versions of similar tests just by filling out a simple form and a click of a button. 

EXAMPLE: When to Use Conductrics Express Macros

One of our national media clients wanted to optimize News headlines for their news homepage. This meant that rather than just provide a single Headline for each news article, the client wanted to try out several potential Headlines for each story, see which worked best, and then use the winning headline going forward. To do this, they wanted to take advantage of Conductrics multi-armed bandits, which automatically discover, and deploy the most effective Headline for each story. 

However, they have scores of articles each day, so they needed to be able to run hundreds of these tests every month. In addition, these tests needed to be set up by multiple, non-technical editors safely, so as to not risk breaking the homepage. 

This is where Express Macros helped. Macros let the client extend and customize the Conductrics test creation process by:

1) creating simple custom forms to make it easy to set up each test; and

2) applying custom JavaScript to that form in order for the test to execute properly. 

How the Express Macro Works

For example, this macro will create a form with two input fields, “Post Id” and “Article Headline”. You can, of course, create any number of fields that are needed.

Now that we have specified what data to request from the test creator, we now just need to provide the JavaScript that will use the values taken in by the Macro’s form to run the test. In this case we will want to use the ‘Post Id’ (alternatively, this could be a URL or some other identifier) to tell Conductrics on which article to run the test. We also include in our JavaScript logic to swap in the alternative Headline(s) for the test. 

Here is an example of what that might look like: 

While this might look complicated if you don’t know JavaScript, don’t worry, this is something any front-end developer can do easily (or you can just ask us for help). 

All that there is left to do is to name and save it. I have named it ‘Headline Optimization’.

There is just one last step before we can let our Headline editors start to run tests, and that is to assign the Macro to a Template. 

Template Example

Express Macros was developed to bridge the workflows of programmers and non-technical users.  So now that the Macro has been created by the programmer, it has been assigned/converted to a template for use by non-technical users. This makes the process easy-to-use, scaleable, reproducible, and secure.  

Creating a Template is just like setting up any Conductrics Agent. The only difference is that by assigning the Agent to a Macro, it will become a Template that can be used to generate new tests easily. 

For example, here I have created an Agent name ‘Headline Optimization’. In the bottom portion of the set-up page, I select ‘Template’. This brings up a list of all of the Macros I am authorized to use. In this case, there is just the ‘Headline Optimization’ Macro we just created. By selecting this Macro, the Agent will be converted into a Template for all of the ‘Headline Optimization’ tests going forward. 

Now comes the amazing part. All the test creator needs to do is go to the Conductrics Agent List Page, and they will see the custom button created for Headline Tests. 

Clicking the ‘Headline Optimization’ button will bring up the custom form. For our simple ‘Headline Optimization’ example, it looks like this:

Notice that it asks for two pieces of information, the Post Id and the alternative Article Headline to test (you can add multiple alternative headlines to each test using the ‘Add Another’ option).

Once the Post Id and the alternative headline are entered, the test author just clicks ‘Create’ and that’s it! The test will be scheduled and pushed live. 

Not only does this make it super simple for non-technical users to set up hundreds of these tests easily, but it also provides guard rails to prevent accidental, or unintended, selections of erroneous page sections. 

Notifications

Communication of test results are automated with Conductrics notification streams.  Users receive top-level results of each ‘Headline Optimization’ test directly to their Slack channel including company members who are not Conductrics users.  So all relevant stakeholders can also be part of the discussion around what types of Headlines seem to be most compelling and effective.

Here is a simple example – once the Conductrics bandit algorithm has selected a winner, a notification like the following is sent to the client’s Slack with the following summary information.

The winning variation is noted, along with the number of visitors and the click through rate for each headline. 

Conclusion

In this example, the client was able to scale from a handful of tests per month to hundreds of tests per month, and the guard rails allowed multiple non-technical users to have more control over the testing while freeing the developers to work on more complex problems. 

Express Macros and Templates are the ideal solution for digital marketers and CX professionals who have multiple, repeatable versions of a particular test design. They streamline the process, allow for set up in an easy-to-use form, and ensure compliance by controlling what can be modified. Express Macros solve the problem of so many ideas, so little time. If you would like to learn more about Conductrics Express Macros and Templates, please contact us.