Fire Safety Engineering – An Audience & Data Group Case Study//

Conditional Content Testing

Team
Technology for Marketing &
Advertising

Challenge

1) For TFM 2009 the team were keen to improve recipient engagement
by conditionalising their emails so that they focused on an aspect of
the show matching recipient’s areas of interest.

2) Produce a conditional content process that the Audience & Data

Group can successfully roll out and support.
Results

1) Test results showed that the TFM audience were more receptive to
an email that covered all aspects of the event rather then an email
that focused on only an area that they had previously expressed
interest in.

2) Test results showed that the TFM audience put greater trust in the
TFM brand name then they did that of a keynote speaker.

3) Problems arose prohibiting Online Execs from completing their

Conclusion

Conditional content:

1) Cannot at present be supported by email polices and procedures

imposed upon ADG

2) Cannot at present by carried out by marketing teams without first

being given training on how to correctly apply the HTML code used
to drive the conditional content, and secondly being willing to
allocate a considerable amount of time to its application and
testing.

3) Can be valuable for gaining better insight into customer behaviour

and hopefully in future once relevant segments have been
For more
identified information,
be valuable get incustomer engagement.
for improved
touch by email
Ian.Strong@ubm.com
A UBMi Case Study:

Conditional Content

FSE Background 2

What is the Newsletter Model and what are its

benefits? 3-5

Who is eligible for migration on to the Newsletter

model? 6

Challenge 6

Solution 7-
8

Identifying crossover opportunities and

measuring their effectiveness
 Page 2 Conditional Content

Page 3 Soft Subscribe

Strategy
Background Soft Subscribe
Strategy

TFM&A is the UK’s only integrated marketing-solutions

event for marketing, media and advertising professionals.

As such in the run up to the 2009 event the Marketing team

were keen to ensure that their own campaigns practice what
they preach, namely clever and insightful segmentation and
targeting of its customers through its marketing
communications.

In order to do this the team were keen to segment their

customer base by areas of the show that people had
previously expressed an interest in; examples included
Online Advertising and CRM. These segments would then be
sent an email that focused on an aspect of the show
matching their areas of interest.

The number of segments they wished to target was

numerous (5 for one driver) and as such the team were
concerned that they would not be able to secure the job
slots to fulfil them or have the time to produce the
numerous different creatives required for each. As a result
the team were keen to see if they could pioneer the use of
conditional content via the OTIS and MessageFocus system.

A conditional content campaign is defined by the Audience &

Data Group as being an email campaign in which multiple
campaign creatives can be sent all driven by a unique code
in the creative that is specific to a segment. This means
Online Advertising people can be sent an email with an
Online Advertising focus and CRM people can be sent an
email with a CRM focus all through the same SMS job and
email template in MessageFocus.

At present the Audience & Data Group does not support

conditional content as part of a standard email job. As a
result if a marketer wishes to send an email to its marketers
with a marketing slant and an email to its sales segment
with a sales slant it would need to book in two jobs and
produce two creative for them.
 The aim therefore of the test is to find out if conditional
content:

A. can be supported by the ADG team

B. can be carried out successfully by marketing teams
C. has a significant impact upon campaign metrics

Page 3 Conditional Content

Success Criteria

The success of the conditional content test will be measured

against four sets of criteria:

1) Does each segment receive a different creative

2) Can a conditional content job be completed under the
agreed email fulfilment procedures?
3) Is the time and effort involved in fulfilment of a
conditional content campaign shorter than that
involved in fulfilling individual campaigns for
marketers
4) Does the use of a different creative for each segment
have a significant impact on campaign results?

The Challenges

The main challenges facing both the TFM and ADG team
were to find a way to;

1) Produce mutually exclusive segments in OTIS. A record

can only appear in one segment so as to ensure that it
receives the correct creative.

2) Use the cell codes created in OTIS to drive the

conditional content in MessageFocus. Cell codes are the
only aspect of a file that can be amended for conditional
content and be referenced in MessageFocus.
 3) Test that the conditional content is correct for each
segment. The current approval process allows users to
review only one email creative per campaign. However for
conditional content campaigns teams would need to
review all the types of email they wish to send.

4) Review results and identify if conditional content had an

impact upon campaign metrics. Campaign metrics in
MessageFocus are reported at campaign or launch ID
level only. As a result for conditional content campaigns
results are consolidated together preventing us from
being able to identify the performance of individual
segments.

Page 4 Conditional Content

5) In addition the ADG team would also, dependent upon

the results, be required to find a way of rolling this
functionality out to the entire business.

The Solution

1) Mutually exclusive segments can be created by the TFM

team in OTIS using exclusion queries to suppress records
higher up in the dedupe hierarchy. (See below) As such
segments and files need to be created in hierarchical
order.

(A query with an exclusion file included)

 2) Conditional content will be driven by the cell code that
the user has appended to the data in OTIS (See below)
and subsequently referenced within the HTML code they
apply to their MessageFocus template.

Contact Ian.strong@ubm.com for further details on how to

Page 5 append cell codes. Conditional Content

3) MessageFocus automatically dedupes by email address,

as such a single user is unable to receive the different
iterations of the same campaign in their inbox to approve.
The only way of reviewing all iterations of a multi creative
campaign is for TFM to book in a specific approval job in
addition to their final email job. This will be done the day
preceding the real email launch and the Online Execs will
be able to amend the MessageFocus dedupe process
enabling the approver to receive all iterations of the multi
creative content and review.

4) The SSRS report “Activity Summary Report- Email” allows

teams to break their email campaigns down to cell code
level. This means that users who append cell codes to
their OTIS files can now go into this report and compare
the performance of their segments. Full details of how to
use this report can be found on the Wiki under SSRS or by
contacting Ian.strong@ubm.com.
 In order to identify if multi creative content had an
impact on campaign metrics or not a control group of 10%
will be created for each segment TFM produce.

The Click Open and Unsub Open results of the 10%

sample will then be compared with the results of the rest
of the segment (the remaining 90%) for statistical
difference by:

a) Calculating an estimated level of difference for each

campaign metric e.g. Click Open Rate.
The estimated difference is the minimum
difference that would need to exist between
the two results for us to infer that differences
can be explained by more then sampling
error.

b) Calculating the actual difference between the segments

i.e. Click rates for control group = 1.7%, Click Rates of
rest of segment = 1.0% so actual difference = 0.7%.

A sampling error is the error caused by observing a sample instead of the whole population;
e.g. it may transpire that because our non personalised email was sent to a random sample
that all these randomly selected people were women. Women may typically not open up
emails as often as men and so we need to factor in an error to account for that possibility.

Contact Ian.strong@ubm.com for more details on

calculating estimated
Page 6 differences. Conditional Content

Results
We carried out 3 conditional content tests with the TFM
team over the course of their 2009 driver campaign and 1
conditional from name test.

In each test we were able to successfully test and deliver a

different creative to each cell code however during the
campaign launch it was identified that:

• No Unsub link was being recognised in the HTML Check

list
• ADG could not run a spam check in MessageFocus
• An error page was encountered when an approval was
sent to teams
• The time taken for TFM to produce the different
segments in OTIS and append the correct code in
MessageFocus was substantial

The nature of segmentation and content within each

creative differed across each test and accordingly we saw
the results differ also.

Conditional content Test 1

In this test the TFM team segmented their data by the

following areas of interest:

a. Digital Marketing
b. Advertising
c. CRM
d. Data
e. Marketing

90% of those records who fell into the above segments were
sent an email in which the:
1) Opening Paragraph
2) Leading suppliers
3) Industry News

aspects of the email were amended dependent upon the

recipient to focus on their expressed area of interest. I.e. a
person who had ticked area of interest as Digital Marketing
received an email where the Opening paragraph, leading
suppliers and industry news sections focused on all Digital
Marketing related content only.
Page 7 Conditional Content

The remaining 10% of records were sent an email in which

the Opening paragraph, leading suppliers and industry news
sections focused on all aspects of the TFM event. I.e. so a
person who had previously indicated they were only
interested in advertising would receive an email that talked
about all aspects of the event.

The consolidated results of the segments are shown below:

Targeted Non Targeted

Content Content
Bounced Rate 1.3% 1.5%
Delivery Rate 98.7% 98.5%
Open Rate 17.7% 17.7%
Click Open Rate 9.5% 9.7%
Unsub Open Rate 5.5% 5.6%
Spam Open Rate 0.5% 0.4%

Driver 1 2009 Metrics

20.0%

15.0%

10.0%

5.0%

0.0%
Bounced Open Rate Click Open Unsub Open Spam Open
Rate Rate Rate Rate
M ulti C rea tiv e C ontent Non M ulti C rea tiv e C ontent

Click Open Rate – Test for statistical significance

Estimated Difference 0.95
Actual Difference -0.20
Significant NO

Unsub Open Rate – Test for statistical significance

Estimated Difference 0.49
Actual Difference 0.20
Significant NO

Results showed that there was no statistically significant

difference in the Click Open or Unsub Open results across
the two strategies (See above) meaning amending the
opening paragraph, leading Suppliers, Industry News
 sections of an email creative to reflect a person’s area of
interest was no more successful at encouraging recipients to
act then a general email.

Page 8 Conditional Content

Conditional content Test 2

For driver 3 the TFM team were sending an email out to

promote their keynote sessions. As such they segmented
their data by the following areas of interest:

a. Those interested in Digital marketing, Data & CRM

b. Those interested in Digital Marketing only
c. Those interested in CRM only

90% of Segment a. received an email creative that promoted

both keynotes,
90% of Segment b received an email creative that promoted
the Digital Marketing keynote only.
90% of Segment c. received an email creative that promoted
the CRM keynote only.

The remaining 10% of records were sent a non targeted

email that promoted both keynote sessions.

The consolidated results of the segments are shown below:

 Driver 3 2009 Metrics

25.0%

20.0%

15.0%

10.0%

5.0%

0.0%
B ounced Rate Open Rate Click Open Rate Unsub Open Rate

Ta rgeted C ontent Non ta rgeted C ontent

Targeted Non targeted

Content Content
Bounced Rate 1.7% 1.1%
Delivery Rate 98.3% 98.9%
Open Rate 21.2% 18.0%
Click Open Rate 9.5% 11.6%
Unsub Open Rate 2.2% 2.4%

Click Open Rate – Test for statistical significance

Estimated Difference 1.23
Actual Difference -2.10
Significant Yes

Unsub Open Rate – Test for statistical significance

Estimated Difference 0.59
Actual Difference -0.20
Significant NO

Page 9 Conditional Content

The difference in Unsub Open Rates across the targeted

emails and non targeted emails was recorded at 0.2% which
was not statistically significant. However the difference in
Click Open Rates was recorded at 2.1% this exceeded the
estimated difference of 1.23% meaning that the difference
was statistically significant and that we can assume that the
conditional content did have a direct impact on the number
of clicks that the email attained.

It transpired though that the non targeted content out

performed the targeted content suggesting that people
were more receptive to the email that told them about both
 keynotes rather then the email that told them only about
the one they had previously expressed an interest in.

Conditional content Test 3

In driver 4 TFM were sending an email out to promote the

seminar halls they had available at the Event and the
seminars that were taking place in each hall. To replicate the
focus of the halls the team segmented the data by the
following areas of interest:

a. Online Advertising
b. CRM
c. Data
d. Web
e. Digital

90% of records in each of the above segments received an

email that presented the times and titles of seminars held in
the hall that covered their area of interest. I.e. anyone who
had expressed an interest in Online Advertising received an
email that contained a table with times and titles for
seminars held in the Online Advertising hall only.

The remaining 10% of records in each segment received a

generic timetable highlighting key talks from each section.

The consolidated results of the segments are shown below:

Targeted Non targeted

Content Content
Bounced 2.0% 2.0%
Delivery Rate % 98.0% 98.0%
Open Rate % 15.7% 14.8%
Click Open Rate % 7.5% 13.0%
Unsub Open Rate % 2.3% 1.7%

Page 10
Newsletter Model

Click Open Rate – Test for statistical significance

Estimated Difference 1.28
Actual Difference -5.50
Significant Yes

Unsub Open Rate – Test for statistical significance

Top Tip
Estimated Difference 0.51
Actual Difference 0.60
Significant YES

The significant difference in Click Open Rates of 5.5% that

existed across the targeted and non targeted content was
sufficient enough for us to deduce that the use of
conditional content did have an impact on campaign
metrics. However the results show that the email that
covered key parts from each seminar hall was more
successful at engaging its audience then the email that
attempted to engage its audience based only on the areas
they had previously expressed interest in.

We also see that the use of conditional content had an

impact on the Unsub Open rates too with Unsub Open Rates
for the targeted Content being significantly higher then
from the non targeted content. The decision to send seminar
specific content based on areas of interest appears to have
reduced Click and Increased Unsubscribes.

Conditional From Name Test 1 –From Name of Driver 3 (6th

January 2009)

In driver 3 the TFM team decided they would also like to test
Conditional “From Name”. In the email they were sending to
the CRM segment they obtained permission to use the
Keynote speakers name in the emails “From Name”.

As such we sent 90% of the CRM segment an email that was

addressed from Stuart Lauchlan and the remaining 10% of
the segment received the same email addressed from TFM.

The consolidated results are shown below:

Non
Conditional Conditional
From Name From Name
Bounced 1.4% 1.2%
Delivered 98.6% 98.8%
Opened 19.2% 18.8%
Clicked Open 11.1% 12.7%
Unsub Open 2.4% 0.6%

Side by side the results show differences across the

campaign metrics with Open and Unsub Open Rates higher
in the Conditional From Name campaign and Click Open
Rates higher in the non conditional from name campaign.

Open Rate – Test for statistical significance

Top Tip
Estimated Difference 2.78
Actual Difference 0.40
Significant NO

Click Open Rate – Test for statistical significance

Estimated Difference 2.35
Actual Difference -1.60
Significant NO

Unsub Open Rate – Test for statistical significance

Estimated Difference 0.63
Actual Difference 1.80
Significant YES

The differences in the Open and Click Open Rates were not
pronounced enough for us to deduce with any confidence
that the use of the keynote speakers name had any impact
upon campaign metrics. However the difference in Unsub
Open Rates of 1.8% was statistically significant and as such
we can say that using the keynote speakers name as the
From Name did result in more unsubscribes than using the
TFM brand name.

Conclusion

The overriding aims of the tests were to find out if

conditional content:

A. can be supported by the ADG team

B. can be carried out successfully by marketing teams
C. has a significant impact upon campaign metrics

We attempted to answer this by using the following success

criteria:

1) Does each segment receive a different creative?

2) Can a conditional content job be completed out under
the agreed email fulfilment policies and procedures?
3) Is the time and effort involved in fulfilment of a multi-
creative campaign shorter than that involved in
fulfilling individual campaigns for marketers?
4) Does use of a different creative for each segment have
a significant impact on campaign results?
It was very pleasing to see that in each of the 3 conditional
content tests the ADG and TFM team were able to
successfully deliver a different email creative to each
segment. However we have seen that this was only
achievable by completing the job outside of the UBMI agreed
email fulfilment policies and procedures. These checks are
in place to ensure that every email UBMi sends out is data
protection compliant and constructed in a way that will not
negatively affect our listing with email service providers
who monitor spam scores. Unfortunately for reasons that
are still unknown, MessageFocus prevented the Online Execs
from completing these on the conditional content
campaigns. As such this means that ADG cannot roll out the
use of conditional content on a wider premise until the bugs
are fixed for fear of launching emails that may contravene
the data protection act.

The concept of conditional content was designed to fulfil in

one email job what had previously only been possible in
multiple jobs. In turn freeing up resource with the ADG team
to complete more jobs for other teams. However because
there is no approval process in MessageFocus for checking
multiple creative a specific approval job was needed to be
booked by teams the day preceding the launch to review the
numerous iterations. As such conditional content is only
worth doing if you have 3 or more segments in your data.

Whether teams segment their data for use in a conditional

content campaign or across a series of individual campaigns
the process is the same. As such the use of conditional
content did not shorten the time it takes for teams to create
their data file nor would it ever. The real difference was
supposed to be experienced in the time it takes to product
the creative with amendments being shorter then the time
taken to produce numerous different creatives. The skills
required by the team to include conditional content were
new and had to be learnt and this combined with the fact
that numerous segments were being tested meant that the
time it took TFM to produce a functional conditional content
campaign was substantail. This process sped up as the test
progressed but was still felt to be very time consuming and
nowhere near as speedy as first hoped and anticipated.

For TFM the decision to segment their data by area of

interest and to use this to drive the conditional content in
their emails yielded some valuable insight. In test 1 the
minor email amendments made to the opening paragraph,
suppliers and industry news had no significant impact upon
campaign metrics and therefore recipient engagement.
However in test 2 and test 3 where more noticeable
conditionalisation was applied the results suggested with
some conviction that the TFM audience were more receptive
to emails that cover all aspects of the event rather then
emails that focus only on areas that recipients have
previously expressed an interest in. This would appear to
validate the previous marketing strategy carried out by the
TFM team and would also indicate that area of interest,
which was a multiple choice question and hence not
exclusive, is not a strong enough segmentation set for
conditional content to be tested on.

So in reflection it is fair to say that conditional content:

A. Cannot at present be supported by email polices and

procedures imposed upon ADG
B. Cannot at present by carried out by marketing teams
without first being given training on how to correctly
apply the HTML code used to drive the conditional
content, and secondly being willing to allocate a
considerable amount of time to its application and
testing.
C. Can be valuable for gaining better insight into
customer behaviour and hopefully in future once
relevant segments have been identified be valuable for
improved customer engagement.

Conclusion

The ADG team are now in dialogue with MessageFocus to

get a better understanding of the problems encountered
in the tests above. Together we will address and attempt
to achieve a solution that can enable conditional content
to be carried out by teams in line with UBMis current
email campaign fulfilment policies and practices.

The ADG team will also begin to formulate a

document/training session that can be used to educate
users around the business in the correct creation of
conditional content.

The ADG team will explore more suitable segments for

conditional content to be undertaken on.
 Newsletter Model

If you decide to do all this using the new newsletter model you
wont have to maintain the integrity of your lists any further as
Top Tip
the system will take care of any new additions/losers. Contact