PacSocial: Field Test Report

Max Nanis max@pacsocial.com @x0xMaximus Ian Pearce ian@pacsocial.com @peeinears November 15, 2011 Tim Hwang tim@pacsocial.com @timhwang

Introduction

The Pacic Social Architecting Corporation (PacSocial) has been focused on the development of technologies that enable large-scale shaping of social groupings and communities online. Our work has centered on the generation of socialbots swarms of automated, intelligent identities on platforms like Facebook and Twitter that interact, encourage, and provoke communities towards certain behaviors. The vision of this technology is to enable operators to actively mold and shape the social topology of human networks online to produce desired outcomes. The present report focuses on data that were collected during PacSocials most recent eld test concerning socialbots designed to operate on Twitter. An integral part of its development cycle, PacSocial conducts such eld tests regularly; the results of these tests help to benchmark performance and also guide further development. While past studies have shown that socialbots are ecient at fostering bot-human interaction on Twitter, this study is the second of two experiments aimed at assessing the ability of socialbots to inuence connection and interaction between two human users. Primarily due to an experimental design aw, we were unable to produce any signicant results from the rst of these human-human connection experiments. We addressed these design issues in the design of the present experiment, and for the rst time we are able to show signicant results concerning the ability of socialbots to inuence humanhuman connection online.

havior within target groups of users on Twitter. To do this, we tracked the tweet and follow activity of 2700 Twitter users over the course of 54 days, from September 19 to November 12, 2011. For the rst 33 days (the control period), no socialbots were deployed. That is, the control period is marked by the condition that users had no contact or interaction with our socialbots. Socialbots were deployed immediately following the control period, and we continued to track users activity over the next 21 days (the experimental period). To determine the socialbots eect on the target group, we compare user activity during the control period to that during the experimental period. Each user in the initial group of 2700 was randomly assigned to one of nine experimental groups. Each experimental group contained 300 users and one socialbot, making a total of nine socialbots: Botai . A socialbots experimental group is called its target group. Socialbots were programmed to operate strategically in ways intended to foster connection between users in their respective target groups. A socialbot assesses the follow network of its target group and operates accordingly, utilizing tactics that involve following, mentioning, tweeting, and retweeting.

Results

Metrics each fall into one of two categories. The rst category (bot-human interaction) concerns the socialbots ability to connect to and interact with 2 Design & Methodology other users. The second category (human-human The purpose of this study is to analyze the extent to interaction) includes metrics that measure the sowhich socialbots can aect the tweet and follow be- cialbots ability to connect users to each other. 1

3.1

Bot-human interaction

We measured the socialbots outgoing and incoming follow and mention activity over the course of the study. While socialbots were programmed to vary tweet and follow activity over the course of a day, variation in day-to-day activity was minimal. Because all socialbots in this study were programmed with the same settings, little variability was seen in outgoing activity across the socialbots as well. On average, socialbots each tweeted about 36 times and followed roughly 19 users each day. Over the course of the experimental period (21 days), socialbots were able to attract a total of 561 followers an average of about 62 followers per socialbot. Some socialbots were more successful at attracting followers than others (Fig. 1). Our most followed socialbot, Bota , was able to garner 92 followers, while our least followed socialbot, Bote , was only able to attract 45 followers over the experimental period. While socialbots focused primarily on gaining follows from users in their respective target groups, socialbots also attracted many users from outside of these groups. On average, socialbots were each able to gain follows from about 16 users in their respective target groups or roughly 5% of users in each target group.

Socialbots received an average of about 33 incoming tweets each, with varying gures across all socialbots. The most mentioned socialbot, Bota , received 75 incoming tweets, while the least mentioned socialbot, Botd , received only 12 such tweets (Fig. 2). Bota s higher number of incoming tweets here is partially due to longer conversations with users. While all but one of Botd s 12 incoming tweets were from dierent users, Bota s 75 incoming tweets were from 41 unique users. In one instance, for example, Bota had 13 exchanges with one single user.

Figure 2: Socialbots mentions and retweets throughout the experimental period.

3.2

Human-human interaction

Figure 1: Socialbot follower counts throughout the experimental period. We also measured the number of each socialbots incoming tweets over time. These are tweets from other users that retweet or mention the socialbots. 2

To gauge the socialbots eectiveness at connecting users to each other in their respective target groups, we tracked the follow activity between these users over the duration of the entire study. Users on Twitter can decide to follow or unfollow other users at any time. Therefore, the structure of social networks on Twitter changes frequently, and for various reasons. By tracking a users follow activity, we can measure the rate of connection between users in each target group during both the control period and experimental period. These data only include follows from one user to another user in the same target group follows between users and socialbots and connections to users outside the target group are not included. These connection rates are generated by comparing the number of connections at

the beginning and end of each period. During the control period, target groups saw an average connection rate of 0.626 new follows per day. In the experimental period, this average rate increased to 0.901 new follows per day, yielding a +43% change from the control period to the experimental period, averaged across all target groups (Fig. 3, 4).

Figure 3: Box-Whisker plot of Control group (no socialbots) compared to Experimental group (with socialbots). Top and bottom line are maximum and minimum observation respectively, box range of interquartile range, median shown by solid line.

Discussion

The purpose of this study was to measure socialbots ability to inuence the tweet and follow behavior within target groups of users on Twitter. In terms of interaction and connection between socialbots and users, results are comparable to and even slightly better than those of previous studies. Every socialbot in this study was able to engage with a handful of other users. Qualitatively, these interactions appear to be more sophisticated than those in previous studies. The friend-follower ratios of socialbots in this study are also greater than those observed in previous studies. In sum, socialbots remain consistently eective at interacting with users and attracting followers. And further, we believe that recent improvements to the codebase have increased socialbots eectiveness. 3

Research and development at PacSocial has only recently shifted from a focus on fostering bot-human connection to one on inuencing human-human connection, and we have been eager to test the performance of our most recent developments. This study marks the rst time we have been able to successfully collect meaningful data that describe the follow activity between other users in socialbots target groups. If the socialbots were successful in this respect that is, eective at connecting other users to each other we would expect to observe a greater rate of connection between users in the experimental period than in the control period. Results suggest that socialbots were indeed successful at inuencing connection in their respective target groups. After the socialbots were deployed, we observed the connection rates between users increase by an average of 43%. A 355% increase in connection rate was observed in the target group of our most successful socialbot, Bota . One tactic socialbots used to try to connect other users was to suggest to one user that he or she might be interested in following another particular user. A deeper look into the data suggests that this tactic was successful in some cases. For example, users U sera and U serb each began following the other after socialbot Bota tweeted, U sera check out my friend U serb . While not all socialbots in this study were quite as successful as Bota , and although each socialbot was able to connect only a relatively small portion of users from its target group, the ndings of this study are extremely signicant. These ndings indicate the rst successful attempts at automatically and programmatically shaping the topology of online communities. Further, while the scale of this study was relatively small, socialbots are designed to be light, ecient, and entirely automatic and thus, easily deployable in large swarms. We believe this study marks the rst step towards demonstrating the ability of such technologies to shape online communities at a large scale.

4.1

Future Work

Future development at PacSocial will likely focus on improving the tactics designed to connect other users. We have already begun to develop functionality that gives socialbots the ability to determine

Figure 4: Following rates in bot target network throughout control and experimental period. E.P. indicates the beginning of the Experimental Period. the general interests of other users. We hope that this new functionality will oer an additional layer of sophistication to existing tactics, and give rise to entirely new tactics as well.

4.2

Further Questions

While average success rates of socialbots in this study was good, performance varied across socialbots. Why some socialbots perform at much higher success rates than others is still unclear. Further study is required to determine what factors cause this variability.