BR1

BR2
BR3
BR4
BR5
BR6
BR7
BR
CM11
CM12
CM13
CM14
CM15
CM16
CM4
CM5
CR
CR2
CR3
CR4
CR44
CR5
CR6
CR7
CR
CR9
DA1
DA1
DA11
DA12
DA13
DA14
DA15
DA16
DA17
DA1
DA2
DA3
DA4
DA5
DA6
DA7
DA
DA9
E1
E1
E11
E12
E13
E14
E15
E16
E17
E1
E1W
E2
E3
E4
E5
E6
E7
E
E9
EC
EN1
EN2 EN3
EN4 EN5
EN6
EN7
EN
EN9
GU15
GU16
GU17
GU1
GU19
GU2
GU21
GU22
GU23
GU24
GU25
GU47
GU95
HA
HA1
HA2
HA3
HA4
HA5
HA6 HA7 HA
HA9
HP
lG1
lG1
lG11
lG2
lG3
lG4
lG5
lG6
lG7
lG
lG9
KT1
KT1
KT11
KT12
KT13
KT14
KT15
KT16
KT17
KT1
KT19
KT2
KT2
KT21
KT22
KT23
KT24
KT3
KT4
KT5
KT6
KT7
KT
KT9
N1
N1
N11
N12
N13
N14
N15
N16
N17
N1
N19
N2
N2
N21
N22 N3
N4
N5
N6
N7
N
N9
NW1
NW1
NW11
NW2
NW3
NW4
NW5
NW6
NW7
NW
NW9
RM1
RM1
RM11
RM12
RM13
RM14
RM15
RM16
RM17
RM1
RM19
RM2
RM2
RM3
RM4
RM5
RM6
RM7
RM
RM9
SE1
SE1
SE11
SE12
SE13
SE14
SE15
SE16
SE17
SE1
SE19
SE1P
SE2
SE2
SE21
SE22
SE23
SE24
SE25
SE26
SE27
SE2
SE3
SE4
SE5
SE6
SE7
SE
SE9
SL
SL1
SL2
SL3
SL4
SL9
SM1
SM2
SM3
SM4
SM5
SM6
SM7
SW1
SW11
SW12
SW13
SW14
SW15
SW16
SW17
SW1
SW19
SW1
SW2
SW2
SW3
SW4
SW5
SW6
SW7
SW
SW9
TN13
TN14
TN15
TN16
TW1
TW1
TW11
TW12
TW13
TW14
TW15
TW16
TW17
TW1
TW19
TW2
TW2
TW3
TW4
TW5
TW6
TW7
TW
TW9
UB1
UB1
UB11
UB1
UB2
UB3
UB4
UB5
UB6
UB7
UB
UB9
W1
W11
W12
W13
W14
W1 W2
W3
W4
W5
W6
W7
W
W9
WC
WD17
WD1
WD19
WD23
WD24
WD25
WD3
WD4
WD5
WD6
WD7
Data Hacking and the City
CASA Workshop
Steven Gray
Thursday 26th September 2013 - 9:30 - 12:30
OrderoftheMorning
9:30-10:00-Showcaseofwhatwedo
10:00-11:00-Coding--Collec8ngtheData
11:00-12:00-MoreCoding-VisualisingData
12:00-12:30-Ques8onandWrapup
Letsstartcoding!
Butrstafewthingsweneedtosortout
SeparateintoGroups
--SocialData(BestforBeginnerstocoding)
--TransportData
--EnvironmentalData
Rules
--Dontbeafraidtoaskques8onsifyoudontunderstand
--HavefunandEnjoyyourself
QuickShowcaseofmywork
Letsstartcoding!
Mac
--InstallXcodeCommandlinetoolsforyourMacOSVersion
--h$ps://developer.apple.com/downloads/
--InstallHomebrew(hVp://brew.sh/)
--Runonterminal:
ruby-e"$(curl-fsSLhVps://raw.github.com/mxcl/homebrew/go)"
--FollowhVp://madebyhoundstooth.com/blog/install-node-with-homebrew-on-os-x/
--brewinstallnode
NPMwillbeinstalledatthesame8measnode!
InstallNodeJS+NPM
Linux(Ubuntu)
--FollowhVps://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager
Windows
--Downloadandfollowinstruc8ons
InstallNodeJS+NPM
hVp://nodejs.org/download/
OpenNode.jsfromStartMenu-Programs-Node.js-Node.jsCommandPrompt
OrsearchforNode.jsCommandPromptonyoursystem(Windows8,8.1)
InstallModulesviaNPM(AllSystems)
UseNPMtoinstallthefollowingmodulesusedbythecollector.
Checkthatyouhavenpminstalledbyrunningnpm-v(itshouldreport1.3.*)
Runthefollowingtoinstall
npminstalloauthcolors
CollecIngSocialMediaData(TwiLer)-ARecipe
Whatyouwillneed:
1xTwiVerAccount
1xDeveloperKeyfromhLps://dev.twiLer.com/
Sprinklewithsomepa8enceand8meandin8meyoullbecollec8ng
datafromTwiVer.
Instruc8onscomingupsoonaierasmallexplanta8on.
Understandingwhatshappening
http://dev.twitter.com/doc
User Pro!les Tweet Database
H
T
T
P

G
E
T

R
e
q
u
e
s
t
R
e
s
u
l
t
s
{"results":[
{"text":"@twitterapi is great!",
"to_user_id":396524,
"to_user":"TwitterAPI",
"from_user":"jkoum",
"metadata":
{
"result_type":"popular",
"recent_retweets": 100
},
"id":1478555574,
"from_user_id":1833773,
"iso_language_code":"nl",
....}
}
Returns tweets that:

match speci!ed query

located in Lat/Lon box

Historical Tweets

User Pro!le Information

Return Public Timeline

Service is rate limited
JSON or Atom
MakingRequeststoTwiLer
AuthenIcaIon
2MethodsofAuthen8ca8on
--BasicHTTPAuthen8ca8on
--oAuth(MoreComplex
MostmoderndaywebservicesuseoAuth(includingTwiVerandGoogleServices)
Keepsinforma8onsafefromthewrongpeoplebutalsoallowsproviderstobanpeople
iftheyareusingtoomanyresourcesormaking.
SohowdoesoAuthwork
Server
Applica8on
SohowdoesoAuthwork
Server
Applica8on
RequestaSessionToken
PassaIDandaSecret
SohowdoesoAuthwork
Server
Applica8on
RequestaSessionToken
PassaIDandaSecret
AsksLoginwithID
ServergivesaPinnumberforsession
SohowdoesoAuthwork
Server
Applica8on
SohowdoesoAuthwork
Server
Applica8on
Loginissuccessful
:)
SohowdoesoAuthwork
Server
Applica8on
Loginissuccessful
:)
SendsAppTokens
Theseneedtobesaved!
SohowdoesoAuthwork
Server
Applica8on
SohowdoesoAuthwork
Server
Applica8on
Requestusingtokens
Serverchecksifitsvalid
SohowdoesoAuthwork
Server
Applica8on
Requestusingtokens
Serverchecksifitsvalid
DataReturned
:)
GotohLps://dev.twiLer.com/
LoginusingyourcurrentTwiVeraccount
Clickyourprolepicture(toprightcorner)andclickMyApplicaIons
ClickCreateNewApplicaIon
Fillinyourdetailsfortheapp
YoullbepresentedwithyourConsumerKeyandConsumerSecret
Savethesekeysforthenextstep
CollecIngSocialMediaData(TwiLer)
CreaIngyourTwiLerApplicaIonKeys
LetsCollectsomedata
DownloadtheCollector
hVps://gist.github.com/sjg/6712144
LetsCollectsomedata
Aierdownloadingthescripts8ckinyourkeysandsecretinthescriptbetweenthe
quota8onmarksandsavethele
ConsumerKey
ConsumerSecret
LetsCollectsomedata
Copythescriptintoafolderthatyoucanrunthescriptfromandnavigatetothat
folderinyoursystemterminal.
Youcanedittheloca8onofthecollectorbychangingthesevaluesinthele:
Torunthecollectortype:
nodetwiLer_collect.js
SearchTag,La8tude,Longitude,Radius
-NB:willreturnalltweetsinthatarea
LetsCollectsomedata
Ifallgoeswellandeverythingisworkingyoushouldseethefollowingscreen
Gotothelinkinabrowser,logintotwiVerandtypethepinnumberin.Pressreturn
andtweetswillstartcollec8nginalecalledtweets.csvinthesamefolder.
Visualisingtheoutput
GotohLps://mapsengine.google.com
ClickonNewMap
NameyourmapbyclickingonUnItledMap
ClickImportanduploadyourCSVle
ClickBaseMaptochangethemapstyle
CreaIngyourrstVisualisaIonwiththeDataCollected
CreaIngyourrstVisualisaIonwiththeDataCollected
Oncethelehasuploaded,choosethecorrectLa8tudeandLongitudecolumnsand
thenclickCon8nue.ClicktheIDeld(leiover)asthemarker8tleandpressnish.
Theupdatedoutputlewill,hopefully,autodetectthelat/lngelds.
CreaIngyourrstVisualisaIonwiththeDataCollected
Styleyourmarkersbypressingthepaintbucket(highlightedinred)andselectthe
circleshapeandcolour.Experimentwithstylingyourmap.
CreaIngyourrstVisualisaIonwiththeDataCollected
Ifallwentwellyourmapwilllooksomethinglikethis.Whynotshareitwiththe
world!
CreaIngyourrstVisualisaIonwiththeDataCollected
Andyoucanclickthemarkersandseethetweets
DierentSourcesyoucanusetoVisualiseyourdata
GoogleFusionTables
-hVp://www.google.com/drive/apps.html#fusiontables
GoogleMapEngine
-hVps://mapsengine.google.com
GoogleEarth
-hVp://www.google.com/earth/index.html
MapTube
-hVp://www.maptube.org
hVp://download.bigdatatoolkit.org
BR1
BR2
BR3
BR4
BR5
BR6
BR7
BR
CM11
CM12
CM13
CM14
CM15
CM16
CM4
CM5
CR
CR2
CR3
CR4
CR44
CR5
CR6
CR7
CR
CR9
DA1
DA1
DA11
DA12
DA13
DA14
DA15
DA16
DA17
DA1
DA2
DA3
DA4
DA5
DA6
DA7
DA
DA9
E1
E1
E11
E12
E13
E14
E15
E16
E17
E1
E1W
E2
E3
E4
E5
E6
E7
E
E9
EC
EN1
EN2 EN3
EN4 EN5
EN6
EN7
EN
EN9
GU15
GU16
GU17
GU1
GU19
GU2
GU21
GU22
GU23
GU24
GU25
GU47
GU95
HA
HA1
HA2
HA3
HA4
HA5
HA6 HA7 HA
HA9
HP
lG1
lG1
lG11
lG2
lG3
lG4
lG5
lG6
lG7
lG
lG9
KT1
KT1
KT11
KT12
KT13
KT14
KT15
KT16
KT17
KT1
KT19
KT2
KT2
KT21
KT22
KT23
KT24
KT3
KT4
KT5
KT6
KT7
KT
KT9
N1
N1
N11
N12
N13
N14
N15
N16
N17
N1
N19
N2
N2
N21
N22 N3
N4
N5
N6
N7
N
N9
NW1
NW1
NW11
NW2
NW3
NW4
NW5
NW6
NW7
NW
NW9
RM1
RM1
RM11
RM12
RM13
RM14
RM15
RM16
RM17
RM1
RM19
RM2
RM2
RM3
RM4
RM5
RM6
RM7
RM
RM9
SE1
SE1
SE11
SE12
SE13
SE14
SE15
SE16
SE17
SE1
SE19
SE1P
SE2
SE2
SE21
SE22
SE23
SE24
SE25
SE26
SE27
SE2
SE3
SE4
SE5
SE6
SE7
SE
SE9
SL
SL1
SL2
SL3
SL4
SL9
SM1
SM2
SM3
SM4
SM5
SM6
SM7
SW1
SW11
SW12
SW13
SW14
SW15
SW16
SW17
SW1
SW19
SW1
SW2
SW2
SW3
SW4
SW5
SW6
SW7
SW
SW9
TN13
TN14
TN15
TN16
TW1
TW1
TW11
TW12
TW13
TW14
TW15
TW16
TW17
TW1
TW19
TW2
TW2
TW3
TW4
TW5
TW6
TW7
TW
TW9
UB1
UB1
UB11
UB1
UB2
UB3
UB4
UB5
UB6
UB7
UB
UB9
W1
W11
W12
W13
W14
W1 W2
W3
W4
W5
W6
W7
W
W9
WC
WD17
WD1
WD19
WD23
WD24
WD25
WD3
WD4
WD5
WD6
WD7
Thanks
Steven Gray
steven.gray@ucl.ac.uk
@frogo+StevenGray