Real Time Analysis of
CrowdBehaviourforAutomatic and
Accurate Surveillance
EPadmalatha KaredlaAnanthaSashiSekhar
Email:padmalatha@cbit.ac.in Email:sekhar.karedla@gmail.com
Abstract—
Surveillanceinthismoderneraisaveryhugedemand.Creatinganal
ertincaseofemergenciesandminordisturbancesisofverymuchimp
ortance.Asthenumberofsimultaneouscamerafeedsincrease,theb
urdenonthehumansupervisoralsoincreases.Thissystemisawayto
aidthehumansupervisorinthesurveillancejob.Creatingalertsinre
altimemayleadtoveryquickresponsewhichmighthelpustoreactqu
ickly.Withthisinmind,weproposethefollowingthings:
(1)GenerationofViF(ViolentFlowDescriptors)ashigh-
levelfeaturesinrealtime.(2)UsinggeneratedViF’sofaVideoDataset
fortraininganeuralnetandtestingitsaccuracy.(3)Developingasyst
emthatcandetectthesignsofdisturbanceamongthecrowdinrealti
meandcanlearnfromthedecisionsitmakes.
Keywords—
RealTimeSurveillance,ViolentFlowDescriptors,NeuralNetwork.
1. Introduction
Costofsurveillanceequipmentinthisdigitaleraisalmostnoth
ing.AlmostallofthepublicplacesareinstallingCCTVcamerase
verywheresoastokeepthepublicsafe.BehindtheseCCTVcamer
asarehumanssittingandwatch-
ingthefootagecontinuouslysoastoreactquicklyincase
ofemergencies.Thisisaroutineandtediousjobforthehumantoco
ntinuouslypayattentiontomultiplescreens.
Asweknowthereisalimittothehumancapacityandability,there
maybesomeerrorsinthehumansurveillance.Nomatterhowman
ysupervisorsarepresent,thereisaninsufficiencyinsuchprocess.
Ifweallocateacomputertodotheabovejob,thealertgenerationm
ayaidthehumantoreactquickly.
GenerationofViF[1]isalreadybeenexperimentedpreviousl
y.SoastogenerateViF,thereisadetailedprocessthathastobefollo
wed.Startingwiththevideos,theyhave
standardaspectratioof3:4andareofverylowquality.As
breakoutsinthecrowdbecomesarealchallenge.Alsothecontent
ofthevideoisconsideredtobeoriginatedfroma
CCTVcamerahenceanyothersourceofinformationsuchassubti
tlesandaudiocannotbeused.Continuoussurveillance
systemisofmuchimportanceandverylessattentionisgiventoit.
978-1-5386-5313-5/18/$31.00 ©2018 IEEE 1469
MudiamDasaradaRamReddy
Email:dachi8958@gmail.com
Inthisproposed system,we tryto implementan al-
gorithmwhichaccuratelydetectsviolenceinreal-time.
Throughthisalgorithmwetrytoobtainsafersurroundingsandhav
eaquickresponsetimetoviolentincidents.
2. PreviousWork
2.1. OpticalFlow
OpticalFlowisthecorepartofViolenceDetection.OpticalFl
owistherelativemotionbetweentwoimageframeswhicharetake
nattimestandt+∆tateverypixelposition.Methodsfordeterminat
ionofOpticalFlowcanbelistedasPhase correlation, Block-
based method, Dif-
ferentialmethodsandDiscreteOptimizationmethods.The
mostcommonlyusedmethodsareLucasKanadeandHorn-
schunckopticalflowmethods,whichcomeunderDifferen-
tialmethodsbasedonsolvingfirstorderderivative.WeusedCLiu’
s[2]opticalflowalgorithmforourtaskwhichwillbeusedtofurther
obtainFlowVectorMagnitude.SupposeVxandVyarethevelociti
esofapixelalongxandyaxisobtainedthroughopticalflowalgorit
hm,thentheflowvector
2 2
magnitudecan be obtained as, mt = Vx ,t + V y ,t .C Liu’s
OpticalflowalgorithmwasoriginallywritteninClanguageandm
exfiles werewrittenfor
compatibilitywithMATLAB.Weusedbobpackage[3]forusingt
hatparticularalgorithminPython.
2.2. ViolentFlowDescriptors
ViF(ViolentFlowDescriptors)[1]havebeenusedpre-
viouslytoobtaingloballevelfeaturesofavideo.Afterobtainingth
eflowvectormagnitude(m),wecalculatethebinaryvector.Thisb
inaryvectoriscalculatedforeachpixelwhichreflectsthechangei
nmagnitude.
1, if | mx , y ,t +1 − mx , y ,t |≥ θ
bxthecro
, y ,t =  wdbehavioris completely random, detecting
(1)
0 otherwise
Afterobtainingthebinaryvectorforeachframe,we

addbinaryvectorswhichareobtainedforalltheframesandnorma
lizethevaluewiththenumberofframestakenunderconsideration
.
1
b x, y = bx, y ,t (2)
T t
 ThisbgeneratedisdividedintoM*Nnon-
overlappingcellsandcollectingmagnitudechangesofeachcellse
p-
arately.Thesemagnitudechangesarethenrepresentedbyafixeds
izehistogram.TheseM*Nhistogramsarefurtherconcatenatedto
obtainasingledescriptorvectorwhichisknownastheViF.
3. Methodology
Doingrealtimeautomaticsurveillanceon
CCTVfootagehasmanychallengesandlimitations.Therearetw
oassumptionswhichwemake,thefirstassumptionistokeeptheca
meraawayfromtheareaundersurveillance,therehastobestandar
ddistancebetweentheCCTVcameraand
theareatobemonitored.Themainchallengewefaceistokeepthep
rocessingintermsofreal-
time,whichmeansalltheprocessinghastobedoneinlessthan1/25
thof
asecond.Oursystemshouldhavetheabilitytohandlemultiplevid
eosourcesatatime.Thishasbeenachievedbycontinuouslyaccep
tingframesthroughmultiprocessing
usingthreads.FirstpartiswecalculatetheOpticalflowwhichisth
emosttimeconsumingofalltheprocesses.Then
weusethecalculatedopticalflowtoobtainflowvectormagnitude.
NextwegenerateViFwhicharefurtherusedfortrainingandclassi
fication.
3.1. Algorithm
Themainpartofbuildingthesystem isto
haveawellbuiltalgorithm.Firstthevideoispreprocessed,thenwe
calculatetheopticalflowwhichismosttimecon-
suming.FeatureExtractionisdonenextwhichisusedfortrainingt
heneuralnet.Soastomakethesystemscalableformultiplevideos
ources,wehaveembeddedthreadingandmulti-
processing.Thebuiltalgorithmisrobustandcanhandlefaultyvid
eosources.
3.2. GlobalFeatureExtraction
3.2.1. VideoPreprocessing.Videocomingfromsourceisprepr
ocessed.Consideringthevideoaspectratioas3:4.Thesurveillanc
efootageisconsidertobestandarddefinition(scale=240:320).T
heinputframescominginareresized
to75:100sizeandthenconvertedtograyscale.Thelengthandbrea
dthofthevideosarealmostreducedbyone-third.
ForvideoprocessingOpenCV[7]packagehasbeenused.
3.2.2. OpticalFlow.Ce.Liu[2]optical
flowalgorithmhasbeenusedtocalculatetheopticalflow.Thisalg
orithmhasbeenusedparticularlysinceitishighlyefficientandrob
ust.Itreturnsthreevalues,vx(velocityvectoralongx-
axis),vy(velocityvectoralongy-
axis)andw(wrap).Thevectorsareinthesameshapeoftheresizedf
ramewidthandheight.
1470
3.2.3. ViolentFlowDescriptors.ViolentFlowDescriptor[1](
GlobalFeatures)Algorithmisbeingusedwhichhas been
already been implemented previously in
MATLAB.Soastoincreasethescalabilityofthealgorithm,syste
m hasbeenimplementedthroughbasicmultiprocessingand
threads.ViolentFlow Descriptors use flowvector
magnitudeoutput.Furtheronthenormalizedbinaryvector,histo
gramequalizationforthewholesetofframesunderconsideration
isperformedtoobtainasinglefeaturevector.TheViF’sob-
tainedarethenusedfortrainingandclassificationpurposes.Ifthe
numberofbinsarefixedas21(0.0to1.0,intervalof0.05)andconsi
derbothM,Nas4,thenforastandard
definitionvideoofscale240:320,336featuresareobtained
exactly(21*4*4).Valuesofthese336featuresrangefrom0.0to1.
0.
3.3. NeuralNetwork
3.3.1. Structure.For
thegivendatasetoncetheviolentflowdescriptorsaregenerated,tr
ainingtheneuralnetusingthesefeaturesisdone.Builtfourlayered
neuralnet,oneinput
layer,twodenselayersandoneoutputlayer.Inputlayeraccepts
350inputsand gives336 outputs.Middle
denselayersaccept336inputsandgive336outputs.Outputlayera
ccepts336inputsandgives1output.ForinputanddenselayersRe
LU(Rectifier Linear Unit) activation function is
usedandfortheoutputlayerSigmoidfunctionisbeing used.
3.3.2. Training.ForbuildingneuralnetKeras[6]andTensorflo
w[5]arebeingused.Initiallythedataneedstobeformattedsothatit
fitsintoinputlayerofNeuralNet.
TheViolentFlowFeaturesgeneratedforeachvideowillbeinthef
ormatofnumpyarrayofdimensions336*1.Arrayisreshapedinto
1*336dimensions,sothatthefeaturesof
singlevideofitintotheinputlayerofneuralnetasonedatatuple.Fu
rther,concatenationofreshapedfeaturearrayofeachvideointoas
inglearrayisdonesothatthefinalinputtotheneuralnetwillbeanar
rayofdimensions246*336as
thereare246videos(violentandnon-violent)inthedataset.
Ontheotherhand,preknownoutputs(0(non-violent)or
1(violent))ofeachvideoarestoredinanarrayofdimensions
246*1totrainneuralnetandtocalculateaccuracy.Afterdataisrea
dywecanproceedtobuildtheneuralnetintothestructuremention
edabove.UsingKerasSequentialneuralnetwithdenselayerscan
bebuilt
.Amodeliscompiledforwhichwemustspecifya
logarithmiclossfunctionwhichevaluatesasetofweightsandalso
anoptimizertosetlearningrate.Kerashasa
logarithmiclossfunctionforbinaryclassificationproblemdefine
dasbinary-
crossentropyandAdamOptimizerwhichisbestoptimizer.Numb
erofepochsforwhichthetrainingmustbecarriedoutisgiven,batc
hsize(numberofinstancesevaluatedtoperformweightchanges)
andtheinputdatafortrainingthemodelareprovidedasparameter
s.Trainedmodelisstoredintoafileofhd5formatusinghdpypytho
npackage.
3.3.3. ViolenceDetection.Thisphaseinvolvesdetectionofdist
urbanceorviolenceinlivecrowdsurveillancevideosinrealtime.
TheinputsurveillancevideoispreprocessedandViolentFlowDe
scriptorsaregenerateddynamicallyinRealTime.Foreachsecon
dofvideo,featuresareextractedandaregivenasinputtothetraine
dmodelforclassificationandviolencedetection.Ifsomedisturba
nceorviolenceis
detected,itwillbereportedasanalertstatingthatitis
violencealongwiththetimeithasoccurredwithinasecond
ofoccurrence.

3.3.4. Feedback.IntheViolenceDetectionphase,astherealtim
esurveillancetakesplace,thefeaturesgeneratedforeverysecond
aretestedagainsttrainedmodelforclassifi-
cationandviolencedetection.Thosefeatures,alongwith
theiractualoutput(providedbyhuman)generatedbythetrainedm
odelaregivenasfeedbacktothemodel.Thisallowscontinuoustra
iningofneuralnetmodelwhichhelpstoincreasetheaccuracyofcl
assificationandalsofasterdetectionofviolence.
Figure1.ProcessingSpeedswithIntelCentrinoProcessor
3.4. ExtractionofInterestingFeatures

AdaBoostisanensembleofweakclassifiers.AdaBoost
isanalgorithmwhichcouldtellustheimportantsetof
featuresthathelpusclassifyourfeatures.ForthistheFeatureSelec
tionAlgorithmthroughAdaBoost[4]isused.
Oncethefeaturesarearrangedinincreasingorderofthe
errorrates,wecanobtainthefeaturesamongtotal336
featureswhicharehighlyefficientinclassifyingvideos.Theweak
classifiersusedherearedecisionstumps(decisiontreeofheight1)
.

4. Implementation

Inthebelowsubsectionsweprovidetheimplementationdetai
lsandtheoutputsanalysis.Clearanalysisofthesystem
willbedoneinthenextsection.

4.1. ContinuousSurveillance
Figure2.ProcessingSpeedswithInteli5Processor

Weusedtwosetsofconfigurationsforcalculatingpro-
cessingspeedsofthesystem.FirstConfigurationconsistsof
4GBRAM,IntelCentrinoProcessorwithUbuntuOS.Sec-
ondConfigurationconsistsof8GBRAM,Inteli5Processor
withDebianOS.
Foravideowhichisnotinitiallyviolentbutlateronbecomesvi
olent,theproposedsystemisabletodetectthe
exactinstancewherethevideoframesgofromviolenttonon-
violent.ConsideringReal-
TimeCCTVfeed,withinasecondofoccurrenceofviolenceoursy
steminabledetect theviolenceandraiseanalert.
Abovearetwofiguresshowingtherunningtimeofthesystem.
Thetotallengthofvideotakenunderconsiderationisnearly200se
conds.Proposedsystemisabletodetectthe
exactsecondofviolenceoccurrencei.ewheretheframesgofromn
on-
violenttoviolent.WithIntelcentrinoprocessorFig.1,theprocess
ingisbeingcompletednearlyin180
seconds(20secondsfasterthanruntimeofvideo).WithInteli5pro
cessorFig.2,theprocessingofentirevideoisbeingdonenearlyin1
40seconds(1minutefasterthanruntimeofvideo).Witheachdete
ctionofviolenceoutbreak,

1471
thecorrespondingtimetakenbythesystemtodetectthatisbeingsh
own.
4.2. Accuracy
TheaccuracyobtainedbyViF’s[1]asglobalfeaturesusingali
nearSVMis81.30%forexistingsystem.Proposedsystemhasana
ccuracyofnearly85%.
Figure3.BarPlotofObtainedAccuracyValuesinNFolds
ThebargraphinFig.3showstheresultofN-foldscross
verificationwithN=7.Therearetotal5runs(executionofn-
foldsonceinarun).Ineachrunweconsider7heapsin
total.Eachheapcontainingequalnumberofvideos.Among
thesevideosviolentandnon-
violentvideosaredistributedevenly.ViolentandNonViolentvid
eosareplacedrandomlyinheaps.Thisgivesusanideahowrobustt
heproposedsystemis.Theminimumaccuracyweobtainforahea
pinanyrunisgreaterthan70%.
Figure4.LinePlotofObtainedAccuracyValuesinNFolds
Fig.4showsthelinegraph,itissame asFig.
4,butgivesustheclearpictureofaccuraciesofeachsetinitscorres
pondingrun.Eachrunhasbeenassignedadifferentcolor.Fromthi
swecanclearlyidentifytheminimumand
1472
maximumaccuracy.Themaximumobtainedisnearly96%andmini
mumaccuracyisof73%.
Figure5.AccuracyObtainedbytrainingwith70%ofdataandtestingwith30%ofdata
p n
P 33 2
N 5 32
TABLE1.C ONFUSION MATRIXFOR 70:30 DATASET
Thedatasetwhichcontains246videosisdividedintheratioof
70:30.70%ofthedataisusedtotraintheneural
net,30%ofthedataisusedtotesttheaccuracyofgeneratedmodel.
OutputinFig.5showsthattheaccuracyobtainedis90.27%.Aswe
canseetheconfusionmatrixinTable1,thenumberofFalseNegati
vesarejust2,thatmeansthereareonly2casesinthetestsetwhichar
eactuallyviolentbutoursystemwasnotabletodetectit.Whereast
herewere5casesinwhichvideoswerenotviolentbutoursystemd
etectedsomeviolence.
Followingaretheresultsobtained:
• Accuracy=TP/(total)=0.9027
• TruePositiveRate=TP/(Positives)=0.9428
• Precision=TP/(Predictedyes)=0.8684
• Specificity=TN/(Actualno)=0.8648
• MisclassificationRate=(FP+FN)/(total)=0.097
Theaboveaccuracytestsweredoneonadataset[1]containing
246videos.ShortestVideoisof1secondandLongestVideoisof6s
econds.Thesecollectionsofvideoshaveequalnumberofviolent
andnon-
violentvideos.Thiskindofdatasetisknownas”inthewild”datase
t.VideospresentinthedatasetareofstandardCCTVresolution(sc
ale
=240:320)andofsimilaraspectratio(3:4).
5. ResultsandDiscussion
Considerthefollowingscenesobtainedthroughsurveillance
footage:
 Figure6.ViolenceNotstarted
Figure7.Violenceabouttostart
Figure8.Violenceslightlystarted
1473
Figure9.Violencefuriouslystarted
Aboveare4figureswhichshowfourdifferentphasesofsurvei
llancevideo.InitiallyinFig.6,Violencehasnotyetstarted.InFig.7
Violenceisabouttostart,peopleareslightlypushingeachother.Fi
g.8showsthestartofviolenceandinFig.9Violencehasstartedfuri
ously.
Figure10.TerminalOutputforthevideo
InFig.10,itshowstheoutputofthesystemfortheparticularvid
eoshowninpreviousfigures.Aswecanseefortheinitialframe,out
putvalueisveryless,asthescenegets tense in Fig. 7, the system
output value increases. When
theviolencestartsinFig.9,outputvalueincreasesto0.999
indicatingviolence.Lateroninthevideoviolencedecreases
graduallyandhencetheoutputvaluefallsdownto0.06.
6. ConclusionandFurtherWork
Timelydetectionofviolenceinrealtimeisofmuchimportanc
e.System’sabilitytodetectviolencewithinasecondisimportantb
ecausethedifferencebetween
thetimeofoccurrenceandthetimeofdetectionmaybethe
differencebetweenlifeanddeath.Resultshaveshownthatsystem
isperformingfastenoughtodetectoutbreaksinrealtime.Thissyst
em’slearningcapabilitiesandaccuracycanbefurtherimprovedt
hroughAdaBoost.Whateverworkhasbeendoneistogiveattentio
nandimportancetoaccuraterealtimesurveillance.Thissystemof
coursemaynotreplacehumansattheirjobbutwillgreatlyreducet
heburdenon
them.Thiswillgivehumanstheabilitytomonitormultipleareasatatim
e.
Furthertheaccuracycanbegreatlyimprovedbyusingtheresu
ltsofFeatureSelectionalgorithmofAdaBoost[4].
Asexplainedabove,oncewearrangethefeaturesinincreas-
ingorderoftheirerrorrate,wegetorderofimportanceof
336features.Inthatparticularorder,wecanassociatetheweights
ofinputlayerinneuralnet.Thefeatureswhichare
highlyimportantcanhaveahigherweightattheirinputnodeandas
theimportancedecreases,weightscanalsobedecreased.Thisma
ymakeadifferenceinincreasingtheaccuracyofneuralnet.

Acknowledgments
WewouldliketothankDr.THassner[1]forhiscon-
tributionstowardsrealtimesurveillance,thisprojectwouldnotha
vebeenpossiblewithoutthegenerationofViolentFlowDescripto
rs(VIFs).

References
[1]T.Hassner,Y.Itcher,O.KliperGross.ViolentFlows:Real-
TimeDetectionofViolentCrowdBehavior,3rdIEEEInternationalWork-
shoponSocially Intelligent Surveillanceand Monitoring
(SISM)attheIEEEConf.onComputerVisionandPatternRecognition(CV
PR),June2012
[2]Ce.Liu.BeyondPixels:ExploringNewRepresentationsandApplica-
tionsforMotionAnalysis,MassachusettsInstituteofTechnology,Ph.D.
Thesis,2009
[3]Anjos,AndréANDElShafey,LaurentANDWallace,RoyAND Günther,
ManuelAND McCool,Christopher ANDMarcel, Sébastien.
Bob:afreesignalprocessingandmachinelearningtoolboxforre-
searchers,20thACMConferenceonMultimediaSystems(ACMMM),
NaraJapan,2012
[4]RuihuWang.AdaBoostforFeatureSelection,ClassificationandItsRelationw
ith SVM,A Review, Department ofScienceand
TechnologyChongqingUniversityofArtsandSciencesYongchuan,Chon
gqing402160,CHINA
[5]MartnAbadi,AshishAgarwal, Paul Barham,EugeneBrevdo,
ZhifengChen,CraigCitro,GregS.Corrado,AndyDavis,JeffreyDean,Mat
thieuDevin,SanjayGhemawat,IanGoodfellow,AndrewHarp,GeoffreyIr
ving,MichaelIsard,RafalJozefowicz,YangqingJia,LukaszKaiser,Manju
nathKudlur,JoshLevenberg,DanMan,MikeSchuster,RajatMonga,Sherr
yMoore,DerekMurray,ChrisOlah,JonathonShlens,BenoitSteiner,IlyaS
utskever,KunalTalwar,PaulTucker,VincentVanhoucke,VijayVasudevan,
FernandaVigas,OriolVinyals,PeteWarden,MartinWattenberg,MartinWi
cke,YuanYu,andXiaoqiangZheng.TensorFlow:Large-
scalemachinelearningonheterogeneoussystems,2015.Sourceavailablea
thttps://www.tensorflow.org
[6] Chollet,Françoisandothers.Keras,2015.Sourceavailableathttps:
//keras.io
[7]
Itseez.OpenSourceComputerVisionLibrary,2015.Sourceavailableathttp
s://github.com/itseez/opencv

1474