25/09/2015

Proxylizer/ConceptsMikroTikWiki

Proxylizer/Concepts
<Proxylizer

Contents[hide]
1CollectedLogData
2Reports
2.1Reportname
2.2Reporttype
2.3Frequency
2.4Recipient
2.5Dateinterval
2.6Dayinterval
2.7Weekdays
2.8Timeinterval
2.9IP
2.10Domain
2.11Top
2.12Generatetime
3Timecalculation
4CreatedReports
5ReportGeneration
6ReportHistory
7InactiveReports
8ReportEditing
9IPUsers
10DatabaseStatistics

CollectedLogData
ReceivedlogsfromMikroTikrouterWebProxyservercontainvariousinformationaboutwebrequests.
Informationstoredfromtheselogsisdomain,IPaddresses,eventtimeandisthesiteloadedfrom
proxyservercacheornot.AllURLinfothatisbehind'?'isdroppedoffandnotstoredindatabase,
becausethisinfoistoolargeanditisunnecessaryformonitoringWebProxyserverrequeststatistics.
Receiveddomainisstoredinothertablethenlogsanddividedin3partssubdomain,domainandtop
domain.Ifdomainhasmorethan3parts,forexampleabc.def.ghi.mikrotik.com,itisdividedin
'abc.def.ghi'+'mikrotik'+'com'.
Ittakesapproximately200KBofspacetostore1000logrecordsinthedatabase.Youcanseedata
basestatisticintheStatussectionontheProxylizerwebpage.

Reports
Reportname
Thisfieldisdesignedtogiveeachreportusercreatedidentifier.

data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color

1/5

25/09/2015

Proxylizer/ConceptsMikroTikWiki

Reporttype
Thereare2reporttypesDomainandUserstatistics.Theformerreportisusedtoanalyzedomain
usageofallwebrequests,whilethelattertoviewwebrequestsofeachuser.
Frequency
Thereare4typesoffrequencyavailableonce,daily,weeklyandmonthly.Thereisonedifference
betweenoncereportandothers.Oncereportisdesignedtocreatecustomreports,thereareno
restrictionsforthistypeofreport.Asthenamestates,itisexecutedonlyonce,whiletheothersare
periodic.Periodicreportsaregeneratedonlywhenalldataarecollected.Forexample,ifdataforMarch
4thareneeded,themareavailableonlyafterMarch5th00:00.
Recipient
Allgeneratedreportscanbesenttoemail.If"Norecipient"isselected,reportsisnotsenttoanyemail.
But,asallreports,itisstillavailabletoseethegeneratedreportinwebinterface,reportsection,
(history).RecipientaddressesareselectedfromIPUserlist,whichcanbeeditedatIPuserspage.
Dateinterval
Thesefieldsareforsettingupdateboundariesforreport.
Foroncereportitmeansthatalldatainthisperiodwillbeselected
Fordailyforeachday,weeklyeachweek,monthlyeachmonth.
Itisposible,tosettheseboundariesinpastandinfuture.ForexampleiftodayisFebruary15thitis
posibletosetthesevaluesfordailyreportJanuary3rdtoFebruary2ndandreceivereportsforeach
dayinthisperiodorforMarch5thtoApril20thandwhenthesedatescomereportswillbegenerated
then.
Dayinterval
Thisfieldisonlyavailableformonthlyreports.Itdenotesneededdataintervalfirstandlastdayof
eachmonthofdatatoincludeinreport.Forexampleifdataforfirstanthensecondhalfofmonthis
needed,2reportsmustbecreatedone1sttill15th,second16thtill31st.Fortheenddateanyvalue
greaterthanthedaycountintheactualmonthisconsideredasthelastdayofthemonth.Forexample,
thereare28/29daysintheFebruary,soanyvaluegreaterthan28meansendofthemonth.Butfor
October,29means"donotselectdataaboutOctober30and31".
Weekdays
Useifcustomdaysofweekfiltrationneeded.Forexampleifonlyworkingdaysneededthenuncheck
weekenddays.
Timeinterval
Useifcustomtimeofdayfiltrationneeded.Multipletimeintervalsavailable,forexample,09:0012:00
and13:0017:00.Click

toaddand

toremoveintervals.

IP
UsetofilterIPaddresses.If"Showall"selectedthenreportforallIPaddressesandapproximatetime
data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color

2/5

25/09/2015

Proxylizer/ConceptsMikroTikWiki

spentoninternetisdisplayed.OtherwisedomainsandtimeforspecificIPuser.
Domain
Usetofilterdomainsandpartsofdomains.Asexplainedpreviouslydomainisdividedin3partssub
domain,domainandtopdomain.Respectivelyusefirstfieldtofilterbysubdomain,secondbydomain
andetc.
Top
Useifonlymostoftengathereddataisneeded.Forexample,select"Viewtop10"toviewonly10most
useddomains.
Generatetime
Thisfieldistosettimewhenreportmustbegenerated.Foreachfrequencyitisdifferent.For"once"it
isdateanddaytime,fordailyonlydaytimeandetc.Howmentionedpreviouslyinthissectiondatacan
begeneratedonlywhenalldataarecollected.Forexample,ifweeklyreportforworkingdaysneeded
thendatacanbegeneratedonlyafterSaturday00:00.Principleasfastaspossible.

Timecalculation
Timeusedforeachdomainoruseriscalculatedbyasimplealgorithmifusermakesatleastoneweb
requestperminute,userhasusedinternetforoneminute.Itmeansthatitisnodifferenceifthereare
tousandsofrequestsperminuteorejustone.Countoftheserequestareselectedandviewedunder
column"Hitcount".Anditisirrelevant,howmanydifferentrequestsaremadeatonetime.

CreatedReports
Reportsconsistoftwoparts:1.Reportconfigurationcontainsinformationaboutwhatdatahastobe
collected,whenthereportswillbegeneratedandwhowillreceivethem2.Thegeneratedreport
containstheactualdatabasedonconfigurationrules.
TheconfigurationiscreatedinthereportsectionontheProxylizerwebpage.
Reportsaregeneratedautomaticallybyabackgroundscriptwhichisrunbythescheduler.
Whenaconfigurationisdeleted,allgeneratedreportsforthatconfigurationaredeletedaswell.

ReportGeneration
Reportgenerationprincipleislikethiseachminutescriptisstartedandcheckediftheremustbesome
reportgenerated.Reportsaregeneratedasfastasposiblewhenalldataarecollectedandgeneration
timehascome.Ifsomereportsaregeneratingmorethenoneminute,thereismaximumsimultaneous
generatingreportamountdefined,whichcanbesetintheconfigsection.Formulticoreprocessorsthis
valueshouldbeequalwiththecorecounttoutilizetheprocessorefficiently.Ifthereissomeproblems
orerrorsduringreportgeneration,emailissent.Todiagnosetheproblem,lookforlogfilesthatare
storedin"/var/log/proxylizerdirectory(mail_send_log.log"isthedefaultlogfile).

ReportHistory
data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color

3/5

25/09/2015

Proxylizer/ConceptsMikroTikWiki

ThewholehistoryofgeneratedreportsareavailableinthereportsectionontheProxylizerwebpage.
Eachreportinthetablehasthe

(history)iconintheactioncolumn.Clickingonitrevealsthelistof

generatedreportsforthisconfiguration.Aparticularreportcanbeaccessedthenbyclickingonthe
(View)icon.

InactiveReports
Ifareportisnotneededforawhile,itcanbedeactivatedinsteadofdeleting.Ithasthefollowing
advantages:
Thehistoryissaved
Reportcanbeactivatedlaterwithoutcreatingnewconfiguration.
Duringthetimewhenareportisinactive,itisnotgenerated.Ifitisactivatedafterwards,reportwillbe
generatedagainstartingfromtheactivationpoint.Forexample,ifadailyreportisinactiveforaperiod
of20.jan.2009.25.jan.2009.thenextreportgeneratedafter19.jan.2009.willbetheoneabout
26.jan.2009.,reportsduringtheinactivityperiodwillbeskipped.
Reportscanbeactivated/deactivatedintheReportssectionontheProxylizerWebpage.

ReportEditing
Reportconfigurationscanbeeditedaftercreating.Butsomerestrictionsapply:
Reportswithfrequency"once"canbemodifiedonlybeforetheyaregenerated.Whenthereportis
processed,theconfigurationisreadonly,becausethisiswhat"once"means
Tootherreporttypesonlythefields,whichdonotchangethesemanticsofthereport,canbe
edited.Forexample:dateinterval,weekdaysandgeneratetimecanbechangedfordailyreportas
theyalldonotaffectthecontentsofdailyreporttheyonlyspecifysomepropertiesofwhenthe
reportmustbegenerated.Reporttype,frequency,timeinterval,IPaddressanddomainareread
only,becausechangingthesevaluesleadstoacompletelydifferentreport.

IPUsers
EachWebproxyrequesthasasourceIPaddressaddressofthehost,whichgeneratedtheweb
request.ReportscanbefilteredusingthisIPaddress.
UsuallyonestaticIPaddresscorrespondstoaspecificstaffmember.TomakethefilteringbyIP
addresseasy,ProxylizerhastheabilitytoassignrealpersonstoIPaddresses.Thiscanbedonein
theIPUserssectionontheProxylizerwebpage.
IPusersareusedonlytoassignpersonnamesandemailaddressestoIPaddresses.IPusersdonot
haveaccesstoProxylizerwebpage.
EachIPUser(person)hasthefollowingattributes:
IPaddressoftheperson'scomputer
Name
Emailperson'saddresstowhichreportemailscanbesent
Adminwhenchecked,thisuseristreatedasadministrator.SomeProxylizerfunctionsaredesigned
foradmins,forexample,carboncopy(CC)ofemailreportscanbesenttoalladminemail
data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color

4/5

25/09/2015

Proxylizer/ConceptsMikroTikWiki

addresses
Userreceivesemptyreportswhenchecked,thisuserreceivesemailreportsevenwhentheyare
empty(forexample,reportsofweekenddatawhennowebrequestsaregenerated).Thisisuseful
foradministratorstoidentifyreportgenerationproblemstheadministratorissurethatreportwillbe
sentanyway,andwhenitisnotreceived,ithappensonlybecauseofsomesystemorinfrastructure
failure.

DatabaseStatistics
Database(DB)statisticsareshowninStatussectionontheProxylizeWebPage.Thefollowingstatistics
areavailable:
HITMISSratio:showsthewebproxyhit/missrationwhatpartofallrequestsarefoundinthe
proxy'scache
Totaldomaincount:totalcountofdifferentdomainsstoredinthedatabase.Notethat,forexample,
www.mikrotik.comandwiki.mikrotik.comarecountedastwodifferentdomains
Totalhitcount:totalcountofrequestsloggedbythewebproxy
OldestrecordoldestrequeststoredintheDBrightnow.Thisdeterminestheoldestpointintime
wehavedataabout
LatestrecordmostrecentwebrequestloggedintheDB.Ifthisfieldcontainsanoldvalue(let's
say,morethanonehourago)thismeansoneofthefollowing:eithertheclientsarenotusingweb
proxyorrequestsarenotloggedtoProxylizerDB.Thisvaluecanbeusedtodetectsystemerrors
Databasesize:howbigisthedatabasecurrently.Thiscanbeusedtoidentifyhowmuchthesizeof
theDBgrowsdaily,monthlyetc.
Asyoucouldnotice,thesevaluesareloadednotimmediatelyafterlogginginthewebinterface,rather
severalsecondslater.Thereasonissimplecalculatingthestatisticstakessometimeandtonotforce
thewebusertowait,theyareloadedinbackground.Oncethevaluesarecalculated,theyarecached
forthewholewebsession.Togettheactualstatistics,Refreshbuttoncanbeusedlater.

data:text/htmlcharset=utf8,%3Ch1%20id%3D%22firstHeading%22%20class%3D%22firstHeading%22%20lang%3D%22en%22%20style%3D%22color

5/5