dataMine Overview

dataMine is a data capture and visualisation plugin for Vera. It includes a Vera plugin to record and process the data, and a web GUI for data visualisation and configuration. The web GUI is shown in Figure 1. The plugin that runs in Lua records all changes for variables that are being logged. The raw data is then written into log files on Vera (preferabl on a U!" stic#$, and the GUI then re%uests this data fro& the plugin. "oth sides are therefore re%uired to operate ' the Vera, providing the data capture and server, and the GUI, providing the data &anipulation and visualisation. This version of the guide was updated on () *a +(1,.

dataMine wor#s b &onitoring all changes in device variables in Vera. -henever a variable changes, dataMine gets called, and it logs this data into a file for later use. The GUI then interacts with this data to provide the user a visualisation interface.

Installation
dataMine is installed through the UI. &ar#etplace. This installs all re%uired files onto our Vera, however so&e configuration is re%uired before dataMine will run. / U!" stic# is highl reco&&ended to store dataMine data. 0epending on the nu&ber of variables ou are logging, dataMine can generate a reasonable a&ount of data, additionall , as this data is written regularl , this &a cause the internal Vera &e&or proble&s if it is used for an e1tended duration. 2age 1

The U!" stic# can be for&atted on -indows and plugged into a free U!" slot on the Vera. 3ou should avoid for&atting the drive with the F/T14 file s ste& as this has li&itations. 5n an &odern version of -indows, it should co&pile as F/T,+ be default so this shouldn6t be a proble&. Linu1 et1+7e1t, file s ste&s are fine. /fter for&atting in -indows, ou should set the dis# label in the dis# properties ' this allows ou to identif it once it is plugged into Vera. 3ou should then run the GUI and ou will be presented with the dialogue shown in Figure 1. This dialogue as#s ou to select the UUI0 (the Universall Uni%ue Identifier$ of the U!" to be used for dataMine. If ou have &ore than 1 U!" stic# installed in our Vera, and ou6re unsure which one is the one for dataMine, then it is suggested to add a label to the drive ou wish to use for dataMine, and refresh the GUI page so that onl one UUI0 is shown. 5nce ou6ve selected the UUI0, press !ave, and dataMine should be configured. 8estart Luup (press 8eload in UI.$, and dataMine should be up and running.

Figure 2: dataMine initialisation dialogue. The configuration dialogue will set the variables shown in Table 1. It is also possible to set these values &anuall if ou please, however the dialogue will be shown an ti&e that the U!" stic# fails to &ount correctl . Configuration Variable !et0ata0irector !et*ount2oint !et*anual*ount !et*ountUUI0 Suggested Value 7data*ine "lan# ( U!" UUI0

Table 1: Suggested configuration for use with Vera USB logging.

2age +

-hen dataMine starts, it will create two further variables 9 :&ountLocation: and :&ountT pe:. It should be noted that these ;ust report the current &ount status, and are not &eant to be used as data entr points b the user. If ou have &anuall set a &ount point with the <!et*ount2oint= setting, then if the &ounting wor#s then :&ountLocation: will be the sa&e as :!et*ount2oint:. :&ountT pe: will reflect the files ste& used b the U!" stic# ' t picall :vfat: when purchased straight fro& the shop. This should loo# si&ilar to what is shown in Figure ,.

Figure 3: dataMine configuration variables.

Using dataMine with the Vera USB logging

For use with the Vera U!" logging, additional configuration &a be re%uired. This is because Vera will &ount the U!" stic#, and if dataMine is also allowed to &ount the sa&e stic# to a different &ount point, corruption can occur. 3ou &ust either use a separate U!" stic# for dataMine and use the configuration described above, or point the dataMine director to a location below the log director . >ote that the preferred option is to use a separate stic# if possible to avoid conflicts. !uggested configuration for use with U!" logging is shown in Table +. Configuration Variable !et0ata0irector !et*ount2oint !et*anual*ount !et*ountUUI0 Suggested Value 7t&p7log7c&h7data*ine >ot re%uired to be set 1 >/

Table 2: Suggested configuration for use with Vera USB logging.

2age ,

Use
Plugin Properties
The data*ine plugin presents a si&ple device in the UI. dashboard ' as shown in Figure ?. This ;ust shows how &an channels are being &onitored, and how &an channels have been configured in all.

Figure ": #evice overview. Figure . shows the &ain properties tab. /gain, this present the sa&e infor&ation as the device overview, however there is also a lin# to the GUI webpage which is outside of the UI. interface.

Figure : dataMine !ro!erties !age.

2age ?

Graphical User Interface

The GUI is a web interface that allows ou to visualise the data fro& Vera. It uses a tabbed interface to select different data views or t pes as shown in Figure 4. /t the right side of the tab bar is an <5nline !tatus= indicator ' this is green if the web interface is co&&unicating with Vera, and gre if Vera is not co&&unicating with the browser. It displa s red if co&&unication is o#, however there are errors within dataMine (eg. Initialisation error, or error writing data to the U!"$.

Figure $: dataMine User %nterface.

Graph Tab The graph tab provides an interface to graph the data being recorded b the dataMine plugin.

2age .

Figure &: dataMine gra!h window. The graph displa is split into two &ain areas ' the configuration area to the left, and the graph itself. In between the two is a slider bar that allows ad;ust&ent of the position between the two. >ote that there6s also a button in the &iddle of this slider with an arrow pointing to the left. @lic#ing on this will hide the configuration area and ou6ll have a screen that loo#s li#e Figure ). @lic#ing on the bar on the left will te&poraril displa the configuration area, while clic#ing on the arrows at the top of the bar will restore the configuration area. This allows the full window area to be &ade available for graphs. If running data*ine on a &obile browser, the bar will default to the left to provide the &ost space possible for s&all screens.

2age 4

Figure ): 'ra!h tab with configuration area hidden. The configuration area includes areas for selecting the channels to graph, and the graph options. In the Graph @hannels area, clic#ing on a variable na&e, the variable will select or deselect. -hen selected, the variables can be graphed b clic#ing on the U!date 'ra!h button (the graph will use the sa&e ti&e period as is on the current graph$. @lic#ing on the (eset 'ra!h button will clear the current selection and reset the ti&e period bac# to default ' it will not affect the current graph, however. @lic#ing on the Save 'ra!h button will allow ou to save the graph for later %uic# recall. !aved graphs will be listed in the Saved 'ra!hs list.

2age A

Figure ,: 'ra!h channel selection Bach channel has an icon on the left side 9 this indicates the t pe of data. The *ast U!dated colu&n also shows if data has been received recentl . / tic# is shown if the variable data has changed recentl , a ellow e1cla&ation is shown if the data is %uite old, and a red e1cla&ation is shown if the data is ver old. The definition of these flags is configurable per channel since different channel t pes e1pect different change rates. @lic#ing on the 'ra!h +!tions navigation bar brings up a nu&ber of para&eters that can be changed in the graph. !ee Figure 1( and Table , for details of the options.

Figure 1-: 'ra!h o!tion window.

2age )

Option 0ispla data &ar#ers

Description 0ispla s a point on the graph where each <real= data point e1ists. Lines are then drawn between the data points to &a#e up the graph. 0ispla s a border and shadow around the chart, 0ispla s the graph legend showing what data is displa ed with each colour. Bnables crosshairs. !hows the tooltips where the cursor or crosshair is near a data point. The tooltip shows the actual data displa ed on the graph. !witches between local ti&e and Universal ti&e. @hanges the line width of graphs. Bnable or disable the chart ani&ation when data is drawn. Table 3: 'ra!h o!tions.

0ispla shadows 0ispla legend 0ispla crosshairs 0ispla tooltip 0ispla local ti&es Line width /ni&ation

The toolbar across the top of the graph screen provides a nu&ber of buttons for interacting with the graph. Bach of the buttons has a tooltip, so if ou put the &ouse over the button, ou will get a hint as to what the button does.

2age C

Icon Doo& in Doo& out 0ispla last da s data 0ispla last wee#s data 0ispla last &onths data 0ispla last ears data !croll left !croll right 0ispla real9ti&e graph Turn off displa ing Vera >otifications

Description

0ispla notifications relevant to the channels being graphed (fro& the sa&e device, or its parent device$ 0ispla all notifications ' note that this can be overwhel&ing when long ti&e9scales are displa ed. 0ispla s gre bands on the chart during night9ti&e. 0ispla graph infor&ation Table ": 'ra!h toolbar buttons. It is possible to Eoo& using the &ouse as shown in Figure 11. !i&pl hold the &ouse button down at the start of the Eoo& period, &ove to the end, and release the &ouse. >ew graph data will be loaded.

2age 1(

Figure 11: 'ra!h window .oo/ The real9ti&e graph displa s data for the past 1. &inutes, and updates ever second. 0ue to the nature of the displa , it displa s data in <absolute for&= 9 ie. it does not interpolate between data points. This gives the displa a &ore stepped nature than the standard graph. (>oteF 8eal9ti&e graphing is not i&ple&ented in the current version ' it will return soon$.
Notifications

dataMine can displa Vera notifications on the graph as shown in Figure 1+. The notifications are updated ever &inute, so ver new notifications &a not be displa ed. If there are &ore than .(( notifications in the graph, dataMine will onl show a selection of .(( b tri&&ing out ever nth notification. In realit , this is far too &an to be able to view an wa G 2utting our &ouse over the notification will displa &ore infor&ation on the notification.

2age 11

Figure 12: 0otification dis!la1

Day-Night banding

dataMine records the sunrise7sunset ti&es calculated b Vera, and then allows the nights to be displa ed as gre bands in the graphs. This is shown in Figure 1,. To activate7deactivate this option, clic# on the &oon icon in the toolbar. >ote that as dataMine does not calculate the sunrise7sunset ti&es, this feature will onl wor# after the date that this was added to dataMine (/pril +(1,$. /lso, if our Vera is turned off for a period, this will also stop the ti&es being logged.

2age 1+

Figure 13: #a120ight banding.

Sa ing Graphs

Fro& the graph selection window, ou can clic# the !ave button to save a graph configuration for later recall. Figure 1? shows the window that is then displa ed. This allows selection of which a1is each channel will be displa ed on, as well as an icon that will be displa ed alongside the graph in the selection list. The a1is selection &ust be a nu&ber between 1 and ?. The Huic#View reference is a reference for displa ing the graph in an e&bedded webpage. For &ore infor&ation, see the section on e&bedding below. >ote that this should >5T contain spaces.

2age 1,

Figure 1": 'ra!h Save 3 4dit window.

Figure 1 : Saved gra!h selection list.

Figure 1. !hows the saved graph selection list. The and the icon allows ou to edit the graph data.

icon allows ou to delete the saved graph,

>ote that when editing a graph, if ou change the na&e, dataMine will save a new cop of the graph and ou will need to delete the one ou don6t want.

Dashboard Tab Figure 14 shows the dashboard. @urrentl this is (ver &uchG$ wor# in progress and it shows the energ currentl being used. There is a gauge showing the current whole house energ , and a donut chart showing the split b device and categor . / trend chart shows the whole9house use over the past +? hours. 2age 1?

(This will change$.

Figure 1$: #ashboard tab The whole9house gauge provides the current whole house energ use. It is a su& of all the whole house energ categories, less the energ generator categories.

The energ consu&ption donut shows energ consu&ers b categor and device. The Unaccounted categor is the difference between the total of all consu&ers, and the current whole9house consu&ers (note that this doesn6t include the energ generators$.

2age 1.

Configuration Tab The configuration tab shown in Figure 1A allows configuration of dataMine. @urrentl , this pri&aril includes the configuration of variable logging, but further configuration will be added in future.

Figure 1&: 5onfiguration Tab

Variable !onfiguration
Two lists are presented for variable configuration. The active devices are listed under #evice Variables, and inactive devices are listed under 'host Variables. Ghost variables are variables that still e1ist in the dataMine configuration, but the devices have been deleted fro& Vera. 3ou can delete the ghost variable, or #eep it for reference. 0eleting the ghost variable will not delete the historical data fro& the archive. 5nce configuration changes have been &ade, ou &ust press the effect. Option 0evice !ervice Variable Vera @ategor Description The Vera device na&e and nu&ber (can not be changed$. The Vera !ervice na&e (can not be changed$. The Vera Variable na&e (can not be changed$. The Vera 0evice @ategor (can not be changed$. 2age 14 Save button for the& to ta#e

Option 0ispla >a&e Units Logging Bnabled Bnerg @ategor

Description The displa na&e that will be used in dataMine graphs. / te1t string to set the units. Indicates if dataMine is logging data for this variable. !et to true to enable logging. The dataMine energ categor . This is #ept separate fro& the Vera device categor . The following categories can be used 9F 1 -hole9house + Lighting , Ieating ? Bntertain&ent . /ppliances 4 @o&puting A Utilities +1 !olar 2V generation ++ -ind generation
(>ote that this will be &ade &ore user friendl in future. It is currentl li&ited b the libraries being used$

Graph T pe

!ets the graph t pe to be used with this channel ( 0efault (spline$ 1 Line + !catter , /rea ? /rea7Line co&bined (for status graphs$
(>ote that this will be &ade &ore user friendl in future. It is currentl li&ited b the libraries being used$

0ata 5ffset

/dds a data offset when graphing out the channel. The value will be added to each value to allow calibration of a channel. This does not affect the logged data which will alwa s be the value. /llows &odification of the drows warning ti&e. This is set in &inutes. The drows warning is the ti&e after which the icon is shown. !etting this to ( will use the s ste& default. /llows &odification of the drows error ti&e. This is set in &inutes. The drows error is the ti&e after which the icon is shown. !etting this to ( will use the s ste& default. Bnables or disables filtering of values. Filtered data outside of the specified li&its will not be displa ed in graphs. This can be used to filter out large spi#es. *a1i&u& filter value. 0ata above this nu&ber will not be displa ed. *ini&u& filter value. 0ata below this nu&ber will not be displa ed. Table : Variable configuration o!tions.

0rows -arning

0rows Brror

Bnable out9of9li&its filter

Filter &a1i&u& Filter &ini&u&

2age 1A

Figure 1) shows the 9a1is loo#up. This is pri&aril useful for alphanu&erical data (wind direction, weather condition etc$. dataMine actuall generates this loo#up auto&aticall for such data, however this configuration window allows ou to change the wa the data is graphed. " changing the nu&bers against the labels, ou will change the order the data is graphed in the vertical a1is. It is also possible to use this configuration for nu&erical data that ou would prefer to be displa ed using labels. >ote however that this &a produce strange results for data with continuall var ing data, and is best used for variables with a li&ited nu&ber of possible points (as per the data &entioned above$.

Figure 1): 62a7is loo8u! configuration

Embedding graphs in a web page -hen saving a graph, there is the option to specif a <Huic#View reference=. This is a short na&e for the graph ' it &ust not contain spaces, and it should be #ept short. 3ou can then use this reference to e&bed a full window page into a web browser to displa the saved graph. The U8L should be Jvera ipK7d&7graph.ht&lLrefMJ%uic#refKNperiodMJda sKNupdateMJ&insK If period is specified, it will override the nu&ber of da s to displa the graph, otherwise it will default to the nu&ber of da s specified when the graph was saved. If update is specified, the graph will auto&aticall be updated periodicall . The update ti&e is in &inutes. B&bedded graphs are &eant to be %uic# and full screen. The do not offer the sa&e interaction as 2age 1)

the full dataMine GUI. 0a 7night boundaries can be displa ed, but notifications are not supported.

Advanced Topics
Under the hood
dataMine wor#s b loo#ing for changes in device variables ' it is not periodicall logging data based on ti&e, it si&pl logs changes. This is i&portant to note when thin#ing about how a graph is rendered ' for status t pe data (eg. on7off$, the graph needs to account for this to avoid diagonal lines. dataMine records data in @!V files (@o&&a !eparated Value$. (@urrentl $ three different files are generated for each variable being recorded 9 a raw, hourl and dail file. Files are then split to avoid the& getting too large (which is a potential proble& for constantl changing sensor data such as te&perature$. Bach ti&e Vera observes a change in the data for a &onitored variable, dataMine will log the current value into a file. Bach da , the long ter& histor files are generated fro& the raw data. Bach line of data in the @!V logfiles includes two nu&bers 9 the ti&e of the data9point (as a uni1 ti&esta&p$, and the data value. For the raw file, the actual changes are logged. For the hourl and dail files, the &ini&u& and &a1i&u& value and ti&e are logged for each ti&e period.

USB and "inu#

Linu1 has a different philosoph to -indows when it co&es to attaching U!" &e&or stic#s, or dis# drives. !uch devices are <&ounted= into a director in the root file9s ste&, rather than being seen as a separate drive. / director is first created in the root file9s ste&, and the U!" stic# is then &ounted into this director . This has the following i&plications 9F "efore the U!" stic# is &ounted, if ou were to loo# in the director , ou would see files that are resident in the &ain file9s ste&. /fter the U!" stic# is &ounted, ou will see files that are in the root director of the U!" stic#. If there were an files in the &ount director on the &ain file9s ste&, these will no longer be visible as this director is effectivel replaced b the U!" stic#.

Setting syste$ defaults

/ UI will be added to set the s ste& defaults, but for now, this can be done in the following wa ... T pe the following U8L into our browser
<<Vera Address>>/port_3480/data_request?id=lr_dmCtrl&control=saveCon i!&

then add the following para&eter to the end of the above line to set variables (note ' configuration na&es are case sensitiveG$.

2age 1C

energ -holeIouse*inMnu& energ -holeIouse*a1Mnu& general0rows -arningMnu&

!ets the &ini&u& value of the whole house gauge @an not be set above (. !ets the &a1i&u& value of the whole house gauge. @an not be set less than (. !ets the drows warning s ste& default. Values in &illiseconds ' defaults to 1 hour (,4((((($

general0rows BrrorMnu&

!ets the drows error s ste& default. Values in &illiseconds ' defaults to 1 da ()4?((((($

graph0efault2eriodMnu&

!ets the default graph period. Values in da s ' defaults to 1 da . *ust be between 1 to ,(.

3ou should get the <5O= response if ever thing went o#. >ote that no error chec#ing is perfor&ed when ou &a#e changes ' these settings are used in the GUI, and when the GUI reads the& bac# fro& Vera, it does perfor& a sanit chec# and &a reset to a default if the are dee&ed to be outside of nor&al bounds.

Bac%ing up dataMine data

The following is a &ethod fro& Pvespa&an on how to auto&aticall #eep a bac#up of our data*ine data director . This runs on linu1. >ote that ou need to change the ip address and directories. Fro& Pvespa&an original post Qanuar (4, +(1,
(httpF77foru&.&icasaverde.co&7inde1.php7topic,1,(A).&sgCA+4,.ht&lR&sgCA+4,$

5O, so I have now whipped so&ething up, which Ssea&sS to do about what I need for now. I thin#, that if the logs get ver big over ti&e, &a be this isn6t the correct &ethod. "ut at least it can run at ti&es where house is at sleep. "asicall three stepsF 1. @reate public ssh #e s for unattended login to our Vera fro& our linu1 server +. Using scp script, cop the whole data*ine log direcor to our linu1 server. ,. 2ut a line in ou crontab so it is done once ever night. It would have been nice to use rs nc instead of scp, but it sea&s that it is not included in the vera dist, so scp will have to do. >ot sure how &uch data we will see in the end, but ti&e will tell. !o in detail for an one that wants to use this t pe of bac#upF 1. @reate our public #e s according to e.g. this guideF httpF77wi#i.openwrt.org7do#u.phpL idMoldwi#iFdropbearpublic#e authenticationhowto 2age +(

+. *odif the script for our path6s and vera ip (if ou don6t have 6vera6 in our dns7hosts file$. /long with our veras I2, ou will want to change both destdir and the source path on our vera.
#!/bin/bash date="`date +%Y-%m-%d`" olddate="`date --date '2 months ago' +%Y-%m`" destdir=~/backu /datamine # get the !iles" #era does not ha$e rs%nc& so 'e use regular sc sc -( r root)$era*/mnt/datamine+logs ,destdir/ # create the backu tar ball tar -c -! ,destdir/datamine+,date"tar"g- ,destdir/datamine+logs # !i. rights"" chmod /00 ,destdir/datamine+,date"tar"g# remo$e tem !iles rm -r! ,destdir/datamine+logs # 1emo$e all backu s created 2 months ago rm -! ,destdir/datamine+,olddate-2"tar"g-

@reate the destdir ( in this case T7bac#up7data&ine $ ,. 5nce ou have tested the above configuration, and ou are sure it wor#s ;ust add line to our own crontab (crontab 9e in & case$ and add a line li#e this (running . o6cloc# in the &orning$.
0 3 2 2 2 ~/bin/backu +$era+datamine

If you ha e a proble$
If ou have a proble&, I6& happ to help. Iowever, please send the Vera logfiles, provide infor&ation on what Vera &odel ou are using, if ou have other U!" stic#s fitted. There is now a %uic# wa to get so&e of this with the following U8L
44#516 7899/ ort+:;<0/data+re(uest=id=lr+dm>trl?control=debug

This will du&p our configuration, U!" &ount data, and an recent log entries. Iowever, this will onl wor# once the basic plugin is loaded and running. 2rior to that, please provide as &uch infor&ation as ou can. U!"F 8ead9onl file s ste&
@ogging error logging $ariable /A**urn*u n -org*ser$ice7d*Bem eratureCensorD**>urrentBem erature Enable to o en !ile !or 'rite /dataFine/@o!t Bem erature - >urrentBem erature G122;HI"t.t 5rror* '/dataFine/@o!t Bem erature - >urrentBem erature G122;HI"t.t* 1ead-onl% !ile s%stem'

@aused b errors in the file s ste&. @op all files off and refor&at.

2age +1