Beruflich Dokumente
Kultur Dokumente
0 SIEMPRE Library
Part I Description
The EyesWeb SIEMPRE Library includes a collection of software modules and patches which have been specically designed and developed to support the requirements from the EU ICT Siempre project. In particular, the SIEMPRE Library integrates the EyesWeb platform with modules which are required for a distributed multimodal data recording, playback, analysis. The SMPTE decoder block received an audio signal in inputs and decodes the SMPTE timecode contained in the signal. See SMPTE time code and the references therin for a description of the SMPTE encoding schema. The SMPTE encoder block generates an audio signal with an encoded time code. The Wave File Writer block writes timeseries in the binary format chosen for the SIEMPRE recordings. The chosen binary format uses a small subset of the Broadcast Wave Format specications. The DeckLink input block add support for the DeckLink family of framegrabbers, which are used in the Casa Paganini setup (see Acquisition setup at Casa Paganini) for synchronized audio/video input from High Denition (HD) videocameras. The Qualisys SMPTE decoder block, which decodes the SMPTE timecode as packed in 32 bits by the Qualisys Track Manager software. This block, toghether with the OSC support already available in EyesWeb, allows integration between EyesWeb and the Qualisys system. This provides the possibility to analyze, in real-time, the data tracked by the Qualisys Motion Tracking system. For the aim of the SIEMPRE project, the integration is particularly useful to be able to synchronize with the Qualisys Software to visualize the recorded data. The Qualisys SMPTE encoder block, which encodes an SMPTE timecode to a packed 32 bits integer number with the same coding schema as the Qualisys Track Manager software. The Timecode generator block, which generates a timecode locked to an audio input signal; it may be useful for data playback. The Qualisys RT Protocol Decoder block, which get the complete data packet, as sent by the Qualisys Track Manager software, and decodes all of the data in it. It let you to get all the data in a compact way; e.g., all the tracked markers are received in an unique labelled set, instead of being received by separate OSC blocks. The Qualisys TSV Reader block, which reads the data exported by the Qualisys Track Manager software in the TSV (Tab Separated Values) format. In the following chapters, some examples are described, which show the overall infrastructure used for many of the recordings which happens in the EU ICT Siempre project and the related EyesWeb patches.
Figure 1.1: The acquisition setup installed at Casa Paganini, InfoMus Lab, Genova, Italy.
Figure 1.4: A patch to save the multichannel audio signal from the microphones on the instruments.
The setup at Casa Paganini uses Adobe Audition as the audio editor Of course, the le must have been recorded with Smpte enabled
Figure 2.1: The qualisys rt.eywx patch shows the integration between EyesWeb and the Qualisys RTM software.
Figure 2.2: The qualisys controlled smpte.eywx patch regenerates an audio Smpte timecode based on the Smpte valued received by Qualisys via OSC.
which adds bones among markers to form some skeleton structures. These structures are nally added to an OpenGl 3d world and visualized to the user. Note the dierence with the patch shown in Figure 2.1: in that patch an OSC receiver is used to get a single coordinate (the Z) of a single marker. That aproach makes sense when you need to show a small number of markers and you know their labels in advance (as the OSC receiver needs the name of the marker as the OSC tag; however, when the number of marker increases, as it is in the String Quartet case, that approach is unfeasible due to the large number of blocks which would be required. To overcame this problem, the Qualisys Visualization patch uses the RT Protocol decoder block: this block receives a raw packet from the OSC block, and decodes all the data in it (the packet is composed of many OSC bundles). The data is sent to the output pins; in particular two output pins provide the labelled set of identied and unidentied markers. Thus, the data can be easily managed with standard EyesWeb blocks working with geometric labelled sets, or it can be visualized to the user as it happens in this patch.
Figure 2.3: The qualisys visualization.eywx patch shows the visualization, in EyesWeb, of the 3D mocap data.
10
Figure 3.1: The qualisys tsv player.eywx patch shows the visualization, in EyesWeb, of the 3D mocap data exported by Qualisys in TSV format.
11
Figure 3.2: The qualisys tsv processing.eywx patch shows a simple processing of data coming from a Qualisys TSV le.
12
Part II Reference
13
Blocks
Qualisys6DTSVReader
bitmap
Read TSV le from Qualisys. The le must contains at least markers name. In the TSV export settings window check: - Include TSV header - Export time data for every frame Export column headers - Include TSV header Outputs OutputDoubleLabelledSet id output labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing points from qualisys.
14
OutputUnidentiedDoubleLabelledSet id output unidentify labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing unidentied points from qualisys. OutputFrameNumber id output frame number type Kernel, Int datatype (Kernel Catalog). type id kernel, int inplace id *no* inherited id *no* Extracted datas frame number. SMPTE id output smpte type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* Extracted datas SMPTE. Parameters FileName id le name type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=false, layout OverwritePrompt=true, Filter=TSV les (*.tsv)*.tsv File name
15
ActivationMode id read mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Bang Line layout Time Polling domain [ 0, 4 ) Activation mode: Bang: external clock connected to the input, is used to activate the block. On line number: external integer value connected to the input (which is interpreted as the line number of the le) is used to activate the block. On time: external time value connected to the input (which is interpreted as one of the time columns of the le) is used to activate the block. Polling: an internal signal is used to activate the block. If the le has at least one time column the signal is generated according to the oset of consecutive times. If there is no time, Eyesweb automatically activates the block. LoopMode id eof mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Loop Stop layout Pause NoOutput domain [ 0, 4 ) Loop mode. It works only if ActivationMode is set to Polling mode. Loop: activates circular loop mode. Stop: not implemented yet. Pause: generates the current value of le. If the input value of line or time is out of bound it generates rst or last value of le. NoOutput: output is not generated. UseSMPTE id use SMPTE type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool It species if the block use SMPTE or time column to read the le.
16
4.1.2
QualisysC3DReader
bitmap
Read C3D le format. The le follows C3D specications(www.C3D.org) Actually the block doesnt read analod data. Outputs OutputDoubleLabelledSet id output labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing points from qualisys. OutputUnidentiedDoubleLabelledSet id output unidentify labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing unidentied points from qualisys.
17
Parameters FileName id le name type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=false, layout OverwritePrompt=true, Filter=C3D les (*.C3D)*.C3D File name ActivationMode id read mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Bang Line layout Time Polling domain [ 0, 4 ) Activation mode: Bang: external clock connected to the input, is used to activate the block. On line number: external integer value connected to the input (which is interpreted as the line number of the le) is used to activate the block. On time: external time value connected to the input (which is interpreted as one of the time columns of the le) is used to activate the block. Polling: an internal signal is used to activate the block. If the le has at least one time column the signal is generated according to the oset of consecutive times. If there is no time, Eyesweb automatically activates the block. LoopMode id eof mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Loop Stop layout Pause NoOutput domain [ 0, 4 ) Loop mode. Loop: activates circular loop mode. Stop: not implemented yet.
18
Pause: generates the current value of le. If the input value of line or time is out of bound it generates rst or last value of le. NoOutput: output is not generated.
19
4.1.3
QualisysRTProtocolDecoder
bitmap
Decodes the Raw Data received from the Qualisys QTM software via the OSC RealTime protocol. Details The OSC Realtime protocol of QTM can also be decoded by using standard EyesWeb OSC blocks. However, receiving data about all the markers is not strainghforward, as the number unidentied QTM trajectories is time-varying. This block gets all such data as once, building labelled-sets as output Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs RawData id type type id input Base, RawData base, raw data required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The Raw Data received from the Qualisys QTM software via the RT protocol
Outputs Timestamp id timestamp output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The time elapsed since the ecordng was started
20
SMTE timecode id smpte output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The value of SMPTE as received by QTM Frame number id frame number type Kernel, Int datatype (Kernel Catalog). type id kernel, int inplace id *no* inherited id *no* The frame number QTM Unidentied markers id unidentied markers type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The set of markers which were not recognized by QTM as belonging to an AIM model Identied markers id identied markers type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The set of markers which were recognized by QTM as belonging to an AIM model Parameters TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 )
21
The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom Sampling rate id subframe sampling rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) The sampling rate of the data in the source system. Since the SMPTE is expressed also using frames and subframes, the sampling rate of the date is required to correctly interpret the frame and subframe elds.
22
4.1.4
QualisysSmpteDecoder
bitmap
Decodes an SMPTE timecode from an integer number. The SMPTE is assumed to be encoded with the same format used in the Qualisys RT protocol (see details). Details SMPTE time code little endian format: Bit 0 ? 4: Hours Bit 5 ? 10: Minutes Bit 11 ? 16: Seconds Bit 17 ? 21: Frame Bit 22 ? 30: Sub frame Bit 31: Valid bit Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Encoded SMPTE timecode id input type Kernel, Int datatype (Kernel Catalog). type id kernel, int required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The encoded SMPTE timecode as provided by the Qualisys RT protocolo. Usually obtained by means of the OSC block Outputs Decoded timecode id output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The decoded timecode
23
Parameters TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 ) The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom Sampling rate id subframe sampling rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) The sampling rate of the data in the source system. Since the SMPTE is expressed also using frames and subframes, the sampling rate of the date is required to correctly interpret the frame and subframe elds.
24
4.1.5
QualisysSmpteEncoder
bitmap
Encodes an SMPTE timecode to an integer number, to make it suitable, for instance, to be sent via OSC. The encoding protocol is the one used by the Qualisys RT communication protocol (see details). Details SMPTE time code little endian format: Bit 0 ? 4: Hours Bit 5 ? 10: Minutes Bit 11 ? 16: Seconds Bit 17 ? 21: Frame Bit 22 ? 30: Sub frame Bit 31: Valid bit Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Timecode id type type id input Kernel, Time datatype (Kernel Catalog). kernel, time required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* Timecode to be encoded to a packed SMPTE integer
Outputs Encoded timecode id output type Kernel, Int datatype (Kernel Catalog). type id kernel, int inplace id *no* inherited id *no* The encoded timecode
25
Parameters TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 ) The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom Sampling rate id subframe sampling rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) The sampling rate of the data in the source system. Since the SMPTE is expressed also using frames and subframes, the sampling rate of the date is required to correctly interpret the frame and subframe elds.
26
4.1.6
QualisysTSVReader
bitmap
Read TSV le from Qualisys. The le must contains at least markers name. In the TSV export settings window check: - Include TSV header - Export time data for every frame Export column headers - Include TSV header Outputs OutputDoubleLabelledSet id output labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing points from qualisys. OutputUnidentiedDoubleLabelledSet id output unidentify labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing unidentied points from qualisys. OutputFrameNumber id output frame number type Kernel, Int datatype (Kernel Catalog). type id kernel, int inplace id *no* inherited id *no* Extracted datas frame number.
27
SMPTE id output smpte type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* Extracted datas SMPTE. Parameters FileName id le name type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=false, layout OverwritePrompt=true, Filter=TSV les (*.tsv)*.tsv File name ActivationMode id read mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Bang Line layout Time Polling Frame domain [ 0, 5 ) Activation mode: Bang: external clock connected to the input, is used to activate the block. On line number: external integer value connected to the input (which is interpreted as the line number of the le) is used to activate the block. On time: external time value connected to the input (which is interpreted as one of the time columns of the le) is used to activate the block. Polling: an internal signal is used to activate the block. If the le has at least one time column the signal is generated according to the oset of consecutive times. If there is no time, Eyesweb automatically activates the block. On frame number: external integer value connected to the input (which is interpreted as the frame number of the le) is used to activate the block.
28
MediaTimeMode id media time type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool It species if the block set time column of the le as media time. If not, interpolated time is used to set the value. If the input is not a time , this parameter is ignored and the block take the column value in the le ad default. LoopMode id eof mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Loop Stop layout Pause NoOutput domain [ 0, 4 ) Loop mode. It works only if ActivationMode is set to Polling mode. Loop: activates circular loop mode. Stop: not implemented yet. Pause: generates the current value of le. If the input value of line or time is out of bound it generates rst or last value of le. NoOutput: output is not generated. UseSMPTE id use SMPTE type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool It species if the block use SMPTE or time column to read the le.
29
4.1.7
QualisysTSVWriter
bitmap
Writes to le a geometric labelled set of 3d points. Points coordinates are written as tab-separated values, using the same format of the Qualisys software. Notes By default, the block writes the data in a Qualisys compatible format. To this aim, an header is written in the le which is needed by Qualisys to correctly read the data; the header is needed even if you plan to read the generated le through the EyesWeb QualisysTsvreader block. However, the user can decide to omit such header; this makes the resulting le not compatible with the Qualisys software, but easier to be imported in third parties software (e.g., Calc, Excel, Matlab). In this case, since the column names are not written: it is up to the user to remember the meaning of each column Labels mode parameter: the column number of the generated le must uniquely identify a marker (label), otherwise reading back the le will be impossible. However, the labels available in the input dataset might change during time (e.g., missing markers when capturing, new markers available, etc), thus, the problem arises to decide which labels (and hence which columns) will be written to le. The user can select among six possible solutions, through the Labels mode parameter: Custom: the user can specify the number of labels and, for each label, its value. Note that the set of user-specied labels is furtherly processed by the block to get rid of empty or duplicate labels Inputs InputGeometricLabelledSet3DDouble id input labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no*
30
Input Geometric Labelled Set of 3D double points. This must be an homogeneous list of 3d geometric points. Parameters Filename id le name type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=true, layout OverwritePrompt=true, Filter=TSV les (*.tsv)*.tsvAll les (*.*)*.* The name of the output le Input type id input type type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Double layout Int domain [ 0, 2 ) Species if the input geometric labelled set is double or int. Write time column id write time type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Species if the block writes the time column on the output le. The value of the time column is extracted from the media time of the input data Write smpte column id write smpte type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Species if the block writes the smpte column on the output le. If set to true, an additional input will be available; the value of the smpte is received through this input Write frame number column id write frame number type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Species whether to block writes the frame number column on the output le
31
First frame number id rst frame number type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) Species the value of the rst frame number. By default, frames start at 1, but a dierent value can be specied. Write header id write header type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Species whether the block writes the le header or not. See the block notes for details NumOfCameras id num of camera type Kernel, Int datatype (Kernel Catalog). type id kernel, int Number of cameras. Its in the Qualisys TSV header. Frequency id frequency type Kernel, Int datatype (Kernel Catalog). type id kernel, int Frequency. Its in the Qualisys TSV header. Number of analog channels id num of analog type Kernel, Int datatype (Kernel Catalog). type id kernel, int Number of analog channels. This value is not used actually by EyesWeb when reading the TSV, however, it is used by the Qualisys software and must match the number of columns in the corresponding * a.tsv le. Analog frequency id analog freq type Kernel, Int datatype (Kernel Catalog). type id kernel, int Analog frequency. This value is not actually used by EyesWeb when reading the TSV, however, it is used by the Qualisys software and refers to the sampling frequency of the analog data which is available in the corresponding * a.tsv le. Description id description type Kernel, String datatype (Kernel Catalog). type id kernel, string 32
A free-text description to be wrote in the TSV header TimeStamp id time stamp type Kernel, String datatype (Kernel Catalog). type id kernel, string A string representing the timestamp (date and time) of the recording session (e.g., Friday, December 21st, 2012 Labels mode id labels mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom First non-empty frame First frame layout Kinect Microsoft SDK Kinect OpenNI File domain [ 0, 6 ) Species how the block decides which labels are written to le. See the block notes for details
33
4.1.8
SmpteDecoder
bitmap
Details Based on the LTC SMPTE library (http://ltcsmpte.sourceforge.net/), with minor changes to build under Win32. The derived source code is available at https://svn.infomus.org/publ Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Audio Stream id type type id input Base, PCMAudioBuer base, pcm audio buer required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The input audio stream which should contain the SMPTE timecode track
Outputs Timecode id output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The decoded timecode
34
Parameters TimeCode Track id time code track type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 0, +innity ) The zero-based index of the timecode track. If it is greater than the number of available channels, the last channel will be used. Output correction mode id output correction mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: None layout Output oset Buer start domain [ 0, 3 ) In general the SMPTE timecode might be not aligned with the audio buer. This parameter species how to cope with this issue. None: the error caused by this misalignemnt is neglected Output oset: an addition output is added to the block. The output species where the SMPTE timecode started with respecte to rst sample of the audio buer. The oset might be negative Buer start: the output timeoeis adjusted such that it can be interpreted as the timecode assigned to the rst sample of the buer Enable Output Locked id enable output locked type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Enable/disable an output providing info about whether the SMPTE signale has beed locked TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 )
35
The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom
36
4.1.9
SmpteEncoder
bitmap
Details Based on the LTC SMPTE library (http://ltcsmpte.sourceforge.net/), with minor changes to build under Win32. The derived source code is available at https://svn.infomus.org/publ Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Audio clock id type type id input Base, Audio clock base, audio clock required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The input audio clock
Outputs Output id output type Base, PCMAudioBuer type id base, pcm audio buer inplace id *no* inherited id *no* The encoded timecode
37
Parameters TimeCode id time code type Kernel, Time datatype (Kernel Catalog). type id kernel, time The initial timeocde. Reset id reset type Kernel, Trigger datatype (Kernel Catalog). type id kernel, trigger Reset the encoder to the initial timecode value TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 ) The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom Enable timecode output id enable timecode output type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool If this parameter is true, an additional output is added to the block which provides the interpolated timecode of the begining of the audiobuer (i.e., the timecode virtually assigned to sample 0 of the buer)
38
4.1.10
TimeCodeGenerator
bitmap
Generatae a timecode starting locked to the input audio clock signal. Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Audio clock id type type id input Base, Audio clock base, audio clock required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The input audio clock
Outputs Timecode output id timecode output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The timecode virtually assigned to the sample 0 of the output buer Parameters TimeCode id time code type Kernel, Time datatype (Kernel Catalog). type id kernel, time
39
The initial timeocde. Reset id reset type Kernel, Trigger datatype (Kernel Catalog). type id kernel, trigger Reset the encoder to the initial timecode value
40
4.1.11
WaveFileWriter
bitmap
Write the input time series as a wav le. Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs TimeSeries id type type id input Kernel, Generic datatype kernel, generic datatype required for initialization required required for execution read only/read write read write referred as inplace *no* referred as inherited *no* Time series to be saved to le
Required interfaces Kernel, StaticTimeSeries Parameters Filename id param lename type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=true, layout OverwritePrompt=true, Filter=Wave les (*wav)*.wavAll les (*.*)*.*All les (*.*)*.* Name of the wav le. Use an empty name to stop the recording
41
param format Kernel, Int datatype (Kernel Catalog). kernel, int Combo Box: 32bit oat (oat) 64bit oat (Double) layout Normalized 32bit oat (oat) Normalized 64bit oat (Double) domain [ 0, 4 ) Denes the format used to write the sample into the le : oat each sample is writed as 32bit oating value; double each sample is writed as 32bit oating value; Title id param title type Kernel, String datatype (Kernel Catalog). type id kernel, string Title tag. It will be mapped to the INAM Exif tag Datetime id param datetime type Kernel, String datatype (Kernel Catalog). type id kernel, string Datetime tag. It will be mapped to the ICRD Exif tag Comment id param comment type Kernel, String datatype (Kernel Catalog). type id kernel, string Comment tag. Free text that will be mapped to the ICMT Exif tag Timecode id param timecode type Kernel, String datatype (Kernel Catalog). type id kernel, string Timecode tag. It will be mapped to the ISMP Exif tag. AddMetaData id add metadata info type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: No metadata layout Using InfoChunk Using iXml domain [ 0, 3 )
42
43
4.1.12
DeckLinkInput
bitmap
Acquire synchronized audio/video from a DeckLink card. Notes With the partial support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Outputs Output id type type id inplace id inherited id video output Base, Image base, image *no* *no*
AudioOutput id audio output type Base, PCMAudioBuer type id base, pcm audio buer inplace id *no* inherited id *no* Parameters Device Index id device index type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 0, +innity ) The zero-base index of the Decklink card, in the case that more than one is installed in the system
44
InputMode id input mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Audio/Video layout Video only domain [ 0, 2 ) Species whether to capture audio, or video, or both Video Mode id video mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: NTSC 720x486 30/1.001fps interlaced NTSC 720x486 30/1.001fps interlaced (3:2 pulldown) PAL 720x576 25fps interlaced NTSC 720x486 30/1.001fps progressive PAL 720x576 25fps progressive HD 1920x1080 24/1.001fps interlaced HD 1920x1080 24fps interlaced HD 1920x1080 25fps progressive HD 1920x1080 30/1.001fps progressive HD 1920x1080 30fps progressive HD 1920x1080 25fps interlaced layout HD 1920x1080 30/1.001fps interlaced HD 1920x1080 30/1.001fps interlaced HD 1920x1080 50fps progressive HD 1920x1080 60/1.001fps progressive HD 1920x1080 60fps progressive HD 1280x720 50fps progressive HD 1280x720 60/1.001fps progressive HD 1280x720 24/1.001fps progressive 2K 2048x1556 60fps interlaced 2K 2048x1556 60fps interlaced 2K 2048x1556 60fps interlaced domain [ 0, 22 ) Select the resolution and framerate of the video stream
45
Pixel Format id pixel format type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: UYVY 4:2:2 packed layout BGRA 4:4:4 raw domain [ 0, 2 ) Select the pixel format of the video stream Num Audio Channels id num channels type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) Species the number of audio channels to capture Audio Sample Format id sample format type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: 16bits signed layout 32bits signed domain [ 0, 2 ) Species the format of each sample of audio
46
47
Monodimensional timeseries will be saved or converted to time-aligned Broadcast Wave Format compatible les (i.e., extended WAVE les). The SIEMPRE project will exploit subset of the Broadcast Wave Format standard. In particular it will x the number of channels to 1 (monodimensional data), the format to WAVE FORMAT IEEE FLOAT (0x03), and the bits per sample to 32. For a general description of Broadcast Wave Format and Wave les you may refer to the Microsoft documentation or start from the following links: Audio File Format Specifications SonicSpot
A.1.1
Format description
The WAVE (sub-)format used for SIEMPRE has the following structure: Field ChunkID ChunkSize WaveID Length 4 4 4 Content RIFF Size, in bytes, of the RIFF chunk. Should be equal to the length, in bytes, of the le minus 8. WAVE Some chunks. Each chunk is identied by a fourCC (four characters identier) followed by the size, in bytes, of the chunk (the size does not include the FourCC and the size itself). Two mandatory chunks are the fmt chunk and the data chunk, which contains the characteristics of the le (sample rate, num channels, sample size, etc.) and the sample data respectively.
Wave chunks
...
48
The fmt chunk and the data chunks have the following structure:
Length 4 4 2 2
NumSamplesPerSeq 4 AvgBytesPerSec 4
BlockAlignment
BitsPerSample
Content fmt For the SIEMPRE case the value is 16, a generic reader should be prepared to face dierent sizes too. For the SIEMPRE case the value is WAVE FORMAT IEEE FLOAT, i.e., 0x03. For the SIEMPRE case the value is xed to 1, i.e., monodimensional channels Sampling rate (e.g., somthing around 250.0 for kinematical sensors) Data rate, in bytes, of the le. Can be computed as NumSamplesPerSeq * BlockAlignment Size of a block (a sample for each channel). Can be computed as NumChannels * (BitsPerSample / 8). For the SIEMPRE case NumChannels is xed to 1, BitsPerSample is xed to 32, thus BlockAlignement is xed to 4. Number of bits for each sample. For the SIEMPRE case this is xed to 32 (single precision oating point numbers)
Length 4 4 ... 0 or 1
Content data For the SIEMPRE case the value is 4 (size of sample) multiplied by the number of samples This is the actal data: sequence of oat numbers in the SIEMPRE case In the SIEMPRE case this is not needed as data is aligned. In the general case this is only needed if data is not aligned to a 16bits boundary
49
A.1.2
Application compatibility
The les produced by the Wave File Writer block have been tested to load in the following applications: Players VLC media player Windows Media Player Quick Time player Music Software Audacity Sony Sound Force Sony Vegas Pro Computing environments Matlab: does not load additional info about the le. This is a general limitation for the WAVE format support in Matlab.
50