Developer Manual

opehD8copy - Developer Mahual
hIIp://opendbcopy.org
openD8copy is an Open Source ProjecI hosIed aI Sourcelorge
hIIp://www.sourceorge.neI/projecIs/opendbcopy/
Release 0.5!
DaIe 26.07.04
Re!erehce developermanual0.5!
2004 by AnIhony SmiIh, Puzzle lTC GmbH all righIs reserved.
ch200407J0w5874
Puzzle lTC GmbH
MaulbeersIrasse !5
CHJ0!! 8ern
SwiIzerland
Phone: 4! (0)J! J98 44 77
lax: 4! (0)J! J98 44 78
www.puzzle.ch
openD8copy Developer Manual Pelease 0.5!
DocumehI HisIory
DaIe Versioh Persoh CommehIs
26/07/04 0.5! A. SmiIh lirsI issue
28/07/04 0.5! A. SmiIh lurIher chapIers added and reviewed
1able o! CohIehIs
! Overview .............................................................................................................................................. J
2 SoIware PequiremenIs ....................................................................................................................... J
J openD8copy Core lramework and Plugins on Ihe Web ..................................................................... 4
4 ArchiIecIure ......................................................................................................................................... 5
4.! MainConIroller .............................................................................................................................. 5
4.2 !obManager .................................................................................................................................. 6
4.J PluginManager ............................................................................................................................. 6
4.4 PluginScheduler ............................................................................................................................ 6
4.5 DynamicPluginThread ................................................................................................................... 6
4.6 A Plugin lmplemenIaIion .............................................................................................................. 6
5 Plugin ComponenIs ............................................................................................................................. 7
5.! Plugin Model ................................................................................................................................. 7
5.!.! Plugin ConiguraIion .............................................................................................................. 7
5.!.2 CreaIing new Plugin Models .................................................................................................. 9
5.2 Graphical User lnIeraces ............................................................................................................ !0
5.2.! GUl ConiguraIion ................................................................................................................ !!
5.2.2 CreaIing your own GUls ....................................................................................................... !!
5.J Plugin Thread .............................................................................................................................. !2
5.4 Language Pesources ................................................................................................................... !2
6 Lxample The Copy Plugin ................................................................................................................ !J
6.! Overview ..................................................................................................................................... !J
6.2 Plugin DaIabaseModel ................................................................................................................ !J
6.2.! Plugin ConiguraIion lile ...................................................................................................... !J
6.2.2 Plugin AIIribuIes .................................................................................................................. !J
6.2.J Con LlemenIs ...................................................................................................................... !4
6.2.4 Threads LlemenI(s) ............................................................................................................... !4
6.2.5 lnpuI / OuIpuI LlemenIs ...................................................................................................... !5
6.2.6 Source & DesIinaIion DaIabase LlemenIs ............................................................................ !5
6.2.7 Mapping LlemenI ................................................................................................................ !6
6.2.8 lilIer LlemenIs ...................................................................................................................... !6
6.J Graphical User lnIeraces ............................................................................................................ !6
6.J.! GUl AIIribuIes ...................................................................................................................... !7
6.4 Plugin Thread .............................................................................................................................. !7
6.5 Language Pesource liles ............................................................................................................. 22
7 8uilding, TesIing and Deploying Plugins ............................................................................................ 2J
2/2J
Puzzle lTC GmbH
1 Overview
openD8copy is an open source projecI by AnIhony SmiIh, Puzzle lTC GmbH. The projecI is hosIed aI
Sourceorge and published under Ihe Ierms o Ihe GNU General Public License.
openD8copy musI be undersIood as a ramework Io conigure and execuIe plugins. A plugin can do
ANYTHlNG. Some plugins are daIabase speciic, oIhers provide eaIures Io do ile manipulaIions and
Iransers eIc. There is no limiIaIion o whaI a plugin can or could do.
The core ramework iIsel is managed and checked in as one separaIe projecI in Ihe CVS PeposiIory aI
Sourcelorge. Lach plugin is managed in a separaIe projecI paIh wiIhin Ihe same CVS PeposiIory.
Please use Ihe public orum Io posI quesIions. There are also links Io requesI new eaIures and Irack
bugs. The mailing lisIs have been deleIed because excepI one person were posIing quesIions using Ihe
orum.
lor simple plugin developmenI and deploymenI use an exisIing plugin.
Name WebsiIe
openD8copy aI Sourcelorge hIIp://sourceorge.neI/projecIs/opendbcopy/
openD8copy Download hIIp://sourceorge.neI/projecI/showiles.php?group_id~9!406
openD8copy lorum hIIp://sourceorge.neI/orum/?group_id~9!406
openD8copy WebsiIe hIIp://opendbcopy.org
hIIp://opendbcopy.ch
GNU General Public License hIIp://www.gnu.org/licenses/gpl.IxI
Puzzle lTC WebsiIe hIIp://www.puzzle.ch
(Puzzle provides a compleIe Sourceorge mirror :)
2 So!Iware RequiremehIs
openD8copy requires a !ava PunIime LnvironmenI. l noI yeI insIalled, please download an acIual
!ava PunIime LnvironmenI or SIandard Developer KiI !.4 rom hIIp:// java.sun.com .
To develop and / or debug openD8copy or plugins you require a !ava lDL. Am currenIly using
Lclipse J.0, see hIIp://www.eclipse.org.
OpenD8copy Core lramework and Plugins are builI and deployed using Apache AnI. Download
Apache AnI (hIIp://anI.apache.org) i you have noI yeI insIalled an acIual version (!.5 or higher).
An acIual !D8C driver or each daIabase sysIem one plans Io access.
OIher libraries required or new plugins
OpenD8copy 0.5 has been IesIed using !2SDK !.4.2_0J.
The laIesI sources can be checked ouI rom Ihe CVS PeposiIory. lor deIails see Ihe projecIs websiIe.
J/2J
Puzzle lTC GmbH
3 opehD8copy Core Framework ahd Plugihs oh Ihe Web
The openD8copy core ramework and plugins are managed and sIored wiIhin separaIe direcIories in
Ihe CVS PeposiIory aI Sourcelorge.
The core rameworks direcIory is named opendbcopy. Lach plugins direcIory is preixed wiIh
opendbcopy. L.g. opendbcopy.copy (as highlighIed) conIains Ihe Copy Plugin or openD8copy.
Please noIe IhaI Ihe sources are someIimes noI commiIed beore a new release is posIed. l you
require an updaIe please do noI hesiIaIe Io conIacI Ihe developer Ieam.
4/2J
Puzzle lTC GmbH
4 ArchiIecIure
Plugins are execuIed wiIhin separaIe Threads. The Ihreads liecycle is managed by openD8copys core
ramework.
The above diagram is noI compleIe. lI shows some imporIanI classes wiIh aIIribuIes and meIhods.
4.! MainConIroller
MainController is Ihe main class Io launch openD8copy. lI is Ihe cenIral class and parenI o all
Iop manager classes. openD8copy is designed using Ihe ModelViewConIroller PaIIern MVC. The
conIrollers job is Io handle requesIs on Ihe model and handover errors rom Ihe model Io any view
(caller o a requesI).
openD8copy can be run in inIeracIive or baIch mode. Graphical User lnIeraces (GUl) provide Ihe
abiliIy Io conigure jobs and plugins, execuIe single plugins and plugin chains, also called jobs. All
coniguraIion parameIers can be sIored in xml iles. Once a job is conigured and ready or producIion,
openD8copy can be execuIed in baIch mode, wiIhouI loading any GUls. This allows saving memory
and place openD8copy on a server noI providing any Window ComponenIs (X Server). A job is sIored
in a single xml ile and conIains plugin meIadaIa o one or several plugin models.
5/2J
Puzzle lTC GmbH
4.2 !obManager
JobManager is owned and creaIed by MainController. JobManager is Ihe owner o a job
model. A job may conIain zero Io n plugins. A JobManager insIance iIsel uses a PluginManager
insIance Io manage plugins. JobManager provides meIhods Io imporI and exporI a job
coniguraIion. An execute() meIhod handles requesIs on Ihe job model or handles requesIs over Io
Ihe appropriaIe plugin model, which is parI o Ihe job model.
4.J PluginManager
PluginManager is owned and creaIed by JobManager. lIs responsibiliIy is Io parse and
dynamically load plugin resources rom direcIories. AIer loading resources and meIadaIa o plugins,
PluginManager is able Io add a plugin Io one o Iwo lisIs. One lisI conIains plugins which are noI
bound Io a job, only available in memory. A second lisI conIains plugins which are parI o a plugin
chain bound Io a job which can be saved as a single uniI.
While PluginManager is responsible or loading plugin resources and models, iI is noI responsible
or loading and managing Graphical User lnIeraces o plugins. This is done by PluginGuiManager.
4.4 PluginScheduler
PluginScheduler is a singleIon, owned by PluginManager. PluginScheduler receives a lisI o
plugins Io execuIe a plugin chain or a single plugin model. lIs responsibiliIy is Io dynamically load a
plugin Ihread, pass ouIpuI parameIers as inpuI Io ollowing plugins, inIerrupI single plugins and
plugin chains when requesIed and inally schedule plugins. A plugin which has been successully
execuIed is removed rom PluginSchedulers lisI. NoI so rom a plugin chain sIored in
PluginManager.
4.5 DynamicPluginThread
DynamicPluginThread exIends java.lang.Thread and implemenIs Ihe lnIerace IExecute.
IExecute only deines one meIhod execute().
The inal public run() meIhod implemenIed by DynamicPluginThread calls setUp(),
execute() and tearDown(). lI also handles Ihe sIaIe o a plugin.
Plugins implemenIed musI overwriIe Ihe execute() meIhod o DynamicPluginThread.
OpIionally setUp() and tearDown() meIhods o DynamicPluginThread can be overwriIIen.
setUp() and tearDown() are called beore, respecIively aIer execute() has been successully
compleIed (no excepIion Ihrown). Use Ihe setUp() meIhod Io read coniguraIion parameIers, seIup
daIabase connecIions eIc. The execuIe meIhod is called aIer successul setUp(), e.g. coniguraIion
parameIers were successully read and validaIed, daIabase connecIion(s) opened eIc. Do noI use Ihe
tearDown() meIhod Io close daIabase connecIions as Ihis meIhod is only called aIer successul
execute(). Use tearDown() or example Io close ile wriIers or whaIever else. Close connecIions
in a finally block o your execute() meIhod body.
4.6 A Plugin lmplemenIaIion
A plugins Ihread class, e.g. CopyMappingPlugin, musI exIend DynamicPluginThread.
The only required meIhod Io implemenI is
public final void execute() throws PluginException
6/2J
Puzzle lTC GmbH
5 Plugih CompohehIs
Lach openD8copy plugin consisIs o Ihe ollowing componenIs:
Plugin Model
Graphical User lnIeraces
Plugin Thread
Language Pesource liles
lor an example please have a look aI Ihe ollowing chapIer.
5.! Plugin Model
The model o a plugin is represenIed using a plugin.xml ile and !ava Model class. All parameIers
speciied wiIhin such a plugin.xml ile can be used when seIIing up, execuIing or Iearing down a
plugin. One can provide iIs own !ava Model, exIend exisIing ones or jusI use exisIing ones.
All plugin.xml iles musI conIain Ihe ollowing elemenIs
ElemehI Name DescripIioh
conf
This elemenI conIains coniguraIion elemenIs. openD8copy provides a
generic graphical user inIerace Io display and manipulaIe plugin
coniguraIion parameIers. See Ihe nexI chapIer, Generic AIIribuIes or GUl
ConiguraIion or deIails.
threads
luIure releases may provide Ihe abiliIy Io speciy more Ihan one Ihread.
CurrenIly Ihe elemenI threads musI conIain one childelemenI called
thread, speciying Ihe Ihread class Io execuIe and a description
aIIribuIe.
input
Used Io place ouIpuI rom a ormer plugin as inpuI or Ihe acIual plugin
output
Used Io sIore possible ouIpuI o a plugin model
Lxample plugin model wiIh minimum required elemenIs (replace aIIribuIe values accordingly)
<plugin identifier="id" model_class="org.abc.ModelClass">
<conf />
<threads>
<thread thread_class="org.abc.ThreadClass" description="key" />
</threads>
<input />
<output />
</plugin>
5.!.! Plugin ConiguraIion
Using Ihe graphical user inIerace opendbcopy.gui.PluginConfiguration Io display and
manipulaIe parameIers is noI mandaIory. One can also provide iIs own graphical user inIerace(s) Io
manipulaIe daIa and coniguraIion parameIers. WiIhin Ihe con elemenI one can add child elemenIs.
To add parameIers which can be conigured using Ihe generic GUl PanelConfiguration, Ihe
ollowing aIIribuIes are currenIly supporIed, see an example below.
AIIribuIe Name DescripIioh
type
WhaI GUl Iype is required Io conigure a parameIers value
7/2J
Puzzle lTC GmbH
value
May conIain a deaulI value or be empIy
required
true or false currenIly noI IesIed auIomaIically
description
key Io resource enIry in resource ile (language speciic descripIions)
The aIIribuIe type currenIly supporIs Ihe ollowing Iypes. Type names are case insensiIive.
type values DescripIioh
string
SIring parameIers
boolean
ParameIers which musI be Irue or alse
int
lnIeger value parameIers. TesIed when user enIers a
value.
file_dir_filelists_selection
Drop down box or selecIion o ile, direcIory or ilelisI
file
lile browser
dir
DirecIory browser
hibernate_dialect
Drop down box or HibernaIe dialecI selecIion
(see
hIIp://www.hibernaIe.org/hib_docs/api/neI/s/hibernaIe/di
alecI/packagesummary.hIml)
An example or selecIion o a direcIory. The deaulI direcIory is dump, which may be relaIive or
absoluIe (e.g. /home/smitha/.opendbcopy/inout/dump, c:\temp).
<conf>

<dir required="true" value="dump" type="dir"
description="plugin.opendbcopy.dump.conf.outputDir" />
</conf>
8/2J
Puzzle lTC GmbH
5.!.2 CreaIing new Plugin Models
A plugins daIa conIainer is a model. OperaIions on a model can be implemenIed in separaIe classes or
Ihe model iIsel. A models ile represenIaIion is plugin.xml providing Ihe basic sIrucIure. LxIending
an exisIing !ava Model class is noI necessary as you can browse Ihe model elemenIs rom Iop Io
boIIom using pure XML APls. lI is up Io you Io decide whaI is simpler and nicer. Providing meIhods
Io seI and geI daIa rom Ihe model is oIen more comorIable and more lexible or uIure exIensions
Ihan direcIly parsing Ihe XML Iree.
The ollowing diagram shows a simpliied hierarchy o Ihe model archiIecIure. Please noIe IhaI
DatabaseModel provides Ions o meIhods which cannoI be shown on on simple diagram.
9/2J
Puzzle lTC GmbH
5.2 Graphical User lnIeraces
Lvery plugin musI provide a gui.xml ile, even i no GUls are required. ln baIch mode no GUls are
loaded aI all i noI enabled. Use gui.xml Io speciy GUls Io load, order wiIhin wizard and GUl IiIle.
To develop your own GUls and add Ihose Io Ihe Iab selecIion, exIend Ihe super class DynamicPanel.
The ollowing diagram shows Ihe relaIion o some imporIanI GUl classes. NoIe IhaI one imporIanI
aIIribuIe is Ielling openD8copy i a GUl shall be regisIered as an observer o Ihe observable model
PluginGuiManager. The diagram is on only an exIracI and noI compleIe.
8oxes in yellow are sIandard GUls which are used in sIandard plugins and can be used wiIhin your
own plugins. To creaIe your own GUls, see orange box on Ihe leI hand side, exIend Ihe class
DynamicPanel.
!0/2J
Puzzle lTC GmbH
5.2.! GUl ConiguraIion
GUl coniguraIion is sIored in a ile called gui.xml. lIs sIrucIure looks like:
<gui identifier="id" display_order="numeric_value"
resource_name="resource_file_name_without_file_ending">
<title value="key for title" />
<panels>
<panel title="key for first gui title">
<class name="class name for first gui" register_as_observer="true" />
</panel>
<panel title="key for second gui title">
<class name="class name for second gui" register_as_observer="false" />
</panel>
</panels>
</gui>
All gui.xml iles musI conIain Ihe ollowing elemenIs as children o Ihe rooI elemenI gui.
title
An aIIribuIe value conIains Ihe key or Ihe language dependenI IiIle.
panels
May conIain 0 Io n panel elemenIs
The elemenI gui musI conIain Ihe ollowing aIIribuIes:
identifier
The unique idenIiier used wiIhin Ihe plugin.xml
display_order
The display_order aIIribuIe musI be a numeric number and allows Io
conIrol Ihe posiIion o Ihis plugins IiIle wiIhin Ihe plugin selecIion menu.
resource_name
The resource properIy name or Ihis plugin wiIhouI ile ending
.properties.
A gui.xml only conIains one elemenI panels (plural). This plugins elemenI may conIain 0 Io n
elemenIs named plugin. Lach plugin elemenI musI provide a title aIIribuIe. The titles
aIIribuIe value is Ihe key or Ihe resources (language dependenI). Lach panel elemenI musI conIain a
child elemenI named class.
The elemenI class musI conIain Ihe ollowing aIIribuIe:
name name musI conIain Ihe java package and class name or Ihe sIandard or
cusIom GUl. This aIIribuIe is required.
regisIer_as_observer Use Ihis aIIribuIe Io Iell PluginGui i Ihis GUl shall be regisIered as observer
o Ihe model observed. This aIIribuIe is noI mandaIory.
5.2.2 CreaIing your own GUls
When creaIing a new plugin iI may be IhaI you require new GUls. To do such is very simple.
openD8copy already provides Ihe ramework Io puI a new GUl inIo a worklow wizard, regisIer Ihe
new GUl as observer, provide a next buIIon or your wizard eIc. l your new plugin is o general
inIeresI, why noI donaIe Ihe source code back Io Ihe openD8copy projecI?
Have a look aI some oIher GUls exIending Ihe DynamicPanel beore you sIarI creaIing your own
GUls Io amiliarise yoursel. lI is up Io you i you wanI Io overwriIe Ihe supermeIhods o
DynamicPanel, such as Ihe meIhod update(...) or onSelect(). onSelect() is
auIomaIically called whenever a user clicks on Ihe appropriaIe Iab or Ihis GUl. OverwriIe Ihis meIhod
when Ihis GUl shall updaIe iIs componenIs, load daIa or whaIever.
!!/2J
Puzzle lTC GmbH
A word abouI GUl design using Swing. Generally you are ree Io use whaIever LayoutManager you
like. There is no limiIaIion pure !ava. All new GUls o openD8copy will be replaced using
TableLayout (see hIIp://www.clearIhoughI.ino/soIware/TableLayouI), i noI yeI already done as
Ihis LayoutManager is very good.
Have a look aI how LxcepIions are caughI, respecIively Ihrown. Do noI caIch LxcepIions and handle
Ihem i iI is noI Ihe GUls responsibiliIy. When Ihrowing LxcepIions Ihose are inally caughI by
FrameMain and presenIed Io Ihe user and also logged properly.
5.J Plugin Thread
Lvery plugin musI implemenI a plugin Ihread class, exIending Ihe super class
DynamicPluginThread. This plugin Ihread class musI implemenI an execute() meIhod.
AddiIional meIhods, classes or oIher !ava libraries, even exIernal programmes can be used.
The plugin Ihread class IogeIher wiIh opIional classes and archives musI be deployed as jar archive(s)
or zip ile(s) preerrably jar ile(s). Using Ihe deaulI plugin build.xml ile, all class iles including
source code are added Io one single jar ile, including language resource iles. When deploying a
plugin Io Ihe openD8copy environmenI, all jar iles o Ihe plugins lib direcIory are deployed Ioo.
5.4 Language Pesources
Lvery plugin musI provide aI leasI one resource ile. Pesource iles are normal .properties iles,
using key / value pairs.
The suix o Ihe resource ile name speciies Ihe language, e.g. de or german (,deuIsch). The
deaulI language resource ile is expecIed Io conIain Lnglish key / value pairs, wiIh no ile suix or
Lnglish.
Pesource values may conIain parameIers speciied using {0}, {1}, {...} as placeholders wiIhin a
value. The number speciies Ihe parameIer number, beginning aI 0 (zero). When reading language
dependenI resource values conIaining dynamic parameIers, such as numbers, names eIc., parameIers
can be replaced by providing SIring arrays.
Here an example: message.model.successful=capturing {0} done
The key is message.model.successful. The Lnglish IexI or Ihis key is capturing {0} done.
The value wiIhin brackeIs {0} is variable.
To read Ihe value according Io Ihe language currenIly seI use Ihe ollowing code.
String[] param = { "testmodel" };
rm.getString("message.model.successful", param);
rm is Ihe PesourceManager owned by MainConIroller, which is available in any GUl exIending
DynamicPanel.
To read values wiIhouI parameIers use Ihe ollowing synIax:
rm.getString("key");
Please noIe IhaI openD8copy has an advanced ResourceManager implemenIed. One does noI have
Io provide Ihe bundle id as preix or keys, Locale eIc. A value or a given key is ound i Ihe key /
value pair was loaded properly, independenI o ile. The only disadvanIage o Ihis model is uniqueness
o keys which musI be guaranIeed Io avoid overwriIing o key / value pairs.
!2/2J
Puzzle lTC GmbH
6 Example - 1he Copy Plugih
6.! Overview
This plugin, called opendbcopy.copy, copies records o selecIed source Iables and columns inIo
selecIed desIinaIion Iables.
l Ihe underlying PD8MS supporIs PeerenIial lnIegriIy ConsIrainIs, Ihe order o Iables Io process is
respecIed, buI Ihis eaIure is parI o Ihe openD8copy core ramework and musI noI be implemenIed
wiIhin Ihis plugin. The plugin direcIly geIs an ordered lisI o Iables Io process.
l copy errors occur, Ihose are logged inIo comma separaIed value iles, i enabled by Ihe user. The
plugin wriIes a new ile or each Iable conIaining copy errors, including Ihe records values and error
message.
6.2 Plugin DaIabaseModel
This plugin uses Ihe DatabaseModel class Io hold and manipulaIe model daIa. DatabaseModel
inheriIs rom iIs superclass Model.
6.2.! Plugin ConiguraIion lile
The plugin coniguraIion ile conIains Ihe ollowing elemenIs and aIIribuIes
<plugin identifier="copy" model_class="opendbcopy.plugin.model.database.DatabaseModel">
<conf>

<dir required="true" value="" type="dir" description="plugin.opendbcopy.copy.conf.outputDirErrorLogs" />
<log_error value="true" type="boolean" description="plugin.opendbcopy.copy.conf.logError" />
<output>
<filelist value="error_logs" type="string"
description="plugin.opendbcopy.copy.conf.output.filelistIdentifier" />
</output>
</conf>
<threads>
<thread thread_class="opendbcopy.plugin.copy.CopyMappingPlugin" description="title.plugin.opendbcopy.copy" />
</threads>
<input />
<output />
<source_db>
<driver />
<metadata />
<connection />
<catalog value="" />
<schema value="%" />
<table_pattern value="%" />
<model />
</source_db>
<destination_db>
<driver />
<metadata />
<connection />
<catalog value="" />
<schema value="%" />
<table_pattern value="%" />
<model />
</destination_db>
<mapping />
<filter>
<string name="trim" process="false" />
<string name="remove_intermediate_whitespaces" process="false" />
<string name="set_null" process="false" />
</filter>
</plugin>
6.2.2 Plugin AIIribuIes
identifier
copy a unique sIring idenIiier
!J/2J
Puzzle lTC GmbH
model_class opendbcopy.plugin.model.database.DatabaseModel
Model class providing elemenIs or source daIabase, desIinaIion daIabase,
daIabase mapping and ilIer elemenIs. Values seI in Ihe above ile are deaulI
values. DatabaseModel can be used or single daIabase mode by removing
Ihe elemenI desIinaIion_db.
6.2.J Con LlemenIs
dir <dir required="true" value="" type="dir"
description="plugin.opendbcopy.copy.conf.outputDirError
Logs" />
Speciy Ihe ouIpuI paIh or error log iles
required ~ ,true means Ihis parameIer musI be seI
value ~ no deaulI direcIory seI
type ~ dir pops up a direcIory browser when clicked by user
description key or resource properIy
log_error <log_error value="true" type="boolean"
description="plugin.opendbcopy.copy.conf.logError" />
Shall errors occuring be logged or noI
value ~ ,Irue by deaulI errors are logged
type ~ ,boolean pops up a dialog box Io selecI Irue or alse
output
filelist elemenI
<filelist value="error_logs" type="string"
description="plugin.opendbcopy.copy.conf.output.filelis
tIdentifier" />
l errors occur during Ihe copy process, Ihis ouIpuI elemenI conIains a ilelisI
wiIh paIh and ilenames Io error log iles or possible urIher processing o
ollowing plugins
value ~ error_logs a sIring name or ilelisI idenIiicaIion
type ~ string pops up a dialog allowing Io speciy a sIring
6.2.4 Threads LlemenI(s)
thread <thread
thread_class="opendbcopy.plugin.copy.CopyMappingPlugin"
description="title.plugin.opendbcopy.copy" />
CurrenIly only one thread_class elemenI is allowed per plugin. The
elemenI threads musI conIain an elemenI called thread.
thread_class ~
opendbcopy.plugin.copy.CopyMappingPlugin
paIh and class name o Ihe Thread class implemenIing Ihe execute()
meIhod
!4/2J
Puzzle lTC GmbH
6.2.5 lnpuI / OuIpuI LlemenIs
The input and output elemenI are seI aI runIime. Do noI use Ihese elemenIs Io save coniguraIion
input or output as Ihese elemenIs may be overwriIIen by PluginScheduler.
6.2.6 Source & DesIinaIion DaIabase LlemenIs
The ollowing elemenIs are illed by reading daIabase meIadaIa and model(s). DaIabase MeIadaIa is
read when IesIing a daIabase connecIion. Models are read when a user clicks on CapIure Source
Model or CapIure DesIinaIion Model. lor more inormaIion abouI Iypes, consIanIs and meIadaIa
available please see Ihe !ava APl, class java.sql.DatabaseMetaData.
driver
ConIains inormaIion abouI Ihe driver used or Ihe appropriaIe daIabase.
CurrenIly a drivers name and version are sIored as aIIribuIes.
metadata
ConIains urIher subelemenIs:
db_product_name e.g. ,Oracle, PostgreSQL
db_product_version daIabase producI version
catalog_separator separaIor beIween caIalog name and Iable
name oIen Ihis is a doI (,.)
identifier_quote_string How are sIrings quoIed
catalog
l available lisIs all caIalogs available as separaIe elemenIs
schema
l available lisIs all schemas available as separaIe elemenIs
type_info
lisIs all available daIa Iypes speciying:
type_name (e.g. ,VARCHAR2)
locale_type_name (e.g. ,VARCHAR2 normally Ihe same as
type_name or null)
data_type (e.g. ,12 java.sql.Types ConsIanI)
precision (e.g. ,4000)
literal_prefix (e.g. , preix or Ihis daIa Iype)
literal_suffix (e.g. , suix or Ihis daIa Iype)
nullable (,true or ,false is Ihis daIa Iype nullable)
case_sensitive (e.g. ,0 or 1 is Ihis daIa Iype case sensiIive)
connection
ConIains aIIribuIes or:
driver_class (e.g. ,oracle.jdbc.driver.OracleDriver)
url (e.g. ,jdbc:oracle:thin:@localhost:1521:test)
username
password
catalog
CaIalog selecIed i supporIed
schema
Schema selecIed i supporIed
table_pattern
Table paIIern allows Io speciy an SQL like paIIern Io use when capIuring Iable
meIadaIa, e.g. ,TA% only capIures Iables beginning wiIh TA ollowed by any
characIer or none. See !ava APl or more deIails.
!5/2J
Puzzle lTC GmbH
model
When a model is capIured, Ihis elemenI conIains an aIIribuIe
capture_date. CapIure a daIabase model, save Ihe plugin as job and Ihen
have a look aI Ihe XML ile creaIed. The model hierarchy should be sel
explaining.
Model conIains children named table. A table elemenI can be o Iype
TABLE or VIEW, idenIiied by Ihe aIIribuIe table_type. An aIIribuIe
process (true or false) allows Io speciy i a Iable shall be processed or
noI.
Table conIains column elemenIs or each column in Ihe underlying Iable /
view column. These column elemenIs conIain aIIribuIes such as name,
type_name, data_type, column_size, decimal_digits,
nullable and process. When primary, oreign keys and indexes are
enabled, appropriaIe elemenIs are added Io Ihe table elemenI. The
aIIribuIes o Ihe Ihese elemenIs are sel explaining, comparing Ihem wiIh Ihe
DaIabaseMeIaDaIa meIhods or reading primary, imporIed and exporIed keys
as well as index inormaIion.
6.2.7 Mapping LlemenI
This elemenI conIains inormaIion Io map source Iables Io desIinaIion Iables. Same or columns. This
elemenIs sIrucIure is likely Io be changed in a uIure version Io allow more complex mapping and
merging rules Io be deined.
6.2.8 lilIer LlemenIs
The sIrucIure o Ihis elemenI is likely Io be changed in uIure versions. CurrenIly iI conIains a sIring
elemenI or each sIring ilIer implemenIed. A name aIIribuIe idenIiies a sIring ilIer and a process
aIIribuIe true or false allow Io enable or disable appropriaIe sIring ilIer.
6.J Graphical User lnIeraces
gui.xml conIains Ihe panels used Io speciy and execuIe Ihis plugin.
The rooI elemenI gui musI conIain Iwo elemenIs. title and panels. Title musI conIain an
aIIribuIe value speciying Ihe resource key or Ihe plugins IiIle Io show wiIhin Ihe GUl o openD8copy.
plugins can conIain children named panel or Ihe individual panels or Ihis plugin. The order o
Ihese plugin elemenIs wiIhin Iheir plugins parenI elemenI speciies Ihe order o Ihe panels added
wiIhin Ihe Iab in Ihe GUl o openD8copy, e.g. Ihe irsI panel elemenI is Ihe irsI GUl in Ihe wizard
creaIed eIc.
A panel elemenI musI conIain a class elemenI. LlemenI class musI conIain one aIIribuIe, name
o Ihe Panel class Io load. The second aIIribuIe, register_as_observer (true or false) is noI
mandaIory, as described in previous chapIers.
The copy plugin uses 7 GUls Io conigure, gaIher daIabase meIadaIa and inally execuIe Ihe plugin. All
excepI PanelConfiguration and PanelExecute are regisIered as observers o Ihe model.
PanelExecute iIsel conIains iIs own regisIraIion mechanism see code o PanelExecute or
deIails.
The ile looks like:
<gui identifier="copy" display_order="20" resource_name="plugin_opendbcopy_copy">
<title value="title.plugin.opendbcopy.copy" />
<panels>
<panel title="title.plugin.opendbcopy.copy.panel.0">
<class name="opendbcopy.gui.PanelConfiguration" />
</panel>
!6/2J
Puzzle lTC GmbH
<class name="opendbcopy.gui.database.PanelConnection" register_as_observer="true" />
</panel>
<class name="opendbcopy.gui.database.PanelModel" register_as_observer="true" />
</panel>
<class name="opendbcopy.gui.database.dual.PanelMappingTable" register_as_observer="true" />
</panel>
<class name="opendbcopy.gui.database.dual.PanelMappingColumn" register_as_observer="true" />
</panel>
<class name="opendbcopy.gui.database.PanelFilter" register_as_observer="true" />
</panel>
<class name="opendbcopy.gui.PanelExecute" />
</panel>
</panels>
</gui>
6.J.! GUl AIIribuIes
identifier
copy a unique sIring idenIiier musI be Ihe same as used wiIhin
plugin.xml
display_order 20
Allows Io speciy Ihe posiIion o Ihis plugin wiIhin Ihe plugin menu selecIion.
When adding new plugins leave some space beIween numbers or addiIions.
Have a look aI Ihe oIher display_order numbers o oIher plugins.
resource_name plugin_opendbcopy_copy
The name o Ihe resource ile name (language dependenI IexIs). NoIe IhaI Ihe
name speciied musI noI conIain Ihe ile ending .properties and Ihe ile
iIsel musI be accessible wiIhin Ihe plugins classpaIh.
6.4 Plugin Thread
CopyMappingPlugin is Ihe Ihread class o Ihe Copy Plugin, see opendbcopy.copy.
CopyMappingPlugin is Ihe only class required Io copy daIa rom a source inIo a desIinaIion
daIabase, independenI o underlying daIabase managemenI sysIem. Thereore Ihe deployed jar ile
currenIly only conIains Ihis one class ile, CopyMappingPlugin. Pesource properIies iles are added
Io Ihis jar ile Ioo inIo Ihe jar archives rooI. lor simpler debugging o Ihis plugin Ihe source code is
also added Io Ihe jar ile. See Ihe anI jar target or deIails.
The source code o Ihe class CopyMappingPlugin is as ollows:
/*
* Copyright (C) 2004 Anthony Smith
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
*
* ----------------------------------------------------------------------------
* TITLE $Id: CopyMappingPlugin.java,v 1.2 2004/04/08 11:21:05 iloveopensource Exp $
* ---------------------------------------------------------------------------
*
* --------------------------------------------------------------------------*/
package opendbcopy.plugin.copy;
import java.io.File;
import java.io.FileOutputStream;
import java.io.OutputStreamWriter;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
!7/2J
Puzzle lTC GmbH
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import opendbcopy.config.XMLTags;
import opendbcopy.connection.DBConnection;
import opendbcopy.connection.exception.CloseConnectionException;
import opendbcopy.controller.MainController;
import opendbcopy.filter.StringConverter;
import opendbcopy.plugin.model.DynamicPluginThread;
import opendbcopy.plugin.model.Model;
import opendbcopy.plugin.model.database.DatabaseModel;
import opendbcopy.plugin.model.exception.PluginException;
import opendbcopy.sql.Helper;
import opendbcopy.util.InputOutputHelper;
import org.jdom.Element;
/**
* Copies records of selected source tables and columns into selected destination tables. If the underlying
* RDBMS supports Referential Integrity Constraints, the order of tables to process is respected. If possible
* errors shall be logged, those are logged into comma separated value files.
*
* @author Anthony Smith
* @version $Revision: 1.2 $
*/
public class CopyMappingPlugin extends DynamicPluginThread {
private static final String fileType = "csv";
private DatabaseModel model; // this plugin's model
private Connection connSource;
private Connection connDestination;
private Statement stmSource;
private PreparedStatement pstmtDestination;
private ResultSet rs;
private StringBuffer recordBuffer; // to hold records contents
private StringBuffer recordErrorBuffer; // in case of errors records are written to file
private String stmSelect = "";
private String stmInsert = "";
private String sourceTableName = "";
private String destinationTableName = "";
private String newLine = "";
private File outputPath = null;
private String fileName = "";
private String delimiter = "";
private boolean log_error = false;
private boolean errorLogSetup = false;
private int counterRecords = 0;
private int counterTables = 0;
private List processColumns;
private List processTables;
boolean trimString = false;
boolean trimAndRemoveMultipleIntermediateWhitespaces = false;
boolean trimAndReturnNullWhenEmpty = false;
/**
* Creates a new CopyMappingPlugin object.
*
* @param controller DOCUMENT ME!
* @param baseModel DOCUMENT ME!
*
* @throws PluginException DOCUMENT ME!
*/
public CopyMappingPlugin(MainController controller,
Model baseModel) throws PluginException {
// Call the super constructor
super(controller, baseModel);

// cast the super base model into a specific database model
this.model = (DatabaseModel) baseModel;
}
/**
* Read configuration and setup database connections
*
*/
protected final void setUp() throws PluginException {
// set the new line character as this differs from OS to OS
newLine = MainController.lineSep;
// get the plugin configuration
Element conf = model.getConf();
if (conf == null) {
throw new PluginException("Missing conf element");
}
try {
// set the output path selected by user
outputPath = new File(conf.getChild(XMLTags.DIR).getAttributeValue(XMLTags.VALUE));
// create the output directory if it does not yet exist
if (!outputPath.exists()) {
boolean mkDirOk = outputPath.mkdir();
!8/2J
Puzzle lTC GmbH
if (!mkDirOk) {
throw new PluginException("Could not create " + outputPath.getAbsolutePath());
}
}
// shall errors be logged?
log_error = Boolean.valueOf(conf.getChild(XMLTags.LOG_ERROR).getAttributeValue
(XMLTags.VALUE)).booleanValue();
if (log_error) {
recordBuffer = new StringBuffer();
recordErrorBuffer = new StringBuffer();
}
// check string filters
if (model.getStringFilterTrim().getAttributeValue(XMLTags.PROCESS).compareTo("true") == 0) {
trimString = true;
}
if (model.getStringFilterRemoveIntermediateWhitespaces().getAttributeValue(XMLTags.PROCESS).compareTo("true")
== 0) {
trimAndRemoveMultipleIntermediateWhitespaces = true;
}
if (model.getStringFilterSetNull().getAttributeValue(XMLTags.PROCESS).compareTo("true") == 0) {
trimAndReturnNullWhenEmpty = true;
}
// get connections
connSource = DBConnection.getConnection(model.getSourceConnection());
connDestination = DBConnection.getConnection(model.getDestinationConnection());
// extract the tables to copy
processTables = model.getDestinationTablesToProcessOrdered();
} catch (Exception e) {
throw new PluginException(e);
}
// now set the number of tables that need to be copied
model.setLengthProgressTable(processTables.size());
}
/**
* Copies records of selected source tables and columns into selected destination tables.
* If requested and occuring, errors are logged
*
*/
public final void execute() throws PluginException {
Iterator itColumns;
Element tableProcess;
Element columnDestination;
int colCounter;
Object input;
try {
stmSource = connSource.createStatement();
Iterator itProcessTables = processTables.iterator();

ArrayList generatedFiles = new ArrayList();
while (!isInterrupted() && itProcessTables.hasNext()) {
tableProcess = (Element) itProcessTables.next();
sourceTableName = tableProcess.getAttributeValue(XMLTags.SOURCE_DB);
destinationTableName = tableProcess.getAttributeValue(XMLTags.DESTINATION_DB);
// file name for error logs
if (log_error) {
fileName = destinationTableName + "_ERRORS" + "." + fileType;
}
// get the columns to process
processColumns = model.getMappingColumnsToProcessByDestinationTable(destinationTableName);
// setting record counter to minimum of progress bar
model.setCurrentProgressRecord(0);
model.setLengthProgressRecord(0);
// Reading number of records for progress bar
model.setLengthProgressRecord(Helper.getNumberOfRecordsFiltered(stmSource, model, XMLTags.SOURCE_DB,
sourceTableName));
// get Select Statement for source model
stmSelect = Helper.getSelectStatement(model, sourceTableName, XMLTags.SOURCE_DB, processColumns);
// get Insert Statement for destination model
stmInsert = Helper.getInsertPreparedStatement(model.getQualifiedDestinationTableName
(destinationTableName), processColumns);
pstmtDestination = connDestination.prepareStatement(stmInsert);
model.setCurrentProgressTable(counterTables);
// Execute SELECT
!9/2J
Puzzle lTC GmbH
rs = stmSource.executeQuery(stmSelect);
// do some logging
model.setProgressMessage("Copying " + model.getQualifiedSourceTableName(sourceTableName) + " into " +
model.getQualifiedDestinationTableName(destinationTableName) + " ...");
logger.info("Copying " + model.getQualifiedSourceTableName(sourceTableName) + " into " +
model.getQualifiedDestinationTableName(destinationTableName) + " ...");
// while there are more records to process and the process is not interrupted by the user
while (!isInterrupted() && rs.next()) {
colCounter = 1;
// process columns
itColumns = processColumns.iterator();
while (itColumns.hasNext()) {
columnDestination = model.getDestinationColumn(destinationTableName, ((Element)
itColumns.next()).getAttributeValue(XMLTags.DESTINATION_DB));
input = rs.getObject(colCounter);
if (input != null) {
input = applyStringFilters(input, Boolean.valueOf(columnDestination.getAttributeValue
(XMLTags.NULLABLE)).booleanValue());
if (input != null) {
if (log_error) {
recordBuffer.append(input + delimiter);
}
pstmtDestination.setObject(colCounter, input);
} else {
if (log_error) {
recordBuffer.append("null" + delimiter);
}
pstmtDestination.setNull(colCounter, Integer.parseInt(columnDestination.getAttributeValue
(XMLTags.DATA_TYPE)));
}
} else {
if (log_error) {
recordBuffer.append("null" + delimiter);
}
pstmtDestination.setNull(colCounter, Integer.parseInt(columnDestination.getAttributeValue
(XMLTags.DATA_TYPE)));
}
colCounter++;
}
// execute the prepared statement and log the error ... and continue without disturbing other
business
try {
// Execute INSERT
pstmtDestination.executeUpdate();
pstmtDestination.clearParameters();
counterRecords++;
model.setCurrentProgressRecord(counterRecords);
} catch (SQLException e) {
connDestination.rollback();
if (log_error && !errorLogSetup) {
initErrorLog(processColumns);
}
if (log_error) {
recordErrorBuffer.append(recordBuffer + e.toString() + newLine);
}
} finally {
// reset recordBuffer
if (log_error) {
recordBuffer = new StringBuffer();
}
}
}
if (!isInterrupted()) {
// commit INSERTs. Commit behaviour depends on RDBMS used
connDestination.commit();

// close the result set
rs.close();
logger.info(counterRecords + " records inserted into table " + destinationTableName);
counterRecords = 0;
// required in case of last table that had to be copied
counterTables++;
model.setCurrentProgressTable(counterTables);
// set processed
tableProcess.setAttribute(XMLTags.PROCESSED, "true");
} else {
// rollback insert in case the user interrupts the process
connDestination.rollback();
20/2J
Puzzle lTC GmbH

// close the result set
rs.close();
counterRecords = 0;
}
if (log_error) {
if (recordErrorBuffer.length() > 0) {
// open file writer
File errorFile = new File(outputPath.getAbsolutePath() + MainController.fileSep +
fileName);
OutputStreamWriter fileWriter = new OutputStreamWriter(new FileOutputStream(errorFile),
MainController.getEncoding());
fileWriter.write(recordErrorBuffer.toString());
fileWriter.close();
generatedFiles.add(errorFile);
logger.error(errorFile + " contains records which could not be processed");
recordErrorBuffer = new StringBuffer();
errorLogSetup = false;
}
}
}
stmSource.close();
pstmtDestination.close();
// close database connections
DBConnection.closeConnection(connSource);
DBConnection.closeConnection(connDestination);
if (!isInterrupted()) {
if (generatedFiles != null && generatedFiles.size() > 0) {
File[] outputFiles = new File[generatedFiles.size()];
outputFiles = (File[]) generatedFiles.toArray(outputFiles);
Element outputConf = model.getConf().getChild(XMLTags.OUTPUT);
model.appendToOutput(InputOutputHelper.createFileListElement(outputFiles, outputConf.getChild
(XMLTags.FILELIST).getAttributeValue(XMLTags.VALUE)));
}
logger.info(counterTables + " table(s) processed");
}
} catch (SQLException sqle) {
throw new PluginException(sqle);
} catch (Exception e1) {
// clean up if required
try {
DBConnection.closeConnection(connSource);
DBConnection.closeConnection(connDestination);
} catch (CloseConnectionException e2) {
// bad luck ... don't worry
}
throw new PluginException(e1);
}
}
/**
* If global string filters were selected by user, those are applied using this method
*
* @param in DOCUMENT ME!
* @param returnNullWhenEmpty DOCUMENT ME!
*
* @return DOCUMENT ME!
*/
private Object applyStringFilters(Object in,
boolean returnNullWhenEmpty) {
if (in instanceof String || in instanceof Character) {
if (trimAndRemoveMultipleIntermediateWhitespaces && trimAndReturnNullWhenEmpty) {
return StringConverter.trimAndRemoveMultipleIntermediateWhitespaces(in, returnNullWhenEmpty);
} else if (trimAndRemoveMultipleIntermediateWhitespaces && !trimAndReturnNullWhenEmpty) {
return StringConverter.trimAndRemoveMultipleIntermediateWhitespaces(in, false);
} else if (trimString && trimAndReturnNullWhenEmpty) {
return StringConverter.trimString(in, returnNullWhenEmpty);
} else if (trimString && !trimAndReturnNullWhenEmpty) {
return StringConverter.trimString(in, false);
} else {
return in;
}
} else {
return in;
}
}
/**
* Called to write a nice row header of column names in possible error logs
*
* @param processColumns DOCUMENT ME!
*/
private void initErrorLog(List processColumns) {
Iterator itColumns = processColumns.iterator();
// set the column headings for the possible error log
while (itColumns.hasNext()) {
recordErrorBuffer.append(((Element) itColumns.next()).getAttributeValue(XMLTags.DESTINATION_DB) + delimiter);
2!/2J
Puzzle lTC GmbH
}
recordErrorBuffer.append("ERROR" + newLine);
errorLogSetup = true;
}
}
6.5 Language Pesource liles
The copy plugin provides resource iles or Lnglish and German.
The source properIies iles are sIored under resource/bundle.
The Lnglish properIies ile is named plugin_opendbcopy_copy.properties. The German
version is named plugin_opendbcopy_copy_de.properties. The exIension de is
according Io Ihe !ava Locale speciicaIion.
The Lnglish version looks as ollows:
title.plugin.opendbcopy.copy=Copy data from a source into a destination database
title.plugin.opendbcopy.copy.panel.0=0. Plugin Configuration
title.plugin.opendbcopy.copy.panel.1=1. Database Connections
title.plugin.opendbcopy.copy.panel.2=2. Models
title.plugin.opendbcopy.copy.panel.3=3. Table Mapping
title.plugin.opendbcopy.copy.panel.4=4. Column Mapping
title.plugin.opendbcopy.copy.panel.5=5. Global String Filters
title.plugin.opendbcopy.copy.panel.6=6. Execute Plugin
plugin.opendbcopy.copy.conf.outputDirErrorLogs=Output dir for error logs
plugin.opendbcopy.copy.conf.logError=Log errors into comma separated files (csv) including
error message
plugin.opendbcopy.copy.conf.appendFileAfterRecords=After how many records shall disk I/O
occur? Depends on available memory and table width
plugin.opendbcopy.copy.conf.output.filelistIdentifier=Filelist identifier for error logs
Please noIe IhaI Ihe IiIle or Ihe panels in a Iab musI be speciied or each plugin as Ihe meaning o
Ihe same GUl may vary, depending on Ihe plugin used.
22/2J
Puzzle lTC GmbH
7 8uildihg, 1esIihg ahd Deployihg Plugihs
OpenD8copy and plugins are compiled and deployed using Apache AnI.
To IesI your plugin(s) your require an openD8copy Core lramework environmenI. The mosI
comorIable soluIion is Io have openD8copy Core lramework and Plugin ProjecI(s) open and linked
wiIhin Ihe same !ava lDL. CurrenIly Ihere are no !UniI IesI classes implemenIed Io IesI openD8copy
Core lramework nor Plugins.
Lach plugin provides a build.xml and build.properties ile. The laIIer conIains plugin speciic
deIails such as plugin name eIc. MosI plugins are released using an idenIical copy o build.xml as
oIher plugins.
Thereore when sIarIing a new plugin projecI, Ihe simplesI way Io seIup your environmenI is Io copy
an exisIing plugin projecI inIo a new direcIory and modiy Ihe new build.properIies ile. This ile
conIains a hardcoded reerence Io your openD8copy projecI locaIion. You musI modiy Ihis
parameIer, nexI Io plugin idenIiier, auIhor name eIc.
The deaulI build IargeI is deploy. See Ihe build.xml ile and Apache AnI User Manual or urIher
deIails.
2J/2J
Puzzle lTC GmbH

Developer Manual

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Developer Manual

Hochgeladen von

Copyright:

Verfügbare Formate

opehD8copy - Developer Mahual

Das könnte Ihnen auch gefallen