| 1 |
joko |
1.1 |
<HTML> |
| 2 |
|
|
<HEAD> |
| 3 |
joko |
1.2 |
<TITLE>Data::Storage::Sync</TITLE> |
| 4 |
joko |
1.1 |
<LINK REV="made" HREF="mailto:"> |
| 5 |
|
|
</HEAD> |
| 6 |
|
|
|
| 7 |
|
|
<BODY> |
| 8 |
|
|
|
| 9 |
|
|
<A NAME="__index__"></A> |
| 10 |
|
|
<!-- INDEX BEGIN --> |
| 11 |
|
|
|
| 12 |
|
|
<UL> |
| 13 |
|
|
|
| 14 |
joko |
1.2 |
<LI><A HREF="#news">NEWS</A></LI> |
| 15 |
|
|
<UL> |
| 16 |
|
|
|
| 17 |
|
|
<UL> |
| 18 |
|
|
|
| 19 |
|
|
<UL> |
| 20 |
|
|
|
| 21 |
joko |
1.3 |
<LI><A HREF="#20030209 backward compatibility with custom application">2003-02-09 - backward compatibility with custom application</A></LI> |
| 22 |
|
|
<LI><A HREF="#20030118 start integration with torus">2003-01-18 - start integration with Torus</A></LI> |
| 23 |
|
|
<LI><A HREF="#20021010 initial commit">2002-10-10 - initial commit</A></LI> |
| 24 |
joko |
1.2 |
</UL> |
| 25 |
|
|
|
| 26 |
|
|
</UL> |
| 27 |
|
|
|
| 28 |
|
|
</UL> |
| 29 |
|
|
|
| 30 |
joko |
1.1 |
<LI><A HREF="#description">DESCRIPTION</A></LI> |
| 31 |
|
|
<LI><A HREF="#requirements">REQUIREMENTS</A></LI> |
| 32 |
joko |
1.3 |
<LI><A HREF="#authors / copyright / license">AUTHORS / COPYRIGHT / LICENSE</A></LI> |
| 33 |
joko |
1.1 |
<LI><A HREF="#support / warranty">SUPPORT / WARRANTY</A></LI> |
| 34 |
joko |
1.3 |
<LI><A HREF="#example output chunks">EXAMPLE OUTPUT CHUNKS</A></LI> |
| 35 |
|
|
<UL> |
| 36 |
|
|
|
| 37 |
|
|
<LI><A HREF="#synchronizing orm objects against entries in a .csv file 17 items modified">synchronizing orm objects against entries in a .csv file - 17 items modified</A></LI> |
| 38 |
|
|
<LI><A HREF="#synchronizing orm objects against entries in a .csv file insync">synchronizing orm objects against entries in a .csv file - in-sync</A></LI> |
| 39 |
|
|
</UL> |
| 40 |
|
|
|
| 41 |
joko |
1.1 |
<LI><A HREF="#bugs">BUGS</A></LI> |
| 42 |
|
|
<LI><A HREF="#user level errors">USER LEVEL ERRORS</A></LI> |
| 43 |
|
|
<UL> |
| 44 |
|
|
|
| 45 |
|
|
<UL> |
| 46 |
|
|
|
| 47 |
|
|
<UL> |
| 48 |
|
|
|
| 49 |
|
|
<LI><A HREF="#mapping">Mapping</A></LI> |
| 50 |
|
|
<LI><A HREF="#dbd::autocsv's rulebase">DBD::AutoCSV's rulebase</A></LI> |
| 51 |
|
|
<LI><A HREF="#check structure of source node">Check structure of source node</A></LI> |
| 52 |
|
|
<LI><A HREF="#modify structure of source node">Modify structure of source node</A></LI> |
| 53 |
|
|
<LI><A HREF="#load source node by ident">Load source node by ident</A></LI> |
| 54 |
|
|
</UL> |
| 55 |
|
|
|
| 56 |
|
|
</UL> |
| 57 |
|
|
|
| 58 |
|
|
</UL> |
| 59 |
|
|
|
| 60 |
|
|
<LI><A HREF="#todo">TODO</A></LI> |
| 61 |
|
|
</UL> |
| 62 |
|
|
<!-- INDEX END --> |
| 63 |
|
|
|
| 64 |
|
|
<HR> |
| 65 |
|
|
<P> |
| 66 |
|
|
<HR> |
| 67 |
joko |
1.2 |
<H1><A NAME="news">NEWS</A></H1> |
| 68 |
|
|
<P> |
| 69 |
joko |
1.3 |
<H4><A NAME="20030209 backward compatibility with custom application">2003-02-09 - backward compatibility with custom application</A></H4> |
| 70 |
joko |
1.2 |
<PRE> |
| 71 |
joko |
1.3 |
- Tangram <-> .csv |
| 72 |
|
|
- Tangram <-> mysql |
| 73 |
|
|
- Tangram <-> perl-code</PRE> |
| 74 |
joko |
1.2 |
<P> |
| 75 |
joko |
1.3 |
<H4><A NAME="20030118 start integration with torus">2003-01-18 - start integration with Torus</A></H4> |
| 76 |
|
|
<PRE> |
| 77 |
|
|
- MAPI <-> LDAP |
| 78 |
|
|
- please look at <A HREF="http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/">http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/</A></PRE> |
| 79 |
|
|
<P> |
| 80 |
|
|
<H4><A NAME="20021010 initial commit">2002-10-10 - initial commit</A></H4> |
| 81 |
|
|
<PRE> |
| 82 |
|
|
- Tangram <-> mysql</PRE> |
| 83 |
joko |
1.2 |
<P> |
| 84 |
|
|
<HR> |
| 85 |
joko |
1.1 |
<H1><A NAME="description">DESCRIPTION</A></H1> |
| 86 |
joko |
1.3 |
<PRE> |
| 87 |
|
|
Data::Transfer::Sync is a module providing a generic synchronization process |
| 88 |
|
|
across arbitrary/multiple storages based on a ident/checksum mechanism. |
| 89 |
|
|
It sits on top of Data::Storage.</PRE> |
| 90 |
joko |
1.1 |
<P> |
| 91 |
|
|
<HR> |
| 92 |
|
|
<H1><A NAME="requirements">REQUIREMENTS</A></H1> |
| 93 |
|
|
<PRE> |
| 94 |
|
|
For full functionality: |
| 95 |
joko |
1.3 |
Data::Compare |
| 96 |
joko |
1.1 |
Data::Storage |
| 97 |
|
|
Data::Transform |
| 98 |
joko |
1.3 |
... and all their dependencies.</PRE> |
| 99 |
joko |
1.1 |
<P> |
| 100 |
|
|
<HR> |
| 101 |
joko |
1.3 |
<H1><A NAME="authors / copyright / license">AUTHORS / COPYRIGHT / LICENSE</A></H1> |
| 102 |
|
|
<PRE> |
| 103 |
|
|
The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl. |
| 104 |
|
|
All rights reserved.</PRE> |
| 105 |
|
|
<PRE> |
| 106 |
|
|
Contributions by Sebastian Utz <su@tunemedia.de> and |
| 107 |
|
|
Jan Hoffmann <jan@romain-volk.com>.</PRE> |
| 108 |
|
|
<PRE> |
| 109 |
|
|
You may distribute it under the terms of either the GNU General Public |
| 110 |
|
|
License or the Artistic License, as specified in the Perl README file.</PRE> |
| 111 |
joko |
1.1 |
<P> |
| 112 |
|
|
<HR> |
| 113 |
|
|
<H1><A NAME="support / warranty">SUPPORT / WARRANTY</A></H1> |
| 114 |
joko |
1.3 |
<PRE> |
| 115 |
|
|
Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.</PRE> |
| 116 |
|
|
<P> |
| 117 |
|
|
<HR> |
| 118 |
|
|
<H1><A NAME="example output chunks">EXAMPLE OUTPUT CHUNKS</A></H1> |
| 119 |
|
|
<P> |
| 120 |
|
|
<H2><A NAME="synchronizing orm objects against entries in a .csv file 17 items modified">synchronizing orm objects against entries in a .csv file - 17 items modified</A></H2> |
| 121 |
|
|
<PRE> |
| 122 |
|
|
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping; |
| 123 |
|
|
notice: OEF::Script::AbstractFeeder: |
| 124 |
|
|
- Loading data of type [n/a] and |
| 125 |
|
|
filtered by * from the storage named base |
| 126 |
|
|
to the storage named backend - filtered by *. |
| 127 |
|
|
- Will attempt to convert the data to Country. |
| 128 |
|
|
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* [] |
| 129 |
|
|
:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
| 130 |
|
|
:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c |
| 131 |
|
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
| 132 |
|
|
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17</PRE> |
| 133 |
|
|
<P> |
| 134 |
|
|
<H2><A NAME="synchronizing orm objects against entries in a .csv file insync">synchronizing orm objects against entries in a .csv file - in-sync</A></H2> |
| 135 |
|
|
<PRE> |
| 136 |
|
|
|
| 137 |
|
|
notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping; |
| 138 |
|
|
notice: OEF::Script::AbstractFeeder: |
| 139 |
|
|
- Loading data of type [n/a] and |
| 140 |
|
|
filtered by * from the storage named base |
| 141 |
|
|
to the storage named backend - filtered by *. |
| 142 |
|
|
- Will attempt to convert the data to Country. |
| 143 |
|
|
info: Data::Transfer::Sync::API->syncNodes: source=backend/* [Country] <- target=base/* [] |
| 144 |
|
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
| 145 |
|
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
| 146 |
|
|
:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c |
| 147 |
|
|
info: Data::Transfer::Sync::Core->_startsync: statistics: in_sync: 197 / total: 197</PRE> |
| 148 |
joko |
1.1 |
<P> |
| 149 |
|
|
<HR> |
| 150 |
|
|
<H1><A NAME="bugs">BUGS</A></H1> |
| 151 |
|
|
<PRE> |
| 152 |
|
|
When in "import" mode for windows file - DBD::AutoCSV may hang. |
| 153 |
joko |
1.2 |
Hint: Maybe the source node contains an ident-, but no checksum-column?</PRE> |
| 154 |
joko |
1.1 |
<P> |
| 155 |
|
|
<HR> |
| 156 |
|
|
<H1><A NAME="user level errors">USER LEVEL ERRORS</A></H1> |
| 157 |
|
|
<P> |
| 158 |
|
|
<H4><A NAME="mapping">Mapping</A></H4> |
| 159 |
|
|
<PRE> |
| 160 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 161 |
|
|
info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup->startSync: Can't access mapping for node "Currency" - please check BizWorks::ResourceMapping. |
| 162 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 163 |
|
|
You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match |
| 164 |
|
|
the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how |
| 165 |
|
|
to access the otherwise generic nodes. |
| 166 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE> |
| 167 |
|
|
<P> |
| 168 |
|
|
<H4><A NAME="dbd::autocsv's rulebase">DBD::AutoCSV's rulebase</A></H4> |
| 169 |
|
|
<PRE> |
| 170 |
|
|
|
| 171 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 172 |
|
|
info: BizWorks::Process::Setup->syncResource( source_node Currency mode PULL erase 0 import 0 ) |
| 173 |
|
|
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv</PRE> |
| 174 |
|
|
<PRE> |
| 175 |
|
|
|
| 176 |
|
|
Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p |
| 177 |
|
|
m line 165, <GEN9> line 1. |
| 178 |
|
|
called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.</PRE> |
| 179 |
|
|
<PRE> |
| 180 |
|
|
|
| 181 |
|
|
DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement |
| 182 |
|
|
notice: Data::Transfer::Sync->syncNodes: No nodes to synchronize. |
| 183 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 184 |
|
|
DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding |
| 185 |
|
|
parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char). |
| 186 |
|
|
If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this "Execution ERROR" which |
| 187 |
|
|
results in a "DBI-Error" afterwards. |
| 188 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE> |
| 189 |
|
|
<P> |
| 190 |
|
|
<H4><A NAME="check structure of source node">Check structure of source node</A></H4> |
| 191 |
|
|
<PRE> |
| 192 |
|
|
|
| 193 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 194 |
|
|
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
| 195 |
|
|
critical: Data::Transfer::Sync->syncNodes: Can not synchronize: No ident found in source node, maybe try to "import" this node first. |
| 196 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 197 |
|
|
If lowlevel detection succeeds, but no other required informations are found, this message is issued. |
| 198 |
|
|
"Other informations" might be: |
| 199 |
|
|
- column-header-row completely missing |
| 200 |
|
|
- ident column is empty |
| 201 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE> |
| 202 |
|
|
<P> |
| 203 |
|
|
<H4><A NAME="modify structure of source node">Modify structure of source node</A></H4> |
| 204 |
|
|
<PRE> |
| 205 |
|
|
|
| 206 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 207 |
|
|
info: Data::Transfer::Sync->syncNodes: source=L/Currency <- target=R/currencies.csv |
| 208 |
|
|
info: Data::Transfer::Sync->_prepareNode_MetaProperties( descent source ) |
| 209 |
|
|
warning: Data::Transfer::Sync->_prepareNode_MetaProperties: node is lacking meta properties - will try to alter... |
| 210 |
|
|
SQL ERROR: Command 'ALTER' not recognized or not supported!</PRE> |
| 211 |
|
|
<PRE> |
| 212 |
|
|
|
| 213 |
|
|
SQL ERROR: Command 'ALTER' not recognized or not supported! |
| 214 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 215 |
|
|
The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing "import" - |
| 216 |
|
|
but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this. |
| 217 |
|
|
This could also appear if your database connection has insufficient rights to modify the database structure. |
| 218 |
|
|
DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually. |
| 219 |
|
|
Hint: Add columns with the names of your "ident" and "checksum" property specifications. |
| 220 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE> |
| 221 |
|
|
<P> |
| 222 |
|
|
<H4><A NAME="load source node by ident">Load source node by ident</A></H4> |
| 223 |
|
|
<PRE> |
| 224 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 225 |
|
|
info: Data::Transfer::Sync->_prepareNode_DummyIdent( descent source ) |
| 226 |
|
|
pcritical: Data::Transfer::Sync->_modifyNode failed: "source" node is empty. |
| 227 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - - |
| 228 |
|
|
The source node could not be loaded. Maybe the ident is missing. Please check manually. |
| 229 |
|
|
Hint: Like above, the ident and/or checksum columns may be missing.... |
| 230 |
|
|
- - - - - - - - - - - - - - - - - - - - - - - - - -</PRE> |
| 231 |
|
|
<P> |
| 232 |
|
|
<HR> |
| 233 |
|
|
<H1><A NAME="todo">TODO</A></H1> |
| 234 |
|
|
<PRE> |
| 235 |
joko |
1.3 |
o sub _resolveIdentProvider |
| 236 |
|
|
x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node |
| 237 |
|
|
x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm |
| 238 |
|
|
o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm |
| 239 |
|
|
x introduce Sync/Compare/MyComparisonImplementation.pm |
| 240 |
|
|
o some generic deferring method - e.g. "$self->defer(action)" - to be able to accumulate a bunch of actions for later processing |
| 241 |
joko |
1.1 |
- this implies everything done is _really_ split up into generic actions - how else would we defer them??? |
| 242 |
|
|
- example uses: |
| 243 |
|
|
- fetch whole checksum list from node |
| 244 |
|
|
- remember source ident retransmits |
| 245 |
|
|
- remember: this is convenient - and maybe / of course faster - but we'll loose "per-node-atomic" operations |
| 246 |
joko |
1.3 |
o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema) |
| 247 |
|
|
o expand statistics / keep track of: |
| 248 |
joko |
1.1 |
- touched/untouched nodes |
| 249 |
joko |
1.3 |
o full sync |
| 250 |
joko |
1.1 |
- just do a push and a pull for now but use stats for touched nodes in between to speed up things |
| 251 |
joko |
1.3 |
o introduce some new metadata flags for a descent |
| 252 |
joko |
1.1 |
- isNewNodePropagator |
| 253 |
joko |
1.3 |
x isWriteProtected</PRE> |
| 254 |
joko |
1.1 |
|
| 255 |
|
|
</BODY> |
| 256 |
|
|
|
| 257 |
|
|
</HTML> |
Parent Directory
| 
Revision Log
