/[cvs]/nfo/perl/libs/Data/Transfer/Sync.README.html
ViewVC logotype

Contents of /nfo/perl/libs/Data/Transfer/Sync.README.html

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.5 - (show annotations)
Tue Feb 11 10:20:12 2003 UTC (22 years, 8 months ago) by joko
Branch: MAIN
CVS Tags: HEAD
Changes since 1.4: +0 -0 lines
File MIME type: text/html
FILE REMOVED 
- removed
1 <HTML>
2 <HEAD>
3 <TITLE>Data::Storage::Sync</TITLE>
4 <LINK REV="made" HREF="mailto:">
5 </HEAD>
6
7 <BODY>
8
9 <A NAME="__index__"></A>
10 <!-- INDEX BEGIN -->
11
12 <UL>
13
14 <LI><A HREF="#news">NEWS</A></LI>
15 <UL>
16
17 <UL>
18
19 <UL>
20
21 <LI><A HREF="#20030209 backward compatibility with custom application">2003-02-09 - backward compatibility with custom application</A></LI>
22 <LI><A HREF="#20030118 start integration with torus">2003-01-18 - start integration with Torus</A></LI>
23 <LI><A HREF="#20021010 initial commit">2002-10-10 - initial commit</A></LI>
24 </UL>
25
26 </UL>
27
28 </UL>
29
30 <LI><A HREF="#description">DESCRIPTION</A></LI>
31 <LI><A HREF="#requirements">REQUIREMENTS</A></LI>
32 <UL>
33
34 <UL>
35
36 <LI><A HREF="#dependencies">dependencies</A></LI>
37 <LI><A HREF="#additional modules required for 'outlook2ldap'">additional modules required for 'outlook2ldap'</A></LI>
38 <LI><A HREF="#additional modules required for 'db2db'">additional modules required for 'db2db'</A></LI>
39 </UL>
40
41 </UL>
42
43 <LI><A HREF="#authors / copyright / license">AUTHORS / COPYRIGHT / LICENSE</A></LI>
44 <LI><A HREF="#support / warranty">SUPPORT / WARRANTY</A></LI>
45 <LI><A HREF="#example output chunks">EXAMPLE OUTPUT CHUNKS</A></LI>
46 <UL>
47
48 <LI><A HREF="#synchronizing orm objects against entries in a .csv file 17 items modified">Synchronizing orm objects against entries in a .csv file - 17 items modified</A></LI>
49 <LI><A HREF="#synchronizing orm objects against entries in a .csv file insync">Synchronizing orm objects against entries in a .csv file - in-sync</A></LI>
50 </UL>
51
52 <LI><A HREF="#bugs">BUGS</A></LI>
53 <LI><A HREF="#user level errors">USER LEVEL ERRORS</A></LI>
54 <UL>
55
56 <UL>
57
58 <UL>
59
60 <LI><A HREF="#mapping">Mapping</A></LI>
61 <LI><A HREF="#dbd::autocsv's rulebase">DBD::AutoCSV's rulebase</A></LI>
62 <LI><A HREF="#check structure of source node">Check structure of source node</A></LI>
63 <LI><A HREF="#modify structure of source node">Modify structure of source node</A></LI>
64 <LI><A HREF="#load source node by ident">Load source node by ident</A></LI>
65 </UL>
66
67 </UL>
68
69 </UL>
70
71 <LI><A HREF="#todo">TODO</A></LI>
72 </UL>
73 <!-- INDEX END -->
74
75 <HR>
76 <P>
77 <HR>
78 <H1><A NAME="news">NEWS</A></H1>
79 <P>
80 <H4><A NAME="20030209 backward compatibility with custom application">2003-02-09 - backward compatibility with custom application</A></H4>
81 <PRE>
82 - Tangram &lt;-&gt; .csv
83 - Tangram &lt;-&gt; mysql
84 - Tangram &lt;-&gt; perl-code</PRE>
85 <P>
86 <H4><A NAME="20030118 start integration with torus">2003-01-18 - start integration with Torus</A></H4>
87 <PRE>
88 - MAPI &lt;-&gt; LDAP
89 - please visit <A HREF="http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/">http://www.netfrag.org/horde/chora/cvs.php/perl/scripts/outlook2ldap/</A></PRE>
90 <P>
91 <H4><A NAME="20021010 initial commit">2002-10-10 - initial commit</A></H4>
92 <PRE>
93 - Tangram &lt;-&gt; mysql</PRE>
94 <P>
95 <HR>
96 <H1><A NAME="description">DESCRIPTION</A></H1>
97 <PRE>
98 Data::Transfer::Sync is a module providing a generic synchronization process
99 across arbitrary/multiple storages based on a ident/checksum mechanism.
100 It sits on top of Data::Storage.</PRE>
101 <P>
102 <HR>
103 <H1><A NAME="requirements">REQUIREMENTS</A></H1>
104 <P>
105 <H3><A NAME="dependencies">dependencies</A></H3>
106 <PRE>
107 from CPAN or (for win32) a ppm-repository:
108 App::Config (CPAN only)
109 Tie::IxHash
110 Log::Dispatch
111 Log::Dispatch::Config
112 mixin
113 Hash::Merge
114 Tie::SecureHash
115 Date::Manip</PRE>
116 <PRE>
117 from nfo/perl/libs:
118 Data::Compare
119 Data::Storage
120 Data::Transform
121 DesignPattern:: (to be renamed into Class::)
122 ... and all their dependencies.</PRE>
123 <P>
124 <H3><A NAME="additional modules required for 'outlook2ldap'">additional modules required for 'outlook2ldap'</A></H3>
125 <PRE>
126 Win32::OLE
127 Net::LDAP</PRE>
128 <P>
129 <H3><A NAME="additional modules required for 'db2db'">additional modules required for 'db2db'</A></H3>
130 <PRE>
131 DBI
132 DBD::mysql
133 DBD::CSV
134 Tangram</PRE>
135 <P>
136 <HR>
137 <H1><A NAME="authors / copyright / license">AUTHORS / COPYRIGHT / LICENSE</A></H1>
138 <PRE>
139 The Data::Transfer::Sync module is Copyright (c) 2002 Andreas Motl.
140 All rights reserved.</PRE>
141 <PRE>
142 Code contributions, suggestions, support and ideas by
143 Sebastian Utz &lt;su@tunemedia.de&gt; and Jan Hoffmann &lt;jan@romain-volk.com&gt;.
144 </PRE>
145 <PRE>
146
147 You may distribute it under the terms of either the GNU General Public
148 License or the Artistic License, as specified in the Perl README file.</PRE>
149 <P>
150 <HR>
151 <H1><A NAME="support / warranty">SUPPORT / WARRANTY</A></H1>
152 <PRE>
153 Data::Transfer::Sync is free software. IT COMES WITHOUT WARRANTY OF ANY KIND.</PRE>
154 <PRE>
155 Please feel free to contact me at &lt;andreas.motl@ilo.de&gt;.</PRE>
156 <P>
157 <HR>
158 <H1><A NAME="example output chunks">EXAMPLE OUTPUT CHUNKS</A></H1>
159 <P>
160 <H2><A NAME="synchronizing orm objects against entries in a .csv file 17 items modified">Synchronizing orm objects against entries in a .csv file - 17 items modified</A></H2>
161 <PRE>
162 notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
163 notice: OEF::Script::AbstractFeeder:
164 - Loading data of type [n/a] and
165 filtered by * from the storage named base
166 to the storage named backend - filtered by *.
167 - Will attempt to convert the data to Country.
168 info: Data::Transfer::Sync::API-&gt;syncNodes: source=backend/* [Country] &lt;- target=base/* []
169 :c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:ctr:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
170 :c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:ctr:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:ctr:c:c:c:c:ctr:c:c:c:c:c:c:c
171 :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:ctr:c:c:ctr:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
172 info: Data::Transfer::Sync::Core-&gt;_startsync: statistics: in_sync: 180 / total: 197 / attempt_modify: 17 / ok: 17</PRE>
173 <P>
174 <H2><A NAME="synchronizing orm objects against entries in a .csv file insync">Synchronizing orm objects against entries in a .csv file - in-sync</A></H2>
175 <PRE>
176
177 notice: PULL INTO backend NODE * TYPE Country SELECT NODE * TYPE [n/a] FROM base USING MODULE BizWorks::ResourceMapping;
178 notice: OEF::Script::AbstractFeeder:
179 - Loading data of type [n/a] and
180 filtered by * from the storage named base
181 to the storage named backend - filtered by *.
182 - Will attempt to convert the data to Country.
183 info: Data::Transfer::Sync::API-&gt;syncNodes: source=backend/* [Country] &lt;- target=base/* []
184 :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
185 :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
186 :c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c:c
187 info: Data::Transfer::Sync::Core-&gt;_startsync: statistics: in_sync: 197 / total: 197</PRE>
188 <P>
189 <HR>
190 <H1><A NAME="bugs">BUGS</A></H1>
191 <PRE>
192 When in &quot;import&quot; mode for windows file - DBD::AutoCSV may hang.
193 Hint: Maybe the source node contains an ident-, but no checksum-column?</PRE>
194 <P>
195 <HR>
196 <H1><A NAME="user level errors">USER LEVEL ERRORS</A></H1>
197 <P>
198 <H4><A NAME="mapping">Mapping</A></H4>
199 <PRE>
200 - - - - - - - - - - - - - - - - - - - - - - - - - -
201 info: BizWorks::Process::Setup-&gt;syncResource( source_node Currency mode PULL erase 0 import 0 )critical: BizWorks::Process::Setup-&gt;startSync: Can't access mapping for node &quot;Currency&quot; - please check BizWorks::ResourceMapping.
202 - - - - - - - - - - - - - - - - - - - - - - - - - -
203 You have to create a sub for each node used in synchronization inside named Perl module. The name of this sub _must_ match
204 the name of the node you want to sync. This sub holds mapping metadata to give the engine hints about how
205 to access the otherwise generic nodes.
206 - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
207 <P>
208 <H4><A NAME="dbd::autocsv's rulebase">DBD::AutoCSV's rulebase</A></H4>
209 <PRE>
210
211 - - - - - - - - - - - - - - - - - - - - - - - - - -
212 info: BizWorks::Process::Setup-&gt;syncResource( source_node Currency mode PULL erase 0 import 0 )
213 info: Data::Transfer::Sync-&gt;syncNodes: source=L/Currency &lt;- target=R/currencies.csv</PRE>
214 <PRE>
215
216 Execution ERROR: Error while scanning: Missing first row or scanrule not applied. at C:/home/amo/develop/netfrag.org/nfo/perl/libs/DBD/CSV.p
217 m line 165, &lt;GEN9&gt; line 1.
218 called from C:/home/amo/develop/netfrag.org/nfo/perl/libs/Data/Storage/Handler/DBI.pm at 123.</PRE>
219 <PRE>
220
221 DBI-Error: DBD::AutoCSV::st fetchrow_hashref failed: Attempt to fetch row from a Non-SELECT statement
222 notice: Data::Transfer::Sync-&gt;syncNodes: No nodes to synchronize.
223 - - - - - - - - - - - - - - - - - - - - - - - - - -
224 DBD::AutoCSV contains a rulebase which is spooled down while attempting to guess the style of the csv file regarding
225 parameters like newline (eol), column-seperation-character (sep_char), quoting character (quote_char).
226 If this spool runs out of entries and no style could be resolved, DBD::CSV dies causing this &quot;Execution ERROR&quot; which
227 results in a &quot;DBI-Error&quot; afterwards.
228 - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
229 <P>
230 <H4><A NAME="check structure of source node">Check structure of source node</A></H4>
231 <PRE>
232
233 - - - - - - - - - - - - - - - - - - - - - - - - - -
234 info: Data::Transfer::Sync-&gt;syncNodes: source=L/Currency &lt;- target=R/currencies.csv
235 critical: Data::Transfer::Sync-&gt;syncNodes: Can not synchronize: No ident found in source node, maybe try to &quot;import&quot; this node first.
236 - - - - - - - - - - - - - - - - - - - - - - - - - -
237 If lowlevel detection succeeds, but no other required informations are found, this message is issued.
238 &quot;Other informations&quot; might be:
239 - column-header-row completely missing
240 - ident column is empty
241 - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
242 <P>
243 <H4><A NAME="modify structure of source node">Modify structure of source node</A></H4>
244 <PRE>
245
246 - - - - - - - - - - - - - - - - - - - - - - - - - -
247 info: Data::Transfer::Sync-&gt;syncNodes: source=L/Currency &lt;- target=R/currencies.csv
248 info: Data::Transfer::Sync-&gt;_prepareNode_MetaProperties( descent source )
249 warning: Data::Transfer::Sync-&gt;_prepareNode_MetaProperties: node is lacking meta properties - will try to alter...
250 SQL ERROR: Command 'ALTER' not recognized or not supported!</PRE>
251 <PRE>
252
253 SQL ERROR: Command 'ALTER' not recognized or not supported!
254 - - - - - - - - - - - - - - - - - - - - - - - - - -
255 The Engine found a node which structure does not match the required. It tries to alter this automatically - only when doing &quot;import&quot; -
256 but the DBD driver (in this case DBD::CSV) gets in the way croaking not to be able to do this.
257 This could also appear if your database connection has insufficient rights to modify the database structure.
258 DBD::CSV croaks because it doesn't implement the ALTER command, so please edit your columns manually.
259 Hint: Add columns with the names of your &quot;ident&quot; and &quot;checksum&quot; property specifications.
260 - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
261 <P>
262 <H4><A NAME="load source node by ident">Load source node by ident</A></H4>
263 <PRE>
264 - - - - - - - - - - - - - - - - - - - - - - - - - -
265 info: Data::Transfer::Sync-&gt;_prepareNode_DummyIdent( descent source )
266 pcritical: Data::Transfer::Sync-&gt;_modifyNode failed: &quot;source&quot; node is empty.
267 - - - - - - - - - - - - - - - - - - - - - - - - - -
268 The source node could not be loaded. Maybe the ident is missing. Please check manually.
269 Hint: Like above, the ident and/or checksum columns may be missing....
270 - - - - - - - - - - - - - - - - - - - - - - - - - -</PRE>
271 <P>
272 <HR>
273 <H1><A NAME="todo">TODO</A></H1>
274 <PRE>
275 o sub _resolveIdentProvider
276 x wrap _doModifySource and _doTransferTarget around a core function which can change virtually any type of node
277 x split this module up into Sync.pm, Sync/Core.pm, Sync/Compare.pm and Sync/Compare/Checksum.pm
278 o introduce _compareNodes as a core method and wrap it around methods in Sync/Compare/Checksum.pm
279 x introduce Sync/Compare/MyComparisonImplementation.pm
280 o some generic deferring method - e.g. &quot;$self-&gt;defer(action)&quot; - to be able to accumulate a bunch of actions for later processing
281 - this implies everything done is _really_ split up into generic actions - how else would we defer them???
282 - example uses:
283 - fetch whole checksum list from node
284 - remember source ident retransmits
285 - remember: this is convenient - and maybe / of course faster - but we'll loose &quot;per-node-atomic&quot; operations
286 o feature: mechanism to implicit inject checksum property to nodes (alter table / modify schema)
287 o expand statistics / keep track of:
288 - touched/untouched nodes
289 o full sync
290 - just do a push and a pull for now but use stats for touched nodes in between to speed up things
291 o introduce some new metadata flags for a descent
292 - isNewNodePropagator
293 x isWriteProtected</PRE>
294
295 </BODY>
296
297 </HTML>
MailToCvsAdmin">MailToCvsAdmin
 ViewVC Help
Powered by ViewVC 1.1.26 RSS 2.0 feed