Changes between Initial Version and Version 1 of Initial Incremental Export

Timestamp:

10/31/11 13:56:51 (13 years ago)

Author:

Nick Holden

Comment:

--

Initial Incremental Export

@@ +1 @@
+In November 2011 we began work exporting data from Onyx for the first time.
+
+Recognising that data was likely to be incorrect at least in some way this is not considered to be the definitive export, and NO PURGE WILL TAKE PLACE.
+
+To manage the load on the server, and limit file sizes, the plan is to manage incremental exports up to date, recognising that this process might need to be repeated later.
+
+The task of splitting the export into increments is managed in the export-destinations.xml configuration file, as described in the [[Onyx Export and Purge]] page.
+
+Each incremental export will require a specific export-destinations.xml file with inclusion /exclusion criteria tailored to suit. Each stage in the interview needs to be specifically referenced with a stanza in the export-destinations.xml file.
+
+
+
+Strategy is to conduct exports as follows:
+
+Include only interviews with the status COMPLETE, exclude any previously exported.
+
+For each increment then have a time window based on the Recruitment Context stage, and specifically the timeStart attribute of the QuestionnaireRun.
+
+First run limit this to January through to June of the year 2010:
+
+{{{
+       <script type="INCLUDE">
+          <javascript>
+            <![CDATA[($('Participants:Admin.Interview.status').any('COMPLETED')).and($('RecruitmentContextQuestionnaire:QuestionnaireRun.timeStart').year().trim().any('2010')).and($('RecruitmentContextQuestionnaire:QuestionnaireRun.timeStart').month().trim().any('0', '1', '2', '3', '4', '5'))]]> // COMPLETE status only and year = 2010 and month 0 = January through to 5 = June
+          </javascript>
+        </script>
+        <script type="EXCLUDE">
+          <javascript>
+            <![CDATA[$('Participants:Admin.Interview.exportLog.destination').any('BRICCS.Participants')]]> // i.e. exclude if previously exported
+          </javascript>
+        </script>
+}}}
+
+
+Second run, extend the month limit through to 6 = July
+
+Third run, extend the month limit through to 7 = August
+
+Fourth run, extend the month limit through to 8 = September
+
+Fifth run, extend the month limit through to 9 = October
+
+Sixth run, extend the month limit through to 10 = November
+
+Seventh run, extend the month limit through to 11 = December
+
+Eighth run, remove the year limit, and set month limit back to 0 = January (should include only January 2011 participants)
+
+Ninth run, extend the month limit through to 1 = February
+
+Tenth run, extend the month limit through to 2 = March
+
+Eleventh run, extend the month limit through to 3 = April
+
+Twelfth run, extend the month limit through to 4 = May
+
+Thirteenth run, extend the month limit through to 5 = June (will start to possibly pick up 2010 cases again, but not if previously exported)
+
+Fourteenth run, extend the month limit through to 6 = July
+
+Fifteenth run, extend the month limit through to 7 = August
+
+Sixteenth run, extend the month limit through to 8 = September
+
+Seventeenth run, extend the month limit through to 9 = October
+
+Eighteenth run, remove both the year and month limits. Should mop up all remaining unexported interviews.
+
+On completion of the 18 runs, the total number of exported interviews should match the total number of COMPLETE interviews in Onyx.
+