Designing Your Application
Data warehouses and/or data marts are created for a
variety of reasons. Knowing the factors leading to the decision
for extracting or moving data will help you identify the issues
for DM management that will have the greatest impact on your success. The
ultimate goal is always to have an automated process that moves
and converts all of your data in a very small period of time. However,
it helps to know what issues are most important in your company's
criteria for a successful data warehouse. Are you consuming large
amounts of time in the management of 3GL extraction programs and
load routines? Are you uncomfortable with ad hoc user access to
production legacy systems? Is your network unable to handle the
performance requirements for decision support and analysis?
If you consider the reasons for warehousing, your attention will
be properly focused on the critical variables. DataMigrator provides
you with the flexibility to control the variables inherent at your
site and in your data.
From an application perspective, you must evaluate what will
direct your overall data movement strategy. Considering the following
questions will help you learn more about the capabilities of DataMigrator
and how to control them.
-
What is the typical volume of data being migrated? One
million bytes, 100 million, or several gigabytes of data?
-
How often will data be migrated? Once
every night or every 15 minutes? A single large load once per month?
-
What time of day or night is optimal for moving the data? Some
sites cannot get access to data until after the nightly production
jobs are complete (which may be at 5:30 AM). When is the warehouse
required? Are different time zones involved? This complex question
impacts not only your performance requirements, but also your scheduling
methodology.
-
Is there a key value in the source data that will identify changed rows? DataMigrator
can locate changed data using an SQL WHERE criteria, unless an extract file
exists with delta changes. Is this file available? Do you know what
the WHERE criteria should look like? Using Change Data
Capture, an optional, add on component available for
selected relational databases, you can load a data target with only
the records that have changed.
-
How many designers will be developing extractions with DataMigrator? Some
sites designate a single DataMigrator administrator who is responsible
for all the extract and load routines that are used to develop their
data marts. (There are rarely any conflicts in such an environment;
one user has complete control over the requests, the source data,
and the target locations that will be written.) Other sites have
a team of developers interacting to create a collective data warehouse.