dbTalk Databases Forums  

Optimization of data migration

microsoft.public.sqlserver.dts microsoft.public.sqlserver.dts


Discuss Optimization of data migration in the microsoft.public.sqlserver.dts forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
fleo
 
Posts: n/a

Default Optimization of data migration - 08-17-2005 , 08:29 AM






Hi, I might work on a DB2 to SQL Server data migration using DTS.
What should I keep in mind to optimize performance?
Thank you

Reply With Quote
  #2  
Old   
Allan Mitchell
 
Posts: n/a

Default Re: Optimization of data migration - 08-17-2005 , 01:25 PM






A few things that run off the top of my head.

Make sure that you have the appropriate indexes on the Source to facilitate retrieval of the data to move,
When inserting to the destination removes indexes and triggers.
Move only the data you ctually need to move
Fast Disks
Fast CPUs + cache
As little distance as possible betwen Source And Destination (fast Network)



On Wed, 17 Aug 2005 06:29:04 -0700, "fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote:

Quote:
Hi, I might work on a DB2 to SQL Server data migration using DTS.
What should I keep in mind to optimize performance?
Thank you

Reply With Quote
  #3  
Old   
fleo
 
Posts: n/a

Default Re: Optimization of data migration - 08-17-2005 , 09:05 PM



"Allan Mitchell" wrote:

Quote:
A few things that run off the top of my head.

Make sure that you have the appropriate indexes on the Source to facilitate retrieval of the data to move,
When inserting to the destination removes indexes and triggers.
Move only the data you ctually need to move
Fast Disks
Fast CPUs + cache
As little distance as possible betwen Source And Destination (fast Network)
Thanks Allan.

I found papers on migrating from Oracle, MySQL, ... but nothing on DB2??
(http://www.microsoft.com/technet/int...ation/sql.mspx)

I am going to be migrating the data only (not the objects). I have seen
multiple options and I am not sure about the best way to go. Should I create
a staging database?


Reply With Quote
  #4  
Old   
Allan Mitchell
 
Posts: n/a

Default Re: Optimization of data migration - 08-18-2005 , 12:30 AM



It could well be worthwhile having a staging area yes. This way your data
can go through a series of screens to clean things up and when it reaches
the end of the process is ready for insertion into the real destination.

do you need to do cleansing or is it a stright pick up and move?


Allan


"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote

Quote:
"Allan Mitchell" wrote:

A few things that run off the top of my head.

Make sure that you have the appropriate indexes on the Source to
facilitate retrieval of the data to move,
When inserting to the destination removes indexes and triggers.
Move only the data you ctually need to move
Fast Disks
Fast CPUs + cache
As little distance as possible betwen Source And Destination (fast
Network)
Thanks Allan.

I found papers on migrating from Oracle, MySQL, ... but nothing on DB2??
(http://www.microsoft.com/technet/int...ation/sql.mspx)

I am going to be migrating the data only (not the objects). I have seen
multiple options and I am not sure about the best way to go. Should I
create
a staging database?



Reply With Quote
  #5  
Old   
fleo
 
Posts: n/a

Default Re: Optimization of data migration - 08-19-2005 , 07:10 AM



Yes I need to perform cleaning/transformation.

All the info I found put the emphasis on migrating the database itself
rather than the data. Do you know where I could get information about
migrating the data? The pitfalls, the things I should check in the source
data, how to design the DTS to be able to go from developpement to test and
production phases, ...

Thank you

"Allan Mitchell" wrote:

Quote:
It could well be worthwhile having a staging area yes. This way your data
can go through a series of screens to clean things up and when it reaches
the end of the process is ready for insertion into the real destination.

do you need to do cleansing or is it a stright pick up and move?


Allan


"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote in message
news:62DC12EE-AC4C-4543-905A-F16257FB6C6D (AT) microsoft (DOT) com...
"Allan Mitchell" wrote:

A few things that run off the top of my head.

Make sure that you have the appropriate indexes on the Source to
facilitate retrieval of the data to move,
When inserting to the destination removes indexes and triggers.
Move only the data you ctually need to move
Fast Disks
Fast CPUs + cache
As little distance as possible betwen Source And Destination (fast
Network)
Thanks Allan.

I found papers on migrating from Oracle, MySQL, ... but nothing on DB2??
(http://www.microsoft.com/technet/int...ation/sql.mspx)

I am going to be migrating the data only (not the objects). I have seen
multiple options and I am not sure about the best way to go. Should I
create
a staging database?




Reply With Quote
  #6  
Old   
Allan Mitchell
 
Posts: n/a

Default Re: Optimization of data migration - 08-19-2005 , 05:33 PM



To move the data you will need to understand the differences between DB2 and
SQL Server. The ranges stored in datatypes may be different, there may be
completely incompatable datatypes. The list that I gave you will help to
optimise the load. What it will not do is tell you the differences between
SQL Server and DB2. This article does it in reverse though

http://www.redbooks.ibm.com/tstudio/...te/index_4.htm






"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote

Quote:
Yes I need to perform cleaning/transformation.

All the info I found put the emphasis on migrating the database itself
rather than the data. Do you know where I could get information about
migrating the data? The pitfalls, the things I should check in the source
data, how to design the DTS to be able to go from developpement to test
and
production phases, ...

Thank you

"Allan Mitchell" wrote:

It could well be worthwhile having a staging area yes. This way your
data
can go through a series of screens to clean things up and when it reaches
the end of the process is ready for insertion into the real destination.

do you need to do cleansing or is it a stright pick up and move?


Allan


"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote in message
news:62DC12EE-AC4C-4543-905A-F16257FB6C6D (AT) microsoft (DOT) com...
"Allan Mitchell" wrote:

A few things that run off the top of my head.

Make sure that you have the appropriate indexes on the Source to
facilitate retrieval of the data to move,
When inserting to the destination removes indexes and triggers.
Move only the data you ctually need to move
Fast Disks
Fast CPUs + cache
As little distance as possible betwen Source And Destination (fast
Network)
Thanks Allan.

I found papers on migrating from Oracle, MySQL, ... but nothing on
DB2??
(http://www.microsoft.com/technet/int...ation/sql.mspx)

I am going to be migrating the data only (not the objects). I have
seen
multiple options and I am not sure about the best way to go. Should I
create
a staging database?






Reply With Quote
  #7  
Old   
fleo
 
Posts: n/a

Default Re: Optimization of data migration - 08-21-2005 , 08:22 PM



Many thanks for your help Alan!
I read the article and got puzzled by the "source physical file". Is there
something equivalent when migrating to SQL Server?

I am definitely not an expert on DTS so maybe I'm missing something and
being too simplistic. I had in mind to create an OLEDB connection to the DB2
database and copy the data "as is" in a staging database. Maybe I could
perform some sort of filtration at the same time??

Then I would transform/copy the data to the real database...

"Allan Mitchell" wrote:

Quote:
To move the data you will need to understand the differences between DB2 and
SQL Server. The ranges stored in datatypes may be different, there may be
completely incompatable datatypes. The list that I gave you will help to
optimise the load. What it will not do is tell you the differences between
SQL Server and DB2. This article does it in reverse though

http://www.redbooks.ibm.com/tstudio/...te/index_4.htm






"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote in message
news:80BB64EE-83F3-4D66-B7AF-614FF4EE80D3 (AT) microsoft (DOT) com...
Yes I need to perform cleaning/transformation.

All the info I found put the emphasis on migrating the database itself
rather than the data. Do you know where I could get information about
migrating the data? The pitfalls, the things I should check in the source
data, how to design the DTS to be able to go from developpement to test
and
production phases, ...

Thank you

"Allan Mitchell" wrote:

It could well be worthwhile having a staging area yes. This way your
data
can go through a series of screens to clean things up and when it reaches
the end of the process is ready for insertion into the real destination.

do you need to do cleansing or is it a stright pick up and move?


Allan


"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote in message
news:62DC12EE-AC4C-4543-905A-F16257FB6C6D (AT) microsoft (DOT) com...
"Allan Mitchell" wrote:

A few things that run off the top of my head.

Make sure that you have the appropriate indexes on the Source to
facilitate retrieval of the data to move,
When inserting to the destination removes indexes and triggers.
Move only the data you ctually need to move
Fast Disks
Fast CPUs + cache
As little distance as possible betwen Source And Destination (fast
Network)
Thanks Allan.

I found papers on migrating from Oracle, MySQL, ... but nothing on
DB2??
(http://www.microsoft.com/technet/int...ation/sql.mspx)

I am going to be migrating the data only (not the objects). I have
seen
multiple options and I am not sure about the best way to go. Should I
create
a staging database?







Reply With Quote
  #8  
Old   
Allan Mitchell
 
Posts: n/a

Default Re: Optimization of data migration - 08-22-2005 , 12:31 AM



Your idea is perfectly workable. You would load the data to a staging DB,
perform a number of screen upon the data and then present it to the outside
world by moving it into your real DB

Allan

"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote

Quote:
Many thanks for your help Alan!
I read the article and got puzzled by the "source physical file". Is
there
something equivalent when migrating to SQL Server?

I am definitely not an expert on DTS so maybe I'm missing something and
being too simplistic. I had in mind to create an OLEDB connection to the
DB2
database and copy the data "as is" in a staging database. Maybe I could
perform some sort of filtration at the same time??

Then I would transform/copy the data to the real database...

"Allan Mitchell" wrote:

To move the data you will need to understand the differences between DB2
and
SQL Server. The ranges stored in datatypes may be different, there may
be
completely incompatable datatypes. The list that I gave you will help to
optimise the load. What it will not do is tell you the differences
between
SQL Server and DB2. This article does it in reverse though

http://www.redbooks.ibm.com/tstudio/...te/index_4.htm






"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote in message
news:80BB64EE-83F3-4D66-B7AF-614FF4EE80D3 (AT) microsoft (DOT) com...
Yes I need to perform cleaning/transformation.

All the info I found put the emphasis on migrating the database itself
rather than the data. Do you know where I could get information about
migrating the data? The pitfalls, the things I should check in the
source
data, how to design the DTS to be able to go from developpement to test
and
production phases, ...

Thank you

"Allan Mitchell" wrote:

It could well be worthwhile having a staging area yes. This way your
data
can go through a series of screens to clean things up and when it
reaches
the end of the process is ready for insertion into the real
destination.

do you need to do cleansing or is it a stright pick up and move?


Allan


"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote in message
news:62DC12EE-AC4C-4543-905A-F16257FB6C6D (AT) microsoft (DOT) com...
"Allan Mitchell" wrote:

A few things that run off the top of my head.

Make sure that you have the appropriate indexes on the Source to
facilitate retrieval of the data to move,
When inserting to the destination removes indexes and triggers.
Move only the data you ctually need to move
Fast Disks
Fast CPUs + cache
As little distance as possible betwen Source And Destination (fast
Network)
Thanks Allan.

I found papers on migrating from Oracle, MySQL, ... but nothing on
DB2??
(http://www.microsoft.com/technet/int...ation/sql.mspx)

I am going to be migrating the data only (not the objects). I have
seen
multiple options and I am not sure about the best way to go. Should
I
create
a staging database?









Reply With Quote
  #9  
Old   
db2team@hotmail.com
 
Posts: n/a

Default Re: Optimization of data migration - 08-22-2005 , 05:54 AM



There is also a nice too called StarQuest Data Replicator:
http://www.starquest.com/Productfolder/infoSQDR.html

Good luck,

Bob


Reply With Quote
  #10  
Old   
fleo
 
Posts: n/a

Default Re: Optimization of data migration - 08-22-2005 , 08:01 PM



Thank you for your help

"fleo" wrote:

Quote:
Many thanks for your help Alan!
I read the article and got puzzled by the "source physical file". Is there
something equivalent when migrating to SQL Server?

I am definitely not an expert on DTS so maybe I'm missing something and
being too simplistic. I had in mind to create an OLEDB connection to the DB2
database and copy the data "as is" in a staging database. Maybe I could
perform some sort of filtration at the same time??

Then I would transform/copy the data to the real database...

"Allan Mitchell" wrote:

To move the data you will need to understand the differences between DB2 and
SQL Server. The ranges stored in datatypes may be different, there may be
completely incompatable datatypes. The list that I gave you will help to
optimise the load. What it will not do is tell you the differences between
SQL Server and DB2. This article does it in reverse though

http://www.redbooks.ibm.com/tstudio/...te/index_4.htm






"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote in message
news:80BB64EE-83F3-4D66-B7AF-614FF4EE80D3 (AT) microsoft (DOT) com...
Yes I need to perform cleaning/transformation.

All the info I found put the emphasis on migrating the database itself
rather than the data. Do you know where I could get information about
migrating the data? The pitfalls, the things I should check in the source
data, how to design the DTS to be able to go from developpement to test
and
production phases, ...

Thank you

"Allan Mitchell" wrote:

It could well be worthwhile having a staging area yes. This way your
data
can go through a series of screens to clean things up and when it reaches
the end of the process is ready for insertion into the real destination.

do you need to do cleansing or is it a stright pick up and move?


Allan


"fleo" <fleo (AT) discussions (DOT) microsoft.com> wrote in message
news:62DC12EE-AC4C-4543-905A-F16257FB6C6D (AT) microsoft (DOT) com...
"Allan Mitchell" wrote:

A few things that run off the top of my head.

Make sure that you have the appropriate indexes on the Source to
facilitate retrieval of the data to move,
When inserting to the destination removes indexes and triggers.
Move only the data you ctually need to move
Fast Disks
Fast CPUs + cache
As little distance as possible betwen Source And Destination (fast
Network)
Thanks Allan.

I found papers on migrating from Oracle, MySQL, ... but nothing on
DB2??
(http://www.microsoft.com/technet/int...ation/sql.mspx)

I am going to be migrating the data only (not the objects). I have
seen
multiple options and I am not sure about the best way to go. Should I
create
a staging database?







Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.