dbTalk Databases Forums  

Best practice for loading data into relational tables

comp.databases.oracle.misc comp.databases.oracle.misc


Discuss Best practice for loading data into relational tables in the comp.databases.oracle.misc forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Anoop
 
Posts: n/a

Default Best practice for loading data into relational tables - 03-16-2008 , 12:19 AM






I am not sure if I can word this right. But I am trying to explore the
best way of loading data into about 40-50 oracle tables. We have
collected the data through various means / formats - some are in text
files, excel, some that we know need to be arranged etc..

The way we are doing it now, is to write insert statements for every
line of data. So the actual arrangement of the data and writing the
insert sql's are happening simultaneously. But there are many issues
in writing insert sql's. The primary being debugging, - once I run the
file containing all inserts, I get "1 row loaded" - doesnt really tell
me which sql ran - each line runs independent of the row above or
below it, the file just helps me run them all together, that is all.
The other issue (related to the first) is if the tables are related
then I need to order my inserts so that the primary key gets inserted
before the child/foreign keys. This is a very manual process and I do
not think this can be automated much.

So I am thinking probably my approach is wrong - maybe I should deal
with the data load in a different manner - is there any best practice
that I could use? How have you guys handled data loads of medium size
~10000 rows going into 40 tables that are related to one another?

Thanks.

Reply With Quote
  #2  
Old   
sybrandb@hccnet.nl
 
Posts: n/a

Default Re: Best practice for loading data into relational tables - 03-16-2008 , 07:24 AM






On Sat, 15 Mar 2008 23:19:16 -0700 (PDT), Anoop
<anoopkumarv (AT) gmail (DOT) com> wrote:

Quote:
I am not sure if I can word this right. But I am trying to explore the
best way of loading data into about 40-50 oracle tables. We have
collected the data through various means / formats - some are in text
files, excel, some that we know need to be arranged etc..

The way we are doing it now, is to write insert statements for every
line of data. So the actual arrangement of the data and writing the
insert sql's are happening simultaneously. But there are many issues
in writing insert sql's. The primary being debugging, - once I run the
file containing all inserts, I get "1 row loaded" - doesnt really tell
me which sql ran - each line runs independent of the row above or
below it, the file just helps me run them all together, that is all.
The other issue (related to the first) is if the tables are related
then I need to order my inserts so that the primary key gets inserted
before the child/foreign keys. This is a very manual process and I do
not think this can be automated much.

So I am thinking probably my approach is wrong - maybe I should deal
with the data load in a different manner - is there any best practice
that I could use? How have you guys handled data loads of medium size
~10000 rows going into 40 tables that are related to one another?

Thanks.
One word: sql*loader.
Can load data into several tables at once.
Reinventing a wheel which Oracle already provides in every release of
Oracle will not only be very costly, but also waste of resources.

--

Sybrand Bakker
Senior Oracle DBA


Reply With Quote
  #3  
Old   
sybrandb@hccnet.nl
 
Posts: n/a

Default Re: Best practice for loading data into relational tables - 03-16-2008 , 07:24 AM



On Sat, 15 Mar 2008 23:19:16 -0700 (PDT), Anoop
<anoopkumarv (AT) gmail (DOT) com> wrote:

Quote:
I am not sure if I can word this right. But I am trying to explore the
best way of loading data into about 40-50 oracle tables. We have
collected the data through various means / formats - some are in text
files, excel, some that we know need to be arranged etc..

The way we are doing it now, is to write insert statements for every
line of data. So the actual arrangement of the data and writing the
insert sql's are happening simultaneously. But there are many issues
in writing insert sql's. The primary being debugging, - once I run the
file containing all inserts, I get "1 row loaded" - doesnt really tell
me which sql ran - each line runs independent of the row above or
below it, the file just helps me run them all together, that is all.
The other issue (related to the first) is if the tables are related
then I need to order my inserts so that the primary key gets inserted
before the child/foreign keys. This is a very manual process and I do
not think this can be automated much.

So I am thinking probably my approach is wrong - maybe I should deal
with the data load in a different manner - is there any best practice
that I could use? How have you guys handled data loads of medium size
~10000 rows going into 40 tables that are related to one another?

Thanks.
One word: sql*loader.
Can load data into several tables at once.
Reinventing a wheel which Oracle already provides in every release of
Oracle will not only be very costly, but also waste of resources.

--

Sybrand Bakker
Senior Oracle DBA


Reply With Quote
  #4  
Old   
sybrandb@hccnet.nl
 
Posts: n/a

Default Re: Best practice for loading data into relational tables - 03-16-2008 , 07:24 AM



On Sat, 15 Mar 2008 23:19:16 -0700 (PDT), Anoop
<anoopkumarv (AT) gmail (DOT) com> wrote:

Quote:
I am not sure if I can word this right. But I am trying to explore the
best way of loading data into about 40-50 oracle tables. We have
collected the data through various means / formats - some are in text
files, excel, some that we know need to be arranged etc..

The way we are doing it now, is to write insert statements for every
line of data. So the actual arrangement of the data and writing the
insert sql's are happening simultaneously. But there are many issues
in writing insert sql's. The primary being debugging, - once I run the
file containing all inserts, I get "1 row loaded" - doesnt really tell
me which sql ran - each line runs independent of the row above or
below it, the file just helps me run them all together, that is all.
The other issue (related to the first) is if the tables are related
then I need to order my inserts so that the primary key gets inserted
before the child/foreign keys. This is a very manual process and I do
not think this can be automated much.

So I am thinking probably my approach is wrong - maybe I should deal
with the data load in a different manner - is there any best practice
that I could use? How have you guys handled data loads of medium size
~10000 rows going into 40 tables that are related to one another?

Thanks.
One word: sql*loader.
Can load data into several tables at once.
Reinventing a wheel which Oracle already provides in every release of
Oracle will not only be very costly, but also waste of resources.

--

Sybrand Bakker
Senior Oracle DBA


Reply With Quote
  #5  
Old   
sybrandb@hccnet.nl
 
Posts: n/a

Default Re: Best practice for loading data into relational tables - 03-16-2008 , 07:24 AM



On Sat, 15 Mar 2008 23:19:16 -0700 (PDT), Anoop
<anoopkumarv (AT) gmail (DOT) com> wrote:

Quote:
I am not sure if I can word this right. But I am trying to explore the
best way of loading data into about 40-50 oracle tables. We have
collected the data through various means / formats - some are in text
files, excel, some that we know need to be arranged etc..

The way we are doing it now, is to write insert statements for every
line of data. So the actual arrangement of the data and writing the
insert sql's are happening simultaneously. But there are many issues
in writing insert sql's. The primary being debugging, - once I run the
file containing all inserts, I get "1 row loaded" - doesnt really tell
me which sql ran - each line runs independent of the row above or
below it, the file just helps me run them all together, that is all.
The other issue (related to the first) is if the tables are related
then I need to order my inserts so that the primary key gets inserted
before the child/foreign keys. This is a very manual process and I do
not think this can be automated much.

So I am thinking probably my approach is wrong - maybe I should deal
with the data load in a different manner - is there any best practice
that I could use? How have you guys handled data loads of medium size
~10000 rows going into 40 tables that are related to one another?

Thanks.
One word: sql*loader.
Can load data into several tables at once.
Reinventing a wheel which Oracle already provides in every release of
Oracle will not only be very costly, but also waste of resources.

--

Sybrand Bakker
Senior Oracle DBA


Reply With Quote
  #6  
Old   
Markus Tazl
 
Posts: n/a

Default Re: Best practice for loading data into relational tables - 03-16-2008 , 09:40 AM



On 16 Mrz., 07:19, Anoop <anoopkum... (AT) gmail (DOT) com> wrote:
How have you guys handled data loads of medium size
Quote:
~10000 rows going into 40 tables that are related to one another?
as Sybrand stated, SQL Loader is number one choice.

http://www.orafaq.com/wiki/SQL%2ALoader_FAQ

http://www.psoug.org/reference/sqlloader.html

best regards
Markus Tazl



Reply With Quote
  #7  
Old   
Markus Tazl
 
Posts: n/a

Default Re: Best practice for loading data into relational tables - 03-16-2008 , 09:40 AM



On 16 Mrz., 07:19, Anoop <anoopkum... (AT) gmail (DOT) com> wrote:
How have you guys handled data loads of medium size
Quote:
~10000 rows going into 40 tables that are related to one another?
as Sybrand stated, SQL Loader is number one choice.

http://www.orafaq.com/wiki/SQL%2ALoader_FAQ

http://www.psoug.org/reference/sqlloader.html

best regards
Markus Tazl



Reply With Quote
  #8  
Old   
Markus Tazl
 
Posts: n/a

Default Re: Best practice for loading data into relational tables - 03-16-2008 , 09:40 AM



On 16 Mrz., 07:19, Anoop <anoopkum... (AT) gmail (DOT) com> wrote:
How have you guys handled data loads of medium size
Quote:
~10000 rows going into 40 tables that are related to one another?
as Sybrand stated, SQL Loader is number one choice.

http://www.orafaq.com/wiki/SQL%2ALoader_FAQ

http://www.psoug.org/reference/sqlloader.html

best regards
Markus Tazl



Reply With Quote
  #9  
Old   
Markus Tazl
 
Posts: n/a

Default Re: Best practice for loading data into relational tables - 03-16-2008 , 09:40 AM



On 16 Mrz., 07:19, Anoop <anoopkum... (AT) gmail (DOT) com> wrote:
How have you guys handled data loads of medium size
Quote:
~10000 rows going into 40 tables that are related to one another?
as Sybrand stated, SQL Loader is number one choice.

http://www.orafaq.com/wiki/SQL%2ALoader_FAQ

http://www.psoug.org/reference/sqlloader.html

best regards
Markus Tazl



Reply With Quote
  #10  
Old   
DA Morgan
 
Posts: n/a

Default Re: Best practice for loading data into relational tables - 03-16-2008 , 12:27 PM



Anoop wrote:
Quote:
I am not sure if I can word this right. But I am trying to explore the
best way of loading data into about 40-50 oracle tables. We have
collected the data through various means / formats - some are in text
files, excel, some that we know need to be arranged etc..

The way we are doing it now, is to write insert statements for every
line of data. So the actual arrangement of the data and writing the
insert sql's are happening simultaneously. But there are many issues
in writing insert sql's. The primary being debugging, - once I run the
file containing all inserts, I get "1 row loaded" - doesnt really tell
me which sql ran - each line runs independent of the row above or
below it, the file just helps me run them all together, that is all.
The other issue (related to the first) is if the tables are related
then I need to order my inserts so that the primary key gets inserted
before the child/foreign keys. This is a very manual process and I do
not think this can be automated much.

So I am thinking probably my approach is wrong - maybe I should deal
with the data load in a different manner - is there any best practice
that I could use? How have you guys handled data loads of medium size
~10000 rows going into 40 tables that are related to one another?

Thanks.
I'd vote SQL*Loader is almost all cases though given the small number of
rows you might want to look at using an External Table.
http://www.psoug.org/reference/externaltab.html
Which is really just another implementation of SQL*Loader.
--
Daniel A. Morgan
Oracle Ace Director & Instructor
University of Washington
damorgan@x.washington.edu (replace x with u to respond)
Puget Sound Oracle Users Group
www.psoug.org


Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.