dbTalk Databases Forums  

Mass Import/Generate PKs

comp.databases.postgresql.general comp.databases.postgresql.general


Discuss Mass Import/Generate PKs in the comp.databases.postgresql.general forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Hunter Hillegas
 
Posts: n/a

Default Mass Import/Generate PKs - 11-06-2004 , 02:00 PM






I have a CSV file with 400,000 lines of email mailing list information that
I need to migrate to a new PostgreSQL database.

Each line has all the info I need except a PK (I usually use an int4 column
for a PK).

If the file were smaller I would probably just use Excel to pop in a PK and
then just load into the table...

Since Excel chokes on files greater than 65k lines, this won't work.

Is there a way to get this done inside psql for instance? Or another route?

Thanks,
Hunter



---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo (AT) postgresql (DOT) org)


Reply With Quote
  #2  
Old   
Peter Eisentraut
 
Posts: n/a

Default Re: Mass Import/Generate PKs - 11-06-2004 , 02:21 PM






Hunter Hillegas wrote:
Quote:
I have a CSV file with 400,000 lines of email mailing list
information that I need to migrate to a new PostgreSQL database.

Each line has all the info I need except a PK (I usually use an int4
column for a PK).
You could import the file into PostgreSQL and add a primary key column
later.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/


---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend



Reply With Quote
  #3  
Old   
Hunter Hillegas
 
Posts: n/a

Default Re: Mass Import/Generate PKs - 11-06-2004 , 02:29 PM



That sounds nice and easy...

So, I would do something like 'ALTER TABLE' to generate the PK column? What
would be the best way to populate it? Is there an area of doco I should be
looking at?

Thanks,
Hunter


Quote:
From: Peter Eisentraut <peter_e (AT) gmx (DOT) net
Date: Sat, 6 Nov 2004 21:21:25 +0100
To: Hunter Hillegas <lists (AT) lastonepicked (DOT) com>, PostgreSQL
pgsql-general (AT) postgresql (DOT) org
Subject: Re: [GENERAL] Mass Import/Generate PKs

Hunter Hillegas wrote:
I have a CSV file with 400,000 lines of email mailing list
information that I need to migrate to a new PostgreSQL database.

Each line has all the info I need except a PK (I usually use an int4
column for a PK).

You could import the file into PostgreSQL and add a primary key column
later.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/



---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html



Reply With Quote
  #4  
Old   
Franco Bruno Borghesi
 
Posts: n/a

Default Re: Mass Import/Generate PKs - 11-06-2004 , 03:13 PM



the simplest way to do it seems to be adding a SERIAL column to your
table, and then adding a primary key constraint:

1)insert data into table
2)ALTER TABLE <table> ADD id SERIAL;
3)ALTER TABLE <table> ADD CONSTRAINT <table>_pk PRIMARY KEY (id);

you can check the docs for the SERIAL type:
http://www.postgresql.org/docs/7.4/s...ATATYPE-SERIAL

On Sat, 2004-11-06 at 17:29, Hunter Hillegas wrote:

Quote:
That sounds nice and easy...

So, I would do something like 'ALTER TABLE' to generate the PK column? What
would be the best way to populate it? Is there an area of doco I should be
looking at?

Thanks,
Hunter


From: Peter Eisentraut <peter_e (AT) gmx (DOT) net
Date: Sat, 6 Nov 2004 21:21:25 +0100
To: Hunter Hillegas <lists (AT) lastonepicked (DOT) com>, PostgreSQL
pgsql-general (AT) postgresql (DOT) org
Subject: Re: [GENERAL] Mass Import/Generate PKs

Hunter Hillegas wrote:
I have a CSV file with 400,000 lines of email mailing list
information that I need to migrate to a new PostgreSQL database.

Each line has all the info I need except a PK (I usually use an int4
column for a PK).

You could import the file into PostgreSQL and add a primary key column
later.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/




---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (FreeBSD)

iD8DBQBBjT5221dVnhLsBV0RAlGvAJ0aK0iF/6oH53xdMiI8YKqhCQsH6ACeKH5F
SK0L3yjPNatg65thWZyOIck=
=xlVQ
-----END PGP SIGNATURE-----



Reply With Quote
  #5  
Old   
Ed L.
 
Posts: n/a

Default Re: Mass Import/Generate PKs - 11-06-2004 , 03:56 PM



On Saturday November 6 2004 2:13, Franco Bruno Borghesi wrote:
Quote:
the simplest way to do it seems to be adding a SERIAL column to your
table, and then adding a primary key constraint:

1)insert data into table
2)ALTER TABLE <table> ADD id SERIAL;
3)ALTER TABLE <table> ADD CONSTRAINT <table>_pk PRIMARY KEY (id);
You may also need to populate the id column with unique values in between
these two steps with something like "

update table set id = nextval('table_id_seq'::text) where id isnull"

I don't think SERIAL does that for you.

Ed


---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly



Reply With Quote
  #6  
Old   
Franco Bruno Borghesi
 
Posts: n/a

Default Re: Mass Import/Generate PKs - 11-06-2004 , 04:12 PM



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (FreeBSD)

iD8DBQBBjUw/21dVnhLsBV0RAgIsAJ0XYXium3Qs+1SPflb9gRs73Ap6SwCgle O0
+djocSv8cNc9eL/VIIiwQnA=
=NJ/7
-----END PGP SIGNATURE-----


Reply With Quote
  #7  
Old   
Goutam Paruchuri
 
Posts: n/a

Default Re: Mass Import/Generate PKs - 11-08-2004 , 12:54 PM



Yes you can use the copy command. Check for


copy TABLE NAME from 'c:\\bcpdata\\Files\\FILENAME.txt' with delimiter
as '\t' NULL as '';

When creating a table, use an incremental column (data type is serial).
Hope the above helps.

- Goutam


Quote:
-----Original Message-----
From: pgsql-general-owner (AT) postgresql (DOT) org
[mailtogsql-general-owner (AT) postgresql (DOT) org] On Behalf Of
Hunter Hillegas
Sent: Saturday, November 06, 2004 3:01 PM
To: PostgreSQL
Subject: [GENERAL] Mass Import/Generate PKs


I have a CSV file with 400,000 lines of email mailing list
information that I need to migrate to a new PostgreSQL database.

Each line has all the info I need except a PK (I usually use
an int4 column for a PK).

If the file were smaller I would probably just use Excel to
pop in a PK and then just load into the table...

Since Excel chokes on files greater than 65k lines, this won't work.

Is there a way to get this done inside psql for instance? Or
another route?

Thanks,
Hunter



---------------------------(end of
broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to
majordomo (AT) postgresql (DOT) org)


Confidentiality Notice
The information contained in this e-mail is confidential and intended for use only by the person(s) or organization listed in the address. If you have received this communication in error, please contact the sender at O'Neil & Associates, Inc., immediately. Any copying, dissemination, or distribution of this communication, other than by the intended recipient, is strictly prohibited.


---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo (AT) postgresql (DOT) org)



Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.