dbTalk Databases Forums  

long deletes :( Pls help

comp.databases.postgresql.novice comp.databases.postgresql.novice


Discuss long deletes :( Pls help in the comp.databases.postgresql.novice forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Noel
 
Posts: n/a

Default long deletes :( Pls help - 01-19-2004 , 07:24 PM






Hi all,

I'm trying to delete from a table which has 42Mill rows, using a foreign
key, which is index.
The delete has been going three days now and is really frustrating.

The scheme is as follows:

region
-------
id
// other fields

region_db_comparison
-----------------------
id
// other fields

alignment // (42Mil) from which I'm trying to delete from
----------
id
region_db_comparison // foreign key to region_db_comparison.id and
indexed (relation: 1(reg_db_cmp) -> N (alignment))
subject_region // foreign key to region.id and indexed (relation:
1(region -> N(alignments))
// other fields

gap //(129Mil entries)
---
id
alignment // foreign key to alignment.id and indexed (relation
N(alignment) -> N(gap))
// other fields

repeat_blastp_block
---------------------
id
alignment // foreign key to alignment.id and indexed (relation
N(alignment) -> N(gap))

I'm doing a delete of a region_db_comparison, in java.
The steps in the code are:
Collect alignments for the region_db_comparison -> For each alignment
delete any gaps associated with it (through the gap.alignment field)
then delete the alignments via the alignment.region_db_comparison field
(WHERE the code takes the most time e.g.. three days)-> delete the
region_db_comparison through id.

When the tables were created we specified NO rules/triggers for DELETE
(such as cascade etc...) only stated which fields were foreign keys and
which tables & fields the referenced to.

Any help / suggestions speeding the delete would be really appreciated.
A colleague suggested disabling any triggers that were automatically
created when the tables were, would this help?
I've looked at pg_triggers but can't make sense of the fields tgtype or
tgargs. What do the values match to? What is the order of tgargs?
For example:
tgrelid | tgname |
tgfoid | tgtype | tgenabled | tgisconstraint | tgconstrname|
17863718 | RI_ConstraintTrigger_35330293 | 1654 | 9 |
t | t | <unnamed> |

tgconstrrelid | tgdeferrable | tginitdeferred | tgnargs | tgattr |
tgargs
17863723 | f | f | 6
Quote:
unnamed>\000gap\000alignment\000UNSPECIFIED\000ali gnment\000id\000
Am I understanding the tgargs fields correct; this is a trigger for the
table gap on the feild alignment to the table alignment for the feild id?

--
Noel Faux
Department of Biochemistry and Molecular Biology
Monash University
Clayton 3168
Victoria
Australia



---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster



Reply With Quote
  #2  
Old   
Noel
 
Posts: n/a

Default Re: long deletes :( Pls help - 01-19-2004 , 07:44 PM






Forgot to mention:
Running psql (PostgreSQL) 7.3.1 and using jdbc driver pg73jdbc3.jar
Cheers
Noel

Noel wrote:

Quote:
Hi all,

I'm trying to delete from a table which has 42Mill rows, using a
foreign key, which is index.
The delete has been going three days now and is really frustrating.

The scheme is as follows:

region
-------
id
// other fields

region_db_comparison
-----------------------
id
// other fields

alignment // (42Mil) from which I'm trying to delete from
----------
id
region_db_comparison // foreign key to region_db_comparison.id and
indexed (relation: 1(reg_db_cmp) -> N (alignment))
subject_region // foreign key to region.id and indexed (relation:
1(region -> N(alignments))
// other fields

gap //(129Mil entries)
---
id
alignment // foreign key to alignment.id and indexed (relation
N(alignment) -> N(gap))
// other fields

repeat_blastp_block
---------------------
id
alignment // foreign key to alignment.id and indexed (relation
N(alignment) -> N(gap))

I'm doing a delete of a region_db_comparison, in java.
The steps in the code are:
Collect alignments for the region_db_comparison -> For each alignment
delete any gaps associated with it (through the gap.alignment field)
then delete the alignments via the alignment.region_db_comparison
field (WHERE the code takes the most time e.g.. three days)-> delete
the region_db_comparison through id.

When the tables were created we specified NO rules/triggers for DELETE
(such as cascade etc...) only stated which fields were foreign keys
and which tables & fields the referenced to.

Any help / suggestions speeding the delete would be really appreciated.
A colleague suggested disabling any triggers that were automatically
created when the tables were, would this help?
I've looked at pg_triggers but can't make sense of the fields tgtype
or tgargs. What do the values match to? What is the order of tgargs?
For example:
tgrelid | tgname |
tgfoid | tgtype | tgenabled | tgisconstraint | tgconstrname|
17863718 | RI_ConstraintTrigger_35330293 | 1654 | 9 |
t | t | <unnamed> |

tgconstrrelid | tgdeferrable | tginitdeferred | tgnargs | tgattr |
tgargs
17863723 | f | f | 6
|
|<unnamed>\000gap\000alignment\000UNSPECIFIED\000a lignment\000id\000

Am I understanding the tgargs fields correct; this is a trigger for
the table gap on the feild alignment to the table alignment for the
feild id?


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster



Reply With Quote
  #3  
Old   
Tom Lane
 
Posts: n/a

Default Re: long deletes :( Pls help - 01-19-2004 , 08:38 PM



Noel <noel.faux (AT) med (DOT) monash.edu.au> writes:
Quote:
I'm trying to delete from a table which has 42Mill rows, using a foreign
key, which is index.
The delete has been going three days now and is really frustrating.
It sounds like you're getting a really bad plan for the queries
triggered by the foreign key constraints. One thing to make sure
of is that the datatype of the referenced and referencing columns
are the same in each foreign key relationship. Also check that the
planner statistics are reasonably up to date (pg_class.relpages and
pg_class.reltuples should be in the vicinity of reality, at least).

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html



Reply With Quote
  #4  
Old   
Tom Lane
 
Posts: n/a

Default Re: long deletes :( Pls help - 01-19-2004 , 08:48 PM



I said:
Quote:
It sounds like you're getting a really bad plan for the queries
triggered by the foreign key constraints. One thing to make sure
of is that the datatype of the referenced and referencing columns
are the same in each foreign key relationship. Also check that the
planner statistics are reasonably up to date (pg_class.relpages and
pg_class.reltuples should be in the vicinity of reality, at least).
Oh, I almost forgot the most important thing: you need an index on the
referencing column in each FK relationship. The system requires you to
have an index on the referenced column, but not the other one.
Unfortunately, deletes in the referenced table are gonna be really slow
if you do not have such an index ...

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend



Reply With Quote
  #5  
Old   
Iandé Coutinho
 
Posts: n/a

Default executing backup remotly - 01-22-2004 , 08:30 AM



Hi guys,

for the past 2 hours or so, i´ve been trying to execute pg_dump remotly,
let me explain better;

I´ve got postgresql 7.4.1 with cygwin and ipc-daemon2, running on winXP,
i´ve created the database, got postmaster running, everything seems fine.
Please correct me if i´m wrong, but to do a backup of a database i only need
the the permission of an OS user and need to execute a command line like
pg_dump.

problem: I have an apllication on one machine running a windows server OS,
and the database is on a linux, RH8.0, how can i execute a command to
perform a backup operation, from the windows aplication machine, both OS
user are disticts, but have admin privileges.

any help, would be apreciated,

thx,

Iandé




---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend


Reply With Quote
  #6  
Old   
Tom Lane
 
Posts: n/a

Default Re: executing backup remotly - 01-22-2004 , 09:49 AM



=?iso-8859-1?Q?Iand=E9_Coutinho?= <iande (AT) br (DOT) inter.net> writes:
Quote:
for the past 2 hours or so, i´ve been trying to execute pg_dump remotly,
let me explain better;
It would help if you'd shown exactly what you tried and what errors
you got.

This should work as long as (a) you specify "-h other-machine" in
pg_dump's command line, and (b) the other machine is set up to allow
connections from your machine --- that means a suitable entry in
pg_hba.conf and TCP connections enabled in postgresql.conf. There's
not enough info in your message to guess exactly where you went wrong,
though.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo (AT) postgresql (DOT) org)



Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.