dbTalk Databases Forums  

Re: DELETE operations on large tables

comp.databases.mysql comp.databases.mysql


Discuss Re: DELETE operations on large tables in the comp.databases.mysql forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Lennart Jonsson
 
Posts: n/a

Default Re: DELETE operations on large tables - 07-16-2011 , 02:54 PM






On 2011-07-16 16:43, Ignoramus29044 wrote:
Quote:
I have had one issue with mysql. I have a couple of projects with
relatively lagre databases (tens of gigabytes).

Sometimes I have to do a DELETE operation, such as to clean up old
(no longer needed) entries.

When I do a DELETE, the table is locked. This may take a very long time,
during which all SELECT queries are blocked. This locks to apache
webserver instances locking in wait, overwhelming apache, etc.

I would like to know if there is a solution for this.

Have you considered range partitioning?

http://dev.mysql.com/doc/refman/5.1/...ing-range.html


/Lennart

[...]

Reply With Quote
  #2  
Old   
Ignoramus29044
 
Posts: n/a

Default Re: DELETE operations on large tables - 07-16-2011 , 07:41 PM






On 2011-07-16, Lennart Jonsson <erik.lennart.jonsson (AT) gmail (DOT) com> wrote:
Quote:
On 2011-07-16 16:43, Ignoramus29044 wrote:
I have had one issue with mysql. I have a couple of projects with
relatively lagre databases (tens of gigabytes).

Sometimes I have to do a DELETE operation, such as to clean up old
(no longer needed) entries.

When I do a DELETE, the table is locked. This may take a very long time,
during which all SELECT queries are blocked. This locks to apache
webserver instances locking in wait, overwhelming apache, etc.

I would like to know if there is a solution for this.


Have you considered range partitioning?

http://dev.mysql.com/doc/refman/5.1/...ing-range.html


That sounds like a good plan. Another idea to keep in mind is to have
a separate table, such as

bigtable_date DATE,
bigtable_id bigint

and select from this table and do single deletes (or group deletes) on
the big table.

Maybe it is better than DELETE ... LIMIT ...?

Reply With Quote
  #3  
Old   
Álvaro G. Vicario
 
Posts: n/a

Default Re: DELETE operations on large tables - 07-18-2011 , 01:23 AM



El 16/07/2011 16:43, Ignoramus29044 escribió/wrote:
Quote:
I have had one issue with mysql. I have a couple of projects with
relatively lagre databases (tens of gigabytes).

Sometimes I have to do a DELETE operation, such as to clean up old
(no longer needed) entries.

When I do a DELETE, the table is locked. This may take a very long time,
during which all SELECT queries are blocked. This locks to apache
webserver instances locking in wait, overwhelming apache, etc.

I would like to know if there is a solution for this.
If the whole table is getting locked, you are probably using MyISAM. In
such case, you might be able to benefit from the LOW_PRIORITY and QUICK
clauses:

http://dev.mysql.com/doc/refman/5.5/en/delete.html

See also:

http://dev.mysql.com/doc/refman/5.5/...ete-speed.html


--
-- http://alvaro.es - Álvaro G. Vicario - Burgos, Spain
-- Mi sitio sobre programación web: http://borrame.com
-- Mi web de humor satinado: http://www.demogracia.com
--

Reply With Quote
  #4  
Old   
John Nagle
 
Posts: n/a

Default Re: DELETE operations on large tables - 07-20-2011 , 03:31 PM



On 7/16/2011 7:43 AM, Ignoramus29044 wrote:
Quote:
I have had one issue with mysql. I have a couple of projects with
relatively lagre databases (tens of gigabytes).

Sometimes I have to do a DELETE operation, such as to clean up old
(no longer needed) entries.

When I do a DELETE, the table is locked. This may take a very long time,
during which all SELECT queries are blocked. This locks to apache
webserver instances locking in wait, overwhelming apache, etc.

I would like to know if there is a solution for this.
One of the simplest solutions is, rather than doing a big delete
infrequently, do a small delete frequently. If you want to delete
entries more than 30 days old, do it every day, or more frequently
if necessary. If the WHERE expression is one that executes fast
(like "mydate < '2011-03-01'", assuming "mydate" is indexed),
this is cheap. If you run more than once a day, try
like "mydate < TIMESTAMPADD(DAY,-31,NOW())", which is exactly
31 days ago.

Partitioning is an option. but may be overkill.

John Nagle

Reply With Quote
  #5  
Old   
Luuk
 
Posts: n/a

Default Re: DELETE operations on large tables - 07-21-2011 , 12:13 PM



On 20-07-2011 22:31, John Nagle wrote:
Quote:
On 7/16/2011 7:43 AM, Ignoramus29044 wrote:
I have had one issue with mysql. I have a couple of projects with
relatively lagre databases (tens of gigabytes).

Sometimes I have to do a DELETE operation, such as to clean up old
(no longer needed) entries.

When I do a DELETE, the table is locked. This may take a very long time,
during which all SELECT queries are blocked. This locks to apache
webserver instances locking in wait, overwhelming apache, etc.

I would like to know if there is a solution for this.

One of the simplest solutions is, rather than doing a big delete
infrequently, do a small delete frequently. If you want to delete
entries more than 30 days old, do it every day, or more frequently
if necessary. If the WHERE expression is one that executes fast
(like "mydate < '2011-03-01'", assuming "mydate" is indexed),
this is cheap. If you run more than once a day, try
like "mydate < TIMESTAMPADD(DAY,-31,NOW())", which is exactly
31 days ago.

Partitioning is an option. but may be overkill.

John Nagle
i tried to suggest that 4 days ago, but i'm not that good in explaining

--
Luuk

Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.