dbTalk Databases Forums  

Re: select count(*) from <table> taking FOREVER

comp.databases.ibm-db2 comp.databases.ibm-db2


Discuss Re: select count(*) from <table> taking FOREVER in the comp.databases.ibm-db2 forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Fan Ruo Xin
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-21-2003 , 07:51 PM






Is it a single db partition or multiple db partitions db server?
If it is EE, you can turn on INTRA_PARALLELISM.
Regards,
FRX

allenj (AT) ndr (DOT) com wrote:

Quote:
DB2 v8 FP2
Red Hat Linux 7.3

I have a large (well, I think its large) <table> containing around 100
million rows.

When I do a
select count(*) from <table
either from JDBC or from the DB2 CLP, things just hang. It never
comes back. Any other queries against the table are pronto, so its
not a locked table or anything. I have 2 unique indexes against
the table.

I have run stats against the table a LOT...

This should be instant. Any idea whats happening?

thanks

Allen


Reply With Quote
  #2  
Old   
Philip Nelson
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-22-2003 , 07:54 AM






On Thu, 21 Aug 2003 21:45:53 +0000, allen wrote:

Quote:
DB2 v8 FP2
Red Hat Linux 7.3

I have a large (well, I think its large) <table> containing around 100
million rows.

When I do a
select count(*) from <table
either from JDBC or from the DB2 CLP, things just hang. It never
comes back. Any other queries against the table are pronto, so its
not a locked table or anything. I have 2 unique indexes against
the table.

I have run stats against the table a LOT...

This should be instant. Any idea whats happening?

thanks

Allen
Allen,

I'm not surprised it's taking some time : you're having to read through
the whole table to return a count of all the rows. All you can do is try
to make the count go as fast as possible by throwing as much hardware and
parallelism at it as possible.

I assume you're not considering such as statement for production ? If you
were then maybe time to reconsider !!!

If so, do you really need the count 100% accurate. If not, then pick up
the ballpark figure from the catalog (CARDF on SYSIBM.SYSTABLES I think).
This of course assumes that you've run RUNSTATS recently.

HTH

Phil


Reply With Quote
  #3  
Old   
Serge Rielau
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-22-2003 , 09:07 AM



I don't think a table scan is needed. It's suffient to scan any index
(ideally the smallest)

Cheers
Serge



Reply With Quote
  #4  
Old   
allenj@ndr.com
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-22-2003 , 09:20 AM



On Fri, 22 Aug 2003 12:54:39 GMT, Philip Nelson <teamdba (AT) scotdb (DOT) com>
wrote:

Quote:
On Thu, 21 Aug 2003 21:45:53 +0000, allen wrote:

DB2 v8 FP2
Red Hat Linux 7.3

I have a large (well, I think its large) <table> containing around 100
million rows.

When I do a
select count(*) from <table

I'm not surprised it's taking some time : you're having to read through
the whole table to return a count of all the rows. All you can do is try
to make the count go as fast as possible by throwing as much hardware and
parallelism at it as possible.

I assume you're not considering such as statement for production ? If you
were then maybe time to reconsider !!!

If so, do you really need the count 100% accurate. If not, then pick up
the ballpark figure from the catalog (CARDF on SYSIBM.SYSTABLES I think).
This of course assumes that you've run RUNSTATS recently.
Shouldn't the "select count(*)" pick up the ballpark figure from the
catalog, rather than literally counting rows?


Reply With Quote
  #5  
Old   
Mark Yudkin
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-22-2003 , 10:26 AM



I'm doing count(*) queries over multi-million rows tables with response
times of a few seconds. For 100M rows, I'd expect around 1-2 minutes on our
system (W2K AS, 4x500MHz Xeon, 2.75GB RAM). There's no I/O involved.

You should examine DB2 I/O (use the performance monitoring tools). If it's
doing I/O to count the rows, that's the problem. Add RAM and increase the
buffer pool until it stops doing I/O (ours is around 1GB). Also enable
multi-page prefretching, add I/O agents, etc, to speed up the initial I/O.

If that doesn't help, you can always migrate from Linux to Windows 2000 /
2003 AS or DC - it has a better I/O model. (Now you can all flame me!)

<allenj (AT) ndr (DOT) com> wrote

Quote:
DB2 v8 FP2
Red Hat Linux 7.3

I have a large (well, I think its large) <table> containing around 100
million rows.

When I do a
select count(*) from <table
either from JDBC or from the DB2 CLP, things just hang. It never
comes back. Any other queries against the table are pronto, so its
not a locked table or anything. I have 2 unique indexes against
the table.

I have run stats against the table a LOT...

This should be instant. Any idea whats happening?

thanks

Allen



Reply With Quote
  #6  
Old   
Serge Rielau
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-22-2003 , 12:53 PM



Only if you do db2set DB2_THUMB_RULE = YES;
*tongue firmly in cheek*

Cheers
Serge



Reply With Quote
  #7  
Old   
MCPHEAL
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-22-2003 , 02:10 PM



Quote:
I don't think a table scan is needed. It's suffient to scan any index
(ideally the smallest)
agreed - you could verify this with explain


Reply With Quote
  #8  
Old   
Philip Nelson
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-22-2003 , 04:38 PM



On Fri, 22 Aug 2003 13:53:26 -0400, Serge Rielau wrote:

Quote:
Only if you do db2set DB2_THUMB_RULE = YES;
*tongue firmly in cheek*

Cheers
Serge
Would it not be possible to do something like this with the dynamic
statistics in DB2 for z/OS V8 ?

Just a thought.


Reply With Quote
  #9  
Old   
PM \(pm3iinc-nospam\)
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-23-2003 , 12:48 PM



count(*) is, if possible, modified by the optimizer with a column index.
(index scan)

SQL Statement:

select count(*)
from dboxs.lineitem
for read only


Section Code Page = 1252

Estimated Cost = 57986,378906
Estimated Cardinality = 1,000000

Access Table Name = DBOXS.LINEITEM ID = 6,11
Quote:
#Columns = 0
Relation Scan
| Prefetch: Eligible
Lock Intents
| Table: Intent Share
| Row : Next Key Share
Predicate Aggregation
| Column Function(s)
Aggregation Completion
Column Function(s)
Return Data to Application
#Columns = 1
End of section


create index
reorg
runstats
explain

SQL Statement:

select count(*)
from dboxs.lineitem
for read only


Section Code Page = 1252

Estimated Cost = 38675,507813
Estimated Cardinality = 1,000000

Access Table Name = DBOXS.LINEITEM ID = 6,11
Quote:
#Columns = 0
Index Scan: Name = DBOXS.X01LINEITEM ID = 1
| Regular Index (Clustered)
| Index Columns:
| | 1: L_ORDERKEY (Ascending)
| #Key Columns = 0
| | Start Key: Beginning of Index
| | Stop Key: End of Index
| Index-Only Access
| Index Prefetch: Eligible 1412
| Predicate Aggregation
| | Column Function(s)
Lock Intents
| Table: Intent Share
| Row : Next Key Share
Aggregation Completion
Column Function(s)
Return Data to Application
#Columns = 1
End of section



PM




Reply With Quote
  #10  
Old   
Paul Vernon
 
Posts: n/a

Default Re: select count(*) from <table> taking FOREVER - 08-27-2003 , 05:43 AM



<allenj (AT) ndr (DOT) com> wrote

Quote:
DB2 v8 FP2
Red Hat Linux 7.3

I have a large (well, I think its large) <table> containing around 100
million rows.

When I do a
select count(*) from <table
either from JDBC or from the DB2 CLP, things just hang. It never
comes back. Any other queries against the table are pronto, so its
not a locked table or anything. I have 2 unique indexes against
the table.
You could use an summary table to hold the total count (or some partitioned count)


Otherwise, in V8 this will give you some estimate of the number of rows

db2 select count(*) * 100 from <table> tablesample system(1)



Regards
Paul Vernon
Business Intelligence, IBM Global Services




Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.