dbTalk Databases Forums  

Performance of mass INSERTS into an MDC

comp.databases.ibm-db2 comp.databases.ibm-db2


Discuss Performance of mass INSERTS into an MDC in the comp.databases.ibm-db2 forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
stavrogin
 
Posts: n/a

Default Performance of mass INSERTS into an MDC - 07-16-2011 , 10:50 AM






Recently 700,000 rows were INSERTED intra-day (while the db was busy
with servicing queries) into an MDC table organized on 4 dimensions.
Some 30 different cells were hit. The INSERTS were not sorted by MDC
key. The performance was terrible. The Informatica job seemed to hang
for hours and then, when a lull in the query workload appeared, all
the rows were quickly inserted.

I believe buffer contention is the primary cause of the poor
performance and that if the INSERTS were done ordered by MDC key, this
performance could be significantly improved.

Is my assumption correct? Will mass INSERTS perform better if sorted
on the MDC key?

BTW, a test was performed by INSERTing the same rows into a non-MDC
and the performance was fine.

Reply With Quote
  #2  
Old   
Helmut Tessarek
 
Posts: n/a

Default Re: Performance of mass INSERTS into an MDC - 07-19-2011 , 03:18 AM






Quote:
Some 30 different cells were hit. The INSERTS were not sorted by MDC
key. The performance was terrible. The Informatica job seemed to hang
for hours and then, when a lull in the query workload appeared, all
the rows were quickly inserted.
Hmm, this sounds like either contention on certain index pages and/or locking
problems. Which version of DB2 or you using? db2level output, please. I
remember some APARs related to that problem, so you might want to consider
upgrading to the latest fixpak. I know that this is easier said than done, but
this is also a reason why test machines should be considered part of the
production environment... Unfortunately most CFOs see this differently.

Quote:
I believe buffer contention is the primary cause of the poor
performance and that if the INSERTS were done ordered by MDC key, this
performance could be significantly improved.
Yes, this might be true in some cases, but also depends on the root cause of
your problem. It's definitely worth a shot.

Quote:
Is my assumption correct? Will mass INSERTS perform better if sorted
on the MDC key?
Yes, this is a reasonable assumption, but depends on several factors. The
easiest would be to try it out. I'd say, your performance will be better.
I could try to elaborate which factors would influence insert performance, but
this is rather a theoretical discussion w/o more detailed information on the
environment.

--
Helmut K. C. Tessarek
DB2 Performance and Development
IBM Toronto Lab

Reply With Quote
  #3  
Old   
stavrogin
 
Posts: n/a

Default Re: Performance of mass INSERTS into an MDC - 07-19-2011 , 09:53 AM



On Jul 19, 4:18*am, Helmut Tessarek <tessa... (AT) evermeet (DOT) cx> wrote:
Quote:
Some 30 different cells were hit. The INSERTS were not sorted by MDC
key. The performance was terrible. The Informatica job seemed to hang
for hours and then, when a lull in the query workload appeared, all
the rows were quickly inserted.

Hmm, this sounds like either contention on certain index pages and/or locking
problems. Which version of DB2 or you using? db2level output, please. I
remember some APARs related to that problem, so you might want to consider
upgrading to the latest fixpak. I know that this is easier said than done, but
this is also a reason why test machines should be considered part of the
production environment... Unfortunately most CFOs see this differently.

I believe buffer contention is the primary cause of the poor
performance and that if the INSERTS were done ordered by MDC key, this
performance could be significantly improved.

Yes, this might be true in some cases, but also depends on the root causeof
your problem. It's definitely worth a shot.

Is my assumption correct? Will mass INSERTS perform better if sorted
on the MDC key?

Yes, this is a reasonable assumption, but depends on several factors. The
easiest would be to try it out. I'd say, your performance will be better.
I could try to elaborate which factors would influence insert performance, but
this is rather a theoretical discussion w/o more detailed information on the
environment.

--
Helmut K. C. Tessarek
DB2 Performance and Development
IBM Toronto Lab
as requested:

DB21085I Instance "v95inst1" uses "64" bits and DB2 code release
"SQL09053"
with level identifier "06040107".
Informational tokens are "DB2 v9.5.0.3", "s081210", "U823474", and Fix
Pack
"3a".
Product is installed at "/opt/IBM/db2/V9.5".

We will endevor to sort the input and monitor execution.

Thanks

Reply With Quote
  #4  
Old   
rmarcosa
 
Posts: n/a

Default Re: Performance of mass INSERTS into an MDC - 07-19-2011 , 05:50 PM



Be careful for lock escalations. If users are doing long queries that
lock many rows, you can have lock escalations that lock entire table,
and the inserts will have to wait to the end of the query. If you have
locktimeout to -1, the lock never ends. Ensure in the db2diag.log and
the nfy file (or windows event viewer if you are using windows) that
don't have lock escalations, and if you have it, avoid them using the
locklist and maxlocks parameters.

Best regards,
Ricardo Marcos Alonso

Reply With Quote
  #5  
Old   
Helmut Tessarek
 
Posts: n/a

Default Re: Performance of mass INSERTS into an MDC - 07-19-2011 , 11:39 PM



Quote:
DB21085I Instance "v95inst1" uses "64" bits and DB2 code release
"SQL09053"
with level identifier "06040107".
Informational tokens are "DB2 v9.5.0.3", "s081210", "U823474", and Fix
Pack
"3a".
Product is installed at "/opt/IBM/db2/V9.5".
You really should try the latest fixpak. There were some perf related fixes
since FP3a for MDCs...

Although those fixes will only help, if you don't have any locking problems
(as Ricardo pointed out).

--
Helmut K. C. Tessarek
DB2 Performance and Development
IBM Toronto Lab

Reply With Quote
  #6  
Old   
stavrogin
 
Posts: n/a

Default Re: Performance of mass INSERTS into an MDC - 07-20-2011 , 05:04 PM



On Jul 19, 6:50*pm, rmarcosa <rmarc... (AT) gmail (DOT) com> wrote:
Quote:
Be careful for lock escalations. If users are doing long queries that
lock many rows, you can have lock escalations that lock entire table,
and the inserts will have to wait to the end of the query. If you have
locktimeout to -1, the lock never ends. Ensure in the db2diag.log and
the nfy file (or windows event viewer if you are using windows) that
don't have lock escalations, and if you have it, avoid them using the
locklist and maxlocks parameters.

Best regards,
Ricardo Marcos Alonso
I monitored this event via db2top. This problem was not locking
related.

Reply With Quote
  #7  
Old   
stavrogin
 
Posts: n/a

Default Re: Performance of mass INSERTS into an MDC - 07-20-2011 , 05:05 PM



On Jul 20, 12:39*am, Helmut Tessarek <tessa... (AT) evermeet (DOT) cx> wrote:
Quote:
DB21085I *Instance "v95inst1" uses "64" bits and DB2 code release
"SQL09053"
with level identifier "06040107".
Informational tokens are "DB2 v9.5.0.3", "s081210", "U823474", and Fix
Pack
"3a".
Product is installed at "/opt/IBM/db2/V9.5".

You really should try the latest fixpak. There were some perf related fixes
since FP3a for MDCs...

Although those fixes will only help, if you don't have any locking problems
(as Ricardo pointed out).

--
Helmut K. C. Tessarek
DB2 Performance and Development
IBM Toronto Lab
I'll check out the MDC fixes. Thanks

Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.