 | |
Execution plan ?
microsoft.public.sqlserver.setup microsoft.public.sqlserver.setup
 |
| | Thread Tools | Display Modes |
#1
 
| |||
| |||
|
Execution plan ? -
07-16-2003
, 07:58 AM
#2
| |||
| |||
|
|
English version Facts: For about 3 months now, when testing, we observed, periodically (3 occurences every 2 weeks), a performance problem with a Store Proc that make an UPDATE of a table in our application. This table has a very high volume of updates. Usually, the execution of this Store Proc take less than a milli- second. When in trouble, the delay is between 200 and 1200 milli-seconds, depending on the volume of transactions. When it happen, it will last all day. The structure of table is as follow: Table1 ---> relation 1 to N toward Table2 - relation 1 to N toward Table3 The problem occur while updating Table3 The Store Proc has 2 SQL instruction. It first do a SELECT that find the ID of Table1 from the ID of Table3 Table3 has a TRIGGER on a simple UPDATE The slowing down seems to occur in the SELECT that look for the ID of Table1 We also observed that a bad EXECUTION PLAN in the cache seems to worsen the problem Every day, the DBCC command is executed to recalcultate the statistics, every day a new EXECTION PLAN is generated. If we empty the cache and force the generation of a new EXECUTION PLAN, the problem is solved. Question: Has a similar problem been reported and what are the factors that can make an EXECTION PLAN become so inefficient. |
#3
| |||
| |||
|
|
-----Original Message----- My guess would be that your running into what Bart describes below or your just getting a value the first time through (after the stats are recreated) that results in a poor plan for all the other values. The reason for the performance difference stems from a feature called "parameter sniffing". Consider a stored proc defined as follows: CREATE PROC proc1 @p1 int AS SELECT * FROM table1 WHERE c1 = @p1 GO Keep in mind that the server has to compile a complete execution plan for the proc before the proc begins to execute. In 6.5, at compile time SQL didn't know what the value of @p1 was, so it had to make a lot of guesses when compiling a plan. Suppose all of the actual parameter values for "@p1 int" that a user ever passed into this stored proc were unique integers that were greater than 0, but suppose 40% of the [c1] values in [table1] were, in fact, 0. SQL would use the average density of the column to estimate the number of rows that this predicate would return; this would be an overestimate, and SQL would might choose a table scan over an index seek based on the rowcount estimates. A table scan would be the best plan if the parameter value was 0, but unfortunately it happens that users will never or rarely pass @p1=0, so performance of the stored proc for more typical parameters suffers. In SQL 7.0 or 2000, suppose you executed this proc for the first time (when the sp plan is not in cache) with the command "EXEC proc1 @p1 = 10". Parameter sniffing allows SQL to insert the known value of parameter @p1 into the query at compile time before a plan for the query is generated. Because SQL knows that the value of @p1 is not 0, it can compile a plan that is tailored to the class of parameters that is actually passed into the proc, so for example it might select an index seek instead of a table scan based on the smaller estimated rowcount -- this is a good thing if most of the time 0 is not the value passed as @p1. Generally speaking, this feature allows more efficient stored proc execution plans, but a key requirement for everything to work as expected is that the parameter values used for compilation be "typical". In your case, the problem is that you have default NULL values for your parameters ("@Today DATETIME = NULL, ...") that are not typical because the parameter values are changed inside the stored proc before they are used -- as a result NULL will never actually be used to search the column. If the first execution of this stored proc doesn't pass in an explicit value for the @Today parameter, SQL believes that its value will be NULL. When SQL compiles the plan for this sp it substitutes NULL for each occurrence of @Today that is embedded within a query. Unfortunately, after execution begins the first thing the stored proc does is change @Today to a non-NULL value if it is found to be NULL, but unfortunately SQL doesn't know about this at compile time. Because NULL is a very atypical parameter value, the plan that SQL generates may not be a good one for the new value of the parameter that is assigned at execution time. So, the bottom line is that if you assign defaults to your sp parameters and later use those same parameters in a query, the defaults should be "typical" because they will be used during plan generation. If you must use defaults and business logic dictates that they be atypical (as may be the case here if app modifications are not an option), there are two possible solutions if you determine that the substitution of atypical parameter values is causing bad plans: 1. "Disable" parameter sniffing by using local DECLARE'd variables that you SET equal to the parameters inside the stored proc, and use the local variables instead of the offending parameters in the queries. This is the solution that you found yourself. SQL can't use parameter sniffing in this case so it must make some guesses, but in this case the guess based on average column density is better than the plan based on a specific but "wrong" parameter value (NULL). 2. Nest the affected queries somehow so that they run within a different context that will require a distinct execution plan. There are several possibilities here. for example: a. Put the affected queries in a different "child" stored proc. If you execute that stored proc within this one *after* the parameter @Today has been changed to its final value, parameter sniffing will suddenly become your friend because the value SQL uses to compile the queries inside the child stored proc is the actual value that will be used in the query. b. Use sp_executesql to execute the affected queries. The plan won't be generated until the sp_executesql stmt actually runs, which is of course after the parameter values have been changed. c. Use dynamic SQL ("EXEC (@sql)") to execute the affected queries. An equivalent approach would be to put the query in a child stored proc just like 2.a, but execute it within the parent proc with EXEC WITH RECOMPILE. Option #1 seems to have worked well for you in this case, although sometimes one of the options in #2 is a preferable choice. Here are some guidelines, although when you're dealing with something as complicated as the query optimizer experimentation is often the best approach <g>: - If you have only one "class" (defined as values that have similar density in the table) of actual parameter value that is used within a query (even if there are other classes of data in the base table that are never or rarely searched on), 2.a. or 2.b is probably the best option. This is because these options permit the actual parameter values to be used during compilation which should result in the most efficient query plan for that class of parameter. - If you have multiple "classes" of parameter value (for example, for the column being searched, half the table data is NULL, the other half are unique integers, and you may do searches on either class), 2.c can be effective. The downside is that a new plan for the query must be compiled on each execution, but the upside is that the plan will always be tailored to the parameter value being used for that particular execution. This is best when there is no single execution plan that provides acceptable execution time for all classes of parameters. HTH - Bart ------------ Bart Duncan Microsoft SQL Server Support Please reply to the newsgroup only - thanks. This posting is provided "AS IS" with no warranties, and confers no rights. -- Andrew J. Kelly SQL Server MVP "Patrick" <bingo (AT) microsoft (DOT) com> wrote in message news:0aae01c34b99$f8f82f80$a401280a (AT) phx (DOT) gbl... English version Facts: For about 3 months now, when testing, we observed, periodically (3 occurences every 2 weeks), a performance problem with a Store Proc that make an UPDATE of a table in our application. This table has a very high volume of updates. Usually, the execution of this Store Proc take less than a milli- second. When in trouble, the delay is between 200 and 1200 milli-seconds, depending on the volume of transactions. When it happen, it will last all day. The structure of table is as follow: Table1 --- relation 1 to N toward Table2 - relation 1 to N toward Table3 The problem occur while updating Table3 The Store Proc has 2 SQL instruction. It first do a SELECT that find the ID of Table1 from the ID of Table3 Table3 has a TRIGGER on a simple UPDATE The slowing down seems to occur in the SELECT that look for the ID of Table1 We also observed that a bad EXECUTION PLAN in the cache seems to worsen the problem Every day, the DBCC command is executed to recalcultate the statistics, every day a new EXECTION PLAN is generated. If we empty the cache and force the generation of a new EXECUTION PLAN, the problem is solved. Question: Has a similar problem been reported and what are the factors that can make an EXECTION PLAN become so inefficient. . |
#4
| |||
| |||
|
|
-----Original Message----- My guess would be that your running into what Bart describes below or your just getting a value the first time through (after the stats are recreated) that results in a poor plan for all the other values. The reason for the performance difference stems from a feature called "parameter sniffing". Consider a stored proc defined as follows: CREATE PROC proc1 @p1 int AS SELECT * FROM table1 WHERE c1 = @p1 GO Keep in mind that the server has to compile a complete execution plan for the proc before the proc begins to execute. In 6.5, at compile time SQL didn't know what the value of @p1 was, so it had to make a lot of guesses when compiling a plan. Suppose all of the actual parameter values for "@p1 int" that a user ever passed into this stored proc were unique integers that were greater than 0, but suppose 40% of the [c1] values in [table1] were, in fact, 0. SQL would use the average density of the column to estimate the number of rows that this predicate would return; this would be an overestimate, and SQL would might choose a table scan over an index seek based on the rowcount estimates. A table scan would be the best plan if the parameter value was 0, but unfortunately it happens that users will never or rarely pass @p1=0, so performance of the stored proc for more typical parameters suffers. In SQL 7.0 or 2000, suppose you executed this proc for the first time (when the sp plan is not in cache) with the command "EXEC proc1 @p1 = 10". Parameter sniffing allows SQL to insert the known value of parameter @p1 into the query at compile time before a plan for the query is generated. Because SQL knows that the value of @p1 is not 0, it can compile a plan that is tailored to the class of parameters that is actually passed into the proc, so for example it might select an index seek instead of a table scan based on the smaller estimated rowcount -- this is a good thing if most of the time 0 is not the value passed as @p1. Generally speaking, this feature allows more efficient stored proc execution plans, but a key requirement for everything to work as expected is that the parameter values used for compilation be "typical". In your case, the problem is that you have default NULL values for your parameters ("@Today DATETIME = NULL, ...") that are not typical because the parameter values are changed inside the stored proc before they are used -- as a result NULL will never actually be used to search the column. If the first execution of this stored proc doesn't pass in an explicit value for the @Today parameter, SQL believes that its value will be NULL. When SQL compiles the plan for this sp it substitutes NULL for each occurrence of @Today that is embedded within a query. Unfortunately, after execution begins the first thing the stored proc does is change @Today to a non-NULL value if it is found to be NULL, but unfortunately SQL doesn't know about this at compile time. Because NULL is a very atypical parameter value, the plan that SQL generates may not be a good one for the new value of the parameter that is assigned at execution time. So, the bottom line is that if you assign defaults to your sp parameters and later use those same parameters in a query, the defaults should be "typical" because they will be used during plan generation. If you must use defaults and business logic dictates that they be atypical (as may be the case here if app modifications are not an option), there are two possible solutions if you determine that the substitution of atypical parameter values is causing bad plans: 1. "Disable" parameter sniffing by using local DECLARE'd variables that you SET equal to the parameters inside the stored proc, and use the local variables instead of the offending parameters in the queries. This is the solution that you found yourself. SQL can't use parameter sniffing in this case so it must make some guesses, but in this case the guess based on average column density is better than the plan based on a specific but "wrong" parameter value (NULL). 2. Nest the affected queries somehow so that they run within a different context that will require a distinct execution plan. There are several possibilities here. for example: a. Put the affected queries in a different "child" stored proc. If you execute that stored proc within this one *after* the parameter @Today has been changed to its final value, parameter sniffing will suddenly become your friend because the value SQL uses to compile the queries inside the child stored proc is the actual value that will be used in the query. b. Use sp_executesql to execute the affected queries. The plan won't be generated until the sp_executesql stmt actually runs, which is of course after the parameter values have been changed. c. Use dynamic SQL ("EXEC (@sql)") to execute the affected queries. An equivalent approach would be to put the query in a child stored proc just like 2.a, but execute it within the parent proc with EXEC WITH RECOMPILE. Option #1 seems to have worked well for you in this case, although sometimes one of the options in #2 is a preferable choice. Here are some guidelines, although when you're dealing with something as complicated as the query optimizer experimentation is often the best approach <g>: - If you have only one "class" (defined as values that have similar density in the table) of actual parameter value that is used within a query (even if there are other classes of data in the base table that are never or rarely searched on), 2.a. or 2.b is probably the best option. This is because these options permit the actual parameter values to be used during compilation which should result in the most efficient query plan for that class of parameter. - If you have multiple "classes" of parameter value (for example, for the column being searched, half the table data is NULL, the other half are unique integers, and you may do searches on either class), 2.c can be effective. The downside is that a new plan for the query must be compiled on each execution, but the upside is that the plan will always be tailored to the parameter value being used for that particular execution. This is best when there is no single execution plan that provides acceptable execution time for all classes of parameters. HTH - Bart ------------ Bart Duncan Microsoft SQL Server Support Please reply to the newsgroup only - thanks. This posting is provided "AS IS" with no warranties, and confers no rights. -- Andrew J. Kelly SQL Server MVP "Patrick" <bingo (AT) microsoft (DOT) com> wrote in message news:0aae01c34b99$f8f82f80$a401280a (AT) phx (DOT) gbl... English version Facts: For about 3 months now, when testing, we observed, periodically (3 occurences every 2 weeks), a performance problem with a Store Proc that make an UPDATE of a table in our application. This table has a very high volume of updates. Usually, the execution of this Store Proc take less than a milli- second. When in trouble, the delay is between 200 and 1200 milli-seconds, depending on the volume of transactions. When it happen, it will last all day. The structure of table is as follow: Table1 --- relation 1 to N toward Table2 - relation 1 to N toward Table3 The problem occur while updating Table3 The Store Proc has 2 SQL instruction. It first do a SELECT that find the ID of Table1 from the ID of Table3 Table3 has a TRIGGER on a simple UPDATE The slowing down seems to occur in the SELECT that look for the ID of Table1 We also observed that a bad EXECUTION PLAN in the cache seems to worsen the problem Every day, the DBCC command is executed to recalcultate the statistics, every day a new EXECTION PLAN is generated. If we empty the cache and force the generation of a new EXECUTION PLAN, the problem is solved. Question: Has a similar problem been reported and what are the factors that can make an EXECTION PLAN become so inefficient. . |
#5
| |||
| |||
|
|
Good news and more questions ? The good news is that with your lead, we reproduced the problem. The first value of the day, for one of the parameter, cause the bad execution plan. - This value is associated to a column on witch there is a index. - The domain of value is 1 to 1019. But we only have 49 differents values. - This value is the only one of the entire domain to cause the problem. - This value is the maximum value of the domain.(1019) *** To correct the SP, we add some hints to the query to force the order of execution. It's working well ! *** Trying to understand why this single value cause the optimizer to choose a different plan, we taked a look at the statistics generated for this particular index and here what we have discovered:: Like a said, everyday we use the command "UPDATE STATISTICS ourTable WITH FULLSCAN, ALL". If we look closely to the stats resulting of this command, we notice that the value causing the problem is NOT in the stats... dbcc SHOW_STATISTICS (detsommairevente,IX_ProduitXID) :: 1 0.0 1431.0 0 0.0 2 0.0 357.0 0 0.0 3 0.0 1553.0 0 0.0 4 0.0 344.0 0 0.0 5 0.0 2152.0 0 0.0 6 0.0 482.0 0 0.0 7 0.0 2620.0 0 0.0 8 0.0 585.0 0 0.0 9 0.0 2400.0 0 0.0 10 0.0 571.0 0 0.0 11 0.0 4057.0 0 0.0 12 0.0 920.0 0 0.0 13 0.0 1567.0 0 0.0 14 0.0 349.0 0 0.0 15 0.0 28473.0 0 0.0 16 0.0 6387.0 0 0.0 17 0.0 28473.0 0 0.0 18 0.0 6387.0 0 0.0 19 0.0 28472.0 0 0.0 20 0.0 6385.0 0 0.0 21 0.0 2455.0 0 0.0 22 0.0 592.0 0 0.0 23 0.0 1424.0 0 0.0 24 0.0 350.0 0 0.0 25 0.0 2603.0 0 0.0 26 0.0 578.0 0 0.0 27 0.0 2097.0 0 0.0 28 0.0 469.0 0 0.0 29 0.0 1047.0 0 0.0 30 0.0 229.0 0 0.0 31 0.0 1571.0 0 0.0 32 0.0 343.0 0 0.0 1001 0.0 145.0 0 0.0 1002 0.0 141.0 0 0.0 1003 0.0 165.0 0 0.0 1004 0.0 78.0 0 0.0 1005 0.0 350.0 0 0.0 1006 0.0 1587.0 0 0.0 1008 0.0 453.0 0 0.0 1009 0.0 103.0 0 0.0 1010 0.0 115.0 0 0.0 1012 514.0 516.0 1 514.0 1013 0.0 114.0 0 0.0 1014 0.0 516.0 0 0.0 1015 0.0 121.0 0 0.0 1016 0.0 581.0 0 0.0 1017 0.0 136.0 0 0.0 ????????****** So we try something...we run the command "UPDATE STATISTICS ourTable WITH FULLSCAN, INDEX". This command results in a brand new set of stats...INCLUDING our famous values(1019) ??? And magically the first query with this new set of stats dont' reproduce the problem !!! dbcc SHOW_STATISTICS (detsommairevente,IX_ProduitXID) :: 1 0.0 1431.0 0 0.0 2 0.0 357.0 0 0.0 3 0.0 1553.0 0 0.0 4 0.0 344.0 0 0.0 5 0.0 2152.0 0 0.0 6 0.0 482.0 0 0.0 7 0.0 2620.0 0 0.0 8 0.0 585.0 0 0.0 9 0.0 2400.0 0 0.0 10 0.0 571.0 0 0.0 11 0.0 4057.0 0 0.0 12 0.0 920.0 0 0.0 13 0.0 1567.0 0 0.0 14 0.0 349.0 0 0.0 15 0.0 28473.0 0 0.0 16 0.0 6402.0 0 0.0 17 0.0 28473.0 0 0.0 18 0.0 6402.0 0 0.0 19 0.0 28472.0 0 0.0 20 0.0 6401.0 0 0.0 21 0.0 2455.0 0 0.0 22 0.0 592.0 0 0.0 23 0.0 1424.0 0 0.0 24 0.0 350.0 0 0.0 25 0.0 2603.0 0 0.0 26 0.0 578.0 0 0.0 27 0.0 2097.0 0 0.0 28 0.0 469.0 0 0.0 29 0.0 1047.0 0 0.0 30 0.0 229.0 0 0.0 31 0.0 1571.0 0 0.0 32 0.0 343.0 0 0.0 1001 0.0 145.0 0 0.0 1002 0.0 141.0 0 0.0 1003 0.0 165.0 0 0.0 1004 0.0 78.0 0 0.0 1005 0.0 350.0 0 0.0 1006 0.0 1587.0 0 0.0 1008 0.0 453.0 0 0.0 1009 0.0 103.0 0 0.0 1010 0.0 115.0 0 0.0 1011 0.0 514.0 0 0.0 1012 0.0 516.0 0 0.0 1013 0.0 114.0 0 0.0 1014 0.0 516.0 0 0.0 1015 0.0 121.0 0 0.0 1016 0.0 581.0 0 0.0 1017 0.0 136.0 0 0.0 1019 0.0 15.0 0 0.0 So ! My question is :: What's the difference between the ALL and INDEX options on UPDATE STATISTICS and WHY does the highest value of the index is not part of the stats when the ALL option is used ? Thank's PATRICK -----Original Message----- My guess would be that your running into what Bart describes below or your just getting a value the first time through (after the stats are recreated) that results in a poor plan for all the other values. The reason for the performance difference stems from a feature called "parameter sniffing". Consider a stored proc defined as follows: CREATE PROC proc1 @p1 int AS SELECT * FROM table1 WHERE c1 = @p1 GO Keep in mind that the server has to compile a complete execution plan for the proc before the proc begins to execute. In 6.5, at compile time SQL didn't know what the value of @p1 was, so it had to make a lot of guesses when compiling a plan. Suppose all of the actual parameter values for "@p1 int" that a user ever passed into this stored proc were unique integers that were greater than 0, but suppose 40% of the [c1] values in [table1] were, in fact, 0. SQL would use the average density of the column to estimate the number of rows that this predicate would return; this would be an overestimate, and SQL would might choose a table scan over an index seek based on the rowcount estimates. A table scan would be the best plan if the parameter value was 0, but unfortunately it happens that users will never or rarely pass @p1=0, so performance of the stored proc for more typical parameters suffers. In SQL 7.0 or 2000, suppose you executed this proc for the first time (when the sp plan is not in cache) with the command "EXEC proc1 @p1 = 10". Parameter sniffing allows SQL to insert the known value of parameter @p1 into the query at compile time before a plan for the query is generated. Because SQL knows that the value of @p1 is not 0, it can compile a plan that is tailored to the class of parameters that is actually passed into the proc, so for example it might select an index seek instead of a table scan based on the smaller estimated rowcount -- this is a good thing if most of the time 0 is not the value passed as @p1. Generally speaking, this feature allows more efficient stored proc execution plans, but a key requirement for everything to work as expected is that the parameter values used for compilation be "typical". In your case, the problem is that you have default NULL values for your parameters ("@Today DATETIME = NULL, ...") that are not typical because the parameter values are changed inside the stored proc before they are used -- as a result NULL will never actually be used to search the column. If the first execution of this stored proc doesn't pass in an explicit value for the @Today parameter, SQL believes that its value will be NULL. When SQL compiles the plan for this sp it substitutes NULL for each occurrence of @Today that is embedded within a query. Unfortunately, after execution begins the first thing the stored proc does is change @Today to a non-NULL value if it is found to be NULL, but unfortunately SQL doesn't know about this at compile time. Because NULL is a very atypical parameter value, the plan that SQL generates may not be a good one for the new value of the parameter that is assigned at execution time. So, the bottom line is that if you assign defaults to your sp parameters and later use those same parameters in a query, the defaults should be "typical" because they will be used during plan generation. If you must use defaults and business logic dictates that they be atypical (as may be the case here if app modifications are not an option), there are two possible solutions if you determine that the substitution of atypical parameter values is causing bad plans: 1. "Disable" parameter sniffing by using local DECLARE'd variables that you SET equal to the parameters inside the stored proc, and use the local variables instead of the offending parameters in the queries. This is the solution that you found yourself. SQL can't use parameter sniffing in this case so it must make some guesses, but in this case the guess based on average column density is better than the plan based on a specific but "wrong" parameter value (NULL). 2. Nest the affected queries somehow so that they run within a different context that will require a distinct execution plan. There are several possibilities here. for example: a. Put the affected queries in a different "child" stored proc. If you execute that stored proc within this one *after* the parameter @Today has been changed to its final value, parameter sniffing will suddenly become your friend because the value SQL uses to compile the queries inside the child stored proc is the actual value that will be used in the query. b. Use sp_executesql to execute the affected queries. The plan won't be generated until the sp_executesql stmt actually runs, which is of course after the parameter values have been changed. c. Use dynamic SQL ("EXEC (@sql)") to execute the affected queries. An equivalent approach would be to put the query in a child stored proc just like 2.a, but execute it within the parent proc with EXEC WITH RECOMPILE. Option #1 seems to have worked well for you in this case, although sometimes one of the options in #2 is a preferable choice. Here are some guidelines, although when you're dealing with something as complicated as the query optimizer experimentation is often the best approach <g>: - If you have only one "class" (defined as values that have similar density in the table) of actual parameter value that is used within a query (even if there are other classes of data in the base table that are never or rarely searched on), 2.a. or 2.b is probably the best option. This is because these options permit the actual parameter values to be used during compilation which should result in the most efficient query plan for that class of parameter. - If you have multiple "classes" of parameter value (for example, for the column being searched, half the table data is NULL, the other half are unique integers, and you may do searches on either class), 2.c can be effective. The downside is that a new plan for the query must be compiled on each execution, but the upside is that the plan will always be tailored to the parameter value being used for that particular execution. This is best when there is no single execution plan that provides acceptable execution time for all classes of parameters. HTH - Bart ------------ Bart Duncan Microsoft SQL Server Support Please reply to the newsgroup only - thanks. This posting is provided "AS IS" with no warranties, and confers no rights. -- Andrew J. Kelly SQL Server MVP "Patrick" <bingo (AT) microsoft (DOT) com> wrote in message news:0aae01c34b99$f8f82f80$a401280a (AT) phx (DOT) gbl... English version Facts: For about 3 months now, when testing, we observed, periodically (3 occurences every 2 weeks), a performance problem with a Store Proc that make an UPDATE of a table in our application. This table has a very high volume of updates. Usually, the execution of this Store Proc take less than a milli- second. When in trouble, the delay is between 200 and 1200 milli-seconds, depending on the volume of transactions. When it happen, it will last all day. The structure of table is as follow: Table1 --- relation 1 to N toward Table2 - relation 1 to N toward Table3 The problem occur while updating Table3 The Store Proc has 2 SQL instruction. It first do a SELECT that find the ID of Table1 from the ID of Table3 Table3 has a TRIGGER on a simple UPDATE The slowing down seems to occur in the SELECT that look for the ID of Table1 We also observed that a bad EXECUTION PLAN in the cache seems to worsen the problem Every day, the DBCC command is executed to recalcultate the statistics, every day a new EXECTION PLAN is generated. If we empty the cache and force the generation of a new EXECUTION PLAN, the problem is solved. Question: Has a similar problem been reported and what are the factors that can make an EXECTION PLAN become so inefficient. . |
|
« Previous Thread
|
Next Thread »
| Thread Tools | |
| Display Modes | |
Linear Mode
| |
Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.