Hello,

We are running ASA 7.0.4.3498 & 3541 on Windows 2000 Server
SP4.
We've got 40 ASA databases servicing 2,500 users that have
just nose dived in performance to varying degrees!
The servers are all HP DL380 machines and apart from McAfee
AV 7.1 and Net Support 9 there is nothing else running on
the machine.

The servers have been running for 3+ years without a hitch
but since Wednesday there has been a massive increase in
disk write % in the Windows performance monitor and we don't
know why but it means the servers (and users) are
struggling.

The app that uses the system is written in Delphi 5 using
NativeDB components and developed in-house and has had no
changes made to it of any significance. The app runs on
several terminal servers that the users log onto over a WAN.

Over the Easter break we installed the latest critical
Windows security patches. We are rolling back the updates on
some of the servers tonight to see if it makes a difference
tomorrow morning.
Unfortunately the users have finished for the day and the
servers have all gone idle so I won't be able to tell until
the morning if any changes I make are worth it.

Has anybody else seen anything similar in the last few days?
Is there an easy way to target what the actual disk activity
is?

Here's hoping....

Thanks,

Adrian.

You might also check to see if there was a McAfee update at the time
your slowdown hit...


In article <444d1e10.7e23.1681692777 (AT) sybase (DOT) com>, Adrian Scanlon says...
.....

--
Remove the ns_ from if replying by e-mail (but keep posts in the
newsgroups if possible).

Hi,

It looks like the last McAfee update was on Friday after the
problem started.

One of our techie chaps has also run some of the ASA stats
in the performance monitor today and it's matching the
windows activity step for step so it looks like it is the
ASA that's generating the writes.

I'm also going to un-install the AV completely on one of the
servers to see if it has any effect tomorrow.

Please keep the ideas coming because our heads are getting
sore from banging them on the wall!

Thanks.

Check for disk fragmentation.

Another possibility is internal database fragmentation which is
possible even if the disk is perfectly defragmented. AFAIK there is no
way to determine if your V7 databases are fragmented, but if you do
any significant amount of inserting and/or deleting and these
databases haven't been dumped and reloaded in a long time, it is quite
likely.

Both kinds of fragmentation can result in heavy disk I/O and poor
performance... and the degradation can be sudden.

The solution to database fragmentation in V7 is to dbunload the
database, dbinit a new one, dbisql to reload the database, and then
defragment the drive to ensure the new .DB file is contiguous. The
builtin Windows defrag is not very good, especially not compared to
Diskeeper or other third-party defrag tools.

V7 is very old and (AFAIK) no longer actively supported with new fixes
etc. V9 is much better; you can measure fragmentation, and defragment
on the fly.

Breck



On 24 Apr 2006 11:50:56 -0700, Adrian Scanlon wrote:

--
Breck Carter [Team iAnywhere]
RisingRoad SQL Anywhere and MobiLink Professional Services
www.risingroad.com
The book: http://www.risingroad.com/SQL_Anywhe...ers_Guide.html
breck.carter (AT) risingroad (DOT) com

Thanks Breck, I'll unload and rebuild a couple of them
tonight but I'd be surprised (amazed!) if all 40 went on the
same day for this reason.
Not all the databases are the same age, some are 4 years old
and others are just over 12 months but they all perform the
same function.

However, I'm willing to try anything at this point!

I'll post back the results tomorrow when we've got the users
back on.


Adrian.

Have any changes occured to the application/queries sent to the server. Is
it possible that a recently introduced query is causing the server to
generate enormous amounts of data either as the end result or due to
intermediate computations?

-anil

<Adrian Scanlon> wrote

On 24 Apr 2006 14:31:47 -0700, Adrian Scanlon wrote:

I agree... it wasn't clear to me that the symptom was universal to all
engines. Anil's suggestion makes more sense.

Breck

--
Breck Carter [Team iAnywhere]
RisingRoad SQL Anywhere and MobiLink Professional Services
www.risingroad.com
The book: http://www.risingroad.com/SQL_Anywhe...ers_Guide.html
breck.carter (AT) risingroad (DOT) com

Hello everyone.

Fortunately (for the users) all the servers are running
perfectly again this morning.
Unfortunatley (for me) it means I don't have an answer as to
what was going wrong so it feels like a little gremlin is
still lurking in the shadows!

I rebuilt some databases, uninstalled the anti virus on
others, rolled back the windows updates on some and
defragged the drives on others. All different combinations
on different servers and even left some exactly as they were
with the same version of the application.

Got up bright and early this morning and watched them all
log in and use the system without a problem. Every user
happy.

I guess I'll just have to wait and see... just my luck it's
the company year end this weekend. Fingers crossed.

Thanks everybody.

Adrian.

To expand on Anil's suggestion: When the thrashing begins, check to
see if the temporary file is growing rapidly. This might be a symptom
of a rogue query such as a cartesian product with no limiting WHERE
clause.

The Help describes where the temporary file is located. For a 7.0.3
database on my laptop it goes where the TEMP environment variable
says; e.g.

C:\Documents and Settings\bcarter\Local Settings\Temp\asat0001.tmp

There probably is one temp file for each server.

Breck

On 24 Apr 2006 14:31:47 -0700, Adrian Scanlon wrote:

--
Breck Carter [Team iAnywhere]
RisingRoad SQL Anywhere and MobiLink Professional Services
www.risingroad.com
The book: http://www.risingroad.com/SQL_Anywhe...ers_Guide.html
breck.carter (AT) risingroad (DOT) com

Breck Carter [Team iAnywhere] wrote:

Filemon may yield interesting results, too.

Dmitri.