dbTalk Databases Forums  

Data Corruption

comp.databases.mysql comp.databases.mysql


Discuss Data Corruption in the comp.databases.mysql forum.



Reply
 
Thread Tools Display Modes
  #11  
Old   
The Natural Philosopher
 
Posts: n/a

Default Re: Data Corruption - 07-02-2010 , 03:53 AM






Tyrone Slothrop wrote:
Quote:
On Fri, 02 Jul 2010 01:26:20 +0100, The Natural Philosopher
tnp (AT) invalid (DOT) invalid> wrote:

Tyrone Slothrop wrote:
<cut to the chase>

Quote:
I am logging to a text file from the PHP script that is called from
the AJAX script. Once the SQL statement is generated and before it is
executed, the query appends to a log file (which is archived daily in
case I need to refer to it in a fit of madness).

Right. Can you CONCLUSIVELY say that the corruption you get does NOT
show up as a malformed SQL query logged by this script?


Quote:
The thing that makes me crazy is that it is one in a million query
triggers this thing. It has occurred 5 times in the last year.

STOP! you don't know the bad query is coming from this area at all.

How many other process communicate with mysqld? how many COULD
communicate with it, in theory?

I mean is there an old cron script or a test script somewhere still
running..

Years ago a story was going round a big software house in Johannesburg.
About te code that always crashed about every year, requiring its prime
author to make a trip to Durban to fix it.

Several year went by before it was noted that the author, a keen racing
fan, always seemed to be 'away in Durban' fixing this code during the
week the Durban Races were on...



Quote:
What happens is that not all fields are overwritten, but all 60,000+
records are.
That is almost definitely an update table ...set fields with no where..



It is always a certain set of fields which are
Quote:
overwritten and not always the same set.
Next question: is the field set that is altered, *every* altered set
that is, possible (and consistent) with the ajax partial update program?

If not something else is.maybe doing it.


You should - and its a vile thought - turn mysqld logging on.

Generally this is helpful when you can reproduce the fault easily. Its
bad news to leave running for a year to trap the odd 'neutrino' SQL
statement ;-)



Since the PHP script called
Quote:
by AJAX does incremental saves as the form is filled out, this is the
script that is suspect. Within the log I can track the queries of an
individual applicant as each query gets a bit longer as more fields of
the form are completed.

And, I take it, that this log is consistent and correct in every way?

which means that either you've missed some code that does a rogue
update, or some other process is doing it..

Reply With Quote
  #12  
Old   
The Natural Philosopher
 
Posts: n/a

Default Re: Data Corruption - 07-02-2010 , 03:57 AM






Axel Schwenke wrote:
Quote:
Tyrone Slothrop <ts (AT) paranoids (DOT) org> wrote:

How are you logging the queries? In terms of what is sent to the mysqld
or is the mysqld itself logging the queries it has recieved?
I am logging to a text file from the PHP script that is called from
the AJAX script.

This is not enough. You should let MySQL log the queries.

If you're fine with just the query, the binlog would be enough. It has
the advantage that it is rotated automatically and that it logs only
queries that modify data.

The general log provides more info, i.e. which MySQL account was used
and from which machine the login was done. It also records *all*
queries and thus grows rapidly. You have to rotate it manually.

Finally I have a bit weird idea: use mysql-proxy to filter your
queries. You just have to write few lines of Lua code to identify
the problem query (i.e. with a regex). Then proxy can log it and/or
suppress the execution of this query.

Can you shed more light on the mysql-proxy?

I had not heard of that..

It looks a most useful tool.


Quote:
XL

Reply With Quote
  #13  
Old   
Captain Paralytic
 
Posts: n/a

Default Re: Data Corruption - 07-02-2010 , 04:21 AM



On 2 July, 09:57, The Natural Philosopher <t... (AT) invalid (DOT) invalid>
wrote:
Quote:
Can you shed more light on the mysql-proxy?

I had not heard of that..
I'm sorry, the temptation is too great!

http://lmgtfy.com/?q=mysql-proxy

where you will find:
http://dev.mysql.com/tech-resources/...ngstarted.html

Reply With Quote
  #14  
Old   
Axel Schwenke
 
Posts: n/a

Default Re: Data Corruption - 07-02-2010 , 04:31 AM



The Natural Philosopher <tnp (AT) invalid (DOT) invalid> wrote:
Quote:
Axel Schwenke wrote:

Finally I have a bit weird idea: use mysql-proxy to filter your
queries. You just have to write few lines of Lua code to identify
the problem query (i.e. with a regex). Then proxy can log it and/or
suppress the execution of this query.

Can you shed more light on the mysql-proxy?
I had not heard of that..
It looks a most useful tool.
http://forge.mysql.com/wiki/MySQL_Proxy

Written by a MySQL^WSun^WOracle employee. All the time critical part
is C but you can plug Lua functions at certain stages of request and
response handling.

Some use cases: load balancing, read-write splitting + load balancing
(to distribute connections on a replication cluster). Statistical
analysis. I.e. proxy is used to collect data for the Enterprise Monitor
http://www.mysql.com/products/enterprise/monitor.html


XL

Reply With Quote
  #15  
Old   
Doug Miller
 
Posts: n/a

Default Re: Data Corruption - 07-02-2010 , 06:04 AM



In article <i0k9de$54p$1 (AT) news (DOT) albasani.net>, The Natural Philosopher <tnp (AT) invalid (DOT) invalid> wrote:
Quote:
Tyrone Slothrop wrote:
On Fri, 02 Jul 2010 01:26:20 +0100, The Natural Philosopher
tnp (AT) invalid (DOT) invalid> wrote:

Tyrone Slothrop wrote:

cut to the chase

I am logging to a text file from the PHP script that is called from
the AJAX script. Once the SQL statement is generated and before it is
executed, the query appends to a log file (which is archived daily in
case I need to refer to it in a fit of madness).


Right. Can you CONCLUSIVELY say that the corruption you get does NOT
show up as a malformed SQL query logged by this script?
More to the point, can he conclusively state that all queries that update the
database are being logged by this script? Of course not. There is no assurance
that the faulty query has any connection at all with the script that's doing
the logging.
[...]
Quote:
What happens is that not all fields are overwritten, but all 60,000+
records are.

That is almost definitely an update table ...set fields with no where..
Or with a WHERE clause that matches all rows.
[...]
Quote:
You should - and its a vile thought - turn mysqld logging on.
That is the only way to be sure that every query gets logged. Right now, the
only queries being logged are those that the application has been told to log.
[...]
Quote:
either you've missed some code that does a rogue
update, or some other process is doing it..
The OP doesn't seem to have considered that possibility...

Reply With Quote
  #16  
Old   
The Natural Philosopher
 
Posts: n/a

Default Re: Data Corruption - 07-02-2010 , 06:14 AM



Captain Paralytic wrote:
Quote:
On 2 July, 09:57, The Natural Philosopher <t... (AT) invalid (DOT) invalid
wrote:
Can you shed more light on the mysql-proxy?

I had not heard of that..

I'm sorry, the temptation is too great!

http://lmgtfy.com/?q=mysql-proxy

where you will find:
http://dev.mysql.com/tech-resources/...ngstarted.html
Cap'n sir, I already did that as soon as I saw it.

It was more 'how in general did you find it? what problems did it solve
for YOU? etc. etc. info I was after..

Reply With Quote
  #17  
Old   
The Natural Philosopher
 
Posts: n/a

Default Re: Data Corruption - 07-02-2010 , 06:15 AM



Axel Schwenke wrote:
Quote:
The Natural Philosopher <tnp (AT) invalid (DOT) invalid> wrote:
Axel Schwenke wrote:
Finally I have a bit weird idea: use mysql-proxy to filter your
queries. You just have to write few lines of Lua code to identify
the problem query (i.e. with a regex). Then proxy can log it and/or
suppress the execution of this query.
Can you shed more light on the mysql-proxy?
I had not heard of that..
It looks a most useful tool.

http://forge.mysql.com/wiki/MySQL_Proxy

Written by a MySQL^WSun^WOracle employee. All the time critical part
is C but you can plug Lua functions at certain stages of request and
response handling.

Some use cases: load balancing, read-write splitting + load balancing
(to distribute connections on a replication cluster). Statistical
analysis. I.e. proxy is used to collect data for the Enterprise Monitor
http://www.mysql.com/products/enterprise/monitor.html


XL
The answer I wanted. Thanks.

Looks like a useful tool if I ever want it.

Reply With Quote
  #18  
Old   
Peter H. Coffin
 
Posts: n/a

Default Re: Data Corruption - 07-02-2010 , 07:55 AM



On Fri, 02 Jul 2010 00:43:23 -0500, Gordon Burditt wrote:
Quote:
What protection do you have against malicious or simply erroneous
queries to your AJAX script? Are there any old and buggy versions
of your AJAX script still on your server?

Since it happens every couple of months, have you considered that
it might be triggered by Google's (or some other search engine's)
web spider?

I once set up an administrative page (accessible from localhost
only) for maintaining a SPAM filter. It (stupidly) used a url like:
http://mysystem.example.org/spamfilt...e.php?id=63683 on a
"DELETE" button when I examined a particular record. Later I ran
a program to check for broken links on my site, doing a web-spider-walk
(Google does much the same thing. However, this one was run from
localhost, bypassing the security protection). Gee, where did all
the spam filter entries go? Don't use GET on URLs intended to make
changes in a database (add/delete records, change passwords, buy
stuff, send email, etc.), except incidentally for things like hit
counters. Use POST. And use SSL where appropriate.
This. And validate the input anyway. And check it again. If your delete
operation is supposed to delete one record, run a select on that
criteria first *every time*, spit back an error, log it, and *don't run
the query*, if the number of records is anything OTHER than exactly what
you're expecting: 1.

--
40. I will be neither chivalrous nor sporting. If I have an unstoppable
superweapon, I will use it as early and as often as possible instead
of keeping it in reserve.
--Peter Anspach's list of things to do as an Evil Overlord

Reply With Quote
  #19  
Old   
Tyrone Slothrop
 
Posts: n/a

Default Re: Data Corruption - 07-02-2010 , 09:46 AM



On Fri, 02 Jul 2010 12:55:03 GMT, "Peter H. Coffin"
<hellsop (AT) ninehells (DOT) com> wrote:

Quote:
On Fri, 02 Jul 2010 00:43:23 -0500, Gordon Burditt wrote:
What protection do you have against malicious or simply erroneous
queries to your AJAX script? Are there any old and buggy versions
of your AJAX script still on your server?

Since it happens every couple of months, have you considered that
it might be triggered by Google's (or some other search engine's)
web spider?

I once set up an administrative page (accessible from localhost
only) for maintaining a SPAM filter. It (stupidly) used a url like:
http://mysystem.example.org/spamfilt...e.php?id=63683 on a
"DELETE" button when I examined a particular record. Later I ran
a program to check for broken links on my site, doing a web-spider-walk
(Google does much the same thing. However, this one was run from
localhost, bypassing the security protection). Gee, where did all
the spam filter entries go? Don't use GET on URLs intended to make
changes in a database (add/delete records, change passwords, buy
stuff, send email, etc.), except incidentally for things like hit
counters. Use POST. And use SSL where appropriate.

This. And validate the input anyway. And check it again. If your delete
operation is supposed to delete one record, run a select on that
criteria first *every time*, spit back an error, log it, and *don't run
the query*, if the number of records is anything OTHER than exactly what
you're expecting: 1.
Thank you all for your opinions and suggestions.

I truly loathe to turn on MySQL logging since this is a production
server with sites which provide the income for my client (who is about
as anal-compulsive person as I have ever known). Unfortunately, I
cannot sandbox the application and place it under the same stress as
in the production environment. I am strongly encouraging the client
to lease a dedicated MySQL server. As load has been increasing, I
have moved the nameserver service from the server and in the process
of moving all mail services.

The AJAX code uses post and SSL, and passes raw data directly from the
form. The real work is done in PHP. I much prefer using 100%
server-side validation but the client had to have this feature and
though I fought against it, in the end, we all do what the client
wants since she/he is the one paying my bills.

Yes, it is very possible that an unescaped binary character is causing
the bad query -- at least that is my present theory. I am testing a
revised script which will be put in production in the next day or two.
The downside is that it will take months to know if the fix worked.

BTW, there is no direct access to the form by Google. The only access
to the form is via another form or by providing a unique query string.
If there is no match on the hash or the form has been completed, the
user is redirected to another page. The query string is used only to
identify the record and, once that record is called, is never used in
any way to provide the basis of an update query.

I will follow up on a couple suggestions in this thread.

Again, thank you all.

Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.