I am working on a script to process a data feed and load it into a
MySQL DB. It mostly works, except that for almost 20,000 lines of
data, I get 25 lines of data skipped, and about 1,500 warnings. I
need to know first what those warnings are, and whether or not they
affected the data being loaded. Second, I need to know what lines
were skipped, and why.

If it matters, the table, and data file, has several dozen fields.

The nature of this data is sensitive and it is important that none of
the data is lost, or altered in a way that changes its meaning, during
this process.

So, how can I find out about all these warnings that came from a
single load statement, and how can I find out the details of what
lines were skipped?

Thanks

Ted

Ted wrote:
The answer to that depends on how you wrote your script

On May 8, 5:23*pm, "Paul Lautman" <paul.laut... (AT) btinternet (DOT) com> wrote:
Actually, for this, the results I saw were from a test of one line of
code submitted directly into the mysql client. Once it is correct,
then it will be added to a SQL script (which will ultimately have only
two statements, each of which loads data from one of two CSV files.

The SQL statements are specifically routine LOAD DATA LOCAL INFILE
statements, with the usual fields terminated by, lines terminated by
clauses, each referring to specific columns in a specific table.

Maybe I have been lucky, but in all the time I've done bulk data
loading, this is the first time I received a result indicating some
records were skipped. I found, hours after I posted, that SHOW
WARNINGS; will give me the warnings, but that told me nothing about
the records that were skipped.

The master script, that will ultimately invoke the short SQL script is
written in Perl, but I don't want to do much with the DB directly from
Perl since the perl interface to MySQL seems unreliable. For example,
while I can make prepared calls to trivial select statements, I have
yet to get a Perl prepared statement (using DBI) to successfully
execute a stored procedure. I am so frustrated with that interface,
I'd rather just have perl spawn a child process, passing my SQL script
to the mysql client.

Thanks for your time.

Ted