Hi,

Yesterday I was in the process of convincing one of my clients to use
MySQL than SQL Server. I had a long fight with the technical team
there and I was told to prove the performance of MySQL to them when
there are more than 50 users do some reporting action.


So I defined a sample database with two master tables and one
transaction table; and wrote a command line application (in C#) which
will do 'n' requests to the server at the sametime (using threading).
But when I did the test from the application, it failed!

I have to demo them a db that works with 5000000 rows in the txn table
with 50 users accessing it in a given point of time.


I tried to call the proc "fetch_all_failed_students_with_subjectname"
from my application from 20 threads. It failed! It failed even when I
called 5 threads!

Then I removed most of the rows and tried it with 500000. It worked
with 15 threads! but failed with 20 threads.

I still believe this will be some configuration change; I still hope
in MySQL. Please share your thoughts. I am don't care about losing
this client; but is very much care about my confidence in MySQL.

My demo system is 64 bit with 3GB RAM running in Windows Server 2008.

Here is the script for the sample DB
------------------------------------------------------------
delimiter $$

drop table if exists student_subject_mapping$$
drop table if exists student$$
drop table if exists subjects$$

drop procedure if exists fetch_all_failed_students_with_subjectname$$
drop procedure if exists insertSubjects$$
drop procedure if exists insertstudents$$

CREATE TABLE `student` (
`StudentId` int(11) NOT NULL AUTO_INCREMENT,
`FirstName` varchar(50) NOT NULL,
`LastName` varchar(50) DEFAULT NULL,
`RollNumber` int(11) DEFAULT NULL,
PRIMARY KEY (`StudentId`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1$$

CREATE TABLE `subjects` (
`SubjectId` int(11) NOT NULL AUTO_INCREMENT,
`Name` varchar(50) NOT NULL,
PRIMARY KEY (`SubjectId`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1$$


CREATE TABLE `student_subject_mapping` (
`mappingid` int(11) NOT NULL AUTO_INCREMENT,
`studentid` int(11) NOT NULL,
`subjectid` int(11) NOT NULL,
`status` int(11) DEFAULT NULL,
`remarks` varchar(1000) DEFAULT NULL,
`mark` int(11) DEFAULT NULL,
PRIMARY KEY (`mappingid`),
KEY `fk_student_student_sub_mapping` (`studentid`),
KEY `fk_subject_student_sub_mapping` (`subjectid`),
CONSTRAINT `fk_student_student_sub_mapping` FOREIGN KEY (`studentid`)
REFERENCES `student` (`StudentId`) ON DELETE NO ACTION ON UPDATE NO
ACTION,
CONSTRAINT `fk_subject_student_sub_mapping` FOREIGN KEY (`subjectid`)
REFERENCES `subjects` (`SubjectId`) ON DELETE NO ACTION ON UPDATE NO
ACTION
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1$$


CREATE DEFINER=`root`@`%` PROCEDURE `insertstudents`()
BEGIN
declare vi int;
set vi = 1;

while (vi < 20000) do

INSERT INTO `student`
(`FirstName`,
`LastName`,
`RollNumber`)
VALUES
(
concat('first name ', cast(vi as char) ),
concat('last name ', cast(vi as char) ),
vi
);

set vi = vi + 1;

end while;

END$$

CREATE DEFINER=`root`@`%` PROCEDURE `insertSubjects`()
BEGIN
declare vi int;
set vi = 1;

while (vi < 200) do

INSERT INTO subjects
(`Name`)
VALUES
(
concat('subject ', cast(vi as char) )
);

set vi = vi + 1;


end while;

END$$

CREATE DEFINER=`root`@`%` PROCEDURE
`fetch_all_failed_students_with_subjectname`()
BEGIN

select st.*, sb.name `subject name`, ssm.mark, ssm.remarks from
student st
inner join student_subject_mapping ssm
on st.studentid = ssm.studentid and ssm.status=2
inner join subjects sb on ssm.subjectid = sb.subjectid
;
END$$

call insertstudents()$$

call insertSubjects()$$

INSERT INTO `student_subject_mapping`(`studentid`,`subjectid`) select
studentid, subjectid from student join subjects $$

update student_subject_mapping set status = 2 , remarks = 'Fail', mark
= 30 - FLOOR(1 + RAND() * 10) where studentid % subjectid = 0$$

update student_subject_mapping set status = 1 , remarks = 'Success',
mark = 30 + FLOOR(10 + RAND() * 20) where studentid % subjectid <> 0 $
$


Here is my C# code
-----------------------------------------

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using MySql.Data.MySqlClient;
using System.Configuration;
using System.Threading;

namespace MySQLDBLoadTestApp
{
class Program
{
static void Main(string[] args)
{

Console.WriteLine("Enter # of threads");
String noThread = Console.ReadLine();

if(String.IsNullOrEmpty(noThread))
noThread = "1";

int threadCnt = Int32.Parse(noThread);

for(int i=0 ; i< threadCnt ; i++)
{
Thread t1 = new Thread(loadTester);
t1.Start();
}


// loadTester();

Console.Read();
}

private static void loadTester()
{
MySqlConnection connection = new
MySqlConnection(ConfigurationManager.AppSettings["connectionString"]);

connection.Open();

MySqlTransaction txn = connection.BeginTransaction();

try
{

int count =
Int32.Parse(ConfigurationManager.AppSettings["commandCount"]);

for (int i = 1; i <= count; i++)
{
MySqlCommand command = connection.CreateCommand();
String commandText = ConfigurationManager.AppSettings[("command" +
i)];

command.CommandText = commandText;

DateTime start = DateTime.Now;
Console.WriteLine("Start : " + commandText + " @ " + start);
command.ExecuteNonQuery();
DateTime end = DateTime.Now;
Console.WriteLine("End : " + commandText + " @ " + end + " Took
(sec):" + end.Subtract(start).Seconds);
}

txn.Commit();

}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
txn.Rollback();
}
finally
{
connection.Close();
}
}
}
}

Thanks

On Mon, 16 Jan 2012 04:40:05 -0800 (PST), mahesh wrote:
Failed how? Wrong results? No results? Connection dropped? Error
message?

Yes, it's probably a configuration problem. So what IS your configuration?

That'll only matter once we know what's wrong.

--
57. Before employing any captured artifacts or machinery, I will
carefully read the owner's manual.
--Peter Anspach's list of things to do as an Evil Overlord

On Jan 16, 10:14*am, "Peter H. Coffin" <hell... (AT) ninehells (DOT) com> wrote:

which .ini file did you use: my-medium.ini? my-large.ini? are you
using named-pipes?
What is the max-connections setting?

The error I am getting is "Timeout Error in IO operation"

I have made some changes in values while trying to resolve the issue.
But no changes resolved my problem.

Here is my.ini
---------------------------------------

# MySQL Server Instance Configuration File
#
----------------------------------------------------------------------
# Generated by the MySQL Server Instance Configuration Wizard
#
#
# Installation Instructions
#
----------------------------------------------------------------------
#
# On Linux you can copy this file to /etc/my.cnf to set global
options,
# mysql-data-dir/my.cnf to set server-specific options
# (@localstatedir@ for this installation) or to
# ~/.my.cnf to set user-specific options.
#
# On Windows you should keep this file in the installation directory
# of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
# make sure the server reads the config file use the startup option
# "--defaults-file".
#
# To run run the server from the command line, execute this in a
# command line shell, e.g.
# mysqld --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y
\my.ini"
#
# To install the server as a Windows service manually, execute this in
a
# command line shell, e.g.
# mysqld --install MySQLXY --defaults-file="C:\Program Files\MySQL
\MySQL Server X.Y\my.ini"
#
# And then execute this in a command line shell to start the server,
e.g.
# net start MySQLXY
#
#
# Guildlines for editing this file
#
----------------------------------------------------------------------
#
# In this file, you can use all long options that the program
supports.
# If you want to know the options a program supports, start the
program
# with the "--help" option.
#
# More detailed information about the individual options can also be
# found in the manual.
#
#
# CLIENT SECTION
#
----------------------------------------------------------------------
#
# The following options will be read by MySQL client applications.
# Note that only client applications shipped by MySQL are guaranteed
# to read this section. If you want your own MySQL client program to
# honor these values, you need to specify it as an option during the
# MySQL client library initialization.
#
[client]

port=3306

[mysql]

default-character-set=latin1


# SERVER SECTION
#
----------------------------------------------------------------------
#
# The following options will be read by the MySQL Server. Make sure
that
# you have installed the server correctly (see above) so it reads
this
# file.
#
[mysqld]

# The TCP/IP Port the MySQL Server will listen on
port=3306

sql_mode=STRICT_ALL_TABLES,NO_ZERO_DATE,NO_ZERO_IN _DATE


#Path to installation directory. All paths are usually resolved
relative to this.
basedir="C:/Program Files/MySQL/MySQL Server 5.5/"

#Path to the database root
datadir="C:/ProgramData/MySQL/MySQL Server 5.5/Data/"

# The default character set that will be used when a new schema or
table is
# created and no character set is defined
character-set-server=latin1

# The default storage engine that will be used when create new tables
when
default-storage-engine=INNODB

# Set the SQL mode to strict
sql-
mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_E NGINE_SUBSTITUTION"

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=100

# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
#query_cache_size=0 //CHANGED BY CKR
query_cache_size=0
query_cache_type=0

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_cache=256

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=369M


# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there
aren't
# more than thread_cache_size threads from before. This greatly
reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=8

#*** MyISAM Specific options

# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA
INFILE.
# If the file-size would be bigger than this, the index will be
created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G

# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer
the
# key cache method. This is mainly used to force long character keys
in
# large tables to use the slower key cache method to create the index.
myisam_sort_buffer_size=738M

# Size of the Key Buffer, used to cache index blocks for MyISAM
tables.
# Do not set it larger than 30% of your available memory, as some
memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=65M

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=1M
read_rnd_buffer_size=256K

# This buffer is allocated when MySQL needs to rebuild the index in
# REPAIR, OPTIMZE, ALTER table statements as well as in LOAD DATA
INFILE
# into an empty table. It is allocated per thread so be careful with
# large settings.
sort_buffer_size=256K


#*** INNODB Specific options ***
innodb_data_home_dir="C:/MySQL Datafiles/"

# Use this option if you have a MySQL server with InnoDB support
enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
#skip-innodb

# Additional memory pool that is used by InnoDB to store metadata
# information. If InnoDB requires more memory for this purpose it
will
# start to allocate it from the OS. As this is fast enough on most
# recent operating systems, you normally do not need to change this
# value. SHOW INNODB STATUS will display the current amount used.
innodb_additional_mem_pool_size=32M

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit=1

# The size of the buffer InnoDB uses for buffering log data. As soon
as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=16M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set
this
# parameter up to 80% of the machine physical memory size. Do not set
it
# too large, though, because competition of the physical memory may
# cause paging in the operating system. Note that on 32bit systems
you
# might be limited to 2-3.5G of user level memory per process, so do
not
# set it too high.
#innodb_buffer_pool_size=2385M - Modified by CKR

#innodb_buffer_pool_size=1G
innodb_buffer_pool_size=2G


# Size of each log file in a log group. You should set the combined
size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for
the
# recovery process.
#innodb_log_file_size=1G
innodb_log_file_size=1G

# Number of threads allowed inside the InnoDB kernel. The optimal
value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=8
max_allowed_packet = 32M



log-slow-queries="C:/slow.log"
long_query_time=4



On Jan 17, 3:21*am, onedbguru <onedbg... (AT) yahoo (DOT) com> wrote:

I have added a new index for the column 'status' in the table
student_subject_mapping. This made a tremendous difference in
performance. Now I am able to execute 40 reqs at a time; but fails
when tries 50 reqs.


On Jan 17, 10:14*am, mahesh <connect.mah... (AT) gmail (DOT) com> wrote:

Hi,

i didn't read the messages in detail but you definetly want to use
innodb for a job like this!

Use innotop for monitoring.

Right. Having correct indexes ist most important.
A missing index is taking your response time from a fraction of a
second to nirvana!

Try with any sort of Linux.

Regards,

Bodo

klch wrote:

I was doing a fairly complex multi-table SELECT and was puzzled that
sometimes it took several seconds, and sometimes it was instant..of
course if the server has the result cached..

Adding some judicious indices got the result down to a few ms in *all*
cases.


around with it is beyond me....