dbTalk Databases Forums  

Using RAM effectively

comp.databases.xbase.fox comp.databases.xbase.fox


Discuss Using RAM effectively in the comp.databases.xbase.fox forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
MollyShark
 
Posts: n/a

Default Using RAM effectively - 08-02-2006 , 06:32 AM






I'm running Foxpro 7.0 on a server recently upgraded from 2 to 4 Gb of
RAM. I still get the insufficient memory error for file mapping which
means the RAM addition isn't helping at all. We are looking at buying
new servers which come with 16 Gb RAM. I'm guessing this overload
isn't going to do a thing for speed or memory errors. Can someone fill
me in on Foxpro's issue using RAM? We may have to change over to a Sql
Server or MySQL based product if this isn't resolved. We're sorting
files of millions of records and it is just slower than we can handle.

Thanks.

MJ


Reply With Quote
  #2  
Old   
Thomas Ganss
 
Posts: n/a

Default Re: Using RAM effectively - 08-03-2006 , 02:48 AM






Vfp*9* has definitely less memory errors, but they still exist.
Look up sys(3050) for SOME info. Perhaps give some detailed info about
your table layout and the problems / errors encountered - I've been
working since 1999 > 1GB and have run into some <g>.

regards

thomas

MollyShark schrieb:
Quote:
I'm running Foxpro 7.0 on a server recently upgraded from 2 to 4 Gb of
RAM. I still get the insufficient memory error for file mapping which
means the RAM addition isn't helping at all. We are looking at buying
new servers which come with 16 Gb RAM. I'm guessing this overload
isn't going to do a thing for speed or memory errors. Can someone fill
me in on Foxpro's issue using RAM? We may have to change over to a Sql
Server or MySQL based product if this isn't resolved. We're sorting
files of millions of records and it is just slower than we can handle.

Thanks.

MJ


Reply With Quote
  #3  
Old   
Cy Welch
 
Posts: n/a

Default Re: Using RAM effectively - 08-03-2006 , 03:34 PM



Thomas Ganss wrote:
Quote:
Vfp*9* has definitely less memory errors, but they still exist.
Look up sys(3050) for SOME info. Perhaps give some detailed info about
your table layout and the problems / errors encountered - I've been
working since 1999 > 1GB and have run into some <g>.

regards

thomas

MollyShark schrieb:
I'm running Foxpro 7.0 on a server recently upgraded from 2 to 4 Gb of
RAM. I still get the insufficient memory error for file mapping which
means the RAM addition isn't helping at all. We are looking at buying
new servers which come with 16 Gb RAM. I'm guessing this overload
isn't going to do a thing for speed or memory errors. Can someone fill
me in on Foxpro's issue using RAM? We may have to change over to a Sql
Server or MySQL based product if this isn't resolved. We're sorting
files of millions of records and it is just slower than we can handle.

Thanks.

MJ

You know, errors can sometimes be misleading. When doing large queries
or large sorts and such, "out of memory" often is out of diskspace on
the drive that the temp files for VFP are being created. More memory
will help some as more will get cached into memory, but if the files
being worked with are larger than the free memory then you will be
working a lot on disk. A good rule of thumb on very large files is that
you need at least 2 to 3 times the size of the files free on the drive
where VFP is creating it's temp files. Of course, if your files are
approaching 2GB in size, you should probably be looking at going to SQL
Server or MySQL anyways since VFP tables are limited to 2GB each. The
other thing to be careful of performance wise, is joining more than
about 3 tables on large queries, as this in my experience slows WAY
down. In many cases with queries of this type, I break them down into 2
or more queries each running off of the previous. I can do large
complex queries faster that way in many cases then with a single query.

--
Cy Welch
89 Camaro RS 5.0 TBI
98 Passport
03 Malibu


Reply With Quote
  #4  
Old   
Thomas Ganss
 
Posts: n/a

Default Re: Using RAM effectively - 08-04-2006 , 11:20 PM



Cy Welch schrieb:

Quote:
You know, errors can sometimes be misleading. When doing large queries
or large sorts and such, "out of memory" often is out of diskspace on
the drive that the temp files for VFP are being created.
That can happen, but with todays disk prices it nore a config prblem or
administration issue. In our local configuration the vfp temp files are
put into a separate dir besides the local data files (and can be sent to
another disk in case of multi-disk machines) without personal user
quotas. Having a too small swap disk can also create such errors.

Quote:
More memory
will help some as more will get cached into memory, but if the files
being worked with are larger than the free memory then you will be
working a lot on disk. A good rule of thumb on very large files is that
you need at least 2 to 3 times the size of the files free on the drive
where VFP is creating it's temp files.
Often much more, as disk speed decreases a lot when only small spaces
are left and defragmentation REALLY put's the brakes on...
Of course, if your files are
Quote:
approaching 2GB in size, you should probably be looking at going to SQL
Server or MySQL anyways since VFP tables are limited to 2GB each.
It sure is a pity MS did not enhance the table size limit the same way
smaller companies were able to increase the dbf format: they can use
terabytes with a slightly different header "most simiar to the vfp
format" <g>.
The
Quote:
other thing to be careful of performance wise, is joining more than
about 3 tables on large queries, as this in my experience slows WAY
down. In many cases with queries of this type, I break them down into 2
or more queries each running off of the previous. I can do large
complex queries faster that way in many cases then with a single query.
This is something I have read often, but never really verified as I tend
to break queries into smaller actions for maintainability often. Still,
sometimes it is easier on simple joins to add more tables.

Have you found any multi-table query slower than your step-wise approach
even when FORCE was applied to keep the yoin order identical to your
approach ?

regards

thomas


Reply With Quote
  #5  
Old   
Cy Welch
 
Posts: n/a

Default Re: Using RAM effectively - 08-07-2006 , 07:49 PM



Thomas Ganss wrote:
Quote:
Cy Welch schrieb:

Of course, if your files are
approaching 2GB in size, you should probably be looking at going to
SQL Server or MySQL anyways since VFP tables are limited to 2GB each.

It sure is a pity MS did not enhance the table size limit the same way
smaller companies were able to increase the dbf format: they can use
terabytes with a slightly different header "most simiar to the vfp
format" <g>.
The
That was done that way on purpose. MS does not want FoxPro competing
with SQL Server even on something like file size. I understand there is
also something to do with interoperability as part of the problem is
that the record locking scheme is part of the limitation, and changing
it would no longer allow interoperability (one of the problem with these
small companies products is the only talk with themselves). I
personally would be just fine with the loss of backwards compatibility,
but I'm sure others would not be.

Quote:
other thing to be careful of performance wise, is joining more than
about 3 tables on large queries, as this in my experience slows WAY
down. In many cases with queries of this type, I break them down into
2 or more queries each running off of the previous. I can do large
complex queries faster that way in many cases then with a single query.

This is something I have read often, but never really verified as I tend
to break queries into smaller actions for maintainability often. Still,
sometimes it is easier on simple joins to add more tables.

Have you found any multi-table query slower than your step-wise approach
even when FORCE was applied to keep the yoin order identical to your
approach ?

I have not worked much with FORCE, but I think even then it works better
with them broken down because you can reduce the I/O if the queries are
intelligently designed.

--
Cy Welch
89 Camaro RS 5.0 TBI
98 Passport
03 Malibu


Reply With Quote
  #6  
Old   
Thomas Ganss
 
Posts: n/a

Default Re: Using RAM effectively - 08-08-2006 , 07:58 AM




Quote:
It sure is a pity MS did not enhance the table size limit the same way
smaller companies were able to increase the dbf format: they can use
terabytes with a slightly different header "most simiar to the vfp
format" <g>.
The


That was done that way on purpose. MS does not want FoxPro competing
with SQL Server even on something like file size.
total agreement on that assumption <g>.

I understand there is
Quote:
also something to do with interoperability as part of the problem is
that the record locking scheme is part of the limitation, and changing
it would no longer allow interoperability (one of the problem with these
small companies products is the only talk with themselves). I
So they need a new table header and 8-Byte locking - they introduced
larger changes in vfp3.

Quote:
personally would be just fine with the loss of backwards compatibility,
but I'm sure others would not be.
It would not have to be the default. But we are probably very much in
synch with our assumptions <bg>.

Quote:
other thing to be careful of performance wise, is joining more than
about 3 tables on large queries, as this in my experience slows WAY
down. In many cases with queries of this type, I break them down
into 2 or more queries each running off of the previous. I can do
large complex queries faster that way in many cases then with a
single query.


This is something I have read often, but never really verified as I
tend to break queries into smaller actions for maintainability often.
Still, sometimes it is easier on simple joins to add more tables.

Have you found any multi-table query slower than your step-wise
approach even when FORCE was applied to keep the yoin order identical
to your approach ?


I have not worked much with FORCE, but I think even then it works better
with them broken down because you can reduce the I/O if the queries are
intelligently designed.
I am not certain I understand what you try to say above: are you perhaps
talking about creating indeces on the in-between cursors different from
the ones vfp will create ? Or forcing a temporary index file to be
created on a different disk for better access/head movement patterns ? I
am always eager to add such tricks to my repertoire <bg>.

curious

thomas


Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.