I have a very large Tracking file with modulo of 437909 and it has been
running for 6 hours on an ACCOUNT-RESTORE. This was sized by F-RESIZE
before the restore so the size should be right.

The file has millions of records with an average size of 200. It tracks
every change to costs or prices made in the system for a multi-store
operation, so this file will continue to grow.

I have thought of creating subfiles by month, but then listing from TCL
is not really usable anymore and writing programs around that scheme is
a pain.

Short of purging data from the file, which is an option but I don't
want to do it because it is good data, what strategy should I use to
make this file more manageable?

Steve

How many million transactions? What version of D3/Linux? At what stage
is it "stuck": - restoring the file? or building indices against it?
Had the RESIZE-FILE command actually finished resizing the file prior
to this backup being done?

On the surface it looks like if the file has less than around 6 million
records the modulo should be OK ... but I'm still suspicious that a
previous resize may not have finished, so every record is now being
re-hashed (if the restore is happening on the same box, maybe do a
CHECK-RESIZING

Not sure the exact count right now. I will give that later.

D3 Release Version 7.4.2.LINUX on RHEL 3

It is in the middle of an ACCOUNT-RESTORE and there was no RESIZE-FILE
done on the file. There is no INDEX on the file.

I type in CHECK-RESIZING and it just drops back to TCL with no output.

So, doing a RESIZE-FILE before the SAVE and RESTORE would make it go
faster because of rehashing?

Steve

Here is the count of the file:

6298677

Steve,

We have several 7.4.2 system restoring single files of 2,000,000+ items
in a few minutes.

we stopped using f-resize many years ago as it slows down restores by
rehashing every item in the file(s).

On very large files 'resize-file' can also be very slow and grind
system performance.

For large files we calculate a new modulo, create-file data
old-file,new modulo. Copy data from old-file to oldfile,new. When
this is completed rename-file old-file,old-file to old-file,old then
rename-file old-file,new to old-file,old-file. We then delete or DX
dict old-file old.

BBB

srs (AT) perfectionsoftware (DOT) com wrote:
I suppose you could split the data by attributes into different smaller
files.
Look at your reports and updates & see if there are groups of
attributes that are used together.
You say your files tracks changes for costs & prices for stores.
Your fields might be ...

Store
date
time
product
old cost
new cost
old bulk1 cost (for qtys >100)
new bulk1 cost (for qtys >100)
old bulk2 cost (qtys >1000)
new bulk2 cost (qtys >1000)
old bulk3 cost (qtys >5000)
new bulk3 cost (qtys >5000)
old price
new price
old promo price
new promo price
old disc1 price (discount regular cust)
new disc1 price (discount regular cust)
old disc2 price (discount large cust)
new disc2 price (discount large cust)
old disc3 price (discount Huge cust)
new disc3 price (discount Huge cust)

So your files could become

StorePriceChg master
Store
Date
time
Sequential key

CostChg (by Seq Key)
old cost
new cost

Bulk1CostChg (by Seq key)
old Bulk1 cost
new Bulk1 cost

and so on...

Means quite a lot of programing changes when building the files
But Access listings need not change because you can change the
dictionaries on the master file to translate to the other files.
Then you could f_resize the smaller files one by one, each one would be
smaller, less disk thrashing. Resizing sound like something youll be
doing a lot of as the data is going to build and build.

I've probably been as guilty as anyone of just adding another field to
the file when some analysis of how the data is used might sugest
information could be kept in seperate files, but with the same key.
This has to weighed against the ability to read everything in one hit
that is Picks big effiency advantage over 1NF type databases.

Jeremy Thomson

I believe the behaviour you describe (F-RESIZE not "sticking") is
'unique' to D3/NT with files in the FSI. AFIK the "restore" issue
doesn't apply to VME or *nix

On Thu, 30 Mar 2006 03:20:30 -0800, srs wrote:

What I have done with our main history file is created an archive file. We
store medical records (billing and patient medical data). Our main file
was 8+ million records, on D3/aix. In the archive file, I use a key of
just the patient ID, and put all records under that. Each record was
smallish, like yours. So what I do is periodically is select older
records, and take each record and convert the system delimiters down one
ascii value, and append the record as a new attribute to the archive
record under the patient key. You could use a part number, for example. I
have an archive subroutine that converts char(254) to char(253), for
example. So a record becomes an attribute. I then have a general read
subroutine, that when it's looking for a record, checks the main file
first, then the archive file if needed. I usually don't have to do access
reports on data old enough to archive. So I wind up with fewer bigger
records, instead of a lot of little records.

Art