Will it hurt our system to resize ABS and the FILE-OF-FILES?

Will it help?

According to ISTAT, my ABS has a size of 13 and should be 1277. It has
6,919 records with a total size of 1,938,433.


Thanks,
Danny

Depending on your version of D3, you may not be able to resize FOF -
give it a try .... this can become a bottleneck. ABS should have been
"the right size" if you didn't disable resing when system was initially
restored

ddspell-m3 wrote:

Hi
I believe that there is a considerable grey area in as to what is a correct
file size. If one considers that all items larger than 1600bytes ( I cannot
remember the exact figure but it is very low) are actually stored as a
pointer in the actual file with the true data off in uncontrollable overflow
then ISTAT et al return incorrect estimates for programs and other large
items. Personally I have always sized program files very small say 19 19 to
store 1000+ programs with no problems. My reasoning being that this
minimises the storage requirement and maximises the speed of retrieval. The
dictionary portion should of course be identical to the data portion in the
program situation as each contains one entry for each item, ie object and
source. I am talking D3 NT FSI of course.
Peter McMurray
"ddspell-m3" <ddspell (AT) yahoo (DOT) com> wrote

Resizing files only makes sense if you use those files often.

FOF gets updated at odd times, IMHO, but not real often. Abs is almost
never written into, and only occasionally read.

So I believe the answers to your questions are: No, it won't hurt; and no,
it probably won't make any noticible difference.


Mark Brown

Also, IIRC, you have to chagne the size of the file of files using the
debuger, becuase it's a "system level" file, not usually accessible to the
common (wo)man, but it CAN be resized. FSI:FOF is a different animal and
should probably best be left alone.


"ddspell-m3" <ddspell (AT) yahoo (DOT) com> wrote

"Peter McMurray" wrote:
I think that number is something like 1750 or 1780. And in a *nix
environment you can create the file with a P command to force all
items to be pointer items. I mention this because this is related to
my notes below.

Sorry Peter, but that logic doesn't necessarily follow. All BASIC
object items are pointer items (regardless of size). The dict file is
only a container for items that are about 50 bytes in size. When you
compile code, the pointer gets created in the main dict space, but the
object code itself immediately goes into overflow. This is the way
all of these systems work.

With the NT FSI, I've noticed that deleting the object (using
Decatalog) seems to only remove the pointers, it doesn't actually
release the space back which was allocated to the overflow files
(.D3O). I don't restore very often but I trust all of that unused
Win32 space will get returned the next time I do a full restore of D3.

Mark, do you think that might have something to do with dirty bits,
object caching, or some other process that needs old items? The space
is still occupied even after a reboot.

The D3 File Manager (aka the file mangler) has a checkbox for
compression, and one optimistic checkbox for unicode too, but alas
these seem permanently disabled and there doesn't seem to be a way to
get D3 to compress FSI files, or to store unicode.

Again, I haven't done a restore in a while, but if the FSI D3O files
similarly don't compress without a restore, then for anyone who finds
their Windows space is less than desirable, a D3 restore might help.

As far as how to pack the frames, note that D3NT allows for custom
frame sizes of 1k to 128k. For those tiny files that are only going
to have a couple small items, create them at 1k. For larger files you
may find performance is better with (for example) 100x128k frames
rather than 3200x4k. I dunno, no one's paying me for benchmarks.

On some of the points above, I'll defer to someone with a clue - I've
forgotten how a lot of this D3 stuff works, and since it breaks from
one release to the next anyway I've stopped trying to keep up.

T

I agree with Mark (no surprise there, eh?). The FOF isn't worth
resizing. However!!! It does gather statistics surprisingly often on
file opens, reads, writes, and other geeky statistics. So if your FOF
is seriously undersized, call RD and ask them about resizing it - the
answer will be dependent on your release.

We've just discussed this recently - one of the things that
artificially inflates the size of the FOF is a record that's kept
every time the file is cleared or recreated. You can delete the fof
data to eliminate all of this history, that will surely make it
smaller. The next file-save will regenerate all stats.

So, to clear the FOF, you can't clear-file, or delete/create, just:
select fof
delete fof
and
select fsidm,FileOfFiles,
delete fsidm,FileOfFiles,


Addenda:
1) The documentation for the FSI FOF is wrong and I don't think
clear/delete data is actually stored in there.
2) Clearing the FOF files periodically is good if you do a lot of
create/deleting of work files, which is the only reason why I bothered
to mention this stuff above.
3) You can use nt_makefof to rebuild the fof file(s?) if you don't
want to do a save.
4) Don't really care about this stuff anymore.

T



"Mark Brown" wrote:

"Tony Gravagno" wrote
items of a workfile just to avoid this. It's sad that one cannot
curtail this limitless obscure logging.

But you should! The unlimited integer-part precision makes
d3 a tool for math research. Basic is a great
language for simple chores, versus C, a great language
for writing myriads of bugs in one small program. It is much
easier to read and train others in Basic than Perl, my next
choice for doing windoze file admin tasks. Accuterm has the
ability to float a .gif file amidst the screen text, and like Xterm,
shares the vector graphics drawing mode seen in the old 'Tank'
video arcade game. If you've already got an office full of terms/
winboxes on a lan, why not program in Scrabble and that new
word game Quiddler for after-hours parties? There are command
line apps for the windoze world that will trigger a flatbed scanner,
and d3/nt can launch noninteractive command line apps, so...

One day I hope to find a win-based email client that lets d3
weed out my at-work Inbox spam better than OE manages...

This is a routine I run before every save:

SOURCE-UT 'SR.UT.CLEAR.FOF' size = 1329
01 SUBROUTINE SR.UT.CLEAR.FOF
02 **********************************
03 * PROGRAM TO CLEAR FLAGS IN THE
04 * FOF FILE.
05 *
06 *
07 * (c) Copyright 1991 - 2002, Key Data Systems Group
08 * All rights reserved. 559-432-3832 USA
09 * JEK 11/12/2002
10 **********************************
11 COMMON /KDS01/ FILES(300),KDS01.INIT
12 COMMON /KDS02/ KPRMS(300),KDS02.INIT
13 PROMPT ""
14 OPEN "DM,FILE-OF-FILES," TO FOF ELSE GOTO 999
15 PRINT ; PRINT "NOW CLEARING FILE-OF-FILES FLAGS..."
16 LINE='SELECT FOF WITH FILECODE # "" AND WITH MD = "KEY]"'
17 EXECUTE LINE
18 EOL=0
19 LOOP
20 READNEXT ID ELSE EOL=1
21 UNTIL EOL=1 DO
22 READ FREC FROM FOF,ID THEN
23 FREC<18>=""
24 FREC<19>=""
25 FREC<20>=""
26 WRITE FREC ON FOF,ID
27 END
28 REPEAT
29 *
30 * EXIT PROGRAM
31 *
32 999 RETURN
33 *
34 END


"Tony Gravagno" <g6q3x9lu53001 (AT) sneakemail (DOT) com.invalid> wrote

<snip>
Hi Tony
You seem to have missed my point. By not allocating many frames to the DICT
and DATA portions I have minimised storage. As I said all items will goto
overflow (of course if you love one line programs then some of your source
will stay in prime space). However one must allow a dictionary space that
is large enough for the 1000+ entries that a typical system may have. As
you have said with standard frame size that is around 80 to a frame ergo 19
is a pretty good size being a prime number.
Unfortunately File SIZER does not recognise this and the P type file
vanished with FSI.
Peter McMurray

"Frank Winans" wrote:

Actually you can. If you put a C or D in atb 17 of the FOF it will
stop that logging. What if you clear the file? Keep a table of files
that need this tweak, run them through a proggie that does a get-fof
command so that you know which fof item to update, and run the prog
whenever you select/delete the items in the file.

[I hope that's correct, again I haven't played with this stuff in a
long time. It might only apply to VME where we shouldn't have files
anyway. If you care about doing this for the FSI then petition RD for
some accurate documentation.]


Frank, I'm in complete agreement with you about the power and
flexibility of D3 and the MV model. Whenever someone has a need to
integrate MV with the world, I'm sure I can satisfy it with D3 or
other MV platforms.

(Going a little deeper than anyone's interest...)
My apathy revolves more about my disappointment with RD. They've
driven the spirit out of this fine database with a complete lack of
business or technical vision. Their documentation doesn't tell you
exactly how things work so people are left to figure it out themselves
or give up trying. I participated in the D3 v7.5 Beta and made lots
of notes on the docs, and while this is the best documentation they've
had in years, there were really too many errors to comment on, so I
gave up. Over time RD has made it so difficult for vendors to get
things done that a lot of vendors don't even support D3 as a primary
platform, if at all. It's impossible for me to fight this anymore, so
my response to a lot of things now is "don't know, don't care".

On the other side, if someone paid me for my knowledge about how these
things work I'd contribute more and keep up my chops on the various
topics. But there is hardly anyone willing to pay for D3
DBMS-specific expertise, so my product-specific knowledge is waning -
if the market doesn't care then why should I? Unless someone is
willing to pay for a solution to FOF woes I'm not inclined to do much
research on topics like this anymore. The rules of supply and demand
are playing out in serious ways in this market.


I built POP3 capabilities into NebulaMail, and it can easily do
something similar to what you're asking.
http:// remove.thisNebula-RnD.com/products/mail.htm
One of these days I might port NebulaMail to QM and sell it purely as
an anti-spam application. That said, Pick people honestly don't buy
tools like this, and there are a thousand tools very similar to it in
the freeware market. The lack of market incentive makes it a
non-starter.

See SpamPal.org for one fine example of software that does exactly
what you're asking, minus the MV middle-tier.

It would be relatively easy to integrate SpamPal or other products
with MV-based rule sets. This could not be written or supported for
free. The number of people who would be willing to pay for such
development are so few that this will probably never be done. This is
ultimately the single factor that stiffles most of my MV development -
everyone wants software for free.

T