I really hate that when I type WHERE {port} it tells the names of the
ABS frames and they don't really tell me a whole lot about what the
process is doing. Reads, writes and dynamic arrays I pick out pretty
easily but what the heck is br.unix.bix among many others. How do I
look at the ABS frame and decode it into assembler? I would love to
see what some of these ABS routines actually do, plus I haven't
written or even seen Pick Assembler since the ADDS MENTOR days and
when I see the WHERE output I was a bit nostalgic.

Hopefully someone out there has some code that can help to decode ABS
a bit and help refresh my memory or update me if D3 is totally
different than the MENTOR was.

Thanks,

Marshall Lucas

In a D3/Linux environment, I'm trying to figure out how to access a
300mb+ sized linux record. Everything I try yields a runaway process
error. I know the runaway process limit can be set higher with the
set-runaway-limit command, but I don't know how high I'd have to go with
this command, and whether I would really want it that big for normal
operations.

If anybody has any tips or tricks to access a very large linux record
from D3, I'd like to hear them.

Art

On Dec 16, 5:43 pm, Pete Jewell <boredbypolit... (AT) gmail (DOT) com> wrote:
If you use the %OPEN and %READ functions in basic as recommended, you
don't need to bother dividing the file into chunks. These functions
"stream" through the file so it doesn't require that large amounts of
data be loaded into workspace. Use a buffer size of about 5000 - no
need to go smaller, no benefit to going larger.

I use this method to load a file of 13 million records into d3. The
last million load just as fast as the first million.

-Luke

art wrote:

If you want tricks, here are three in decreasing order of performance.
I haven't any of them, so I'm curious how they work. ;b

In short, skip the %functions approach and try the code below which
makes use of commands in the host OS. This is for *nix only though
you can find or write code that does the same functions in Windows.

First try: Generate a command line this:
sed -n '1,100p;101q' filename

FPATH = "/tmp/myfile" ; * change as appropriate
LINE=0
LOOP
CMD = "!" ; * change for your platform to "! ", "SH -C", etc
CMD = CMD:"sed -n '"LINE+1):","LINE+500):"p"
CMD = CMD:";"LINE+501):"q' ":FPATH
EXECUTE CMD CAPTURING BLOCK
UNTIL BLOCK = "" DO ; * please check this!!
GOSUB PROCESS.BLOCK
LINE = LINE + 500
REPEAT
* ...

At the UNTIL/DO, I don't know what to expect, either an error that
there aren't as many lines as we expect, or my guess is null. (I just
wrote it, I didn't try it. Welcome to OSS.)
Be careful about LF delimiters. Also, the syntax for sed may be
different for GNU, AIX or HP, etc.


Second try: You could use awk, or a combination of head and tail, but
sed is optimized for streams. Here is the awk version of the above
sed command.

awk 'NR >= 1 && NR <= 3' filename
CMD = "!"
CMD = CMD:"awk ' NR >= "LINE+1):" && NR <= "LINE+500)
CMD = CMD:" ' ":FPATH


Third try: Here is the heads n tails version, though it's bound to be
much less efficient as the file gets larger:

FPATH = "/tmp/myfile" ; * change as appropriate
FOR LINE = 1 TO 9999999 STEP 100 ; * sloppy, don't care.
CMD = "!" ; * change for your platform to "! ", "SH -C", etc
CMD = CMD:"head -":LINE:" ":FPATH:" | tail -100"
EXECUTE CMD CAPTURING BLOCK
GOSUB PROCESS.BLOCK
NEXT N
* ...

This gets lines 1-100, then gets the last 100 lines. Easy enuf.
Then it gets lines 1-200, and takes off the last 100 lines, which is
101-200. Your system may groan in agony as it gets deeper into the
file.

I hope that's enough tricks for one day.

T

Tony Gravagno wrote:
Thanks all. I'd forgotten about the % functions. I used to be quite
conversant with them. Tony, those are just the ideas I was looking for.
Off to my "man sed"!

Art

"art" wrote
and issue one cd /tmp/bob ; cat /tmp/myfile | split -1000
command to populate it with a gazillion small files.
Then in D3
SSELECT unix:/tmp/bob
and have a basic program open up D3 file
"unix:/tmp/bob" and read each item using that
active select list. -- Untested; look for problems where
split ran out of xzz names {where z is any letter}
and moved on to xzzz names -- make sure SSELECT works ok
there.

Don't forget to tidy up by !rm -r /tmp/bob when no longer needed.

Frank Winans wrote:
Tony, the sed works as advertised, although a little cryptic. I also
took a look at the split command. Since disk space is not a big issue
for me, I like this way, it's simple. I've got SET-RUNAWAY-LIMIT 30000
in user-coldstart. The following command:
!cd /home/art/Documents/ProMed/test;split -C 50m -d root_bak mail
makes 50meg sized blocks, which work just fine with 30000 frames. The
output looks like:
[root@art test]# ls -lha
total 612M
drwxrwxrwx 2 art art 4.0K 2007-12-18 00:02 ./
drwxrwxr-x 9 art art 1.0K 2007-12-17 23:14 ../
-rw-rw-r-- 1 pick pick 50M 2007-12-18 00:01 mail00
-rw-rw-r-- 1 pick pick 50M 2007-12-18 00:01 mail01
-rw-rw-r-- 1 pick pick 50M 2007-12-18 00:02 mail02
-rw-rw-r-- 1 pick pick 50M 2007-12-18 00:02 mail03
-rw-rw-r-- 1 pick pick 50M 2007-12-18 00:02 mail04
-rw-rw-r-- 1 pick pick 50M 2007-12-18 00:02 mail05
-rw-rw-r-- 1 pick pick 4.7M 2007-12-18 00:02 mail06
-rwxrwxrwx 1 art art 305M 2007-12-15 19:47 root_bak*

That is real easy to write a loop for. Just out of curiosity, I tried
100m, and that sized block triggered the run-away process. 50m is just
fine.


Thanks all for the suggestions. Now I can bullet-proof some code.
Art

"art" wrote
Don't forget, if you want to do heavy
maintenance work during office hours,
you can avoid gripes from the staff about
"why is the system so slow?"
by using the *nix feature " nice," as in
nice split
or nice d3

You can also use renice on an existing d3
process if you forgot to launch it as
nice d3. Irreversable though if you ain't root.