Universe Basic Internal Select

#1

I posted this on U@UG, but so far, nothing:

I believe I'm up to speed on most other platforms' use of BASIC internal
selects, but I haven't seen anything conclusive for Universe. We are using a
lot of external selects to avoid the "problem" (I think it's a myth) of
updating/deleting with an internal select, having the group shuffle
underneath those selects, then experiencing either data corruption or
unexpected results.

AFAIK, these days, the only caveat for avoiding internal selects is if you
change the ID, and it gets written into a subsequent group and then
re-selected (unwantedly).

Are there any Universe internals gurus out there who can shed some
compelling light on this subject? I want to take this to the bosses and stop
traversing the files twice if we don't have to. (Actually, I think both
methods traverse twice - but the internal method is much more likely to
select and read a frame without having it flushed between the select and the
read.)

Thanks for your time.

Ed

#2

Well, depending on WHAT you are doing with the select list, you could
always have the best of both worlds !?!?

If you are considering an internal select, I'm assuming "order" isn't
important (perhaps you are building a temporary summary file from a
detail ?). Anyway, you could always harness the "Drumheller Trick" and
have an I-type that actually does the "work" - fire off a select (or
SSelect, LIST etc) & then simply reference the dictionary item --> one
pass through the file PLUS the 'expressiveness' of TCL/ACCESS

Same sort of thing can be done with any mv flavout these days that
supports some mechanism for calling Basic code from a dictionary.

AFAIK, the internal select does NOT traverse twice ... simply has a
pointer into a group (starts at base of file), and move through records
in current frame, when exhausted goes to next frame etc --> 1 pass,
quite quick.

Need to be "careful" though - there was a thread on jbase group in last
6 months that indicated some "tuneables" that could actually cause the
sorts of problems with group shuffling that you are talking about

Ed Sheehan wrote:

Quote:

I posted this on U@UG, but so far, nothing:

I believe I'm up to speed on most other platforms' use of BASIC
internal
selects, but I haven't seen anything conclusive for Universe. We are
using a
lot of external selects to avoid the "problem" (I think it's a myth)
of
updating/deleting with an internal select, having the group shuffle
underneath those selects, then experiencing either data corruption or

unexpected results.

AFAIK, these days, the only caveat for avoiding internal selects is
if you
change the ID, and it gets written into a subsequent group and then
re-selected (unwantedly).

Are there any Universe internals gurus out there who can shed some
compelling light on this subject? I want to take this to the bosses
and stop
traversing the files twice if we don't have to. (Actually, I think
both
methods traverse twice - but the internal method is much more likely
to
select and read a frame without having it flushed between the select
and the
read.)

Thanks for your time.

Ed

#3

Ed, I think it is a myth too.

Back in R83 & AP days, I had a some basic utilities that were
replacements for istat, but displayed results as each group was
traversed, rather than waiting until the end of file is reached to
display the results of the first and subsequent groups (which can be a
really long time on a big file). These programs worked by interrogating
the select list variable, e.g.

open "file" to fv
select fv to list
loop
?? = list<1>
?? = list<2>
etc
readnext id from list else exit
repeat

The stuff in list is partly in hex (the file base, the current group,
etc) and partly a list of ids in the next group; as each group is used
up the next group is read in. Unfortunately, the internal dynamics of
this select list variable kept changing in D3 versions, thus my
utilities stopped working and I lost interest in the reverse engineering
process.

But the point is, for R83, AP, D3 and the like, a basic internal select
grabs the item-ids one group at a time; writing back to the current
group may shuffle the order of items in the group, but does not affect
the ids from the group in the select list variable, which were
all read in once when the new group was first encountered. I have to
believe that the U2 products work the same way.

Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006

Ross Ferris wrote:

Quote:

Well, depending on WHAT you are doing with the select list, you could
always have the best of both worlds !?!?

If you are considering an internal select, I'm assuming "order" isn't
important (perhaps you are building a temporary summary file from a
detail ?). Anyway, you could always harness the "Drumheller Trick" and
have an I-type that actually does the "work" - fire off a select (or
SSelect, LIST etc) & then simply reference the dictionary item --> one
pass through the file PLUS the 'expressiveness' of TCL/ACCESS

Same sort of thing can be done with any mv flavout these days that
supports some mechanism for calling Basic code from a dictionary.

AFAIK, the internal select does NOT traverse twice ... simply has a
pointer into a group (starts at base of file), and move through records
in current frame, when exhausted goes to next frame etc --> 1 pass,
quite quick.

Need to be "careful" though - there was a thread on jbase group in last
6 months that indicated some "tuneables" that could actually cause the
sorts of problems with group shuffling that you are talking about

Ed Sheehan wrote:

I posted this on U@UG, but so far, nothing:

I believe I'm up to speed on most other platforms' use of BASIC

internal

selects, but I haven't seen anything conclusive for Universe. We are

using a

lot of external selects to avoid the "problem" (I think it's a myth)

of

updating/deleting with an internal select, having the group shuffle
underneath those selects, then experiencing either data corruption or

unexpected results.

AFAIK, these days, the only caveat for avoiding internal selects is

if you

change the ID, and it gets written into a subsequent group and then
re-selected (unwantedly).

Are there any Universe internals gurus out there who can shed some
compelling light on this subject? I want to take this to the bosses

and stop

traversing the files twice if we don't have to. (Actually, I think

both

methods traverse twice - but the internal method is much more likely

to

select and read a frame without having it flushed between the select

and the

read.)

Thanks for your time.

Ed

#4

I tend to agree about U2 selects, because there is a variable "@selected"
which contains the number of ids selected for the particular group you're in
when using an internal SELECT. I was just wondering if there was a white
paper or something to nail it down.

As to number of passes, I think both internal and external methods make two
passes. The external is easy to visualize: one pass to gather ids, and one
more to read/process. But the internal does the same thing, albeit one group
at a time: It traverses the first group entirely. This is to gather all the
ids in the group. This first traversal loads up the entire group into RAM,
just as if you had done a READ. That's the first pass. Then it gives over to
the read process, and all the items in the group are read in again, this
time by the READ statement. This repeats for all the groups in the file.

The advantage realized by using the internal method is not the saving of one
pass of the file; it's the efficiency gained by increasing the odds (by a
huge amount) that the read process will obtain the group from memory before
it is flushed.

The external method will make one pass thru all the groups first, and that
increases the odds that when you come back around for your first READ, the
group will be flushed back to disk (especially if the file is larger than
RAM and/or other processes are using RAM). This will cause a second frame
fault, which is hundreds of times longer than a frame attachment from RAM.

It would be nice to know when the group select methodology changed from
string pointer to group select. Seems hard to pin it down though. I think
that for procs, however, it's still the string pointer method. If your proc
chains to one that's in a group which gets shuffled, you'll get a "proc
moved" type error message.

"Scott Ballinger" <scott (AT) paretoTiredOfSpam (DOT) net> wrote

Quote:

Ed, I think it is a myth too.

Back in R83 & AP days, I had a some basic utilities that were replacements
for istat, but displayed results as each group was traversed, rather than
waiting until the end of file is reached to display the results of the
first and subsequent groups (which can be a really long time on a big
file). These programs worked by interrogating the select list variable,
e.g.

open "file" to fv
select fv to list
loop
?? = list<1
?? = list<2
etc
readnext id from list else exit
repeat

The stuff in list is partly in hex (the file base, the current group, etc)
and partly a list of ids in the next group; as each group is used up the
next group is read in. Unfortunately, the internal dynamics of this select
list variable kept changing in D3 versions, thus my utilities stopped
working and I lost interest in the reverse engineering process.

But the point is, for R83, AP, D3 and the like, a basic internal select
grabs the item-ids one group at a time; writing back to the current group
may shuffle the order of items in the group, but does not affect the ids
from the group in the select list variable, which were
all read in once when the new group was first encountered. I have to
believe that the U2 products work the same way.

Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006

Ross Ferris wrote:
Well, depending on WHAT you are doing with the select list, you could
always have the best of both worlds !?!?

If you are considering an internal select, I'm assuming "order" isn't
important (perhaps you are building a temporary summary file from a
detail ?). Anyway, you could always harness the "Drumheller Trick" and
have an I-type that actually does the "work" - fire off a select (or
SSelect, LIST etc) & then simply reference the dictionary item --> one
pass through the file PLUS the 'expressiveness' of TCL/ACCESS

Same sort of thing can be done with any mv flavout these days that
supports some mechanism for calling Basic code from a dictionary.

AFAIK, the internal select does NOT traverse twice ... simply has a
pointer into a group (starts at base of file), and move through records
in current frame, when exhausted goes to next frame etc --> 1 pass,
quite quick.

Need to be "careful" though - there was a thread on jbase group in last
6 months that indicated some "tuneables" that could actually cause the
sorts of problems with group shuffling that you are talking about

Ed Sheehan wrote:

I posted this on U@UG, but so far, nothing:

I believe I'm up to speed on most other platforms' use of BASIC

internal

selects, but I haven't seen anything conclusive for Universe. We are

using a

lot of external selects to avoid the "problem" (I think it's a myth)

of

updating/deleting with an internal select, having the group shuffle
underneath those selects, then experiencing either data corruption or

unexpected results.

AFAIK, these days, the only caveat for avoiding internal selects is

if you

change the ID, and it gets written into a subsequent group and then
re-selected (unwantedly).

Are there any Universe internals gurus out there who can shed some
compelling light on this subject? I want to take this to the bosses

and stop

traversing the files twice if we don't have to. (Actually, I think

both

methods traverse twice - but the internal method is much more likely

to

select and read a frame without having it flushed between the select

and the
read.)

Thanks for your time.

Ed

#5

Afaik, all current Pick products work one group at a time, caching the ids,
so you can neither have data nor file corruption using an Internal Select.
What can happen, though, is that a new item written by someone else will not
be selected, or one deleted will not be present when you Readu (you are
using Readu, aren't you?) Which is pretty much what you should expect, since
you are not locking the file from read/writes.

The jbase issue was that, in the *default* state from some release on, *the
data as well as the ids were cached* in someone's wisdom that this
would speed things up. So you would, in the scenario above, get a stale
version of a record someone had updated (instead of the current) or find a
record that someone has already deleted. Luckily, sanity prevailed and you
can turn off this abominable beahavior-long live jbase environmental
variables.

Chandru Murthi

"Scott Ballinger" <scott (AT) paretoTiredOfSpam (DOT) net> wrote

Quote:

Ed, I think it is a myth too.

Back in R83 & AP days, I had a some basic utilities that were replacements
for istat, but displayed results as each group was traversed, rather than
waiting until the end of file is reached to display the results of the
first and subsequent groups (which can be a really long time on a big
file). These programs worked by interrogating the select list variable,
e.g.

open "file" to fv
select fv to list
loop
?? = list<1
?? = list<2
etc
readnext id from list else exit
repeat

The stuff in list is partly in hex (the file base, the current group, etc)
and partly a list of ids in the next group; as each group is used up the
next group is read in. Unfortunately, the internal dynamics of this select
list variable kept changing in D3 versions, thus my utilities stopped
working and I lost interest in the reverse engineering process.

But the point is, for R83, AP, D3 and the like, a basic internal select
grabs the item-ids one group at a time; writing back to the current group
may shuffle the order of items in the group, but does not affect the ids
from the group in the select list variable, which were
all read in once when the new group was first encountered. I have to
believe that the U2 products work the same way.

Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006

Ross Ferris wrote:
Well, depending on WHAT you are doing with the select list, you could
always have the best of both worlds !?!?

If you are considering an internal select, I'm assuming "order" isn't
important (perhaps you are building a temporary summary file from a
detail ?). Anyway, you could always harness the "Drumheller Trick" and
have an I-type that actually does the "work" - fire off a select (or
SSelect, LIST etc) & then simply reference the dictionary item --> one
pass through the file PLUS the 'expressiveness' of TCL/ACCESS

Same sort of thing can be done with any mv flavout these days that
supports some mechanism for calling Basic code from a dictionary.

AFAIK, the internal select does NOT traverse twice ... simply has a
pointer into a group (starts at base of file), and move through records
in current frame, when exhausted goes to next frame etc --> 1 pass,
quite quick.

Need to be "careful" though - there was a thread on jbase group in last
6 months that indicated some "tuneables" that could actually cause the
sorts of problems with group shuffling that you are talking about

Ed Sheehan wrote:

I posted this on U@UG, but so far, nothing:

I believe I'm up to speed on most other platforms' use of BASIC

internal

selects, but I haven't seen anything conclusive for Universe. We are

using a

lot of external selects to avoid the "problem" (I think it's a myth)

of

updating/deleting with an internal select, having the group shuffle
underneath those selects, then experiencing either data corruption or

unexpected results.

AFAIK, these days, the only caveat for avoiding internal selects is

if you

change the ID, and it gets written into a subsequent group and then
re-selected (unwantedly).

Are there any Universe internals gurus out there who can shed some
compelling light on this subject? I want to take this to the bosses

and stop

traversing the files twice if we don't have to. (Actually, I think

both

methods traverse twice - but the internal method is much more likely

to

select and read a frame without having it flushed between the select

and the
read.)

Thanks for your time.

Ed

#6

Inline...

"murthi" <c_xyz_murthi (AT) seeing_xyz_green (DOT) net> wrote

Quote:

Afaik, all current Pick products work one group at a time, caching the
ids,
so you can neither have data nor file corruption using an Internal Select.
What can happen, though, is that a new item written by someone else will
not
be selected,

This would be the case even with an external select, no?

or one deleted will not be present when you Readu (you are

Quote:

using Readu, aren't you?)

also the same for externals...

Which is pretty much what you should expect, since

Quote:

you are not locking the file from read/writes.

snip

We do a lot of processing, purging etc. Most of these processes do external
selects, even when there is no selection criteria. Several of these files
may have 1 million to 9 million records (pretty soon you're talking real
numbers). I think the potential time savings would be impressive.

Thanks for your thoughts.

Ed

#7

"Ed Sheehan" <NOedsSPAM (AT) xmission (DOT) com> wrote

Quote:

Inline...

"murthi" <c_xyz_murthi (AT) seeing_xyz_green (DOT) net> wrote in message
news:QTpje.4754$BF5.1720 (AT) trndny06 (DOT) ..
Afaik, all current Pick products work one group at a time, caching the
ids,
so you can neither have data nor file corruption using an Internal
Select.
What can happen, though, is that a new item written by someone else will
not
be selected,

This would be the case even with an external select, no?

or one deleted will not be present when you Readu (you are
using Readu, aren't you?)

also the same for externals...

Yes to both, and the window is larger for external selects since you 'cache'
all the ids at once. iae, I suppose if you really wanted a static traverse
of the file, you'd lock something to prevent any changes while you select.

Quote:

Which is pretty much what you should expect, since
you are not locking the file from read/writes.

snip

We do a lot of processing, purging etc. Most of these processes do
external selects, even when there is no selection criteria. Several of
these files may have 1 million to 9 million records (pretty soon you're
talking real numbers). I think the potential time savings would be
impressive.

Absoultely. If sequence is irrelevant, use an Internal Select. Remeber about
the jbase env variable if you're using it.

Quote:

Thanks for your thoughts.
Y'r Welcome... Chandru Murthi

Ed

#8

"Ed Sheehan" <NOedsSPAM (AT) xmission (DOT) com> wrote

Quote:

Inline...

"murthi" <c_xyz_murthi (AT) seeing_xyz_green (DOT) net> wrote in message
news:QTpje.4754$BF5.1720 (AT) trndny06 (DOT) ..
Afaik, all current Pick products work one group at a time, caching the
ids,
so you can neither have data nor file corruption using an Internal
Select.
What can happen, though, is that a new item written by someone else will
not
be selected,

This would be the case even with an external select, no?

As I recall the internal and external selects work slightly differently at
least in Pick/D3. An I'm willing to be corrected if I'm wrong.

As I remember, the external select gets all the ID's within the file, while
the internal select gets the id's of each group as it moves from group to
group. So in theory a person could be working with an internal select at
the beginning of a file while another user add/deletes items at the end of
the file. The internal select will get the latest ID's when the process
moves into the said group.

Quote:

We do a lot of processing, purging etc. Most of these processes do
external
selects, even when there is no selection criteria. Several of these files
may have 1 million to 9 million records (pretty soon you're talking real
numbers). I think the potential time savings would be impressive.

I'm told that if you are traversing all/most items in a file, the internal
select list is suppose to be faster. If you want to skip processing only a
few percentage of records, I believe it is faster to make the decision
inside the program using the internal select.

If I'm wrong on any of this, someone please sound off, 'cause I'm wasting a
lot of processor cycles if I am.

Best regards,

Dale

#9

I think you're right on all counts. The only drawback (other than selection
criteria) I can think of with internal selects is if you change the key and
write to a subsequent group, then it reselects it (unwantedly) later.

As to efficiency, I just did an exhaustive benchmark on a huge file,
comparing an internal select against an external. There were some selection
criteria argued (IF SOMETHING = "1""2""3"... etc), and the external actualy
beat the internal. The internal was building a select dynamic array as it
compared SOMETHING to a validation list. The external was actually
relatively more efficient as elements were added to the criteria, by several
magnitudes. But when all criteria were eliminated, the internal was about
twice as fast.

Moral: If no selection criteria then use internal
If selection criteria then probably use external (but verify on large
files)

Ed

Dale Benedict wrote:

Quote:

"Ed Sheehan" <NOedsSPAM (AT) xmission (DOT) com> wrote in message
news:d6lfbr$sc1$1 (AT) news (DOT) xmission.com...
Inline...

"murthi" <c_xyz_murthi (AT) seeing_xyz_green (DOT) net> wrote in message
news:QTpje.4754$BF5.1720 (AT) trndny06 (DOT) ..
Afaik, all current Pick products work one group at a time, caching
the ids,
so you can neither have data nor file corruption using an Internal
Select. What can happen, though, is that a new item written by
someone else will not
be selected,

This would be the case even with an external select, no?

As I recall the internal and external selects work slightly
differently at least in Pick/D3. An I'm willing to be corrected if
I'm wrong.

As I remember, the external select gets all the ID's within the file,
while the internal select gets the id's of each group as it moves
from group to group. So in theory a person could be working with an
internal select at the beginning of a file while another user
add/deletes items at the end of the file. The internal select will
get the latest ID's when the process moves into the said group.

We do a lot of processing, purging etc. Most of these processes do
external selects, even when there is no selection criteria. Several
of these files may have 1 million to 9 million records (pretty soon
you're talking real numbers). I think the potential time savings
would be impressive.

I'm told that if you are traversing all/most items in a file, the
internal select list is suppose to be faster. If you want to skip
processing only a few percentage of records, I believe it is faster
to make the decision inside the program using the internal select.

If I'm wrong on any of this, someone please sound off, 'cause I'm
wasting a lot of processor cycles if I am.

Best regards,

Dale

#10

"Ed Sheehan" <NOedsSPAM (AT) xmission (DOT) com> wrote

Quote:

I think you're right on all counts. The only drawback (other than
selection
criteria) I can think of with internal selects is if you change the key
and
write to a subsequent group, then it reselects it (unwantedly) later.

The moral here is to never-ever use an internal select list to change keys
in the same file. I've been told, but never seen, that you can get into an
infinite loop doing such a thing. Not sure how!?

Quote:

As to efficiency, I just did an exhaustive benchmark on a huge file,
comparing an internal select against an external. There were some
selection
criteria argued (IF SOMETHING = "1""2""3"... etc), and the external
actualy
beat the internal. The internal was building a select dynamic array as it
compared SOMETHING to a validation list. The external was actually
relatively more efficient as elements were added to the criteria, by
several
magnitudes. But when all criteria were eliminated, the internal was about
twice as fast.

My intent about using an internal select list and letting the program weed
out the few, was that the selection criteria had to be easy. eg. if
item<5> = 'y' then do not process

Sorry I wasn't clear on that.

Quote:

Moral: If no selection criteria then use internal
If selection criteria then probably use external (but verify on
large
files)

Ed

Dale Benedict