D3 Gurus - Strange Problem

#11

Many thanks to everyone for the input. A resolution of sorts has been
found and I will post as much detail as possible for the next poor
schmuck looking for an answer to this.

Updating the system patches made no difference (but surely wasn't
wasted time <g>).

The original symptom of writing 130,000 attributes into the FDI's was
indeed as Mark suggested a scaling issue but a PRECISION statement did
not cure the problem.

In debugging this, it became apparent that at some point in the
f-resize main loop the system would go into some weird "MR4" mode. Once
in this mode, it continued through completion. This affected the
representation of constants even so that a check such as:
TST = 3 + 4
IF TST # 7 THEN DEBUG

Would never fail (even though TST would actually be 70000) because the
constant "7" was being rendered as if it were ICONV(7, "MR4"). The
check that would work was:
TST = 3 + 4
IF LEN(TST) # LEN("7") THEN DEBUG

So PICK's typelessness was quite useful there.

After too much chasing of wild geese, I finally discovered that the
cluprit was a single file's D-Pointer and when I skipped the writev (or
write after modifying the program to read the item, place the value in
att<13> then write the whole item), everything became Kosher again.

While I was unable to put a finger on the precise point in writev and
write where this occurred, it did indeed happen in the write or writev
call. Putting a simple counter in the BASIC line after the write, the
counter would go from say 3813 to 38130000 after the write(v) call.

Further noticing that this file had two B-Trees defined, I deleted the
indices - and suddenly the problem disappeared.

This answered a question that has nagged me since this discovery: "Why
would this have suddenly begun happening?". Within the last month,
staff were experimenting with getting rid of some old R83 homebrew
index files and had been testing D3's B-Trees in this file.

The irony is that one of the big disappointments for me in AP was the
ongoing problem of B-Trees never working as expected ... back in 1992
or so. Live and (don't) learn I guess. LOL

Does this mean that one should never attempt to use D3's B-Trees?

Since I would expect few application programs to be writing to the FDI,
maybe it isn't an issue in production.

On the other hand such completely erratic behavior is pretty
frightening regardless of whether we will intentionally go where it is
known to happen. Where might other hidden treasure reside? Your CFO
might like having some GL entries multiplied by 10,000 but the auditors
would most likely frown upon this. <g>

Thanks again everyone,
-Tom

Quote:

Hello All,

One of my clients' systems has exhibited an odd problem. After the RD
supplied utility F-RESIZE is run, some (many) of the updated D-Pointers
contain 130000 (one hundred thirty thousand) attributes and the
suggested modulo has been placed in attribute<130000>. I am attempting
to find a pattern in which are written this way and which aren't but
with no luck so far.

Suspecting that someone had played with the program, I have recompiled
it. When that didn't fix anything, I decataloged it, moved the existing
program and restored dm,bp,f-resize from the FILES supplied with the
system. I have compared our program with the one from the FILES dm
account as well as with the program from a D3/NT system that someone
sent me. They are identical.

The line in the program that writes the suggested modulo is:
147 if dot else writev odd on fv.q,writem,13

I have removed the line and typed it manually; exported it and
hexdumped it ... there are no hidden characters that I can find. Also,
placing a debug statement before this line shows no garbage in the
"odd" variable; just the modulo enclosed in parens.

As another test, I added a line:
148 if dot else writev odd on fv.q,writem,17

Now those items contain 170000 attributes. Yikes, now _that_ is what I
call scalability! <g

Thinking that something is broken in the system and causing this, I
wrote a little test program:

FN = 'ACCOUNT,TESTFILE,'
OPEN '',FN TO TEST ELSE STOP 'TEST'

READ ITM FROM TEST, 'TESTING' THEN
WRITEV '(4101)' ON TEST, 'TESTING', 13
END ELSE
ITM = 'TEST'
ITM<2> = 'ATT 2'
ITM<3> = 'ATT 3'
ITM<17> = 'Comment'
WRITE ITM ON TEST, 'TESTING'
END

END

Running this twice (once to populate the item, the other to do a
WRITEV), it acts exactly as one would expect it to: the item 'TESTING'
contains 17 attributes, with att<13> = '(4101)'

Has anyone else experienced this? Has anyone any ideas or suggestions?

The system is:
D3 7.4.0
But the ABS and FILES are:
D3_abs-lx-7.4.7-.Z
D3_datafiles-lx-7.4.7-.Z

So upgrades have apparently been performed.

RH 9 2.4.20-8smp

TIA,
-Tom

#12

Tom deL wrote:

Quote:

Does this mean that one should never attempt to use D3's B-Trees?

We gave up on them years ago. Too flakey. I think some people trust
them now in certain versions, but it just wasn't worth it to us.

--
Kevin Powick

#13

Tom,

I think you deserve some kind of award for this comprehesive debugging coup
(and kudos to Mark Brown too)!

I suggest to RD that you get a free year's service for finding this
disaster.

Chandru

"Tom deL" <ted (AT) blackflute (DOT) com> wrote

Quote:

Many thanks to everyone for the input. A resolution of sorts has been
found and I will post as much detail as possible for the next poor
schmuck looking for an answer to this.

Updating the system patches made no difference (but surely wasn't
wasted time <g>).

The original symptom of writing 130,000 attributes into the FDI's was
indeed as Mark suggested a scaling issue but a PRECISION statement did
not cure the problem.

In debugging this, it became apparent that at some point in the
f-resize main loop the system would go into some weird "MR4" mode. Once
in this mode, it continued through completion. This affected the
representation of constants even so that a check such as:
TST = 3 + 4
IF TST # 7 THEN DEBUG

Would never fail (even though TST would actually be 70000) because the
constant "7" was being rendered as if it were ICONV(7, "MR4"). The
check that would work was:
TST = 3 + 4
IF LEN(TST) # LEN("7") THEN DEBUG

So PICK's typelessness was quite useful there.

After too much chasing of wild geese, I finally discovered that the
cluprit was a single file's D-Pointer and when I skipped the writev (or
write after modifying the program to read the item, place the value in
att<13> then write the whole item), everything became Kosher again.

While I was unable to put a finger on the precise point in writev and
write where this occurred, it did indeed happen in the write or writev
call. Putting a simple counter in the BASIC line after the write, the
counter would go from say 3813 to 38130000 after the write(v) call.

Further noticing that this file had two B-Trees defined, I deleted the
indices - and suddenly the problem disappeared.

This answered a question that has nagged me since this discovery: "Why
would this have suddenly begun happening?". Within the last month,
staff were experimenting with getting rid of some old R83 homebrew
index files and had been testing D3's B-Trees in this file.

The irony is that one of the big disappointments for me in AP was the
ongoing problem of B-Trees never working as expected ... back in 1992
or so. Live and (don't) learn I guess. LOL

Does this mean that one should never attempt to use D3's B-Trees?

Since I would expect few application programs to be writing to the FDI,
maybe it isn't an issue in production.

On the other hand such completely erratic behavior is pretty
frightening regardless of whether we will intentionally go where it is
known to happen. Where might other hidden treasure reside? Your CFO
might like having some GL entries multiplied by 10,000 but the auditors
would most likely frown upon this. <g

Thanks again everyone,
-Tom

Hello All,

One of my clients' systems has exhibited an odd problem. After the RD
supplied utility F-RESIZE is run, some (many) of the updated D-Pointers
contain 130000 (one hundred thirty thousand) attributes and the
suggested modulo has been placed in attribute<130000>. I am attempting
to find a pattern in which are written this way and which aren't but
with no luck so far.

Suspecting that someone had played with the program, I have recompiled
it. When that didn't fix anything, I decataloged it, moved the existing
program and restored dm,bp,f-resize from the FILES supplied with the
system. I have compared our program with the one from the FILES dm
account as well as with the program from a D3/NT system that someone
sent me. They are identical.

The line in the program that writes the suggested modulo is:
147 if dot else writev odd on fv.q,writem,13

I have removed the line and typed it manually; exported it and
hexdumped it ... there are no hidden characters that I can find. Also,
placing a debug statement before this line shows no garbage in the
"odd" variable; just the modulo enclosed in parens.

As another test, I added a line:
148 if dot else writev odd on fv.q,writem,17

Now those items contain 170000 attributes. Yikes, now _that_ is what I
call scalability! <g

Thinking that something is broken in the system and causing this, I
wrote a little test program:

FN = 'ACCOUNT,TESTFILE,'
OPEN '',FN TO TEST ELSE STOP 'TEST'

READ ITM FROM TEST, 'TESTING' THEN
WRITEV '(4101)' ON TEST, 'TESTING', 13
END ELSE
ITM = 'TEST'
ITM<2> = 'ATT 2'
ITM<3> = 'ATT 3'
ITM<17> = 'Comment'
WRITE ITM ON TEST, 'TESTING'
END

END

Running this twice (once to populate the item, the other to do a
WRITEV), it acts exactly as one would expect it to: the item 'TESTING'
contains 17 attributes, with att<13> = '(4101)'

Has anyone else experienced this? Has anyone any ideas or suggestions?

The system is:
D3 7.4.0
But the ABS and FILES are:
D3_abs-lx-7.4.7-.Z
D3_datafiles-lx-7.4.7-.Z

So upgrades have apparently been performed.

RH 9 2.4.20-8smp

TIA,
-Tom

#14

Quote:

The line in the program that writes the suggested modulo is:
147 if dot else writev odd on fv.q,writem,13

Did you check the value of "odd" ?
It should not have any @FMs in it.

#15

Thanks, but a couple of questions still remain.

What were the index a-correlative(s) that caused this to happen.

For you who don't know, when D3 started to support calling sub routines from
dictionary items and CALLX and whatnot, they had to create an interface
between English and Basic so the subroutine could be called and there was a
structure that looked like a Basic mainline program for it to return to.

I would guess (totally without proof) that they put the Basic ScaleFactor
variable into this common basic control block to make it available to all
code called within the system instead of having to carry it (and other
variables) twice.

And if an index caused the problem, then it's probably because the index
correlative compilation overflowed its space.

Just conjecture.

Tom, glad things worked out.

Mark Brown

"murthi" <c_xyz_murthi (AT) seeing_xyz_green (DOT) net> wrote

Quote:

Tom,

I think you deserve some kind of award for this comprehesive debugging
coup (and kudos to Mark Brown too)!

I suggest to RD that you get a free year's service for finding this
disaster.

Chandru

#16

Hi Kevin,

Quote:

Does this mean that one should never attempt to use D3's B-Trees?

We gave up on them years ago. Too flakey. I think some people trust
them now in certain versions, but it just wasn't worth it to us.

Yes, I think that especially after this I will suggest my clients stay
away from this apparently dangerous technology.
-Tom

#17

Hi Chandru,

Quote:

I think you deserve some kind of award for this comprehesive debugging coup
(and kudos to Mark Brown too)!

I suggest to RD that you get a free year's service for finding this
disaster.

Thank you for the sentiment. For me, not having to live in a cold sweat
over what might be happening in the bowls of a client's DBMS is reward
enough for me - at least until the next bit of weird behavior rears its
head.

<Current Events>
Today was a day without migration ... soon perhaps we will have
migration. <g>
</Current Events>
-Tom

#18

Hi Mark,

Quote:

What were the index a-correlative(s) that caused this to happen.

From memory because I don't have them right here, they were both fairly
common text extraction/lookup correlatives along the lines of:

A1:"-":0R(G2-1)(TORDERS;X;80;80)

One to extract the vendor number, the other a product description as I
remember. This was the line item file and the lookups were into the
orders and product files.

Quote:

For you who don't know, when D3 started to support calling sub routines from
dictionary items and CALLX and whatnot, they had to create an interface
between English and Basic so the subroutine could be called and there was a
structure that looked like a Basic mainline program for it to return to.

I would guess (totally without proof) that they put the Basic ScaleFactor
variable into this common basic control block to make it available to all
code called within the system instead of having to carry it (and other
variables) twice.

And if an index caused the problem, then it's probably because the index
correlative compilation overflowed its space.

That is possible and I don't pretend to know how the internals work.
But would this be a factor when writing to the FDI? It doesn't seem as
if an index would be updated or its correlative would be compiled
during a writev to the FDI?

At least during my testing over the weekend I am sure that the indexed
attributes were not being updated while I was working because no one
else was on the system.

Quote:

Tom, glad things worked out.

Thank you, me too <g>
-Tom

#19

Hi Lucian,

Quote:

The line in the program that writes the suggested modulo is:
147 if dot else writev odd on fv.q,writem,13

Did you check the value of "odd" ?
It should not have any @FMs in it.

Thank you. Yes that was one of the first things that I checked. A
couple of posts back I outlined the factors causing this problem.
-Tom

#20

Hi
While we are on the complexities of file resizing - and I must admit I am
glad I am not the one to have had this horror happen so thank you all. When
the FSI was introduced I Understood that all items over a particular size ,
around 900 bytes if I recall, would be replaced by an address item in the
proper position in the hash file and the real item would be flipped off to
some more permanent store position in overflow. This raises the question
when dealing with a significant number of large items such as a BP file
should one size it for the actual bytes stored giving a modulo of 6000 to
10000 or more or should one size it based on the number of items say 1000 at
about 10 or 20 bytes each and giving a modulo of say 19.
If one could dump an FSI block one could check but unfortunately that
feature did not happen. I still use dump if I need to locate a funny
character by copying the item into a VME file but that of course does not
allow me to look at the structure of an FSI file.
Peter McMurray
"Mark Brown" <mbrown (AT) drexelmgt (DOT) com> wrote

Quote:

Thanks, but a couple of questions still remain.

What were the index a-correlative(s) that caused this to happen.

For you who don't know, when D3 started to support calling sub routines
from
dictionary items and CALLX and whatnot, they had to create an interface
between English and Basic so the subroutine could be called and there was
a
structure that looked like a Basic mainline program for it to return to.

I would guess (totally without proof) that they put the Basic ScaleFactor
variable into this common basic control block to make it available to all
code called within the system instead of having to carry it (and other
variables) twice.

And if an index caused the problem, then it's probably because the index
correlative compilation overflowed its space.

Just conjecture.

Tom, glad things worked out.

Mark Brown

"murthi" <c_xyz_murthi (AT) seeing_xyz_green (DOT) net> wrote in message
news:JAq5g.2143$HN3.201 (AT) trndny01 (DOT) ..
Tom,

I think you deserve some kind of award for this comprehesive debugging
coup (and kudos to Mark Brown too)!

I suggest to RD that you get a free year's service for finding this
disaster.

Chandru