dbTalk Databases Forums  

D3 b-tree problem

comp.databases.pick comp.databases.pick


Discuss D3 b-tree problem in the comp.databases.pick forum.



Reply
 
Thread Tools Display Modes
  #41  
Old   
stefano
 
Posts: n/a

Default Re: D3 b-tree problem - 01-17-2006 , 05:15 AM






Luke Webber ha scritto:
Quote:
Scott Ballinger wrote:
stefano wrote:
The limit is for duplicate keys within a single item.
ftp://ftp.rainingdata.com/pub/Linux/...ux73xa110.html
Action # 29364
Corrected a problem with indexes that caused index corruption
if a single item contained more than 4095 identical values.
The new limit is 65535.

Regards,
Stefano Maran

This makes sense! I also understood the limit to be 64K unique ids per
indexed value, which made the D3 indexes somewhat less useful. But I
have a site where some btree indexed files have way exceeded 64K ids
per indexed value without any problems. I thought perhaps we were just
lucky? But maybe there isn't a problem in the first place. Some
clarification from RD would be helpful here.

OK, I have some clarification from RDTA via T Data, and the news ain't
great. The limit of 64K keys is as originally interpreted. That is to
say that a file with >64K items with the same non-unique key.

And as to what happens when you exceed that limit, all they can tell me
is that you get "unpredictable results".

What is this, the freakin' Stone Age? Let me out of here!

OK, I have my workaround, but I'm pretty pissed off here.

And while I'm on the subject, why is it that the CREATE-INDEX verb only
shows six digits in its count? Don't they realise that there are going
to be files > 10 million items in size? Thinking small, that's the whole
problem here.

Luke
Luke,
which version of D3 are You running?

Based on my tests:
D3/Linux 7.2.0 - 6 digits
D3/Nt 7.3.6 - 7 digits


On D3/Nt 7.3.6 I conducted the following test:

CREATE-FILE TEST 1 1999

CREATE-INDEX TESTFILE A1

Polulated the file with 4,000,000 items (item-id 1..4000000) with
a single byte in attribute 1 in the range A..Z to have, more or
less, 130,000 items with the same repeated key.

Executed 500,000 readu/write and/or readu/delete/write operations
randomly changing attribute 1 (always in the range A..Z)

Executed select and sselect with and without the use of indexies
and then compared the results: no differences.

Maybe I'm very lucky. ;-)

I know this test is a proof of nothing but, if the limit is for the
whole file, I'll expect some kind of corruption.


Stefano


Reply With Quote
  #42  
Old   
murthi
 
Posts: n/a

Default Re: D3 b-tree problem - 01-17-2006 , 08:38 AM






And of course, compared to Assembler, the UP is not a stone ax(e)? No wonder
RD is so behind the times!

As one who had dealt with the Pick assembler for 15+ years, I must say it's
boring to hear it being blamed for not fixing anything. The fact of the
matter is that any competent programmer can learn the old innards given a
little time. And the entire system can, indeed, be reprogrammed in whatever
language you choose. Vide Universe, Unidata, Univision et al.(aside- maybe
RD needs to change their name again?) So let's not confuse possibility with
inability and inaction, as RD is solidly in the latter camp.

Chandru

"Tony Gravagno" <g6q3x9lu53001 (AT) sneakemail (DOT) com.invalid> wrote

Quote:
"Excalibur" wrote:
I am fascinated by the remark about the company running on UPDATE because
I
understood Rick Davies to say that they were dumping it, maybe he just
meant the VME.

Rick is there, I'm not, whatever he says is gospel ... except for the
rare instance when he's really wrong. They might _want_ to dump
it, but over 10 years of development effort has been invested in the
RD apps based on UP. They might be looking at about as much time to
dump /replace it. No clue, don't care.

T



Reply With Quote
  #43  
Old   
Dale Benedict
 
Posts: n/a

Default Re: D3 b-tree problem - 01-17-2006 , 02:48 PM



I've been reading this thread for a while and I guess I'll put on my flame
suit and put in my 2 cents worth.

But let me preface this with two things...

1. I don't like the poor documentation regarding limitations within RD's
Pick system.
2. Blowing off steam in a public form is not conducive to spreading the
message of how great a MV system is.

People need to learn the limitations of any tool they use, and then don't
try to push the limits of the tool. Pushing a tool past the limits of what
it can handle give 'unexpected results'!

As for Luke's problem, better documentation may have prevented him from
blowing off steam...

I learned very early in the release of AP and D3 the limits of the indexing
function and did my best to setup my indexes to best exploit this feature.
I been able to take the selection process of access statement from 15 to 20
minutes down to seconds. I mostly use indexes just in a pre-selection
process to gather a list of keys that offer the possibility of holding the
information I need. Then I use a second access (AQL) statement to further
refine the selection. This process works well.

When I need to scan through indexes spanning dates, I usually resort to
using a BASIC program and the root and key statements to cycle through the
dates to gather the required keys. This is done because AQL requires an
equal sign (=) in order to use the Pick indexes.

Some people have put forth a good approach for creating an index where the
index key ends in YYYY or YYYYMMDD, and thereby allow the selection of a
years, months, etc worth of keys in short order and allows a second AQL
statement to further refine the select. This is a very smart idea.

I've found in my work that many people are overlooking the possibility of
using D3's ability to see the spooler entries as a regular item. The
spooler an easy and very fast way of building a huge string. I've found
that using the spooler in this fashion can speed up the processing of
building reports for export by a huge amount.

Is anybody using the ASSIGNFQ function? There is an example in the manual
of spooler reports directly to a operating system file, and it works well
under Linux.

Ya gotta know the limits of the tools you work with!!!

Come on folks! Let's stop throwing dirt! It just makes us look like we
have a dirty system. And who wants to buy a dirty system. People keep
complaining about the marketing effort of the system makers and the VAR's.
But there seems to be a whole lot of anti-marketing crap being spread
through the newsgroup.

Whether or not possible clients ever look at CDP to see if we have a good
back-end, you should be writing your posting like if the do. You will never
know if you've blown off a possible client and income by the rhetoric that
gets posted here.

I'm sounding too much like Tony!

Regards,

Dale

"Luke Webber" <luke (AT) webber (DOT) com.au> wrote

Quote:
I have a client who is experiencing some problems with SELECTs on a very
large file with two B-tree indices.

This baby has around 28 million items in it, and when we try to select
them based on the client code (which is indexed), we come up with too
few items. The indices were rebuilt just a couple of weeks ago, and I'd
rather not have to go through that again (big file, takes a while). So I
thought I'd check with the group to see if there are any known problem
with huge files and indexed selects.

I'm running a VERIFY-INDEX at present, but I'm not sure that the damned
thing works all that well.

Any thoughts?

Cheers,
Luke



Reply With Quote
  #44  
Old   
Luke Webber
 
Posts: n/a

Default Re: D3 b-tree problem - 01-17-2006 , 04:17 PM



stefano wrote:

Quote:
Luke,
which version of D3 are You running?

Based on my tests:
D3/Linux 7.2.0 - 6 digits
D3/Nt 7.3.6 - 7 digits
Whups! I said six digits, but it was actually seven. Still not enough,
though, because the file had 28 million records in it, and that's eight
digits. Weird that they should extend the size by one lousy digit. Why
not just remove the limit entirely?

Quote:
On D3/Nt 7.3.6 I conducted the following test:

CREATE-FILE TEST 1 1999

CREATE-INDEX TESTFILE A1

Polulated the file with 4,000,000 items (item-id 1..4000000) with
a single byte in attribute 1 in the range A..Z to have, more or
less, 130,000 items with the same repeated key.

Executed 500,000 readu/write and/or readu/delete/write operations
randomly changing attribute 1 (always in the range A..Z)

Executed select and sselect with and without the use of indexies
and then compared the results: no differences.

Maybe I'm very lucky. ;-)

I know this test is a proof of nothing but, if the limit is for the
whole file, I'll expect some kind of corruption.
The words "unpredictable results" are kind of chilling, aren't they?
Could take in any result from "working perfectly" to "Hey, where has my
data gone?"

Luke


Reply With Quote
  #45  
Old   
Tony Gravagno
 
Posts: n/a

Default Re: D3 b-tree problem - 01-18-2006 , 03:19 PM



"Dale Benedict" wrote:
Quote:
I'm sounding too much like Tony!
LOL, maybe that's why until I got down to this line I was thinking to
myself "way to go Dale" "yeah, you tell um bud" ... Gee, only one
dinner with this choir boy and you're singin the gospel, maybe I've
been approaching this forum thing all wrong.

Seriously, all good advice, and something to remember right before we
click that Send button.

BTW, I'm not sure if anyone noticed but RD did add some enhancements
around 7.4.x to allow indexed date range selection and some other
goodies. I haven't used the mods, can't comment.

T


Reply With Quote
  #46  
Old   
Bill H
 
Posts: n/a

Default Re: D3 b-tree problem - 01-19-2006 , 01:02 AM



Tony:

I'm not sure this works, as I still cannot access indexes except with an
equality. :-(

But then I don't index dates. :-)

Bill

"Tony Gravagno" <g6q3x9lu53001 (AT) sneakemail (DOT) com.invalid> wrote ...

[snipped]

Quote:
BTW, I'm not sure if anyone noticed but RD did add some enhancements
around 7.4.x to allow indexed date range selection and some other
goodies. I haven't used the mods, can't comment.

T



Reply With Quote
  #47  
Old   
Dale
 
Posts: n/a

Default Re: D3 b-tree problem - 01-23-2006 , 08:02 PM



You know, the company I'm with doesn't even sell applications, so I have
little vested interest at this time to ensure that Pick databases get a
bigger foothold in the market place. Who knows maybe in the future I'll be
trying to sell software or be with a company who does sell software built
with a Pick database as the back end.

All that has been said in this thread disappoints me. I've worked with
other databases on and off (mostly off) over my career and haven't found any
other database/application programming system that can create a highly
effective and flexible system. I still hear chuckles from around a 'water
cooler' where a company creating their own software to replace the software
that was built in Pick requesting a change. The Pick system was altered in
less than a week, and the flashy software that was being built for them was
going to take an extra 6 month to accommodate the changes.

I learned within weeks of AP/DOS first showing up much of the limitations of
the indexing. On just has to learn the limits of the tool, step back from
the problem to see the big picture and then use the tools at hand to get the
job done as best as possible. The final process may not be the ideal
solution as seen from the programmer's standpoint, but as long as the
solution works and is maintainable, that all that really matters.

Regards,

Dale

"Tony Gravagno" <g6q3x9lu53001 (AT) sneakemail (DOT) com.invalid> wrote

Quote:
"Dale Benedict" wrote:
I'm sounding too much like Tony!

LOL, maybe that's why until I got down to this line I was thinking to
myself "way to go Dale" "yeah, you tell um bud" ... Gee, only one
dinner with this choir boy and you're singin the gospel, maybe I've
been approaching this forum thing all wrong.

Seriously, all good advice, and something to remember right before we
click that Send button.

BTW, I'm not sure if anyone noticed but RD did add some enhancements
around 7.4.x to allow indexed date range selection and some other
goodies. I haven't used the mods, can't comment.

T



Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.