dbTalk Databases Forums  

How wasteful is READV... WRITEV...?

comp.databases.pick comp.databases.pick


Discuss How wasteful is READV... WRITEV...? in the comp.databases.pick forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Bob Dubery
 
Posts: n/a

Default How wasteful is READV... WRITEV...? - 01-27-2010 , 12:45 AM






I am working in Universe 10.something. OS is Linux.

I noticed this in a program I was working on....
READVU BILL.AMT FROM ADCREDINFO,KEY,39 ELSE BILL.AMT = 0
BILL.AMT<1,1> += DIFF
WRITEV BILL.AMT ON ADCREDINFO,KEY,39

That got me tut-tutting because the writev implies another read
anyway, so we're reading that record and that group that it is in
TWICE.

So I started thinking about something like...
READU ADREC FROM ADCREDINFO,KEY ELSE ADREC = ""
BILL.AMT = ADREC<39,1>
BILL.AMT += DIFF
ADREC<39,1> = BILL.AMT
WRITE ADREC ON ADCREDINFO,KEY

But then I thought about it, and wondered if that would make such a
difference anyway.

Surely when the WRITE is executed we again read the group into memory,
update the group and write it away again.

Long ago I was taught that this kind of READV.... WRITEV.... is
wasteful in terms of disc I/O. But now I'm not so sure.

Reply With Quote
  #2  
Old   
Bob Dubery
 
Posts: n/a

Default Re: How wasteful is READV... WRITEV...? - 01-27-2010 , 12:57 AM






On Jan 27, 8:45*am, Bob Dubery <megap... (AT) gmail (DOT) com> wrote:

Quote:
I noticed this in a program I was working on....
READVU BILL.AMT FROM ADCREDINFO,KEY,39 ELSE BILL.AMT = 0
BILL.AMT<1,1> += DIFF
WRITEV BILL.AMT ON ADCREDINFO,KEY,39

That got me tut-tutting because the writev implies another read
anyway, so we're reading that record and that group that it is in
TWICE.

So I started thinking about something like...
* * * READU ADREC FROM ADCREDINFO,KEY ELSE ADREC = ""
* * * BILL.AMT = ADREC<39,1
* * * BILL.AMT += DIFF
* * * ADREC<39,1> = BILL.AMT
* * * WRITE ADREC ON ADCREDINFO,KEY

But then I thought about it, and wondered if that would make such a
difference anyway.

Surely when the WRITE is executed we again read the group into memory,
update the group and write it away again.

Long ago I was taught that this kind of READV.... WRITEV.... is
wasteful in terms of disc I/O. But now I'm not so sure.
VLIST doesn't shed much light.

The READVU... WRITEVU generates
0B1CE : 274 reapvu FILEVAR [19] KEY 39 => BILL.AMT
0B1D8 : 0C8 jumpt 0B1E6:
0B1E0 : 0F8 move 0 => BILL.AMT
0B1E6 : 060 dyn_extract BILL.AMT 1 1 0 => $R607
0B1F2 : 004 add $R607 DIFF => $R608
0B1FA : 064 dyn_replace BILL.AMT 1 1 0 $R608 => BILL.AMT
0B208 : 212 writev BILL.AMT FILEVAR [19] KEY 39

The READU.... WRITEU generates
0B212 : 270 reapu FILEVAR [19] KEY => ADREC
0B21A : 0C8 jumpt 0B226:
0B220 : 0F8 move "" => ADREC
0B226 : 060 dyn_extract ADREC 39 1 0 => BILL.AMT
0B232 : 004 add BILL.AMT DIFF => BILL.AMT
0B23A : 064 dyn_replace ADREC 39 1 0 BILL.AMT => ADREC
0B248 : 208 write ADREC FILEVAR [19] KEY

So again it comes down to the amount of I/O that a writev and a write
require.

Reply With Quote
  #3  
Old   
Charlie Noah
 
Posts: n/a

Default Re: How wasteful is READV... WRITEV...? - 01-27-2010 , 04:50 AM



On Jan 27, 12:57*am, Bob Dubery <megap... (AT) gmail (DOT) com> wrote:
Quote:
On Jan 27, 8:45*am, Bob Dubery <megap... (AT) gmail (DOT) com> wrote:



I noticed this in a program I was working on....
READVU BILL.AMT FROM ADCREDINFO,KEY,39 ELSE BILL.AMT = 0
BILL.AMT<1,1> += DIFF
WRITEV BILL.AMT ON ADCREDINFO,KEY,39

That got me tut-tutting because the writev implies another read
anyway, so we're reading that record and that group that it is in
TWICE.

So I started thinking about something like...
* * * READU ADREC FROM ADCREDINFO,KEY ELSE ADREC = ""
* * * BILL.AMT = ADREC<39,1
* * * BILL.AMT += DIFF
* * * ADREC<39,1> = BILL.AMT
* * * WRITE ADREC ON ADCREDINFO,KEY

But then I thought about it, and wondered if that would make such a
difference anyway.

Surely when the WRITE is executed we again read the group into memory,
update the group and write it away again.

Long ago I was taught that this kind of READV.... WRITEV.... is
wasteful in terms of disc I/O. But now I'm not so sure.

VLIST doesn't shed much light.

The READVU... WRITEVU generates
0B1CE : 274 reapvu * * * * FILEVAR [19] KEY 39 *=> BILL.AMT
0B1D8 : 0C8 jumpt * * * * *0B1E6:
0B1E0 : 0F8 move * * * * * 0 *=> BILL.AMT
0B1E6 : 060 dyn_extract * *BILL.AMT 1 1 0 *=> $R607
0B1F2 : 004 add * * * * * *$R607 DIFF *=> $R608
0B1FA : 064 dyn_replace * *BILL.AMT 1 1 0 $R608 *=> BILL.AMT
0B208 : 212 writev * * * * BILL.AMT FILEVAR [19] KEY 39

The READU.... WRITEU generates
0B212 : 270 reapu * * * * *FILEVAR [19] KEY *=> ADREC
0B21A : 0C8 jumpt * * * * *0B226:
0B220 : 0F8 move * * * * * "" *=> ADREC
0B226 : 060 dyn_extract * *ADREC 39 1 0 *=> BILL.AMT
0B232 : 004 add * * * * * *BILL.AMT DIFF *=> BILL.AMT
0B23A : 064 dyn_replace * *ADREC 39 1 0 BILL.AMT *=> ADREC
0B248 : 208 write * * * * *ADREC FILEVAR [19] KEY

So again it comes down to the amount of I/O that a writev and a write
require.
There is another consideration here. We have thousands of programs,
not all very well written. If we want to know which programs update a
particular attribute, we use a finder paragraph to search for the item
name and attribute number. Ex: "FIND BP CUSTOMER.ITEM(39)
CUSTOMER.ITEM<39> CUSTOMER.ITEM<39,". Of course we then have to look
for CUSTOMER.REC, CUST.REC, CUSTREC, CUST, CUS, ad nauseum. My
favorite (not) is just REC. Yep, we have that, too. Over time we've
identified most all the different ways our major files are referenced.
The real bear is WRITEV. Since the file handle and ID could be called
anything, it's very difficult to locate updates programmatically.
Whether READV and WRITEV are more efficient by microseconds or are
less, for us they are the pits.

I'd welcome any solutions that don't involve poultry sacrifice or
dancing nude in the moonlight (you REALLY don't want to see that).

Regards,
Charlie Noah
Inland Truck Parts

Reply With Quote
  #4  
Old   
Martin Phillips
 
Posts: n/a

Default Re: How wasteful is READV... WRITEV...? - 01-27-2010 , 05:52 AM



Actual database access is at the record level. Even though it happens
deep inside the database engine, READV is effectively equivalent to a
READ followed by a field extraction.

As far as I am aware (and I am open to argument), none of the
multivalue platforms is clever enough to spot that something like
READV F2 FROM FVAR, ID, 2 ELSE ...
READV F7 FROM FVAR, ID, 7 ELSE ...
could actually do just one read and extract the two fields. It does,
of course, potentially gain from record/group level caching where this
is available.

As a general rule, if you want to extract two or more fields, do the
READ and extract the fields yourself.

The one place where READV is really valuable is with field 0 to
determine whether a record exists. Although this still has to go to
the file to look for the record, it doesn't actually read it into
memory. This becomes really important with huge records, especially
those sometimes found in directory type files that might be many
megabytes long.

Understanding WRITEV is even more important. A WRITEV is effectively a
READ followed by a field replacement and a WRITE. This is fine for a
single use but multiple WRITEVs become very inefficient.


Martin Phillips, Ladybridge Systems

Reply With Quote
  #5  
Old   
Kevin King
 
Posts: n/a

Default Re: How wasteful is READV... WRITEV...? - 01-27-2010 , 08:57 AM



I just want to second what Martin said. WRITEV is a syntactic
convenience and a performance nightmare. The problem isn't one
WRITEV, it's the slippery slope that happens six months later when you
find crap like this in the program:

READV X1 FROM F.CUST,CUST.ID,21 ELSE X1 = 0
READV X2 FROM F.CUST,CUST.ID,22 ELSE X2 = 0
READV X3 FROM F.CUST,CUST.ID,23 ELSE X3 = 0
READV X4 FROM F.CUST,CUST.ID,24 ELSE X4 = 0
*
....some calculations
*
WRITEV X1 ON F.CUST,CUST.ID,21
WRITEV X2 ON F.CUST,CUST.ID,22
WRITEV X3 ON F.CUST,CUST.ID,23
WRITEV X4 ON F.CUST,CUST.ID,24

At some point someone says "well, the last guy used a WRITEV so it
must be okay" and then you end up with stuff like the above crap that
performs badly and doesn't read so hot either.

Reply With Quote
  #6  
Old   
Bob Dubery
 
Posts: n/a

Default Re: How wasteful is READV... WRITEV...? - 01-27-2010 , 10:03 AM



On Jan 27, 12:50*pm, Charlie Noah <cwn... (AT) comcast (DOT) net> wrote:

Quote:
There is another consideration here. We have thousands of programs,
not all very well written. If we want to know which programs update a
particular attribute, we use a finder paragraph to search for the item
name and attribute number. Ex: "FIND BP CUSTOMER.ITEM(39)
CUSTOMER.ITEM<39> CUSTOMER.ITEM<39,". Of course we then have to look
for CUSTOMER.REC, CUST.REC, CUSTREC, CUST, CUS, ad nauseum. My
favorite (not) is just REC. Yep, we have that, too.
You'd love this program then. Somewhere up near the top of this
listing it has something like

EQU ADCREDINFO TO FILEVAR(18).

Reply With Quote
  #7  
Old   
Ed Sheehan
 
Posts: n/a

Default Re: How wasteful is READV... WRITEV...? - 01-27-2010 , 10:28 AM



See below...

"Martin Phillips" <MartinPhillips (AT) ladybridge (DOT) com> wrote

Quote:
Actual database access is at the record level. Even though it happens
deep inside the database engine, READV is effectively equivalent to a
READ followed by a field extraction.

As far as I am aware (and I am open to argument), none of the
multivalue platforms is clever enough to spot that something like
READV F2 FROM FVAR, ID, 2 ELSE ...
READV F7 FROM FVAR, ID, 7 ELSE ...
could actually do just one read and extract the two fields. It does,
of course, potentially gain from record/group level caching where this
is available.

As a general rule, if you want to extract two or more fields, do the
READ and extract the fields yourself.

The one place where READV is really valuable is with field 0 to
determine whether a record exists. Although this still has to go to
the file to look for the record, it doesn't actually read it into
memory. This becomes really important with huge records, especially
those sometimes found in directory type files that might be many
megabytes long.
snip

Do all platforms support READV of attr 0? I'm thinking D3 doesn't, or at
least didn't.

And as far as limiting memory usage by reading attr 0, I believe the entire
record is always read into memory, and just the requested attribute is
copied into process workspace. Maybe it's just a slight improvement after
all.

Ed

Reply With Quote
  #8  
Old   
Bob Dubery
 
Posts: n/a

Default Re: How wasteful is READV... WRITEV...? - 01-27-2010 , 03:20 PM



On Jan 27, 6:28*pm, "Ed Sheehan" <NOedsS... (AT) xmission (DOT) com> wrote:

Quote:
And as far as limiting memory usage by reading attr 0, I believe the entire
record is always read into memory, and just the requested attribute is
copied into process workspace.
There is no attribute zero. At least, if I understand correctly, not
on UniVerse. The key is stored separately from the record. When you do
a READ UV finds the key and then finds the record AFTER the key, so
READV with a zero might be the quickest way to test for the presence
of a record. I've heard that it is, but I've never tested it.

Reply With Quote
  #9  
Old   
Tony Gravagno
 
Posts: n/a

Default Re: How wasteful is READV... WRITEV...? - 01-27-2010 , 03:43 PM



"Ed Sheehan" wrote:
Quote:
Do all platforms support READV of attr 0? I'm thinking D3 doesn't, or at
least didn't.

And as far as limiting memory usage by reading attr 0, I believe the entire
record is always read into memory, and just the requested attribute is
copied into process workspace. Maybe it's just a slight improvement after
all.
D3 does not support READV/0.

I hope I have this right - I'll defer to someone with a clue...

We need to separate the concept of memory and workspace here.

If you have large items then READV/WRITEV prevent reading the entire
buffer into BASIC workspace. When this happens with a whole item,
BASIC is going to suck in as many frames as it needs to hold the
variable. If you're working with FlashBASIC then this is all done in
memory. But at the DBMS level, all of the data needs to be pulled
from disk into memory anyway (not BASIC workspace) so that it can be
scanned for attribute marks. So while you save something in post-read
frame manipulation, you gain nothing in disk IO access time.

With WRITEV, a smarter MV system will not read the item into workspace
before the update, it will count the number of bytes in the current
attribute on disk and replace the attribute in-line if it matches the
number of new bytes. If the byte count is different then the DBMS
needs to shift frames up or down, pretty much negating the imagined
economy of a WRITEV. Every platform does this differently.

HTH
T

Reply With Quote
  #10  
Old   
Tony Gravagno
 
Posts: n/a

Default Re: How wasteful is READV... WRITEV...? - 01-27-2010 , 03:43 PM



Bob Dubery wrote:
Quote:
You'd love this program then. Somewhere up near the top of this
listing it has something like

EQU ADCREDINFO TO FILEVAR(18).
Two things:
1) The above EQUATE to a dimensioned array is very efficient for
handling a large number of files in bulk compared to dealing with each
one individually by name. It's usually used in code generators. Same
goes for referencing attributes:
EQU AR.CURRENT.BALANCE TO 43
EQU CUST.LAST.PAYMENT TO 44
EQU CUST.FILE to FILEVAR(18)
FOR FNUM = 1 TO 99
OPEN FNAMES<FNUM> TO FILEVAR(FNUM) ELSE STOP
NEXT FNUM ; * Note 3 lines, not 99
INVOICE<AR.CURRENT.BALANCE> = BAL
WRITEV BAL ON CUST.FILE,CUST.ID,CUST.LAST.PAYMENT

I'm not saying it's pretty. I'm saying it's a valid technique for
specific kinds of work.

2) To find file usage, check mvScan for U2 at brianleach.co.uk.

T

Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.