OPEN all files

#11

I suspect that you're really only doing the
"expensive" read once, and the next 9,9999
times the code takes a "shortcut."

--
frosty

"Scott Ballinger" <scott.ballinger (AT) gmail (DOT) com> wrote

Hi Peter,

I think it is a myth that file opens are expensive. Try this:

:ct bp open.test

open.test
001 t = time()
002 for n = 1 to 10000
003 open "sb.temp" to sb.temp else stop 201,"sb.temp"
004 next n
005 print time()-t

On D3 or UV, you can open a file 10,000 times in less than a second.

I would argue that:

A) If your program is slow it's not because you are (re-) opening the
same files too many times, and

B) whatever the minuscule speed advantage you may gain from either
opening all files at once, or caching the opens in common, you will
loose more in increased complexity and opportunity for confusion, and

C) if you implement your "open all files once" scheme via a standard
subroutine call, the CALL itself is probably more expensive than the
OPEN.

That being the case, Tony's rule-of-thumb:

Quote:

1) Open what you need when you need it,
2) but open what you know you are going to need up-front.

.... is the best approach.

Just my opinion.

/Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006

On Jan 29, 5:58 am, mdsi2000 <mdsi2... (AT) yahoo (DOT) com> wrote:

Quote:

Hello Group,
Is there a disadvantage to OPEN all files when a user logs in? Would
there be a relationship or formula based on the system memory?

Thanks,
-Peter G.

#12

On Jan 31, 2:49*am, mdsi2000 <mdsi2... (AT) yahoo (DOT) com> wrote:

Quote:

Ok Guys,
My original question was asked get opinions. I was thinking that if I
redesign a MV package to be all SUBROUTINEs modular, I would be
opening the same files left and right. This may not be noticed on a
system with 100 users using terminals. But if I use those same
SUBROUTINEs in a background process or an overnight process with
millions of records, then it will have a big impact.

After reading these opinions, I'm leaning more that OPENing all files
is the best solution. Hardware speed is becoming less of an issue
these days. Thanks for the opinions and I'll report any pit falls that
may arise.

-Peter G.

Peter,

I have a system that I've been running for about 10 years on various
size client sites which is all subroutines run from a main entry
program which opens all files on the system (about 180) when the user
logs on. I have found that on D3 (Windows), the first user who logs
on for the day has to wait about 10 to 20 seconds, but that subsequent
logons are almost instantaneous. The real plus is that for 10 years I
haven't had to think about opening files.

Geoff

#13

"frosty" wrote:

Quote:

I suspect that you're really only doing the
"expensive" read once, and the next 9,9999
times the code takes a "shortcut."

That's correct. Doing repeated read operations always shows a hit on
the first operation and near-zero access time thereafter. The
shortcut is cached memory. If nothing else, the disk sectors are read
in and still in hardware or OS cache. In a VME environment like D3
the frames are in memory and traversed without having to re-read them
from disk.

As I said on 1/29, I'm not sure how well it does this but I believe D3
goes one step further and caches references to file control blocks.
Any user accessing the same file will get a reference to the FCB frame
so that the system doesn't need to chase MD frames, Q-pointers, or
D-pointers.

T

#14

It wasn't a myth on UD up to version 6 on Aix or windows (as the
really bad code has been optimized I haven't really tested since).

I had a data conversion program (that I inherited) running that was
about 60% through after more than 2 days. It was doing a lot of
subroutine calls that opened the files in the subroutines. I only had
one more day so I re-wrote the routines to open the files into named
common and then started the process over. It ran in only a couple of
hours. The only difference was the subroutines weren't continually
reopening the files.

We also had a lot of dictionaries that called one subroutine that
opened two files. Some reports could call this routine 30 to 50 times
per row. On slower servers watching the reports was like they were
coming over a 1200 baud modem. Putting the files into named common
made them nearly instantaneous.

Try putting the open in an external subroutine so that the compiler
can't optimize it and run your test (do an initial one without doing
the open so you can establish the base line for the call).

I still wouldn't put everything into common up front (our app has over
600 files) but I certainly make sure if I'm calling something a lot I
open things up first and pass them.

my .02 (albeit CDN)
Colin Alfke
Calgary, Canada

On Jan 30, 4:49*pm, Scott Ballinger <scott.ballin... (AT) gmail (DOT) com> wrote:

Quote:

Hi Peter,

I think it is a myth that file opens are expensive. Try this:

:ct bp open.test

* * open.test
001 t = time()
002 for n = 1 to 10000
003 * open "sb.temp" to sb.temp else stop 201,"sb.temp"
004 next n
005 print time()-t

On D3 or UV, you can open a file 10,000 times in less than a second.

I would argue that:

A) If your program is slow it's not because you are (re-) opening the
same files too many times, and

B) whatever the minuscule speed advantage you may gain from either
opening all files at once, or caching the opens in common, you will
loose more in increased complexity and opportunity for confusion, and

C) if you implement your "open all files once" scheme via a standard
subroutine call, the CALL itself is probably more expensive than the
OPEN.

That being the case, Tony's rule-of-thumb:

1) Open what you need when you need it,
2) but open what you know you are going to need up-front.

... is the best approach.

Just my opinion.

/Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006

On Jan 29, 5:58*am, mdsi2000 <mdsi2... (AT) yahoo (DOT) com> wrote:

Hello Group,
Is there a disadvantage to OPEN all files when a user logs in? Would
there be a relationship or formula based on the system memory?

Thanks,
-Peter G.

#15

On Jan 29, 3:58*pm, mdsi2000 <mdsi2... (AT) yahoo (DOT) com> wrote:

Quote:

Hello Group,
Is there a disadvantage to OPEN all files when a user logs in? Would
there be a relationship or formula based on the system memory?

Thanks,
-Peter G.

I think it would depend on the MV platform and O/S that you are using.
Certainly in UV you'd need to tweak some parameters depending on your
estimates of number of users and number of files.

#16

On Feb 7, 8:10*am, Bob Dubery <megap... (AT) gmail (DOT) com> wrote:

Quote:

On Jan 29, 3:58*pm, mdsi2000 <mdsi2... (AT) yahoo (DOT) com> wrote:

Hello Group,
Is there a disadvantage to OPEN all files when a user logs in? Would
there be a relationship or formula based on the system memory?

Thanks,
-Peter G.

I think it would depend on the MV platform and O/S that you are using.
Certainly in UV you'd need to tweak some parameters depending on your
estimates of number of users and number of files.

If you have all of your I/O going through the same subroutine(s) you
can maintain a dynamic array of open file names (which really should
be resolved down to the full path) and an associated dimensioned array
of file variables. This way you don't need to worry about opening
files in your application code - ever - and you can reference files by
name rather than by file variable. The time needed to do a locate of
the file name in your dynamic array to pick up the relevant file
variable is negligible. You will only ever open a file once - the
first time it is used. Also - a world of possibilities open up...
Think of triggers fired when you perform a write (which you're
familiar with), then think of similar capabilites on reads... etc. You
can also venture into the world of data-abstraction - along the lines
of jBASE's jEDI (or whatever it was renamed recently).

#17

On Feb 8, 12:56*pm, Mike Preece <mich... (AT) preece (DOT) net> wrote:

Quote:

If you have all of your I/O going through the same subroutine(s) you
can maintain a dynamic array of open file names (which really should
be resolved down to the full path) and an associated dimensioned array
of file variables. This way you don't need to worry about opening
files in your application code - ever - and you can reference files by
name rather than by file variable.
But surely you'd need to include all the file variables in a common

block - or else your I/O routines are going to be opening files all
the time.

The one problem I can see with having all files opened for all users
all the time is that you now have a lot of file variables open in
memory and not necessarily doing anything.

#18

On Feb 8, 2:56*am, Mike Preece <mich... (AT) preece (DOT) net> wrote:

Quote:

On Feb 7, 8:10*am, Bob Dubery <megap... (AT) gmail (DOT) com> wrote:

On Jan 29, 3:58*pm, mdsi2000 <mdsi2... (AT) yahoo (DOT) com> wrote:

Hello Group,
Is there a disadvantage to OPEN all files when a user logs in? Would
there be a relationship or formula based on the system memory?

Thanks,
-Peter G.

I think it would depend on the MV platform and O/S that you are using.
Certainly in UV you'd need to tweak some parameters depending on your
estimates of number of users and number of files.

If you have all of your I/O going through the same subroutine(s) you
can maintain a dynamic array of open file names (which really should
be resolved down to the full path) and an associated dimensioned array
of file variables. This way you don't need to worry about opening
files in your application code - ever - and you can reference files by
name rather than by file variable. The time needed to do a locate of
the file name in your dynamic array to pick up the relevant file
variable is negligible. You will only ever open a file once - the
first time it is used. Also - a world of possibilities open up...
Think of triggers fired when you perform a write (which you're
familiar with), then think of similar capabilites on reads... etc. You
can also venture into the world of data-abstraction - along the lines
of jBASE's jEDI (or whatever it was renamed recently).

I tried this trick with D3/Linux with 50 files. In my testing, It
turns out that the locate took longer than the open.

As for the original query, it may be easier to open all the files into
a named common. But be wary of what the system does with named
commons. In D3 the named common is retained as you change accounts.
Changing accounts may not be an issue for the original poster, but for
those people that do change accounts and the named common is retained,
you must be sure the clear the named common and re-open the files, or
you may be accessing files in an account the should not be accessed.

Dale

#19

On Feb 14, 5:35*pm, Ross Ferris <ro... (AT) stamina (DOT) com.au> wrote:

Quote:

From what you are saying, the problem lies not with the feature, but
with the people that developed it.

Isn't this usually the case? Many languages have features, such as
global variables, but using them is widely considered poor programming
practice. Persistence of common variables across accounts, in D3
anyway, leads to even greater risks.

Quote:

We have always used COMMON in our business software, and use Global
Common within Visage

Yes, but you have been around since the beginning of your software
development, so I imagine you have a solid understanding of how Common
is used throughout your systems. Many of us are not so lucky.

Since Pick was geared towards "business people" being able to build
their own information systems with little background or skill in
computer programming, that's exactly what you got a lot of the time.

I found an old CDP thread from 2001, that I started, regarding the use
of common blocks. It seems people still feel the way they did 9 years
ago.

http://groups.google.com/group/comp....40eef1dbf6 fc

There are the names of some people that I miss around here in that
thread.

--
Kevin Powick

#20

On Feb 14, 7:15*pm, Kevin Powick <kpow... (AT) gmail (DOT) com> wrote:

Quote:

On Feb 14, 5:35*pm, Ross Ferris <ro... (AT) stamina (DOT) com.au> wrote:

From what you are saying, the problem lies not with the feature, but
with the people that developed it.

Isn't this usually the case? *Many languages have features, such as
global variables, but using them is widely considered poor programming
practice. *Persistence of common variables across accounts, in D3
anyway, leads to even greater risks.

We have always used COMMON in our business software, and use Global
Common within Visage

Yes, but you have been around since the beginning of your software
development, so I imagine you have a solid understanding of how Common
is used throughout your systems. *Many of us are not so lucky.

Since Pick was geared towards "business people" being able to build
their own information systems with little background or skill in
computer programming, that's exactly what you got a lot of the time.

I found an old CDP thread from 2001, that I started, regarding the use
of common blocks. *It seems people still feel the way they did 9 years
ago.

http://groups.google.com/group/comp....se_frm/thread/...

There are the names of some people that I miss around here in that
thread.

--
Kevin Powick

Ross,

There is a vast difference between the COMMON and named COMMON, at
least in D3. I personally try to stay away from the named common, as
the block of variables remain intact when logging from one account to
another. The regular common only exists within the main program and
within its subroutines.

The regular COMMON statement I think is a good thing. The NAMED
COMMON is a disaster waiting to happen. Should something go sideways
when logging from one account to another and the named common does not
get clear and reset properly, one could be writing/deleting (CRUD) to/
from files that should not open.

Dale