dbTalk Databases Forums  

Where has all the space gone?

comp.databases.pick comp.databases.pick


Discuss Where has all the space gone? in the comp.databases.pick forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
bontemps
 
Posts: n/a

Default Where has all the space gone? - 01-26-2007 , 09:45 AM






After doing a save with the x and y option thereby saving what I
thought was every piece of data on our system, I get a total byte count
in the File-of-Files significantly smaller than what comes up on the
data usage row of the FREE verb. The FOF Byte total is 42.8 gig.
FREE verb shows 67.2 gig used. Frame count is much closer. 30M frames
is FOF and 33M frames FREE verb. Where are these bytes hiding?


Reply With Quote
  #2  
Old   
Scott Ballinger
 
Posts: n/a

Default Re: Where has all the space gone? - 01-26-2007 , 11:47 AM






1. I don't think the fof byte count includes the "unused" bytes at the
end of every group.
2. fof does not include bytes/frames used by indexes.
3. Frames "lost" due to gfe repairs, etc, are not included in the fof
totals.
4. I think the free space avail count is the same as the overflow count
(from the "available" field in what). free's data used is then just
maxfid - available. (You can check out the free source in dm,bp,)
5. I am not sure where spooler frames are counted (or not).
6. The bottom line is that the available space reported by free (aka
available overflow) is the real number to pay attention to.
7. A interesting comparison might be to total up the number of filesave
data sets (mysave, mysave-1, mysave-2, etc) times 2GB each (I always
set the compressed pseudo-floppy tape device to roll over at 2GB) and
compare to your free totals.On my systems I use the (b option with the
save verb to include the indexes, so the save data sets are a pretty
accurate reflection of the total amount of actual data (not empty
space) on the system.

Here is one example:

(I suggest viewing with a fixed font)

free:
MB Bytes Frames
System usage 0% 15 15,480,000 3,870
Data usage 86% 59,205 59,204,612,000 14,801,153
Available disk space 14% 9,908 9,907,532,000 2,476,883

20 filesave compressed pseudo floppy data files = 40GB of saved data
(includes indexes)

Implies that there is almost 16GB of allocated but unused disk space on
this system ?!

However, I know that this system had a full restore less than 2 months
ago, and the number of "recovered" frames was minimal (less than
100,000). So there doesn't seem to be a pattern of losing frames here,
thus I don't think the discrepancy in this case can be blamed on gfes
or other lost frames. I also don't think that there are any large files
that are grossly oversized (could be another culprit?).

There is one large file w/ 4M records that is defined as a "p" type
file (every item is a pointer); if on average there is 1/2 frame of
unused "pad" at the end of every item in this file, that is 4M x 2000
bytes = 8 billion bytes of unused space on this file alone. So maybe
the 16GB "wasted" disk space is not so crazy after all? (All I can say
is "it seemed like a good idea at the time" to make the file a "p"
type!)

/Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006




On Jan 26, 7:45 am, "bontemps" <jontan_... (AT) yahoo (DOT) com> wrote:
Quote:
After doing a save with the x and y option thereby saving what I
thought was every piece of data on our system, I get a total byte count
in the File-of-Files significantly smaller than what comes up on the
data usage row of the FREE verb. The FOF Byte total is 42.8 gig.
FREE verb shows 67.2 gig used. Frame count is much closer. 30M frames
is FOF and 33M frames FREE verb. Where are these bytes hiding?


Reply With Quote
  #3  
Old   
Mark Brown
 
Posts: n/a

Default Re: Where has all the space gone? - 01-26-2007 , 03:09 PM



There is a table called the SAFE table which contains up to 200 blocks of
frames, each over 200 frames long. If the system crashes, it throws away
the old overflow table which is usually corrupted by the crash and rebuilds
it using this safe table.

There are options on the POVF command which will show these groups. I don't
remember exactly, but I believe A works.

Mark


"bontemps" <jontan_mlr (AT) yahoo (DOT) com> wrote

Quote:
After doing a save with the x and y option thereby saving what I
thought was every piece of data on our system, I get a total byte count
in the File-of-Files significantly smaller than what comes up on the
data usage row of the FREE verb. The FOF Byte total is 42.8 gig.
FREE verb shows 67.2 gig used. Frame count is much closer. 30M frames
is FOF and 33M frames FREE verb. Where are these bytes hiding?




Reply With Quote
  #4  
Old   
Ross Ferris
 
Posts: n/a

Default Re: Where has all the space gone? - 01-26-2007 , 09:53 PM





On Jan 27, 4:47 am, "Scott Ballinger" <scott.ballin... (AT) gmail (DOT) com>
wrote:
Quote:
5. I am not sure where spooler frames are counted (or not).
Spooler frames will be taked OUT of the FREE report, and will NOT be
included in the backup --> anyone who has restored a system knows that
you loose the spooler entries in the process
Quote:
6. The bottom line is that the available space reported by free (aka
available overflow) is the real number to pay attention to.
YEP



Reply With Quote
  #5  
Old   
bontemps
 
Posts: n/a

Default Re: Where has all the space gone? - 01-29-2007 , 09:32 AM



Thanks Scott, Mark and Ross for your help. Will add the option to
save indexes and investigate the P type files. So far I've uncovered
about 1 gig of spooler and P file frames. The rest must be indexes
and unused space within data groups. Cant believe we have over 20 gig
of lost frames. I should do a full restore on the system and see what
space we get back. Again, thank for your responses.

On Jan 26, 12:47 pm, "Scott Ballinger" <scott.ballin... (AT) gmail (DOT) com>
wrote:
Quote:
1. I don't think the fof byte count includes the "unused" bytes at the
end of every group.
2. fof does not include bytes/frames used by indexes.
3. Frames "lost" due to gfe repairs, etc, are not included in the fof
totals.
4. I think the free space avail count is the same as the overflow count
(from the "available" field in what). free's data used is then just
maxfid - available. (You can check out the free source in dm,bp,)
5. I am not sure where spooler frames are counted (or not).
6. The bottom line is that the available space reported by free (aka
available overflow) is the real number to pay attention to.
7. A interesting comparison might be to total up the number of filesave
data sets (mysave, mysave-1, mysave-2, etc) times 2GB each (I always
set the compressed pseudo-floppy tape device to roll over at 2GB) and
compare to your free totals.On my systems I use the (b option with the
save verb to include the indexes, so the save data sets are a pretty
accurate reflection of the total amount of actual data (not empty
space) on the system.

Here is one example:

(I suggest viewing with a fixed font)

free:
MB Bytes Frames
System usage 0% 15 15,480,000 3,870
Data usage 86% 59,205 59,204,612,000 14,801,153
Available disk space 14% 9,908 9,907,532,000 2,476,883

20 filesave compressed pseudo floppy data files = 40GB of saved data
(includes indexes)

Implies that there is almost 16GB of allocated but unused disk space on
this system ?!

However, I know that this system had a full restore less than 2 months
ago, and the number of "recovered" frames was minimal (less than
100,000). So there doesn't seem to be a pattern of losing frames here,
thus I don't think the discrepancy in this case can be blamed on gfes
or other lost frames. I also don't think that there are any large files
that are grossly oversized (could be another culprit?).

There is one large file w/ 4M records that is defined as a "p" type
file (every item is a pointer); if on average there is 1/2 frame of
unused "pad" at the end of every item in this file, that is 4M x 2000
bytes = 8 billion bytes of unused space on this file alone. So maybe
the 16GB "wasted" disk space is not so crazy after all? (All I can say
is "it seemed like a good idea at the time" to make the file a "p"
type!)

/Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006

On Jan 26, 7:45 am, "bontemps" <jontan_... (AT) yahoo (DOT) com> wrote:



After doing a save with the x and y option thereby saving what I
thought was every piece of data on our system, I get a total byte count
in the File-of-Files significantly smaller than what comes up on the
data usage row of the FREE verb. The FOF Byte total is 42.8 gig.
FREE verb shows 67.2 gig used. Frame count is much closer. 30M frames
is FOF and 33M frames FREE verb. Where are these bytes hiding?- Hide quoted text -- Show quoted text -


Reply With Quote
  #6  
Old   
Richard S
 
Posts: n/a

Default Re: Where has all the space gone? - 01-30-2007 , 04:55 AM



bontemps wrote:
Quote:
After doing a save with the x and y option thereby saving what I
thought was every piece of data on our system, I get a total byte count
in the File-of-Files significantly smaller than what comes up on the
data usage row of the FREE verb. The FOF Byte total is 42.8 gig.
FREE verb shows 67.2 gig used. Frame count is much closer. 30M frames
is FOF and 33M frames FREE verb. Where are these bytes hiding?

The subject line reminds me of when I was looking after a Reality system
back in '79. The office manager was used to me talking about 'space'
and "running a 'claim' to get some space back".

One day I had my head in the back of the Reality cabinet and he peered
over my shoulder, pointed at a largish metal box at the base of the
cabinet and asked "Is that where all the space is?"

I didn't have the heart to tell him it was the PSU.


richard.


Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.