All,

I am now using QM's Distributed Files for the storage of session
records in Pavuk. This has been an interesting exercise since Martin
introduced the technology to me at Spectrum.

In the Pavuk web server, every successful login session is identifed
buy a Globally Unique Identifier or GUID. Please consult Wikipedia of
you are unfamiliar with these for a complete discussion on them. A
utility such as UUIDGEN is used to return a GUID which is a 33-byte
hex string such as: 30C7F94E-2ACC-40A9-B20A-C1DEEF14C8CB. The purpose
of GUIDs is to ensure that no system ever have a repeating key - EVER.
This makes the use of GUIDs very handy for things like logfiles.

Because the session IDs are stored perpetually (the records are
small), an efficient storage mechanism had to be devised for a
potential client who is looking to license 5000 users of Pavuk. I
chose to tackle this problem and learn about the QM distributed files.

Distributed files are made up of a collection of regular hash files. A
distributed file is a group of real hash files that have the same
structure. A DF may have 1 or more "part files" that is a hash file to
which it refers. For my example, I'll use the P.SESSIONS.DFL as my
Distributed File and P.SESSIONS.PART as my actual data storage area.

DFs are processed using their primary keys to sort the item into the
appropriate space. In my example, the GUIDs all begin with a hex
string and they are fairly evenly distributed across the 00-FF space
in the first byte. So, let's take the first byte and say we're going
to create 256 physical files. Of course, the files could be anywhere.

I chose to write a program to create the multi-file P.SESSIONS.PART,00
through P.SESSIONS.PART,FF. This provides the storage area.

Next, we need to create an I-TYPE dictionary in P.SESSIONS.PART that
will return a ***NUMERIC PART NUMBER*** based upon our record IDs.
We'll call this dictionary PARTNUMBER.

0001 I
0002 OCONV(@ID[1,2],"MCX")
0005 3R

This dictionary takes the first 2-character hex portion of the file
and returns a number from 0 - 255. 0 - 00, 1b=27, FF=255.

This handy dictionary is used by the Distributed File processor to
find which file to store our data. Now, we need to tell QM to build
the Distributed File itself:

ADD.DF P.SESSIONS.DFL P.SESSIONS.PART,00 0 PARTNUMBER

This first time we run this command, the P.SESSIONS.DFL control
component will be created. We are assigning the P.SESSIONS.PART,00
based upon the Hex ID to be component 0. We're also telling QM that
the method used to compute the part number is the dictionary
"PARTNUMBER"

Next we do:

ADD.DF P.SESSIONS.DFL P.SESSIONS.PART,01 1

Note, we don't have to tell QM the dictionary every time.

We're done at:

ADD.DF P.SESSIONS.DFL P.SESSIONS.PART,FF 255

Of course, I wrote a BASIC program to build this in a loop!

I now have a file that I can OPEN and use called P.SESSIONS.DFL and it
is comprised of 256 physical hash files. Because each QM hash file may
be up to 16,384GB, this creates a file that is theoretically capable
of holding 4.19 Exabytes of data. That's 4,194,304,000,000,000,000
bytes of data. More than enough for my purposes!

If I actually LIST P.SESSIONS.PART,00, I see the session records that
begin with 00. LIST P.SESSIONS.PART,3A shows those beginning with 3A,
etc.

This is a massive, yet simple way to deploy DFs. There is an easy 1-1
correlation between the first byte of the primary key and the storage
location. Remember, the actual index to find the storage location is
an integer beginning with 0.

***

Performance note: If you are going to use massive files like this, I
strongly recommend that the NUMFILES setting in your qmconfig file be
larger than the number of parts of the DF. I set my NUMFILES to 400
and the performance is excellent. This has to do with the number of
physical file handles present.

For Mac Server users, I also recommend that you do

$sudo lauchctl limit maxfiles 4096

Upon system restart. OS X 10.6 (Snow Leopard) has the per-process file
handle limit set absurdly low.

***

The way that you distribute your files is a function of your
imagination and the structure of the primary keys.

In Pavuk 2.0, the archival files will have the option to switch from a
normal hash file to a DF as needed.

Bill Crowell

On 2010-05-03 15:31:16 -0400, Bill Crowell <bcrowell (AT) pavuk (DOT) com> said:

I believe that your conversion is off. It's much smaller than that. 1
Exabyte is a billion GB.

--
Kevin Powick

Bill's excellent notes should be posted to PickWiki.com.

Just to play my role as the contentious skeptic I would argue that QM
is not unique in its ability to support these features or
"theoretical" storage capacities, and that we could have an entire
discussion about the resources and performance concerns related to
processing some number of distributed files.

However, I've argued that one of my two only beefs with QM is about
its ability to support large sites, and Bill's notes do help to
support the claim that QM is indeed capable of supporting very large
and complex data structures. It would be good to see more tech notes
like this.

BTW this is the sort of response I was hoping to see on a thread I
started recently that got absolutely no responses. For that large
project, with no comments in favor of another DBMS, I'm inclined to
port the project into QM and see how it does with distributed files.

T

Hi Bill,

Since you have posted this on a public list, I will reply via the same
list.

I am not convinced that your use of distributed files is the best way
to do this. Perhaps I have mis-understood the problem.

Scattering a set of record ids across a distributed file set is great
as a way to do load balancing across disk drives for very heavily
accessed files but I doubt that you have 256 drives. Simply scattering
the records over many files for the sake of it is likely to reduce
performance because the partitioning mechanism is effectively a higher
level of hashing to determine which part file should be accessed and
therefore adds extra processing. For a file that is not accessed
intensively in a random manner, keeping everything in the one file and
letting the hashing algorithm find things is likely to be much better.

Furthermore, as we discussed privately yesterday, having 256 part
files alongside all the other application files means you will almost
certainly hit the limit on the number of files that can be open
simultaneously at the operating system level. QM includes a mechanism
to handle this (known in other environments as file descriptor
sharing, rotating file pool, etc. We stuck with FDS). This mechanism
gives the programmer the illusion that there is no limit by quietly
closing files at the o/s level if the limit is reached and then
reopening them when you next try to access a file that we have closed.

By scattering what is probably a fairly small record set over 256
files, you are likely to find that many of your reads/writes require
the file to be reopened - a relatively expensive process in terms of
performance.

I am also puzzled why session ids live forever. Surely they become
redundant after some time and can be deleted. The general concept used
by many web sites (including the openqm.com site) is to assign a
temporary unique and random session id that will time out after some
period of inactivity. This period can be made long enough that no one
will ever hit it in a realistic use of the system, perhaps even 24
hours. Defunct session ids can then be deleted when no long active,
freeing disk space.

Keeping session data forever not only fills the file with defunct data
but also means that backups become enormously large. Going back to the
general concepts of distributed files, one of the main uses of these
is for historic data. By breaking this data into, perhaps, monthly
files, we have a file for the current month that is changing and hence
gets backed up every night but the historic data that is effectively
now read-only never need appear in an incremental backup.

Distributed files also allow you to have a single file that is larger
than a disk drive, to exceed operating system file size limits, or to
span a file over geographically separated systems.


Martin Phillips, Ladybridge Systems

Hi Martin et al.

First to Kevin's observation. I did slip an order of magnitude and the
proper result is 4.194x10E15 or 4.194 Petabytes.

Martin's point is well taken. Here's why I let the session records
live forever.

The business cases that I've worked on require a great deal of
tracking and security. We must know, for regulatory compliance,
exactly when a record was changed and by whom, from where, etc. Pavuk
stores all of the requisite information inside the P.SESSIONS file.
During the login process, the user information is verified and subject
to security checks, a session record is set up and starts running. The
system then performs a LOGTO to take them to their assigned database
account(s).

As a user makes updates to the various databases, updates records,
uploads attachments, etc. Pavuk stores the session ID in the tracking
records instead of the direct user name and all of the other data
that is required. 2 critical items are stored: The session ID and the
timestamp of the change which is the Unix Epoch value.

When someone is examining when an update was made, they can retrieve
all of the information about the source of the update from the session
record. We store, among other things, the users name, present group
permissions, IP address, link to the IP2location database, computer
and browser type and other items necessary for forensic examination of
the session. It's not a lot of data, but it is required for web
applications in my present business spaces.

At present, the 16.384Tb limit for a hash file is more than sufficient
and is more than efficient enough for the present load. The purpose of
this exercise was to determine the practicality for log files such as
this one in a very large environment with say, 5,000 users. The
particular inquiry came from a heavily regulated entity for which the
session detail is critical.

Martin's points on the number of file descriptors being open is also
very good. This type of use is very specialized and is only an example
of an enormously scalable solution to this particular problem.

Internally, Pavuk distributes the individual databases in a
heirarchial tree structure that is designed to limit the size of
records in any given database by a logical distribution of the data.

***

The next point that I would argue is the whole concept of the
traditional File Save and Restore. With the amount of data coming into
systems, our tradition of doing a backup and restore breaks down. How
does one effectively back up and restore Terabytes of data? Simply the
I/O performance is a choke point. Then there are the human factors of
simply getting people to perform a backup on a regular basis. Magnetic
tape isn't reliable nor are the humans who handle it.

A number of months back, Wired magazine did an article on the
economics of abundance. What this article argued, and with which I
agree, is that system storage space keeps decreasing in price at a
rate that is cheaper than the effort to archive and restore. The
author "did the math" to show the true cost of this vs. the cost of
simply adding more storage. It's an excellent and thought-provoking
argument. Please read it before firing missiles back at me!!!

From the point of practical storage of data, overcoming the I/O issues
is the primary choke point. This is best done, in my opinion, by
utilizing another storage system - preferably in a different location
and utilizing something like RSYNC or QM's network files to maintain a
live, hot backup. This provides a completely redundant system and not
just a "file save" tape that one must deal with. I can already hear
the moaning about the cost of a redundant system. Again, compute the
economics of what it costs to do a manual backup, store it somewhere
and HOPE that it is still useable some time in the future.

With that, I leave the discussion to try to recover some files that
are on DAT tapes from 2000 for a customer who has long since changed
systems and doesn't have a DAT drive in the present system. The cost
to restore these files will likely exceed $1,000 for just this one
incident.

Cheers!

Bill