That is the question which...

So, I have the following use-case:

A user-request comes in and is processed. This already happens normally.
We want to record what the user-request was, along with a (small) set
of resources that were used during the request processing.

In another process, we are handling updating those resources. When a
resource is updated, we want to look up which user-requests used that
resource, and do "something" accordingly.

We only care about user-requests that happened less than two weeks ago.

We'd like to minimize user-request latency (eg. adding this recording
shouldn't slow down user-request processing more than a few milliseconds).

The resource updating happens far less frequently than user-requests.

My first inclination is to use MySQL, since that is what we use
most-often for data-stores around here. I was considering buffering the
whole thing with a JMS queue (we're a Java shop BTW), so that any DB
latency would be offloaded away from the user-request processing.

Then, I was thinking that some nosql solutions might fit better, and
provide better performance. I don't know much about the various nosql
solutions available, so I'm not really sure where to start.

For what its worth, the user-request can be represented as one
string+several key-value pairs, and the resources can be identified with
two 32bit integers OR another user-request (depending on the resource type).




--
Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>

On 2010-04-12, Daniel Pitts <newsgroup.spamfilter (AT) virtualinfinity (DOT) net> wrote:
writing a records to a table is usually fast.

what's going to happen if that queue grows to some silly size?

I don't know what you want to do.

that's kind of vague too.

a resource is a 64 bits of data or a request

and a request is a string plus several key-value pairs

initially you implied it had same but now you don't mention it

but how many resources does a request have?
can a resource be a value or a key ?




--- news://freenews.netfront.net/ - complaints: news (AT) netfront (DOT) net ---

On 04/13/2010 01:52 AM, Daniel Pitts wrote:
I do not know what Shakespeare would say to this - these are my 0.02 EUR:

Sounds like a perfect use case for either an index on a timestamp column
or using partitioning. In any case you should plan for a regular (e.g.
nightly) cleanout operation.

I'd use MySQL if you use that regularly and can handle the concurrency
issue (i.e. concurrent writing and reading) since it is a well tested
product and apparently you are familiar with it. I would only start
using a JMS queue if you know already that it will be slow, meaning
there is conceptual reasoning or measurement that proves it. Rather
start out simple and make it more complex only if needed. You will, of
course, modularize the code so you can easily shift the writing to an
MDB. :-)

From what I am reading recently [1] I conclude that the promise of
NoSQL has not yet unfolded. Hence I'd rather rely on a technology with
a proven track record and knowledge in house.

Even in a relational database that should be written pretty fast -
especially in those databases whose writing is done in a separate
process like Oracle's DBWriter. Then you have the offloading of IO
there already.

Kind regards

robert


[1]
http://arstechnica.com/business/data...he-rise-of.ars

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/