large-scale log loading & parsing

#1

Hi,
I'm looking for a solution for the following scenario:
- System running on multiple servers (could be dozens)
- Each server is running IIS and contains logs

The solution is a way to easily provide analysis reports on a combined
information from these logs. For example - "What's the IP Geo
distribution between DateX and DateY"

I am afraid that traditional methods (such as bulk-load everything
into mssql) will not work because of the big load.
I was also looking at solutions such as Hive (over Hadoop) but our
environment is Win32 and I'm not sure it's the right path.

So, I'm looking for ideas... Need to easily being able to load those
logs and then easily analyze them

Thanks!

reply to adisapir [at] gmail dot com

#2

"Docco" <adisapir (AT) gmail (DOT) com> wrote

Quote:

Hi,
I'm looking for a solution for the following scenario:
- System running on multiple servers (could be dozens)
- Each server is running IIS and contains logs

The solution is a way to easily provide analysis reports on a combined
information from these logs. For example - "What's the IP Geo
distribution between DateX and DateY"

I am afraid that traditional methods (such as bulk-load everything
into mssql) will not work because of the big load.
I was also looking at solutions such as Hive (over Hadoop) but our
environment is Win32 and I'm not sure it's the right path.

So, I'm looking for ideas... Need to easily being able to load those
logs and then easily analyze them

Thanks!

reply to adisapir [at] gmail dot com

Have you looked into any of the standard web analytics and clickstream
analysis solutions? I suggest you consider buying one before you build it
yourself.

--
David Portas

#3

On Aug 16, 6:12*am, "David Portas"
<REMOVE_BEFORE_REPLYING_dpor... (AT) acm (DOT) org> wrote:

Quote:

"Docco" <adisa... (AT) gmail (DOT) com> wrote in message

news:10caf2ce-9449-4d3c-b99e-87c37c10a0e6 (AT) t13g2000yqt (DOT) googlegroups.com...

Hi,
I'm looking for a solution for the following scenario:
- System running on multiple servers (could be dozens)
- Each server is running IIS and contains logs

The solution is a way to easily provide analysis reports on a combined
information from these logs. For example - "What's the IP Geo
distribution between DateX and DateY"

I am afraid that traditional methods (such as bulk-load everything
into mssql) will not work because of the big load.
I was also looking at solutions such as Hive (over Hadoop) but our
environment is Win32 and I'm not sure it's the right path.

So, I'm looking for ideas... Need to easily being able to load those
logs and then easily analyze them

Thanks!

reply to adisapir [at] gmail dot com

Have you looked into any of the standard web analytics and clickstream
analysis solutions?

Like http://splunk.com just to name one at random.

Quote:

I suggest you consider buying one before you build it
yourself.

--
David Portas

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

large-scale log loading & parsing

comp.databases.theory comp.databases.theory

large-scale log loading & parsing - 08-16-2009 , 04:02 AM

Re: large-scale log loading & parsing - 08-16-2009 , 05:12 AM

Re: large-scale log loading & parsing - 08-17-2009 , 10:05 PM