APM: Another log analyzer

Brian Litke brian.litke at sedl.org
Thu Jul 1 13:52:46 PDT 2010 

Hi,

I've been using the "wusage" access_log analyzer by Boutelle for years.   I believe it is a Perl-based software. It works on UNIX and Windows
http://www.boutell.com/wusage/

I recently had to upgrade to version 8, because my 10-year-old wusage binary did not run on the new virtualized server my site was moved to.

Wusage has a 
 - single domain option ($25) for commercial sites and 
 - 5-domain ($75) and 
 - unlimited number of domains options ($295), too. 

Non-profits price is one-third of the prices shown above.

Brian Litke
Web Administrator http://www.sedl.org



On Jul 1, 2010, at 2:00 PM, austin-request at pm.org wrote:

> Send Austin mailing list submissions to
> 	austin at pm.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://mail.pm.org/mailman/listinfo/austin
> or, via email, send a message with subject or body 'help' to
> 	austin-request at pm.org
> 
> You can reach the person managing the list at
> 	austin-owner at pm.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Austin digest..."
> 
> 
> Today's Topics:
> 
>   1. Musings: Current state of log capture and analysis...
>      (jameschoate at austin.rr.com)
>   2. Re: Musings: Current state of log capture and analysis...
>      (Montgomery Conner)
>   3. Re: [Lopsa-us-tx-austin] Musings: Current state of log
>      capture and analysis... (jameschoate at austin.rr.com)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Thu, 1 Jul 2010 15:47:03 +0000
> From: <jameschoate at austin.rr.com>
> Subject: APM: Musings: Current state of log capture and analysis...
> To: Austin Area Leauge of Pro Sysadmins <lopsa-us-tx-austin at lopsa.org>
> Cc: "Austin-Hacking-Society at googlegroups.com"
> 	<Austin-Hacking-Society at googlegroups.com>, "Austin: pm.org"
> 	<Austin at pm.org>
> Message-ID: <20100701154703.37JJL.74098.root at hrndva-web02-z01>
> Content-Type: text/plain; charset=utf-8
> 
> I'm looking into a solution to collecting logs on at least a hundred or so servers, and possibly somewhere in the neighborhood of 5 million endpoints (and that could grow 2-3x).
> 
> I've been googling around and found:
> 
> Snare - mix of proprietary and open source solution, is based around a central collection service/server which is very appealing
> AWStats - this one is more for single server analysis and just doesn't feel right
> MindTreeInsight - Jave and open source, will likely look a little deeper into this one
> LASSO - Open Source and seems to be Windows only
> syslog-ng - this has been around forever and is scripted based, doesn't scale the way I'd like
> Analog - this one I'm not familiar with, currently researching
> Webalizer - is more focused on single server analysis and may have scaling issues, currently researching
> Yaala - not familiar with this one at all, still researching
> 
> Any that you know if that I missed? If you have a favorite can you share in 3-5 sentences why? Scaling is important.
> 
> I was also looking at a JASON based log analysis tool but didn't find any. This tech looks like a good way to approach this problem. Scaling might be an issue.
> 
> --
> -- -- -- --
> Venimus, Vidimus, Dolavimus
> 
> jameschoate at austin.rr.com
> james.choate at g.austincc.edu
> james.choate at twcable.com
> h: 512-657-1279
> w: 512-845-8989
> http://hackerspaces.org/wiki/Confusion_Research_Center
> 
> Adapt, Adopt, Improvise
> -- -- -- --
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Thu, 1 Jul 2010 11:03:52 -0500
> From: Montgomery Conner <montgomery.conner at gmail.com>
> Subject: Re: APM: Musings: Current state of log capture and
> 	analysis...
> To: jameschoate at austin.rr.com
> Cc: Austin Area Leauge of Pro Sysadmins
> 	<lopsa-us-tx-austin at lopsa.org>,
> 	"Austin-Hacking-Society at googlegroups.com"
> 	<Austin-Hacking-Society at googlegroups.com>, "Austin: pm.org"
> 	<Austin at pm.org>
> Message-ID:
> 	<AANLkTik3dQVHuA22zPn8_2d9OZ4dxxmyQNS5STinVMmw at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> I'm looking into using the Spread Toolkit (http://www.spread.org/) which may
> be more complex than your needs dictate but has some real advantages for my
> use-cases. It has an excellent description of it's use as a logging
> mechanism in Theo Schlossnagel's 'Scalable Internet Architectures' (
> http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X/ref=sr_1_1?ie=UTF8&s=books&qid=1278000081&sr=8-1),
> which I highly recommend; he also wrote the first of many Perl modules that
> speak Spread (all available via the CPAN).
> 
> If you're at all concerned about deriving value (via analysis) of the
> collected data at the scale you're dealing with you might want to consider
> Hadoop (and the Hadoop file system: HDFS) as an end point for storage as
> well as an analysis platform. There are some tools in various states of
> development designed to import massive amounts of data into Hadoop: Scribe,
> Chukwa, and Flume, which was open-sourced just this Monday by Cloudera, are
> among the growing list of alternates in this space.
> 
> Hope that helps,
> Montgomery
> 
> On Thu, Jul 1, 2010 at 10:47 AM, <jameschoate at austin.rr.com> wrote:
> 
>> I'm looking into a solution to collecting logs on at least a hundred or so
>> servers, and possibly somewhere in the neighborhood of 5 million endpoints
>> (and that could grow 2-3x).
>> 
>> I've been googling around and found:
>> 
>> Snare - mix of proprietary and open source solution, is based around a
>> central collection service/server which is very appealing
>> AWStats - this one is more for single server analysis and just doesn't feel
>> right
>> MindTreeInsight - Jave and open source, will likely look a little deeper
>> into this one
>> LASSO - Open Source and seems to be Windows only
>> syslog-ng - this has been around forever and is scripted based, doesn't
>> scale the way I'd like
>> Analog - this one I'm not familiar with, currently researching
>> Webalizer - is more focused on single server analysis and may have scaling
>> issues, currently researching
>> Yaala - not familiar with this one at all, still researching
>> 
>> Any that you know if that I missed? If you have a favorite can you share in
>> 3-5 sentences why? Scaling is important.
>> 
>> I was also looking at a JASON based log analysis tool but didn't find any.
>> This tech looks like a good way to approach this problem. Scaling might be
>> an issue.
>> 
>> --
>> -- -- -- --
>> Venimus, Vidimus, Dolavimus
>> 
>> jameschoate at austin.rr.com
>> james.choate at g.austincc.edu
>> james.choate at twcable.com
>> h: 512-657-1279
>> w: 512-845-8989
>> http://hackerspaces.org/wiki/Confusion_Research_Center
>> 
>> Adapt, Adopt, Improvise
>> -- -- -- --
>> _______________________________________________
>> Austin mailing list
>> Austin at pm.org
>> http://mail.pm.org/mailman/listinfo/austin
>> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.pm.org/pipermail/austin/attachments/20100701/e4b8485a/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 3
> Date: Thu, 1 Jul 2010 18:19:32 +0000
> From: <jameschoate at austin.rr.com>
> Subject: Re: APM: [Lopsa-us-tx-austin] Musings: Current state of log
> 	capture and analysis...
> To: Matt Ray <mray at zenoss.com>
> Cc: Austin Area Leauge of Pro Sysadmins
> 	<lopsa-us-tx-austin at lopsa.org>,
> 	"Austin-Hacking-Society at googlegroups.com"
> 	<Austin-Hacking-Society at googlegroups.com>, "Austin: pm.org"
> 	<Austin at pm.org>
> Message-ID: <20100701181932.SEFU2.75087.root at hrndva-web02-z01>
> Content-Type: text/plain; charset=utf-8
> 
> Hi Matt,
> 
> You won't remember me but I talked to you after your talk here in Austin a few months ago at the Linux expo (don't remember it's actual name now). We discussed your tools ability to capture large SNMP trap populations.
> 
> I'll give this a look.  Thanks.
> 
> ---- Matt Ray <mray at zenoss.com> wrote: 
>> I actually discussed this with one of the admins from Twitter at Velocity.  They were using Splunk, but ran into scaling issues eventually and replaced it with a home-grown solution of Scribe + Hadoop File System (http://hadoopblog.blogspot.com/2009/06/hdfs-scribe-integration.html).  If you're going to large-scale installations, that might be a path to explore.
> 
> --
> -- -- -- --
> Venimus, Vidimus, Dolavimus
> 
> jameschoate at austin.rr.com
> james.choate at g.austincc.edu
> james.choate at twcable.com
> h: 512-657-1279
> w: 512-845-8989
> http://hackerspaces.org/wiki/Confusion_Research_Center
> 
> Adapt, Adopt, Improvise
> -- -- -- --
> 
> 
> ------------------------------
> 
> _______________________________________________
> Austin mailing list
> Austin at pm.org
> http://mail.pm.org/mailman/listinfo/austin
> 
> End of Austin Digest, Vol 80, Issue 1
> *************************************

Brian Litke
Web Administrator
SEDL
4700 Mueller Blvd.
Austin, TX 78723
512-391-6529 (voice)
512-476-2286 (fax)
http://www.sedl.org
"Advancing Research, Improving Education"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/austin/attachments/20100701/a17dbcc7/attachment-0001.html>

More information about the Austin mailing list