<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi,<div><br></div><div>I've been using the "wusage" access_log analyzer by Boutelle for years. I believe it is a Perl-based software. It works on UNIX and Windows</div><div><a href="http://www.boutell.com/wusage/">http://www.boutell.com/wusage/</a></div><div><br></div><div>I recently had to upgrade to version 8, because my 10-year-old wusage binary did not run on the new virtualized server my site was moved to.</div><div><br></div><div>Wusage has a </div><div> - single domain option ($25) for commercial sites and </div><div> - 5-domain ($75) and </div><div> - unlimited number of domains options ($295), too. </div><div><br></div><div>Non-profits price is one-third of the prices shown above.</div><div><br></div><div>Brian Litke</div><div>Web Administrator <a href="http://www.sedl.org">http://www.sedl.org</a></div><div><br></div><div><br></div><div><br><div><div>On Jul 1, 2010, at 2:00 PM, <a href="mailto:austin-request@pm.org">austin-request@pm.org</a> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Send Austin mailing list submissions to<br><span class="Apple-tab-span" style="white-space:pre"> </span><a href="mailto:austin@pm.org">austin@pm.org</a><br><br>To subscribe or unsubscribe via the World Wide Web, visit<br><span class="Apple-tab-span" style="white-space:pre"> </span><a href="http://mail.pm.org/mailman/listinfo/austin">http://mail.pm.org/mailman/listinfo/austin</a><br>or, via email, send a message with subject or body 'help' to<br><span class="Apple-tab-span" style="white-space:pre"> </span>austin-request@pm.org<br><br>You can reach the person managing the list at<br><span class="Apple-tab-span" style="white-space:pre"> </span>austin-owner@pm.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of Austin digest..."<br><br><br>Today's Topics:<br><br> 1. Musings: Current state of log capture and analysis...<br> (jameschoate@austin.rr.com)<br> 2. Re: Musings: Current state of log capture and analysis...<br> (Montgomery Conner)<br> 3. Re: [Lopsa-us-tx-austin] Musings: Current state of log<br> capture and analysis... (jameschoate@austin.rr.com)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Thu, 1 Jul 2010 15:47:03 +0000<br>From: <jameschoate@austin.rr.com><br>Subject: APM: Musings: Current state of log capture and analysis...<br>To: Austin Area Leauge of Pro Sysadmins <lopsa-us-tx-austin@lopsa.org><br>Cc: "Austin-Hacking-Society@googlegroups.com"<br><span class="Apple-tab-span" style="white-space:pre"> </span><Austin-Hacking-Society@googlegroups.com>, "Austin: pm.org"<br><span class="Apple-tab-span" style="white-space:pre"> </span><Austin@pm.org><br>Message-ID: <20100701154703.37JJL.74098.root@hrndva-web02-z01><br>Content-Type: text/plain; charset=utf-8<br><br>I'm looking into a solution to collecting logs on at least a hundred or so servers, and possibly somewhere in the neighborhood of 5 million endpoints (and that could grow 2-3x).<br><br>I've been googling around and found:<br><br>Snare - mix of proprietary and open source solution, is based around a central collection service/server which is very appealing<br>AWStats - this one is more for single server analysis and just doesn't feel right<br>MindTreeInsight - Jave and open source, will likely look a little deeper into this one<br>LASSO - Open Source and seems to be Windows only<br>syslog-ng - this has been around forever and is scripted based, doesn't scale the way I'd like<br>Analog - this one I'm not familiar with, currently researching<br>Webalizer - is more focused on single server analysis and may have scaling issues, currently researching<br>Yaala - not familiar with this one at all, still researching<br><br>Any that you know if that I missed? If you have a favorite can you share in 3-5 sentences why? Scaling is important.<br><br>I was also looking at a JASON based log analysis tool but didn't find any. This tech looks like a good way to approach this problem. Scaling might be an issue.<br><br>--<br> -- -- -- --<br>Venimus, Vidimus, Dolavimus<br><br>jameschoate@austin.rr.com<br>james.choate@g.austincc.edu<br>james.choate@twcable.com<br>h: 512-657-1279<br>w: 512-845-8989<br>http://hackerspaces.org/wiki/Confusion_Research_Center<br><br>Adapt, Adopt, Improvise<br> -- -- -- --<br><br><br>------------------------------<br><br>Message: 2<br>Date: Thu, 1 Jul 2010 11:03:52 -0500<br>From: Montgomery Conner <montgomery.conner@gmail.com><br>Subject: Re: APM: Musings: Current state of log capture and<br><span class="Apple-tab-span" style="white-space:pre"> </span>analysis...<br>To: jameschoate@austin.rr.com<br>Cc: Austin Area Leauge of Pro Sysadmins<br><span class="Apple-tab-span" style="white-space:pre"> </span><lopsa-us-tx-austin@lopsa.org>,<br><span class="Apple-tab-span" style="white-space:pre"> </span>"Austin-Hacking-Society@googlegroups.com"<br><span class="Apple-tab-span" style="white-space:pre"> </span><Austin-Hacking-Society@googlegroups.com>, "Austin: pm.org"<br><span class="Apple-tab-span" style="white-space:pre"> </span><Austin@pm.org><br>Message-ID:<br><span class="Apple-tab-span" style="white-space:pre"> </span><AANLkTik3dQVHuA22zPn8_2d9OZ4dxxmyQNS5STinVMmw@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>I'm looking into using the Spread Toolkit (http://www.spread.org/) which may<br>be more complex than your needs dictate but has some real advantages for my<br>use-cases. It has an excellent description of it's use as a logging<br>mechanism in Theo Schlossnagel's 'Scalable Internet Architectures' (<br>http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X/ref=sr_1_1?ie=UTF8&s=books&qid=1278000081&sr=8-1),<br>which I highly recommend; he also wrote the first of many Perl modules that<br>speak Spread (all available via the CPAN).<br><br>If you're at all concerned about deriving value (via analysis) of the<br>collected data at the scale you're dealing with you might want to consider<br>Hadoop (and the Hadoop file system: HDFS) as an end point for storage as<br>well as an analysis platform. There are some tools in various states of<br>development designed to import massive amounts of data into Hadoop: Scribe,<br>Chukwa, and Flume, which was open-sourced just this Monday by Cloudera, are<br>among the growing list of alternates in this space.<br><br>Hope that helps,<br>Montgomery<br><br>On Thu, Jul 1, 2010 at 10:47 AM, <jameschoate@austin.rr.com> wrote:<br><br><blockquote type="cite">I'm looking into a solution to collecting logs on at least a hundred or so<br></blockquote><blockquote type="cite">servers, and possibly somewhere in the neighborhood of 5 million endpoints<br></blockquote><blockquote type="cite">(and that could grow 2-3x).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I've been googling around and found:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Snare - mix of proprietary and open source solution, is based around a<br></blockquote><blockquote type="cite">central collection service/server which is very appealing<br></blockquote><blockquote type="cite">AWStats - this one is more for single server analysis and just doesn't feel<br></blockquote><blockquote type="cite">right<br></blockquote><blockquote type="cite">MindTreeInsight - Jave and open source, will likely look a little deeper<br></blockquote><blockquote type="cite">into this one<br></blockquote><blockquote type="cite">LASSO - Open Source and seems to be Windows only<br></blockquote><blockquote type="cite">syslog-ng - this has been around forever and is scripted based, doesn't<br></blockquote><blockquote type="cite">scale the way I'd like<br></blockquote><blockquote type="cite">Analog - this one I'm not familiar with, currently researching<br></blockquote><blockquote type="cite">Webalizer - is more focused on single server analysis and may have scaling<br></blockquote><blockquote type="cite">issues, currently researching<br></blockquote><blockquote type="cite">Yaala - not familiar with this one at all, still researching<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Any that you know if that I missed? If you have a favorite can you share in<br></blockquote><blockquote type="cite">3-5 sentences why? Scaling is important.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I was also looking at a JASON based log analysis tool but didn't find any.<br></blockquote><blockquote type="cite">This tech looks like a good way to approach this problem. Scaling might be<br></blockquote><blockquote type="cite">an issue.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--<br></blockquote><blockquote type="cite"> -- -- -- --<br></blockquote><blockquote type="cite">Venimus, Vidimus, Dolavimus<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">jameschoate@austin.rr.com<br></blockquote><blockquote type="cite">james.choate@g.austincc.edu<br></blockquote><blockquote type="cite">james.choate@twcable.com<br></blockquote><blockquote type="cite">h: 512-657-1279<br></blockquote><blockquote type="cite">w: 512-845-8989<br></blockquote><blockquote type="cite">http://hackerspaces.org/wiki/Confusion_Research_Center<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Adapt, Adopt, Improvise<br></blockquote><blockquote type="cite"> -- -- -- --<br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Austin mailing list<br></blockquote><blockquote type="cite">Austin@pm.org<br></blockquote><blockquote type="cite">http://mail.pm.org/mailman/listinfo/austin<br></blockquote><blockquote type="cite"><br></blockquote>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <http://mail.pm.org/pipermail/austin/attachments/20100701/e4b8485a/attachment-0001.html><br><br>------------------------------<br><br>Message: 3<br>Date: Thu, 1 Jul 2010 18:19:32 +0000<br>From: <jameschoate@austin.rr.com><br>Subject: Re: APM: [Lopsa-us-tx-austin] Musings: Current state of log<br><span class="Apple-tab-span" style="white-space:pre"> </span>capture and analysis...<br>To: Matt Ray <mray@zenoss.com><br>Cc: Austin Area Leauge of Pro Sysadmins<br><span class="Apple-tab-span" style="white-space:pre"> </span><lopsa-us-tx-austin@lopsa.org>,<br><span class="Apple-tab-span" style="white-space:pre"> </span>"Austin-Hacking-Society@googlegroups.com"<br><span class="Apple-tab-span" style="white-space:pre"> </span><Austin-Hacking-Society@googlegroups.com>, "Austin: pm.org"<br><span class="Apple-tab-span" style="white-space:pre"> </span><Austin@pm.org><br>Message-ID: <20100701181932.SEFU2.75087.root@hrndva-web02-z01><br>Content-Type: text/plain; charset=utf-8<br><br>Hi Matt,<br><br>You won't remember me but I talked to you after your talk here in Austin a few months ago at the Linux expo (don't remember it's actual name now). We discussed your tools ability to capture large SNMP trap populations.<br><br>I'll give this a look. Thanks.<br><br>---- Matt Ray <mray@zenoss.com> wrote: <br><blockquote type="cite">I actually discussed this with one of the admins from Twitter at Velocity. They were using Splunk, but ran into scaling issues eventually and replaced it with a home-grown solution of Scribe + Hadoop File System (http://hadoopblog.blogspot.com/2009/06/hdfs-scribe-integration.html). If you're going to large-scale installations, that might be a path to explore.<br></blockquote><br>--<br> -- -- -- --<br>Venimus, Vidimus, Dolavimus<br><br>jameschoate@austin.rr.com<br>james.choate@g.austincc.edu<br>james.choate@twcable.com<br>h: 512-657-1279<br>w: 512-845-8989<br>http://hackerspaces.org/wiki/Confusion_Research_Center<br><br>Adapt, Adopt, Improvise<br> -- -- -- --<br><br><br>------------------------------<br><br>_______________________________________________<br>Austin mailing list<br>Austin@pm.org<br>http://mail.pm.org/mailman/listinfo/austin<br><br>End of Austin Digest, Vol 80, Issue 1<br>*************************************<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="font-size: 12px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="2" style="font: normal normal normal 10px/normal 'Lucida Grande'; ">Brian Litke</font></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="2" style="font: normal normal normal 10px/normal 'Lucida Grande'; ">Web Administrator</font></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="'Lucida Grande'" size="2"><span class="Apple-style-span" style="font-size: 10px; ">SEDL</span></font></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="'Lucida Grande'" size="2"><span class="Apple-style-span" style="font-size: 10px; ">4700 Mueller Blvd.</span></font></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="'Lucida Grande'" size="2"><span class="Apple-style-span" style="font-size: 10px; ">Austin, TX 78723</span></font></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="2" style="font: normal normal normal 10px/normal 'Lucida Grande'; ">512-391-6529 (voice)</font></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Lucida Grande" size="2" style="font: normal normal normal 10px/normal 'Lucida Grande'; ">512-476-2286 (fax)</font></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><span class="Apple-style-span" style="font-family: 'Lucida Grande'; font-size: 10px; "><a href="http://www.sedl.org">http://www.sedl.org</a></span></p><p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="'Lucida Grande'" size="2"><span class="Apple-style-span" style="font-size: 10px; ">"Advancing Research, Improving Education"</span></font></p></span></span></span>
</div>
<br></div></body></html>