I'm looking into using the Spread Toolkit (<a href="http://www.spread.org/">http://www.spread.org/</a>) which may be more complex than your needs dictate but has some real advantages for my use-cases. It has an excellent description of it's use as a logging mechanism in Theo Schlossnagel's 'Scalable Internet Architectures' (<a href="http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X/ref=sr_1_1?ie=UTF8&s=books&qid=1278000081&sr=8-1">http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X/ref=sr_1_1?ie=UTF8&s=books&qid=1278000081&sr=8-1</a>), which I highly recommend; he also wrote the first of many Perl modules that speak Spread (all available via the CPAN).<br>
<br>If you're at all concerned about deriving value (via analysis) of the collected data at the scale you're dealing with you might want to consider Hadoop (and the Hadoop file system: HDFS) as an end point for storage as well as an analysis platform. There are some tools in various states of development designed to import massive amounts of data into Hadoop: Scribe, Chukwa, and Flume, which was open-sourced just this Monday by Cloudera, are among the growing list of alternates in this space.<br>
<br>Hope that helps,<br>Montgomery<br><br><div class="gmail_quote">On Thu, Jul 1, 2010 at 10:47 AM, <span dir="ltr"><<a href="mailto:jameschoate@austin.rr.com">jameschoate@austin.rr.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I'm looking into a solution to collecting logs on at least a hundred or so servers, and possibly somewhere in the neighborhood of 5 million endpoints (and that could grow 2-3x).<br>
<br>
I've been googling around and found:<br>
<br>
Snare - mix of proprietary and open source solution, is based around a central collection service/server which is very appealing<br>
AWStats - this one is more for single server analysis and just doesn't feel right<br>
MindTreeInsight - Jave and open source, will likely look a little deeper into this one<br>
LASSO - Open Source and seems to be Windows only<br>
syslog-ng - this has been around forever and is scripted based, doesn't scale the way I'd like<br>
Analog - this one I'm not familiar with, currently researching<br>
Webalizer - is more focused on single server analysis and may have scaling issues, currently researching<br>
Yaala - not familiar with this one at all, still researching<br>
<br>
Any that you know if that I missed? If you have a favorite can you share in 3-5 sentences why? Scaling is important.<br>
<br>
I was also looking at a JASON based log analysis tool but didn't find any. This tech looks like a good way to approach this problem. Scaling might be an issue.<br>
<br>
--<br>
-- -- -- --<br>
Venimus, Vidimus, Dolavimus<br>
<br>
<a href="mailto:jameschoate@austin.rr.com">jameschoate@austin.rr.com</a><br>
<a href="mailto:james.choate@g.austincc.edu">james.choate@g.austincc.edu</a><br>
<a href="mailto:james.choate@twcable.com">james.choate@twcable.com</a><br>
h: 512-657-1279<br>
w: 512-845-8989<br>
<a href="http://hackerspaces.org/wiki/Confusion_Research_Center" target="_blank">http://hackerspaces.org/wiki/Confusion_Research_Center</a><br>
<br>
Adapt, Adopt, Improvise<br>
-- -- -- --<br>
_______________________________________________<br>
Austin mailing list<br>
<a href="mailto:Austin@pm.org">Austin@pm.org</a><br>
<a href="http://mail.pm.org/mailman/listinfo/austin" target="_blank">http://mail.pm.org/mailman/listinfo/austin</a><br>
</blockquote></div><br>