From jameschoate at austin.rr.com Wed Jul 1 05:34:25 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Wed, 1 Jul 2009 12:34:25 +0000 Subject: APM: CaPerl Message-ID: <20090701123425.XUAAA.434509.root@hrndva-web20-z01> CaPerl is a system that implements capability discipline in Perl. It does this by introducting a few extra language constructs, and by restricting what untrusted code is allowed to do. The CaPerl language is compiled into standard Perl which can then be run by a completely standard Perl interpreter. However, untrusted code is prevented from doing anything its trusted (or untrusted) wrapper did not intend it to do. http://caperl.links.org/ -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Wed Jul 1 06:12:58 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Wed, 1 Jul 2009 13:12:58 +0000 Subject: APM: On Buffon Machines and Numbers Message-ID: <20090701131258.OSIIW.435286.root@hrndva-web20-z01> Buffon's needle experiment is well-known: take a plane on which parallel lines at unit distance one from the next have been marked, throw a needle of unit length at random, and, finally, declare the experiment a success if the needle intersects one of the lines. Basic calculus implies that the probability of success is 2/pi~0.63661, and the experiment can be regarded as an analog (i.e., continuous) device that stochastically "computes'' 2/pi. Generalizing the experiment and simplifying the computational framework, we ask ourselves which probability distributions can be produced perfectly, from a discrete source of unbiased coin flips. We describe and analyse a few simple Buffon machines that can generate geometric, Poisson, and logarithmic-series distributions (these are in particular required to transform continuous Boltzmann samplers of classical combinatorial structures into purely discrete random generators). Say that a number is Buffon if it is the probability of success of a probabilistic experiment based on discrete coin flippings. We provide human-accessible Buffon machines, which require a dozen coin flips or less, on average, and produce experiments whose probabilities are expressible in terms of numbers such as pi, exp(-1), log2, sqrt(3), cos(1/4), zeta(5). More generally, we develop a collection of constructions based on simple probabilistic mechanisms that enable one to create Buffon experiments involving compositions of exponentials and logarithms, polylogarithms, direct and inverse trigonometric functions, algebraic and hypergeometric functions, as well as functions defined by integrals, such as the Gaussian error function. http://arxiv.org/abs/0906.5560 -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Wed Jul 1 06:14:00 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Wed, 1 Jul 2009 13:14:00 +0000 Subject: APM: Group theory in cryptography Message-ID: <20090701131400.YDRGX.435309.root@hrndva-web20-z01> This paper is a guide for the pure mathematician who would like to know more about cryptography based on group theory. The paper gives a brief overview of the subject, and provides pointers to good textbooks, key research papers and recent survey papers in the area. http://arxiv.org/abs/0906.5545 -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Wed Jul 1 06:16:06 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Wed, 1 Jul 2009 13:16:06 +0000 Subject: APM: Cryptanalysis of SDES via evolutionary computation techniques Message-ID: <20090701131606.7F7HK.435354.root@hrndva-web20-z01> The cryptanalysis of simplified data encryption standard can be formulated as NP-Hard combinatorial problem. The goal of this paper is two fold. First we want to make a study about how evolutionary computation techniques can efficiently solve the NP-Hard combinatorial problem. For achieving this goal we test several evolutionary computation techniques like memetic algorithm, genetic algorithm and simulated annealing for the cryptanalysis of simplified data encryption standard problem (SDES). And second was a comparison between memetic algorithm, genetic algorithm and simulated annealing were made in order to investigate the performance for the cryptanalysis on SDES. The methods were tested and extensive computational results show that memetic algorithm performs better than genetic algorithms and simulated annealing for such type of NP-Hard combinatorial problem. This paper represents our first effort toward efficient memetic algorithm for the cryptanalysis of SDES. http://arxiv.org/abs/0906.5123 -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Wed Jul 1 17:19:56 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Thu, 2 Jul 2009 0:19:56 +0000 Subject: APM: Neuromancer at 25: What It Got Right, What It Got Wrong Message-ID: <20090702001956.8LWNE.448944.root@hrndva-web22-z01> The novel, published on July 1, 1984, predicted the World Wide Web, cyberspace, and a lot of other things. Which of William Gibson's predictions have come true, and which still seem far off? http://www.pcworld.com/article/167670/neuromancer_at_25_what_it_got_right_what_it_got_wrong.html "The sky above the port was the color of television, tuned to a dead channel..." http://lib.ru/GIBSON/neuromancer.txt -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From JPolache at texasmutual.com Wed Jul 1 17:43:31 2009 From: JPolache at texasmutual.com (Jonathan S. Polacheck) Date: Wed, 1 Jul 2009 19:43:31 -0500 Subject: APM: Jonathan S. Polacheck/AUSTIN/THE_FUND is out of the office. Message-ID: I will be out of the office starting 07/01/2009 and will not return until 07/10/2009. I am on vacation until Aug. 22th, I will respond to your message when I return. From jameschoate at austin.rr.com Thu Jul 2 11:34:23 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Thu, 2 Jul 2009 18:34:23 +0000 Subject: APM: Two Centuries On, a Cryptologist Cracks a Presidential Code Message-ID: <20090702183424.7B8KP.461630.root@hrndva-web09-z01> http://online.wsj.com/article/SB124648494429082661.html -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Thu Jul 2 11:45:12 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Thu, 2 Jul 2009 18:45:12 +0000 Subject: APM: Why am I writing Padre? Message-ID: <20090702184512.PKHAT.461853.root@hrndva-web09-z01> An editor with special consideration for Perl... http://szabgab.com/blog/2009/07/1246515871.html -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Fri Jul 3 18:50:18 2009 From: jameschoate at austin.rr.com (James Choate) Date: Fri, 3 Jul 2009 20:50:18 -0500 Subject: APM: The Robot Group - Maker SIG - Invitation & 2009 Schedule Message-ID: With the support of The Robot Group board and general member support we are glad to announce the beginning of a new SIG. It will be focused on a variety of topics and skills, using hands-on techniques to build mastery. The group will meet on the 2nd and 4th Thursday of each month from 7pm to 10pm. Location will vary and should be checked either on the TRG mailing list or visit their webpage, http://wiki.therobotgroup.org/wiki/Main_Page. Please feel free to spread this information around to other groups and individuals who may have an interest in hands-on technology exploration! The Robot Group - Maker SIG - 2009 Calendar July 9 7-8pm - Soldering: Basic Safety & Operations 8-8:30pm - TRG General Business 8:30-10pm - Lego Mindstorms: Build basic crawler & check build environment 23 7-8pm - Soldering: Basic types of joints and splices 8-8:30 - TRG General Business 8:30-10pm - Lego Mindstorms: Motor control and reading sensors August 13 7-8pm - Soldering: Desoldering using vacuum and braid 8-8:30pm - TRG General Business 8:30-10pm - Lego Mindstorms: Basic autonomy and obstacle avoidance 27 7-8pm - Soldering: Review & skills focus/practice 8-8:30pm - TRG General Business 8:30-10pm - Lego Mindstorms: Review of available sensors and how to read them September 10 7-8pm - Multimeter: Basic Safety & Operations 8-8:30pm - TRG General Business 8:30-10pm - Lego Mindstorms: Upgrade basic crawler sensor suite 24 7-8pm - Multimeter: Practice, practice, practice 8-8:30pm - TRG General Business 8:30-10pm - Lego Mindstorms: Upgrade basic crawler sensor suite October 8 7-8pm - Oscilloscope: Basic Safety & Operations 8-8:30pm - TRG General Business 8:30-10pm - Lego Mindstorms: Enhanced Crawler obstacle avoidance testing 22 7-8pm - Oscilloscope: Practice, practice, practice 8-8:30pm - TRG General Business 8:30-10pm - Lego Mindstorms: Enhanced Crawler obstacle avoidance testing November 12 7-8pm - Discussion of projects and goals for 2010 8-8:30pm - TRG General Business 8:30-10pm - Lego Mindstorms: Individual team projects 26 No meet - Thanksgiving December 10 7-8pm - Open 8-8:30pm - TRG General Business 8:30-10pm - Lego Mindstorms: Individual team projects 24 No meet - Christmas -- -- -- -- Soldering - Basic Mechanics and Techniques We will focus on the basic mechanics of soldering. We will be using the following document as the basis of this discussion. It does contain a small toolset and if you're budget will allow we strongly suggest you get your own tools. However, we will have sufficient supplies to provide everyone opportunity to explore their new skills. It is expected that this series will last 4-6 weeks in total depending on group progress. If you are going to participate it is expected that you have read the following document before the first class. We suggest you download it to your laptop. http://www.physics.unc.edu/~fredb/techresource/docs/soldering.pdf You can buy the basic soldering iron, solder, desolder braid, and a sponge at Fry's for under $20. We'll have several soldering irons available for participants to share if they don't have their own. If you have old electronics (radios and audio equipment especially) that you don't mind sacrificing we'd very much appreciate the gift/practice (they will be sacrificed, not fixed!). -- -- -- -- Lego Mindstorms Resources If you're interested in participating in the Lego Mindstorm component of our effort you should probably visit each of these sites and download to your local laptop all the resources. We will be using 'Not Quite C' as our primary development tool. It is suggested that you have a 3-ring binder, writing instruments, and a laptop running Windows of some flavor. We have at least two RCX bricks and that should be sufficient for the class. If you have your own please bring it so we can break into smaller groups. We will be building a standard tracked crawler as our hardware testbed. NQC FAQ bricxcc.sourceforge.net/nqc/doc/faq.html NQC Tutorial cs.redeemer.ca/derek/tutorials/nqc.pdf NQC User's Manual www.cursolego.elo.utfsm.cl/.../NQC_User_Manual-Dave_Baum%5BEN%5D.pdf NQC Programmer's Guide neuron.eng.wayne.edu/LEGO_ROBOTICS/nqc_guide.pdf Programming Lego Robots Using NQC oz.plymouth.edu/~wjt/Foundations/Labs/markovManual.pdf NQC - Not Quite C Output Commands (Motors) www.cs.binghamton.edu/~reckert/480/intro-to-NQC.pdf NQC - Not Quite C http://bricxcc.sourceforge.net/nqc/ RCX Command Center http://people.cs.uu.nl/markov/lego/rcxcc/index.html Bricx Command Center 3.3 http://bricxcc.sourceforge.net/ References on the H8 processor http://www.google.com/#hl=en&q=Lego+RCX+Hitachi+H8&aq=f&oq=&aqi=&fp=leBsIIJAIN0 http://www.google.com/#hl=en&q=Hitachi+H8&aq=f&oq=&aqi=g7&fp=leBsIIJAIN0 http://www.google.com/#hl=en&q=Hitachi+H8+microprocessor&aq=f&oq=&aqi=&fp=leBsIIJAIN0 -- -- James Choate Solar Soyuz Zaibatsu jameschoate at austin.rr.com 512-657-1279 -- -- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jameschoate at austin.rr.com Fri Jul 3 18:55:21 2009 From: jameschoate at austin.rr.com (James Choate) Date: Fri, 3 Jul 2009 20:55:21 -0500 Subject: APM: Fw: Geek Austin: 30 days until Linux Against Poverty installfest. Message-ID: <9DFC94A5DB3B44428E93245FC80C1098@dell72dbc8897c> ----- Original Message ----- From: Geek Austin To: James Choate Sent: Tuesday, June 30, 2009 10:00 PM Subject: Geek Austin: 30 days until Linux Against Poverty installfest. LinkedIn Groups a.. Group: Geek Austin b.. Subject: Geek Austin: 30 days until Linux Against Poverty installfest. Fellow Austin Geeks, As you know, I've cut back on the GeekAustin parties because I've been working on the GeekAustin free IT classes and Linux Against Poverty. I've just published the July update at the Linux Against Poverty website: http://linuxagainstpoverty.org/july-update The first Austin install fest will be on Saturday August 1st. We can still use a few more volunteers. We can also use your leftover computers (tax receipts can be provided). If your employer has a stack of decommissioned computers (you know -- those windows machines that became virus ridden from surfing myspace and whatever else). We'll put Linux on them and distribute them to children who need them. We are now starting to solicit sponsors. Some of the items we will need to rent for the install fest are work tables and chairs, extension cables and network cables. We'd also like to provide beverages and snacks for the volunteers. If you or your company would like to be a sponsor for LinuxAgainstPoverty, send me a note at lynnbender at geekaustin.org We are also seeking translators to help render the LAP docs in to other languages. If you're a native or near native speaker of Spanish, French, German, or Dutch, and would be interested in helping out, send me a note. After the install fest, there will be a party. Everyone knows how I like parties, and I'm looking to make this one double plus good. Having a volunteer badge at the party will most likely have some benefits. wink wink. You can find all the details at: http://linuxagainstpoverty.org -Lynn Bender GeekAustin Posted By Lynn Bender View or add comments ? Don't want to hear from the manager? Unsubscribe here LinkedIn values your privacy. At no time has LinkedIn made your email address available to any other LinkedIn user without your permission. ?2009, LinkedIn Corporation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jameschoate at austin.rr.com Tue Jul 7 16:06:03 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Tue, 7 Jul 2009 23:06:03 +0000 Subject: APM: Meet tomorrow night at 7pm Message-ID: <20090707230603.11N1M.255785.root@hrndva-web03-z01> pm I stopped by the Mangia on the way home and spoke w/ the manager, we're all set for 7-9pm. Only one hitch, I will be in Round Rock with my favorite spectrum analyzer at the time. Found out about it earlier today. Sorry. So, whomever goes should probably take a Perl book, especially if you figure you'll be there near 7pm. -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Wed Jul 8 20:30:15 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Thu, 9 Jul 2009 3:30:15 +0000 Subject: APM: austin.pm.org not working -- Who to contact? Message-ID: <20090709033015.SH7CI.576344.root@hrndva-web12-z01> Hi, Over the last couple of days the APM webpage is not providing a page. Who should be contacted for support? -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From wwalker at solid-constructs.com Wed Jul 8 21:05:27 2009 From: wwalker at solid-constructs.com (Wayne Walker) Date: Wed, 8 Jul 2009 23:05:27 -0500 Subject: APM: austin.pm.org not working -- Who to contact? In-Reply-To: <20090709033015.SH7CI.576344.root@hrndva-web12-z01> References: <20090709033015.SH7CI.576344.root@hrndva-web12-z01> Message-ID: <20090709040527.GC8325@solid-constructs.com> Oops, that would be me. AND the admins for pm.org. Had to move the server from one colo facility to another, so the IP has changed and the apache instance is probably also not running. I'll see if I can raise one of the pm.org admins on IM. On Thu, Jul 09, 2009 at 03:30:15AM +0000, jameschoate at austin.rr.com wrote: > Hi, > > Over the last couple of days the APM webpage is not providing a page. Who should be contacted for support? > > -- > -- -- -- -- > Venimus, Vidimus, Dolavimus > > James Choate > jameschoate at austin.rr.com > james.choate at twcable.com > 512-657-1279 > www.ssz.com > http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu > http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center > > Adapt, Adopt, Improvise > -- -- -- -- > _______________________________________________ > Austin mailing list > Austin at pm.org > http://mail.pm.org/mailman/listinfo/austin -- Wayne Walker wwalker at solid-constructs.com (512) 633-8076 Senior Consultant Solid Constructs, LLC From jameschoate at austin.rr.com Wed Jul 8 22:30:28 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Thu, 9 Jul 2009 5:30:28 +0000 Subject: APM: Ask Slashdot: What Would You Want In a Large-Scale Monitoring System? on Wednesday July 08, @05:30PM Message-ID: <20090709053028.BAHT6.390649.root@hrndva-web06-z01> http://ask.slashdot.org/story/09/07/08/210241/What-Would-You-Want-In-a-Large-Scale-Monitoring-System?art_pos=8 -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From wwalker at solid-constructs.com Thu Jul 9 08:13:00 2009 From: wwalker at solid-constructs.com (Wayne Walker) Date: Thu, 9 Jul 2009 10:13:00 -0500 Subject: APM: austin.pm.org not working -- Who to contact? In-Reply-To: <20090709040527.GC8325@solid-constructs.com> References: <20090709033015.SH7CI.576344.root@hrndva-web12-z01> <20090709040527.GC8325@solid-constructs.com> Message-ID: <20090709151300.GC5775@solid-constructs.com> On Wed, Jul 08, 2009 at 11:05:27PM -0500, Wayne Walker wrote: > Oops, that would be me. AND the admins for pm.org. > > Had to move the server from one colo facility to another, so the IP has > changed and the apache instance is probably also not running. > > I'll see if I can raise one of the pm.org admins on IM. I mailed support at pm.org asking for the DNS to be changed. When they do the DNS change we should be back online. > > > On Thu, Jul 09, 2009 at 03:30:15AM +0000, jameschoate at austin.rr.com wrote: > > Hi, > > > > Over the last couple of days the APM webpage is not providing a page. Who should be contacted for support? > > > > -- > > -- -- -- -- > > Venimus, Vidimus, Dolavimus > > > > James Choate > > jameschoate at austin.rr.com > > james.choate at twcable.com > > 512-657-1279 > > www.ssz.com > > http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu > > http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center > > > > Adapt, Adopt, Improvise > > -- -- -- -- > > _______________________________________________ > > Austin mailing list > > Austin at pm.org > > http://mail.pm.org/mailman/listinfo/austin > > -- > > Wayne Walker > wwalker at solid-constructs.com > (512) 633-8076 > Senior Consultant > Solid Constructs, LLC -- Wayne Walker wwalker at solid-constructs.com (512) 633-8076 Senior Consultant Solid Constructs, LLC From jameschoate at austin.rr.com Tue Jul 14 14:15:23 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Tue, 14 Jul 2009 21:15:23 +0000 Subject: APM: Hacking Society - Tonights meeting Message-ID: <20090714211523.F0FGW.673516.root@hrndva-web02-z01> Just a quick reminder that the Hacking Society meeting is tonight at the HEB Central Market Cafe on N. Lamar between 38th and 45th. Look for the table with the red covered "Applied Cryptography" book. Starts at 7pm and we'll be done by 9pm. See you there hopefully. -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Tue Jul 14 19:25:24 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Wed, 15 Jul 2009 2:25:24 +0000 Subject: APM: Affordable SMD/SMT Workstations Message-ID: <20090715022525.XIAJB.675938.root@hrndva-web09-z01> http://www.soldertoolsdepot.com/productsdirect.asp?dept=479&main=79&item1=15143+TL&item2=16121+TL -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Fri Jul 17 05:32:47 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Fri, 17 Jul 2009 12:32:47 +0000 Subject: APM: Visual cryptography Message-ID: <20090717123247.90T1G.19284.root@hrndva-web28-z01> http://en.wikipedia.org/wiki/Visual_cryptography -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Fri Jul 17 05:41:46 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Fri, 17 Jul 2009 12:41:46 +0000 Subject: APM: Managing Hierarchical Data in MySQL Message-ID: <20090717124146.GHSZ5.19452.root@hrndva-web28-z01> http://dev.mysql.com/tech-resources/articles/hierarchical-data.html -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Fri Jul 17 05:45:20 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Fri, 17 Jul 2009 12:45:20 +0000 Subject: APM: Relationships among Scientific Paradigms Message-ID: <20090717124520.H5ELJ.19521.root@hrndva-web28-z01> http://informationesthetics.org/documents/scienceMapPrintMockupEd2.jpg -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Fri Jul 17 09:21:42 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Fri, 17 Jul 2009 16:21:42 +0000 Subject: APM: "Go" Bag... Message-ID: <20090717162142.UX9PW.24186.root@hrndva-web28-z01> A long, long time ago...in a galaxy far, far away...I used to do geophysics work and learned to live out of a "Go" bag. It's a habit that I've never lost, and if anything I've learned to appreciate even more. I'm always looking for ways to carry all the crap I seem to always be needing as I go from here to there, doing this or that... This is a good site to keep in your hip pocket if you're similarly afflicted... http://www.milspecmonkey.com/ -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From tshinnic at io.com Fri Jul 17 12:08:59 2009 From: tshinnic at io.com (Thomas L. Shinnick) Date: Fri, 17 Jul 2009 14:08:59 -0500 Subject: APM: "Go" Bag... In-Reply-To: <20090717162142.UX9PW.24186.root@hrndva-web28-z01> References: <20090717162142.UX9PW.24186.root@hrndva-web28-z01> Message-ID: <6.2.5.6.2.20090717140126.06930a18@io.com> At 11:21 AM 7/17/2009, jameschoate at austin.rr.com wrote: >A long, long time ago...in a galaxy far, far away...I used to do >geophysics work and learned to live out of a "Go" bag. It's a habit >that I've never lost, and if anything I've learned to appreciate >even more. I'm always looking for ways to carry all the crap I seem >to always be needing as I go from here to there, doing this or that... > >This is a good site to keep in your hip pocket if you're similarly >afflicted... > >http://www.milspecmonkey.com/ Bad monkey sulk - not play with firefox or opera. Plays with shiny metal (Chrome) and dull rocks (IE). -------------- next part -------------- An HTML attachment was scrubbed... URL: From jameschoate at austin.rr.com Sun Jul 19 07:56:47 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Sun, 19 Jul 2009 14:56:47 +0000 Subject: APM: Engineers working on Apollo 11 LEM computer could only guess Message-ID: <20090719145648.OKCP6.91798.root@hrndva-web01-z01> http://www.njnnetwork.com/njn/?p=19947 -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Mon Jul 20 06:14:20 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Mon, 20 Jul 2009 13:14:20 +0000 Subject: APM: The Exploding Internet 2008 Message-ID: <20090720131421.VP05L.55457.root@hrndva-web18-z01> http://img132.imageshack.us/img132/2988/27062201.jpg -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Mon Jul 20 09:40:18 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Mon, 20 Jul 2009 16:40:18 +0000 Subject: APM: Planes, Trains And Ant Hills Computer Scientists Simulate Activity Of Ants To Reduce Airline Delays Message-ID: <20090720164018.176CR.60673.root@hrndva-web18-z01> http://www.sciencedaily.com/videos/2008/0406-planes_trains_and_ant_hills.htm -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Mon Jul 20 12:14:23 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Mon, 20 Jul 2009 14:14:23 -0500 Subject: APM: How They Built it: The Software of Apollo 11 Message-ID: <20090720191423.FN187.64669.root@hrndva-web18-z01> http://www.linux.com/news/software/developer/29068-apollo-11-story -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Mon Jul 20 12:17:50 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Mon, 20 Jul 2009 14:17:50 -0500 Subject: APM: APOLLO GUIDANCE COMPUTER (AGC) Schematics Message-ID: <20090720191751.R4BBH.64749.root@hrndva-web18-z01> http://klabs.org/history/ech/agc_schematics/index.htm -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Mon Jul 20 12:21:03 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Mon, 20 Jul 2009 14:21:03 -0500 Subject: APM: Build your own Apollo Guidance Computer Message-ID: <20090720192103.C9A2L.64851.root@hrndva-web18-z01> http://echoesofapollo.com/resources/apollo-guidance-computer/ -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Mon Jul 20 18:39:13 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Tue, 21 Jul 2009 1:39:13 +0000 Subject: APM: TRG Maker SIG - Thu. July 23 Schedule -- Soldering & Lego Mindstorms Message-ID: <20090721013914.04KWX.121790.root@hrndva-web01-z01> The next (first) meet of the TRG Maker SIG will happen this Thursday. 7-8pm Basic Soldering (if you have soldering irons and/or basic tools like wire cutters and long nose please bring them) 8-8:30pm TRG Weekly Business 8:30 - 10pm Lego Mindstorms - Bring your laptop and any Lego Mindstorms you have...we'll be building our basic robot and getting the tool chain setup. If we're lucky we'll get some bump switch code written. -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Wed Jul 22 06:27:40 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Wed, 22 Jul 2009 8:27:40 -0500 Subject: APM: Brute Force Programming Message-ID: <20090722132740.E5RO8.151242.root@hrndva-web11-z01> http://melikyan.blogspot.com/2009/07/brute-force-programming.html -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Wed Jul 22 06:38:58 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Wed, 22 Jul 2009 13:38:58 +0000 Subject: APM: Perl on a Stick Message-ID: <20090722133858.TUONM.151560.root@hrndva-web11-z01> http://oylenshpeegul.vox.com/library/post/perl-on-a-stick.html -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Wed Jul 22 12:00:21 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Wed, 22 Jul 2009 19:00:21 +0000 Subject: APM: Nobody Hates Software More Than Software Developers Message-ID: <20090722190021.ODFSU.112705.root@hrndva-web14-z01> http://www.codinghorror.com/blog/archives/001289.html I can't agree with this more, a day doesn't go buy that I don't want to physically kill somebody over some design/implementation decision with a piece of software. A good example is SuSE since it's been purchased by Novell. I started Linux about two weeks after it was released, tsx-11.mit.edu and many hours and floppies later... I started using Slackware when it released it's first distro and kept using it. One picky issue for me was there was simply no way to upgrade when a new release came out, you had to wipe the whole machine and start over. This was a problem for me in spreading Linux (but not for me personally as I rip my machines apart and rebuild them about once a year anyway). So these bright boys in Germany decided to do something about it and took Slackware and munged it until you could do upgrades, and it was good, very good for many years. Then Novell bought them and over the last two major releases, SuSE 10 and 11, my frustration level at the plethora of 'corporate mindset' (ie bullshit) build decisions has taken an excellent product and eviscerated it. For something like two years now I've been slowly building resources and processes for Confusion Research Center and one of these is the servers this will require. SuSE has gotten so bad that I've decided that when I tear my current boxes down in the near term SuSE will not be going on them. I've looked at Red Hat, Ubuntu, etc. and I've pretty much started to lean on going back to something like a cross between Damn Small Linux and Linux from Scratch. I've been buying various distros from OSDisc.com (which I highly recommend!!!!) and none of them seem to scratch my itch. Enough whinning... -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Wed Jul 22 14:19:04 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Wed, 22 Jul 2009 21:19:04 +0000 Subject: APM: Approximating Pi number using Genetic Programming Message-ID: <20090722211904.GN13N.163917.root@hrndva-web11-z01> http://pyevolve.sourceforge.net/wordpress/?p=618 -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Thu Jul 23 08:29:11 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Thu, 23 Jul 2009 15:29:11 +0000 Subject: APM: Cloud-Sourcing's Long-Term Impact On IT Careers Message-ID: <20090723152911.DFCFI.127133.root@hrndva-web02-z01> "InfoWorld provides a reality check on the impact cloud computing will have on IT jobs, the overall effects of which will likely resemble those of outsourcing, automation, and utility computing ? in other words, a movement away from the nuts and bolts of technology toward the business end of the organization. This shift from 'blue-collar IT to white-collar IT' will be accompanied by greater demand for IT pros experienced with virtualization and Web scale-out deployments, even among midlevel organizations, and greater emphasis on SaaS integration among in-house development teams, analysts say. And though the large-scale impact of 'cloud-sourcing' is likely a decade away, those not versed in vendor contract management, cloud integration, analytics, and RIA and mobile development may find themselves pushed toward the less technical jobs to come, those that will require days full of conference calls and putting out fires caused by doing business in the cloud." http://it.slashdot.org/article.pl?sid=09/07/23/1237258 -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jwarner at texas.net Thu Jul 23 10:20:19 2009 From: jwarner at texas.net (John Warner) Date: Thu, 23 Jul 2009 12:20:19 -0500 Subject: APM: Perl, Win32 OLE, and Excel In-Reply-To: <20090722190021.ODFSU.112705.root@hrndva-web14-z01> References: <20090722190021.ODFSU.112705.root@hrndva-web14-z01> Message-ID: <002701ca0bb9$dadfdd40$909f97c0$@net> All, I have a project where I am trying to filter through a large amount of data from an Excel spreadsheet. Since I don't have access to the databases where the data actually resides, I have to use a spreadsheet that was given to me. The spreadsheet contains 79 columns and approximately 113k rows. The data are customer satisfaction survey results along with a plethora of other garbage I don't need. I am only interested in a few columns. My code goes like this... Create an Excel Object Use Object to open Source and Destination spreadsheets Find the column and row boundaries of where data is within the source. my @ArrayOfNames = ('Bill', 'Bob', 'Jane', 'Tom', 'Dick', 'Harry'); #Columns # Source Destination Description # Column Column # 28 3 Responsible Tech # 55 5 Tech Sat Score # 57 6 Overall Sat Score # foreach my $row (2..$LastRow) #skip header row on row 1 { #check the responsible tech foreach my $t (@ArrayOfNames) { my $cellObj = $srcSheet->Cells($row,28); print "Current: $t \t Incident tech: $cellObj->{Value} "; if ($t =~ m/$srcSheet->Cells($row,28)->{Value}/) { print "found a match!\n"; if ($srcSheet->Cells($row,55)->{Value} < 7 || $srcSheet->Cells($row,57)->{Value} < 7) { #copy data from source to destination } }else{ #print "not a match \n"; next; } } } My question: With 113k rows to go through, Perl runs out of memory and the processing takes quite a while. How can I be more efficient? John Warner jwarner at texas.net H: 512.251.1270 C: 512.426.3813 From howanitz at gmail.com Thu Jul 23 10:51:06 2009 From: howanitz at gmail.com (Keith Howanitz) Date: Thu, 23 Jul 2009 12:51:06 -0500 Subject: APM: Perl, Win32 OLE, and Excel In-Reply-To: <002701ca0bb9$dadfdd40$909f97c0$@net> References: <20090722190021.ODFSU.112705.root@hrndva-web14-z01> <002701ca0bb9$dadfdd40$909f97c0$@net> Message-ID: On Thu, Jul 23, 2009 at 12:20 PM, John Warner wrote: > All, > > I have a project where I am trying to filter through a large amount of data > from an Excel spreadsheet. ?Since I don't have access to the databases where > the data actually resides, I have to use a spreadsheet that was given to me. > The spreadsheet contains 79 columns and approximately 113k rows. ?The data > are customer satisfaction survey results along with a plethora of other > garbage I don't need. ?I am only interested in a few columns. [SNIP] Have you tried putting a simple output when reading the xls file to show you how far you are getting in the file before having problems - maybe you are really going beyond 113k records, or it is choking on unusual data in one particular record. I wonder if you simply saved the xls file as a csv file and used the TEXT::CSV_XS module if you would still have troubles. If you want to read the whole thing into memory, you could always read each line, put the 3 important fields in array, and then read the next line so that you only end up with an array that is 113k x 3 rather than the whole spreadsheet in memory. From jameschoate at austin.rr.com Thu Jul 23 11:03:31 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Thu, 23 Jul 2009 18:03:31 +0000 Subject: APM: Perl, Win32 OLE, and Excel In-Reply-To: Message-ID: <20090723180331.9M485.40762.root@hrndva-web17-z01> Besides only processing each sub-set of each row via an input filter -- Assuming each row is not dependent on the other rows, why use an output array at all? Just write it to a file. The only reason I can see writing this to an array is to keep it in memory for subsequence processing. ---- Keith Howanitz wrote: > On Thu, Jul 23, 2009 at 12:20 PM, John Warner wrote: > > All, > > > > I have a project where I am trying to filter through a large amount of data > > from an Excel spreadsheet. ?Since I don't have access to the databases where > > the data actually resides, I have to use a spreadsheet that was given to me. > > The spreadsheet contains 79 columns and approximately 113k rows. ?The data > > are customer satisfaction survey results along with a plethora of other > > garbage I don't need. ?I am only interested in a few columns. > [SNIP] > > Have you tried putting a simple output when reading the xls file to > show you how far you are getting in the file before having problems - > maybe you are really going beyond 113k records, or it is choking on > unusual data in one particular record. > > I wonder if you simply saved the xls file as a csv file and used the > TEXT::CSV_XS module if you would still have troubles. > > If you want to read the whole thing into memory, you could always read > each line, put the 3 important fields in array, and then read the next > line so that you only end up with an array that is 113k x 3 rather > than the whole spreadsheet in memory. > _______________________________________________ > Austin mailing list > Austin at pm.org > http://mail.pm.org/mailman/listinfo/austin -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jwarner at texas.net Thu Jul 23 11:16:00 2009 From: jwarner at texas.net (John Warner) Date: Thu, 23 Jul 2009 13:16:00 -0500 Subject: APM: Perl, Win32 OLE, and Excel In-Reply-To: <20090723180331.9M485.40762.root@hrndva-web17-z01> References: <20090723180331.9M485.40762.root@hrndva-web17-z01> Message-ID: <002d01ca0bc1$a2299bf0$e67cd3d0$@net> In my actual code, I am copying from source file to destination file and the rows are independent of each other. One thought that occurred to me was to not use an array of names using something like the code below. This should reduce this process from an order n^2 operation to an order n operation. my @index; foreach my $row (2..$LastRow) #skip header row on row 1 { my $cellObj = $srcSheet->Cells($row,28); print "Incident tech: $cellObj->{Value} "; if ($cellObj->{Value} =~ m/name1|name2|name3/) { push $row, @index; }else{ print ?not a match \n?; next; } } foreach my $i (@index) { copy target data from source file to destination file } I'll have to test to see what the difference in CPU and memory utilization are once I get into the office. Time to start my day! Thanks for all the suggestions! John Warner jwarner at texas.net H: 512.251.1270 C: 512.426.3813 -----Original Message----- From: austin-bounces+jwarner=texas.net at pm.org [mailto:austin-bounces+jwarner=texas.net at pm.org] On Behalf Of jameschoate at austin.rr.com Sent: Thursday, July 23, 2009 1:04 PM To: austin Subject: Re: APM: Perl, Win32 OLE, and Excel Besides only processing each sub-set of each row via an input filter -- Assuming each row is not dependent on the other rows, why use an output array at all? Just write it to a file. The only reason I can see writing this to an array is to keep it in memory for subsequence processing. ---- Keith Howanitz wrote: > On Thu, Jul 23, 2009 at 12:20 PM, John Warner wrote: > > All, > > > > I have a project where I am trying to filter through a large amount of data > > from an Excel spreadsheet. Since I don't have access to the databases where > > the data actually resides, I have to use a spreadsheet that was given to me. > > The spreadsheet contains 79 columns and approximately 113k rows. The data > > are customer satisfaction survey results along with a plethora of other > > garbage I don't need. I am only interested in a few columns. > [SNIP] > > Have you tried putting a simple output when reading the xls file to > show you how far you are getting in the file before having problems - > maybe you are really going beyond 113k records, or it is choking on > unusual data in one particular record. > > I wonder if you simply saved the xls file as a csv file and used the > TEXT::CSV_XS module if you would still have troubles. > > If you want to read the whole thing into memory, you could always read > each line, put the 3 important fields in array, and then read the next > line so that you only end up with an array that is 113k x 3 rather > than the whole spreadsheet in memory. > _______________________________________________ > Austin mailing list > Austin at pm.org > http://mail.pm.org/mailman/listinfo/austin -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- _______________________________________________ Austin mailing list Austin at pm.org http://mail.pm.org/mailman/listinfo/austin From havoc at boldo.com Thu Jul 23 11:29:10 2009 From: havoc at boldo.com (Pat Ludwig) Date: Thu, 23 Jul 2009 13:29:10 -0500 Subject: APM: Perl, Win32 OLE, and Excel In-Reply-To: <002d01ca0bc1$a2299bf0$e67cd3d0$@net> References: <20090723180331.9M485.40762.root@hrndva-web17-z01> <002d01ca0bc1$a2299bf0$e67cd3d0$@net> Message-ID: <1fe8332f0907231129h64fffb20i324b8cb5a78283ee@mail.gmail.com> Some ideasa) throw the techs into a hash and use -- if (exists $techhash{$cellObj->{Value}} b) dump the excel file into a text file if possible c) after b) run the text file thru the unix utility "cut" to grab just the columns you need The amount of data you are parsing isn't overly large so I suspect that the perl interface to excel you are using is the memory hog. --Pat On Thu, Jul 23, 2009 at 1:16 PM, John Warner wrote: > In my actual code, I am copying from source file to destination file and > the rows are independent of each other. One thought that occurred to me was > to not use an array of names using something like the code below. This > should reduce this process from an order n^2 operation to an order n > operation. > > my @index; > > foreach my $row (2..$LastRow) #skip header row on row 1 > { > my $cellObj = $srcSheet->Cells($row,28); > print "Incident tech: $cellObj->{Value} "; > > if ($cellObj->{Value} =~ m/name1|name2|name3/) > { > push $row, @index; > }else{ > print ?not a match \n?; > next; > } > } > > foreach my $i (@index) > { > copy target data from source file to destination file > } > > I'll have to test to see what the difference in CPU and memory utilization > are once I get into the office. Time to start my day! > > Thanks for all the suggestions! > > > John Warner > jwarner at texas.net > H: 512.251.1270 > C: 512.426.3813 > > > > > -----Original Message----- > From: austin-bounces+jwarner=texas.net at pm.org [mailto: > austin-bounces+jwarner =texas.net at pm.org] On > Behalf Of jameschoate at austin.rr.com > Sent: Thursday, July 23, 2009 1:04 PM > To: austin > Subject: Re: APM: Perl, Win32 OLE, and Excel > > Besides only processing each sub-set of each row via an input filter -- > > Assuming each row is not dependent on the other rows, why use an output > array at all? Just write it to a file. The only reason I can see writing > this to an array is to keep it in memory for subsequence processing. > > ---- Keith Howanitz wrote: > > On Thu, Jul 23, 2009 at 12:20 PM, John Warner wrote: > > > All, > > > > > > I have a project where I am trying to filter through a large amount of > data > > > from an Excel spreadsheet. Since I don't have access to the databases > where > > > the data actually resides, I have to use a spreadsheet that was given > to me. > > > The spreadsheet contains 79 columns and approximately 113k rows. The > data > > > are customer satisfaction survey results along with a plethora of other > > > garbage I don't need. I am only interested in a few columns. > > [SNIP] > > > > Have you tried putting a simple output when reading the xls file to > > show you how far you are getting in the file before having problems - > > maybe you are really going beyond 113k records, or it is choking on > > unusual data in one particular record. > > > > I wonder if you simply saved the xls file as a csv file and used the > > TEXT::CSV_XS module if you would still have troubles. > > > > If you want to read the whole thing into memory, you could always read > > each line, put the 3 important fields in array, and then read the next > > line so that you only end up with an array that is 113k x 3 rather > > than the whole spreadsheet in memory. > > _______________________________________________ > > Austin mailing list > > Austin at pm.org > > http://mail.pm.org/mailman/listinfo/austin > > -- > -- -- -- -- > Venimus, Vidimus, Dolavimus > > James Choate > jameschoate at austin.rr.com > james.choate at twcable.com > 512-657-1279 > www.ssz.com > http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu > http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center > > Adapt, Adopt, Improvise > -- -- -- -- > _______________________________________________ > Austin mailing list > Austin at pm.org > http://mail.pm.org/mailman/listinfo/austin > > > _______________________________________________ > Austin mailing list > Austin at pm.org > http://mail.pm.org/mailman/listinfo/austin > -- Pat Ludwig AIM: HaVoCPaT MSN:pludwigtx at hotmail.com GTalk:havoclad at gmail.com YiM:havoclad "Having a public that actually knows something is our best defense against ever again electing a president who knows nothing." -- Bill Maher 5/8/2009 -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.ellington at gmail.com Thu Jul 23 12:21:49 2009 From: e.ellington at gmail.com (Eric Ellington) Date: Thu, 23 Jul 2009 14:21:49 -0500 Subject: APM: Perl, Win32 OLE, and Excel In-Reply-To: <002701ca0bb9$dadfdd40$909f97c0$@net> References: <20090722190021.ODFSU.112705.root@hrndva-web14-z01> <002701ca0bb9$dadfdd40$909f97c0$@net> Message-ID: I used to do this a bunch. You mention 133k rows. Excel used to max out around something like 65k rows. Maybe I am out of date but how is so much data crammed into a single worksheet? What packages are you using? Thanks, Eric On Thu, Jul 23, 2009 at 12:20 PM, John Warner wrote: > All, > > I have a project where I am trying to filter through a large amount of data > from an Excel spreadsheet. ?Since I don't have access to the databases where > the data actually resides, I have to use a spreadsheet that was given to me. > The spreadsheet contains 79 columns and approximately 113k rows. ?The data > are customer satisfaction survey results along with a plethora of other > garbage I don't need. ?I am only interested in a few columns. > > My code goes like this... > > Create an Excel Object > Use Object to open Source and Destination spreadsheets > Find the column and row boundaries of where data is within the source. > > my @ArrayOfNames = ('Bill', 'Bob', 'Jane', 'Tom', 'Dick', 'Harry'); > > #Columns > # ? ? ? Source ? ? ? ? ?Destination ? ? ? ? ? ? Description > # ? ? ? Column ? ? ? ? ?Column > # ? ? ? ? ? ? ? 28 ? ? ? ? ? ? ? ? ? ? ?3 ? ? ? ? ? ? ? ? ? ? ? Responsible > Tech > # ? ? ? ? ? ? ? 55 ? ? ? ? ? ? ? ? ? ? ?5 ? ? ? ? ? ? ? ? ? ? ? Tech Sat > Score > # ? ? ? ? ? ? ? 57 ? ? ? ? ? ? ? ? ? ? ?6 ? ? ? ? ? ? ? ? ? ? ? Overall Sat > Score > # > foreach my $row (2..$LastRow) #skip header row on row 1 > { > ? ? ? ?#check the responsible tech > ? ? ? ?foreach my $t (@ArrayOfNames) > ? ? ? ?{ > ? ? ? ? ? ? ? ?my $cellObj = $srcSheet->Cells($row,28); > ? ? ? ? ? ? ? ?print "Current: ?$t ?\t Incident tech: ?$cellObj->{Value} "; > > ? ? ? ? ? ? ? ?if ($t =~ m/$srcSheet->Cells($row,28)->{Value}/) > ? ? ? ? ? ? ? ?{ > ? ? ? ? ? ? ? ? ? ? ? ?print "found a match!\n"; > ? ? ? ? ? ? ? ? ? ? ? ?if ($srcSheet->Cells($row,55)->{Value} < 7 || > $srcSheet->Cells($row,57)->{Value} < 7) > ? ? ? ? ? ? ? ? ? ? ? ?{ > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?#copy data from source to destination > ? ? ? ? ? ? ? ? ? ? ? ?} > ? ? ? ? ? ? ? ?}else{ > ? ? ? ? ? ? ? ? ? ? ? ?#print "not a match \n"; > ? ? ? ? ? ? ? ? ? ? ? ?next; > ? ? ? ? ? ? ? ?} > ? ? ? ?} > } > > My question: ?With 113k rows to go through, Perl runs out of memory and the > processing takes quite a while. ?How can I be more efficient? > > > John Warner > jwarner at texas.net > H: ?512.251.1270 > C: ?512.426.3813 > > > > > _______________________________________________ > Austin mailing list > Austin at pm.org > http://mail.pm.org/mailman/listinfo/austin > -- Eric Ellington e.ellington at gmail.com From rhennig at gmail.com Thu Jul 23 13:33:12 2009 From: rhennig at gmail.com (Randall Smith) Date: Thu, 23 Jul 2009 15:33:12 -0500 Subject: APM: Perl, Win32 OLE, and Excel In-Reply-To: References: <20090722190021.ODFSU.112705.root@hrndva-web14-z01> <002701ca0bb9$dadfdd40$909f97c0$@net> Message-ID: <5528c5fe0907231333p7426895eg8636d68d19627c2@mail.gmail.com> Is it possible for you to export the XLS file to a CSV and then process it that way without having to go through the OLE modules? I used to process Word documents using Perl and at a certain point I would run into issues with the OLE, or it would just take a long time since Perl was spending most of its time actually waiting on the OLE stuff to do its thing. If you can export it to a CSV. I haven't had issues with processing just CSV data. If you do need to write things back to a destination of some sort while you're processing, maybe importing it into a databse (MySQL, PostgreSQL) might be good, since you could create a database with a table holding the data you're processing and then create whatever other tables you need to store the results of your work. You could then dump the final product out into a CSV file (or files) and reprocess it as need be. Randy On Thu, Jul 23, 2009 at 2:21 PM, Eric Ellington wrote: > I used to do this a bunch. You mention 133k rows. Excel used to max > out around something like 65k rows. Maybe I am out of date but how is > so much data crammed into a single worksheet? > > What packages are you using? > > Thanks, > > Eric > > On Thu, Jul 23, 2009 at 12:20 PM, John Warner wrote: > > All, > > > > I have a project where I am trying to filter through a large amount of > data > > from an Excel spreadsheet. Since I don't have access to the databases > where > > the data actually resides, I have to use a spreadsheet that was given to > me. > > The spreadsheet contains 79 columns and approximately 113k rows. The > data > > are customer satisfaction survey results along with a plethora of other > > garbage I don't need. I am only interested in a few columns. > > > > My code goes like this... > > > > Create an Excel Object > > Use Object to open Source and Destination spreadsheets > > Find the column and row boundaries of where data is within the source. > > > > my @ArrayOfNames = ('Bill', 'Bob', 'Jane', 'Tom', 'Dick', 'Harry'); > > > > #Columns > > # Source Destination Description > > # Column Column > > # 28 3 > Responsible > > Tech > > # 55 5 Tech Sat > > Score > > # 57 6 Overall > Sat > > Score > > # > > foreach my $row (2..$LastRow) #skip header row on row 1 > > { > > #check the responsible tech > > foreach my $t (@ArrayOfNames) > > { > > my $cellObj = $srcSheet->Cells($row,28); > > print "Current: $t \t Incident tech: $cellObj->{Value} > "; > > > > if ($t =~ m/$srcSheet->Cells($row,28)->{Value}/) > > { > > print "found a match!\n"; > > if ($srcSheet->Cells($row,55)->{Value} < 7 || > > $srcSheet->Cells($row,57)->{Value} < 7) > > { > > #copy data from source to destination > > } > > }else{ > > #print "not a match \n"; > > next; > > } > > } > > } > > > > My question: With 113k rows to go through, Perl runs out of memory and > the > > processing takes quite a while. How can I be more efficient? > > > > > > John Warner > > jwarner at texas.net > > H: 512.251.1270 > > C: 512.426.3813 > > > > > > > > > > _______________________________________________ > > Austin mailing list > > Austin at pm.org > > http://mail.pm.org/mailman/listinfo/austin > > > > > > -- > Eric Ellington > e.ellington at gmail.com > _______________________________________________ > Austin mailing list > Austin at pm.org > http://mail.pm.org/mailman/listinfo/austin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jameschoate at austin.rr.com Fri Jul 24 07:12:29 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Fri, 24 Jul 2009 14:12:29 +0000 Subject: APM: TRG Maker SIG - Current Event Schedule Message-ID: <20090724141229.2WZ99.150183.root@hrndva-web14-z01> Thanks to everyone who participated in last nights first TRG Maker SIG, it was a full house! The next event is Aug. 13, 2009 and the schedule looks like: 7-8pm Soldering (if you have tools please bring them) 8-8:30pm TRG Regular Business 8:30 - 10:00 pm Lego Mindstorms We got three Lego bricks working last night so we're ready to move on to the next step. We'll be building a basic rover and creating our first development skeleton code. Once we can make the rovers follow a hard wired navigation plan we'll be adding a bump switch and some sort of optical sensor(s). Our playfield will be a 4ft x 4ft plywood arena. The background color is black and the circle (ie live area) is white. The initial rule is wander around the white area and avoid other rovers, but don't cross into the black region. ps I'll bring my Robosapien v1.0 and i-Sobot again, so if you have similar bots please bring them and you can use the ring for boxing practice :) -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jwarner at texas.net Mon Jul 27 08:10:26 2009 From: jwarner at texas.net (John Warner) Date: Mon, 27 Jul 2009 10:10:26 -0500 Subject: APM: Perl, Win32 OLE, and Excel In-Reply-To: <5528c5fe0907231333p7426895eg8636d68d19627c2@mail.gmail.com> References: <20090722190021.ODFSU.112705.root@hrndva-web14-z01> <002701ca0bb9$dadfdd40$909f97c0$@net> <5528c5fe0907231333p7426895eg8636d68d19627c2@mail.gmail.com> Message-ID: <006501ca0ecc$5fd23790$1f76a6b0$@net> Thank you all for the suggestions. I will try them out later this week. I especially like the idea of converting to CSV as it would be nice to work with something other than a 75MB Excel spreadsheet. Getting rid of the array with the names helped a lot but given that I am only using Win32::OLE and Win32::OLE::Const, I think bypassing OLE will give me more bang for the buck. Thanks! John From: austin-bounces+jwarner=texas.net at pm.org [mailto:austin-bounces+jwarner=texas.net at pm.org] On Behalf Of Randall Smith Sent: Thursday, July 23, 2009 3:33 PM To: Eric Ellington Cc: Austin: pm.org Subject: Re: APM: Perl, Win32 OLE, and Excel Is it possible for you to export the XLS file to a CSV and then process it that way without having to go through the OLE modules? I used to process Word documents using Perl and at a certain point I would run into issues with the OLE, or it would just take a long time since Perl was spending most of its time actually waiting on the OLE stuff to do its thing. If you can export it to a CSV. I haven't had issues with processing just CSV data. If you do need to write things back to a destination of some sort while you're processing, maybe importing it into a databse (MySQL, PostgreSQL) might be good, since you could create a database with a table holding the data you're processing and then create whatever other tables you need to store the results of your work. You could then dump the final product out into a CSV file (or files) and reprocess it as need be. Randy On Thu, Jul 23, 2009 at 2:21 PM, Eric Ellington wrote: I used to do this a bunch. You mention 133k rows. Excel used to max out around something like 65k rows. Maybe I am out of date but how is so much data crammed into a single worksheet? What packages are you using? Thanks, Eric On Thu, Jul 23, 2009 at 12:20 PM, John Warner wrote: > All, > > I have a project where I am trying to filter through a large amount of data > from an Excel spreadsheet. Since I don't have access to the databases where > the data actually resides, I have to use a spreadsheet that was given to me. > The spreadsheet contains 79 columns and approximately 113k rows. The data > are customer satisfaction survey results along with a plethora of other > garbage I don't need. I am only interested in a few columns. > > My code goes like this... > > Create an Excel Object > Use Object to open Source and Destination spreadsheets > Find the column and row boundaries of where data is within the source. > > my @ArrayOfNames = ('Bill', 'Bob', 'Jane', 'Tom', 'Dick', 'Harry'); > > #Columns > # Source Destination Description > # Column Column > # 28 3 Responsible > Tech > # 55 5 Tech Sat > Score > # 57 6 Overall Sat > Score > # > foreach my $row (2..$LastRow) #skip header row on row 1 > { > #check the responsible tech > foreach my $t (@ArrayOfNames) > { > my $cellObj = $srcSheet->Cells($row,28); > print "Current: $t \t Incident tech: $cellObj->{Value} "; > > if ($t =~ m/$srcSheet->Cells($row,28)->{Value}/) > { > print "found a match!\n"; > if ($srcSheet->Cells($row,55)->{Value} < 7 || > $srcSheet->Cells($row,57)->{Value} < 7) > { > #copy data from source to destination > } > }else{ > #print "not a match \n"; > next; > } > } > } > > My question: With 113k rows to go through, Perl runs out of memory and the > processing takes quite a while. How can I be more efficient? > > > John Warner > jwarner at texas.net > H: 512.251.1270 > C: 512.426.3813 > > > > > _______________________________________________ > Austin mailing list > Austin at pm.org > http://mail.pm.org/mailman/listinfo/austin > -- Eric Ellington e.ellington at gmail.com _______________________________________________ Austin mailing list Austin at pm.org http://mail.pm.org/mailman/listinfo/austin -------------- next part -------------- An HTML attachment was scrubbed... URL: From jameschoate at austin.rr.com Tue Jul 28 06:05:45 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Tue, 28 Jul 2009 13:05:45 +0000 Subject: APM: Confusion Research Center bi-Monthly "Research Safari" - Sat. Aug. 1 Message-ID: <20090728130546.GN9YJ.213451.root@hrndva-web14-z01> The initial meetup will be at 10am in front of the Student Union, West Campus. This is across the street from the Co-Op. Parking can be a problem so visit the UT Parking page. We will then move to a more appropriate library for our R&D. http://www.lib.utexas.edu/ http://www.utexas.edu/parking/ If you know of other developer, maker, builder, hacker, etc. groups that might be interested; please pass it along. -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Tue Jul 28 14:56:39 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Tue, 28 Jul 2009 21:56:39 +0000 Subject: APM: Sandia Studies Botnets In 1M OS Digital Petri Dish on Tuesday July 28, @05:44PM Message-ID: <20090728215639.XB55U.223617.root@hrndva-web15-z01> http://it.slashdot.org/story/09/07/28/1958237/Sandia-Studies-Botnets-In-1M-OS-Digital-Petri-Dish -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Thu Jul 30 11:03:53 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Thu, 30 Jul 2009 13:03:53 -0500 Subject: APM: J.UCS -- Journal of Universal Computer Science Message-ID: <20090730180353.8X69P.267813.root@hrndva-web18-z01> http://www.jucs.org/jucs -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- -- From jameschoate at austin.rr.com Thu Jul 30 14:43:40 2009 From: jameschoate at austin.rr.com (jameschoate at austin.rr.com) Date: Thu, 30 Jul 2009 21:43:40 +0000 Subject: APM: Computational Biology and Chemistry / Computers & Chemistry Message-ID: <20090730214340.BRPZ8.272836.root@hrndva-web18-z01> http://www.informatik.uni-trier.de/~ley/db/journals/candc/ -- -- -- -- -- Venimus, Vidimus, Dolavimus James Choate jameschoate at austin.rr.com james.choate at twcable.com 512-657-1279 www.ssz.com http://www.twine.com/twine/1128gqhxn-dwr/solar-soyuz-zaibatsu http://www.twine.com/twine/1178v3j0v-76w/confusion-research-center Adapt, Adopt, Improvise -- -- -- --