Thiago, muito obrigado pela excelente resposta!<div>Realmente, há muita coisa a ser considerada, todavia, ainda estou longe de chegar no gargalo do TCP/IP em si.</div><div>O que eu percebi é que, infelizmente, não existe nenhum módulo no CPAN apropriado para gerenciar downloads de listas longas de URLs (as minhas listas são de ~500 mil URLs de arquivos estáticos por sessão). Sim, é possível usar LWP e apelar para um fork(), mas aí vi que consumia RAM/CPU demais, *quando comparado a um processo externo*.</div>
<div>E, para mim, como programador Perl, é deveras frustrante delegar os downloads ao lftp (<a href="http://lftp.yar.ru/">http://lftp.yar.ru/</a>)... Afinal, o que o lftp tem de tão bom que eu não possa fazer em Perl?! ;)</div>
<div>Então, implementei o AnyEvent::Net::Curl::Queued, que é capaz de gerenciar filas bem grandes de downloads, de forma assíncrona. Mas será que compensa usá-lo ao invés do lftp? Para isso, comecei fazendo benchmarks.</div>
<div>Agradeço a todos pelas sugestões! Seguem as minhas observações:</div><div><ol><li>Eden, você está certo, os "filhotes" do LWP tem uma inicialização muito lenta; e o mais curioso é que boa parte da inicialização não está no new() propriamente dito, mas é feita durante o primeiro request(). Então, para ser justo, dividi a minha fila de downloads em 10 filas separadas, e aloquei um fork() para cada uma;</li>
<li>LWP::Curl e AnyEvent::Multi::Curl são ambos derivados do WWW::Curl. Não vale a pena testar o WWW::Curl "cru", na minha opinião;</li><li>LWP::Curl + forks é mais rápido do que a interface CLI do curl (ambos paralelizados do mesmo jeito que o LWP, com 10 filas independentes)!!! Atribuo isso a uma esquisitice do curl na hora de extrair as informações sobre o download: por alguma razão, esse processo é MUITO lento;</li>
<li>wget tem o menor overhead de todas as alternativas por ser ridiculamente simples. Quase não tem dependências, por isso a sua inicialização é quase que instantânea;</li><li>AnyEvent::HTTP é a melhor alternativa para acessar uma quantidade limitada de URLs *em paralelo*. Porém, para listas razoavelmente grandes, gerenciamento via callbacks e closures é meio tenebroso...</li>
<li>Reiterando: o *MEU* interesse se restringe a pegar meio milhão de URLs e baixar de forma mais eficiente possível (isso é, o gargalo deve ser minha bandwidth, não minha CPU/RAM). Nesse caso, pouco me importa a possibilidade de fazer parsing de headers, gerenciar cookies ou processamento de HTML. Isso é, quando eu quiser, devo poder habilitar tudo isso; mas o default deve ser OFF. Portanto, o meu "Holy Grail" seria um libwget, que gerencia filas, extrai links e faz downloads recursivos :)</li>
</ol><div>Para fechar, segue a tabela de benchmark atualizada:</div></div><div><br></div><div><span class="Apple-style-span" style="font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace; line-height: 15px; background-color: rgb(250, 250, 250); "><font class="Apple-style-span" size="1"><div class="line number1 index0 alt2" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl spaces" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; "> </code><code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">URL/s WWW::Mechanize LWP::UserAgent HTTP::Lite HTTP::Tiny AnyEvent::Curl::Multi lftp AnyEvent::Net::Curl::Queued AnyEvent::HTTP curl LWP::Curl wget</code></div>
<div class="line number2 index1 alt1" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">WWW::Mechanize 196 -- -60% -80% -85% -86% -88% -89% -92% -97% -97% -100%</code></div>
<div class="line number3 index2 alt2" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">LWP::UserAgent 484 148% -- -51% -63% -66% -70% -72% -80% -93% -93% -99%</code></div>
<div class="line number4 index3 alt1" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">HTTP::Lite 989 405% 104% -- -25% -32% -39% -42% -59% -85% -86% -99%</code></div>
<div class="line number5 index4 alt2" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">HTTP::Tiny 1312 569% 170% 33% -- -9% -19% -23% -46% -80% -82% -99%</code></div>
<div class="line number6 index5 alt1" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">AnyEvent::Curl::Multi 1446 638% 198% 46% 10% -- -10% -16% -41% -78% -80% -98%</code></div>
<div class="line number7 index6 alt2" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">lftp 1609 722% 232% 63% 23% 11% -- -6% -34% -75% -77% -98%</code></div>
<div class="line number8 index7 alt1" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">AnyEvent::Net::Curl::Queued 1713 773% 253% 73% 30% 18% 6% -- -30% -74% -76% -98%</code></div>
<div class="line number9 index8 alt2" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">AnyEvent::HTTP 2437 1144% 403% 146% 86% 69% 51% 42% -- -63% -66% -97%</code></div>
<div class="line number10 index9 alt1" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">curl 6512 3228% 1244% 559% 397% 351% 305% 281% 167% -- -8% -93%</code></div>
<div class="line number11 index10 alt2" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">LWP::Curl 7110 3524% 1364% 618% 442% 391% 341% 315% 191% 9% -- -92%</code></div>
<div class="line number12 index11 alt1" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 1em !important; padding-bottom: 0px !important; padding-left: 0em !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-family: Consolas, 'Bitstream Vera Sans Mono', 'Courier New', Courier, monospace !important; font-style: normal !important; min-height: inherit !important; white-space: pre !important; ">
<code class="pl plain" style="margin-top: 0px !important; margin-right: 0px !important; margin-bottom: 0px !important; margin-left: 0px !important; padding-top: 0px !important; padding-right: 0px !important; padding-bottom: 0px !important; padding-left: 0px !important; border-top-width: 0px !important; border-right-width: 0px !important; border-bottom-width: 0px !important; border-left-width: 0px !important; border-style: initial; border-color: initial; font: inherit; vertical-align: baseline !important; font-family: monospace; border-top-left-radius: 0px 0px !important; border-top-right-radius: 0px 0px !important; border-bottom-right-radius: 0px 0px !important; border-bottom-left-radius: 0px 0px !important; background-image: none !important; background-attachment: initial !important; background-origin: initial !important; background-clip: initial !important; background-color: initial !important; border-style: initial !important; border-color: initial !important; bottom: auto !important; float: none !important; height: auto !important; left: auto !important; line-height: 1.2em !important; outline-width: 0px !important; outline-style: initial !important; outline-color: initial !important; overflow-x: visible !important; overflow-y: visible !important; position: static !important; right: auto !important; text-align: left !important; top: auto !important; width: auto !important; box-sizing: content-box !important; font-style: normal !important; min-height: inherit !important; color: black !important; ">wget 88875 45240% 18215% 8877% 6675% 6045% 5418% 5092% 3544% 1262% 1151% --</code></div>
</font></span><div><br></div>ABS()<br><br>
<br><br><div class="gmail_quote">2011/10/20 Thiago Rondon <span dir="ltr"><<a href="mailto:thiago@aware.com.br">thiago@aware.com.br</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Thu, Oct 20, 2011 at 05:25:20PM -0200, Stanislaw Pusep wrote:<br>
> OK, aqui vai o spoiler: estou fazendo benchmark (de overhead) de todos os<br>
</div>> HTTP agents que conheAS:o. E oA LWP estA! *MUITO* feio na fita:<br>
<div class="im">> <a href="https://metacpan.org/module/AnyEvent::Net::Curl::Queued#OVERHEAD" target="_blank">https://metacpan.org/module/AnyEvent::Net::Curl::Queued#OVERHEAD</a><br>
> ABS()<br>
<br>
</div>Stan,<br>
<br>
Eu vou apontar para outro lado, já que me parece que teu objetivo é<br>
perfomance para efetuar download de arquivos via http (tcp), e você<br>
já esta testando as alternativas que há como módulo para realizar<br>
tua tarefa. :)<br>
<br>
Me parece que teu caso é:<br>
<br>
- Uso do protocolo HTTP para baixar arquivos ;<br>
- Encontrar o melhor algoritimo para trabalhar em paralelo no seu<br>
aplicativo;<br>
- Verificar se você tem "banda" necessária para este trabalho.<br>
<br>
Primeiramente, é interessante saber, quanto de banda você tem<br>
disponível ? Qual o tamanho dos arquivos que você vai efetuar download ?<br>
Ou seja, qual sua capacidade para buscar os arquivos, e qual<br>
'concorrencia' ?<br>
<br>
Como é um caso para conectividade em paralelo, é interessante levantar<br>
alguns pontos da tua rede e do teu sistema operacional para saber se ele<br>
suporta e esta preparado para esta demanda. Isto pode afetar o comportamento<br>
de alguns módulos.<br>
<br>
Apesar do kernel 2.6 já ter muita coisa para o "autotuning" na pilha<br>
TCP, é interessante olhar para alguns parametros, tais como:<br>
<br>
- net/core/[r,w]mem_max e net/ipv4/tcp_[r,w]mem pode ser muito interessante em<br>
casos onde há conexões em paralelo, com excesso uso de conexões e<br>
dados.<br>
<br>
- net/ipv4/tcp_available_congestion_control - Este é um assunto que vale<br>
ler, dependendo do local, da 'qualidade do link' você pode optar por<br>
um algoritimo, se tiver banda em excesso e etc, pode ser outra. Há<br>
muita atualização destes algoritimos ao longo do desenvolvimento do<br>
kernel 2.6, vale verificar e atualizar o kernel se você for alterar<br>
este parametro.<br>
<br>
- txqueuelen é um bom parametro também para ser trabalhado, pois pode te<br>
oferecer uma boa perfomance neste cenário também, este parametro é<br>
efetuado no device, por exemplo 'ifconfig eth0 txqueuelen 100000'<br>
<br>
Em relação ao algoritimo de concorrencia, dependendo da quantidade que<br>
você quer, e se você esta de olho em perfomance, é interessante olhar<br>
para o epoll no Linux. Eu ainda não usei os módulos que trabalham<br>
diretamente com ele, como o IO::Epool, mas há muita coisa surgindo<br>
baseado nesta caracteristica "nova" que esta disponível no kernel,<br>
tais como nginx e etc. Fica uma pergunta.. Será que isto vai fazer<br>
realmente diferença ? Eu não sei, mas gostaria de ver seus resultados<br>
aqui. :)<br>
<br>
Agora, o HTTP você pode utilizar o keep alive, dependendo também das<br>
suas requisições, são para o mesmo host ? Se você tiver uma lista, é<br>
uma boa ordenar por host, e dividir elas entre os processos que você<br>
vai separar, ou seja, se você tiver 30 arquivos no host X, e 30 no<br>
arquivo Y, organize eles para se aproveitar do keep alive, ou se for<br>
tudo diferente desative ele.<br>
<br>
Se os arquivos não foram compactados, e dependendo do tipo, e se<br>
banda for um 'problema', talvez vale apena ativar a compactação para<br>
buscar os arquivos, onde os servidores suportarem.<br>
<br>
Se a lista for 'grande' e 'diversificada', use o dns cache, e vá<br>
resolvando os nomes antes, para economizar uma etapa na hora que<br>
for para 'buscar'.<br>
<br>
Tome cuidado com roteadores bizarros que 'zoem' o MTU, isto pode<br>
afetar bastante, um MTU alto pode te ajudar com transferir pacotes<br>
com dados grandes.<br>
<br>
Enfim, foram ideias que tive agora sobre o problema. :)<br>
<br>
Acredito que o restante da discussão, é o que você já tá procurando,<br>
que é como os módulos trabalham, e efetuam a tarefa que você quer,<br>
mas poste os resultados aqui, pois também estou curioso. :-)<br>
<br>
Ps.: Lembre-se que todos os valores no /proc são volateis, então use o sysctl<br>
para manter ele persistente no teu sistema. :)<br>
<br>
Abs!<br>
<font color="#888888">-Thiago Rondon<br>
</font><div><div></div><div class="h5">=begin disclaimer<br>
Sao Paulo Perl Mongers: <a href="http://sao-paulo.pm.org/" target="_blank">http://sao-paulo.pm.org/</a><br>
SaoPaulo-pm mailing list: <a href="mailto:SaoPaulo-pm@pm.org">SaoPaulo-pm@pm.org</a><br>
L<<a href="http://mail.pm.org/mailman/listinfo/saopaulo-pm" target="_blank">http://mail.pm.org/mailman/listinfo/saopaulo-pm</a>><br>
=end disclaimer<br>
</div></div></blockquote></div><br></div>