[Rio-pm] Ajuda com Threads - tempo de abertura
Thiago Glauco Sanchez
thiagoglauco em ticursos.net
Segunda Julho 12 11:52:38 PDT 2010
Um pequeno exemplo que eu criei apenas para ilustrar... o pessoal que
desejar incrementar, por favor, be my guest.
use 5.12.0;
use POE;
use Net::Ping;
POE::Session->create(
inline_states => {
_start => sub {
my ($kernel) = $_[KERNEL];
print "Setting up a session\n";
$kernel->yield("ping");
},
ping => sub{my ($kernel) = $_[KERNEL];
my @ping = `ping -n 1 www.ticursos.net`;
map{say "ping externo ok" if $_ =~ /Perdidos = 0/ } @ping;
$kernel->delay_set("net_ping" => 3);
},
net_ping => sub{ my ($kernel) = $_[KERNEL];
my $p = Net::Ping->new();
$p->port_number(80);
my ($ret, $duration, $ip) = $p->ping('ticursos.net');
say "net::ping ok" if $ret;
$p->close();
$kernel->delay_set("ping" => 3)
},
},
);
say "Iniciando o Kernel...";
$poe_kernel->run( );
exit(0);
sub start {
my ($kernel) = $_[KERNEL];
say "Iniciando os testes com POE";
$kernel->yield("ping");
}
Em 12/07/2010 15:38, João André Simioni escreveu:
> O pessoal comentou - eu vou dar uma olhada nele com atenção.
>
> Obrigado pela dica.
>
> 2010/7/12 Thiago Glauco Sanchez<thiagoglauco em ticursos.net>:
>
>> O Perl tem um framework para aplicações em rede e Multitarefa chamada POE
>> que
>> "Provide a cooperative scheduling and multitasking environment rivalling
>> threads and IPC"
>> Não seria o caso de estudarmos a possibilidade de usar esse framework no seu
>> sistema???
>>
>> Em 11/07/2010 23:57, João André Simioni escreveu:
>>
>>> Foi um typo - erro de digitação - o Net::Ping testa TCP e UDP. O ping
>>> externo ICMP.
>>>
>>> Mas assim, eu preciso resolver o problema com esse teste em Perl -
>>> como eu disse, essa parte do código é só um pedaço de um sistema
>>> maior. SNMP já é usado para várias outras coisas, mas dependem de
>>> access lists em routers e eu não tenho como liberar.
>>>
>>> []'s
>>>
>>> 2010/7/11 Thiago Glauco Sanchez<thiagoglauco em ticursos.net>:
>>>
>>>
>>>> O que eu quero dizer: você testou com o ping externo que é um teste
>>>> icmp...
>>>> depois fez outro teste icmp com o Net::Ping...
>>>>
>>>> Não seria mais interessante testar de outra forma?
>>>>
>>>> $p = Net::Ping->new("tcp", 2);
>>>> $p->port_number(getservbyname("http", "tcp"));
>>>>
>>>> Mais umas coisas: Suas máquinas/equipamentos em teste estão via internet
>>>> ou
>>>> rede local?
>>>> Já pensou em usar SNMP? Já tive um problema parecido e iria utilizar um
>>>> processo para cada equipamento por desempenho e acabai optando por
>>>> utilizar
>>>> o Net:SNMP
>>>> que permite utilizar uma função de callback em vez de utilizar um
>>>> processo
>>>> para cada equipamento. Eu mando uma requisição para todos os equipamentos
>>>> e
>>>> conforme eles respondem
>>>> a função de call-back trata a requisição.
>>>>
>>>> Porém, não conheço o seu ambiente. Apenas acho que criar 4000 threads não
>>>> seja a melhor solução.
>>>>
>>>>
>>>> Em 11/07/2010 23:18, João André Simioni escreveu:
>>>>
>>>>
>>>>> É typo - já está certo no código. ICMP vai via comando externo, UDP e
>>>>> TCP via Net::Ping.
>>>>>
>>>>> Com relação aos testes, é normal um momento de sobrecarga causar a não
>>>>> resposta de testes - essas sobrecargas não devem duram mais que 5
>>>>> segundos. Por isso o sleep - assim eu faço o mesmo teste 3x, com
>>>>> intervalos de 3 segundos.
>>>>>
>>>>> []'s
>>>>>
>>>>> On Sun, Jul 11, 2010 at 10:31 PM, Blabos de Blebe<blabos em gmail.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Não seria um typo?
>>>>>>
>>>>>> icmp -> ping;
>>>>>> tcp e udp -> Net::Ping
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Jul 11, 2010 at 6:42 PM, Thiago Glauco Sanchez
>>>>>> <thiagoglauco em ticursos.net> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Sim sim, entendi sua preocupação.
>>>>>>> Mas em vez de:
>>>>>>>
>>>>>>> for (1..3) {
>>>>>>> my $ping = `/bin/ping -c 1 -w 1 $ip`;
>>>>>>>
>>>>>>> não seria melhor já mandar pingar as 3 vezes no comando ou apenas 1?
>>>>>>> Você
>>>>>>> está gerando 3 processos externos sem necessidade real.
>>>>>>>
>>>>>>> outra... o comando ping já não é icmp?
>>>>>>> my $p = Net::Ping->new('icmp')???
>>>>>>>
>>>>>>>
>>>>>>> Em 11/07/2010 17:04, João André Simioni escreveu:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Eu quero fazer o ping de 3 formas - UDP Echo / TCP Echo e ICMP Echo.
>>>>>>>> Alguns clientes tem filtros e assim eu evito falsos positivos.
>>>>>>>>
>>>>>>>> Net::Ping mesmo com root tem resultados incoerentes quando uso ICMP e
>>>>>>>> Threads.
>>>>>>>>
>>>>>>>> []'s
>>>>>>>>
>>>>>>>> On Sun, Jul 11, 2010 at 4:56 PM, Thiago Glauco Sanchez
>>>>>>>> <thiagoglauco em ticursos.net> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Endosso a solução do Lorn. Use o POE.
>>>>>>>>>
>>>>>>>>> E cara... pra que isso??
>>>>>>>>>
>>>>>>>>> sub checkIp {
>>>>>>>>> my $ip = shift;
>>>>>>>>> my $pingOk = 0;
>>>>>>>>>
>>>>>>>>> eval {
>>>>>>>>> for (1..3) {
>>>>>>>>> my $ping = `/bin/ping -c 1 -w 1 $ip`;
>>>>>>>>>
>>>>>>>>> if (grep { / 0% packet loss/ } $ping) {
>>>>>>>>> $pingOk = 1;
>>>>>>>>> return 1;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> if ($pingOk == 0) {
>>>>>>>>> my $p = Net::Ping->new('icmp');
>>>>>>>>> $p->service_check(1);
>>>>>>>>> if ($p->ping($ip, 1)) {
>>>>>>>>> $pingOk = 1;
>>>>>>>>> return 1;
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> if ($pingOk == 0) {
>>>>>>>>> my $p = Net::Ping->new('udp');
>>>>>>>>> $p->service_check(1);
>>>>>>>>> if ($p->ping($ip, 1)) {
>>>>>>>>> $pingOk = 1;
>>>>>>>>> return 1;
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> sleep 3;
>>>>>>>>> }
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> Para pingar com ping externo, depois com o Net::Ping via icmp e
>>>>>>>>> depois
>>>>>>>>> com
>>>>>>>>> net::Ping udp? Ah... se não me engano, Net::Ping precisa de
>>>>>>>>> permissão
>>>>>>>>> root
>>>>>>>>> para retornar um icmp de forma correta...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Em 09/07/2010 17:48, João André Simioni escreveu:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Caros,
>>>>>>>>>>
>>>>>>>>>> tenho um script para testar cerca de 4700 equipamentos, usando
>>>>>>>>>> ping.
>>>>>>>>>> Queria usar cerca 250 threads para isso, então o que eu fiz foi
>>>>>>>>>> quebrar o grupo de 4300 em grupos 18 elementos e abri uma thread
>>>>>>>>>> para
>>>>>>>>>> cada.
>>>>>>>>>>
>>>>>>>>>> Não há nenhum problema com o código, o que acontece é que o tempo
>>>>>>>>>> de
>>>>>>>>>> abertura das threads é muito alto. E quanto mais threads eu abro,
>>>>>>>>>> maior o tempo para abertura das próximas. As primeiras abrem em
>>>>>>>>>> 30ms,
>>>>>>>>>> mas perto da thread 70 esse tempo já vai para quase 1 segundo.
>>>>>>>>>>
>>>>>>>>>> Vocês sabem como otimizar esse tempo de abertura? Ou alguma outra
>>>>>>>>>> solução?
>>>>>>>>>>
>>>>>>>>>> Segue o output do script como exemplo:
>>>>>>>>>> Para 4677 cpes, e 250 threads, tenho no final 246 grupos, com 19
>>>>>>>>>> elementos
>>>>>>>>>> cada
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:50 2010.61858
>>>>>>>>>> Created thread 1 (1) - took 0.030138 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:50 2010.92080
>>>>>>>>>> Created thread 2 (2) - took 0.042668 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:50 2010.134968
>>>>>>>>>> Created thread 3 (3) - took 0.064245 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:50 2010.199430
>>>>>>>>>> Created thread 4 (4) - took 0.082642 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:50 2010.282278
>>>>>>>>>> Created thread 5 (5) - took 0.128257 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:50 2010.410769
>>>>>>>>>> Created thread 6 (6) - took 0.136458 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:50 2010.547383
>>>>>>>>>> Created thread 7 (7) - took 0.208085 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:50 2010.755594
>>>>>>>>>> Created thread 8 (8) - took 0.173014 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:50 2010.928756
>>>>>>>>>> Created thread 9 (9) - took 0.132713 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:51 2010.61608
>>>>>>>>>> Created thread 10 (10) - took 0.127795 seconds to create
>>>>>>>>>> Trying to create thread in Fri Jul 9 17:46:51 2010.189536
>>>>>>>>>> Created thread 11 (11) - took 0.301836 seconds to create
>>>>>>>>>>
>>>>>>>>>> Obrigado
>>>>>>>>>>
>>>>>>>>>> João André Simioni
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #!/usr/bin/perl
>>>>>>>>>>
>>>>>>>>>> use strict;
>>>>>>>>>> use DBI;
>>>>>>>>>> use Net::Ping;
>>>>>>>>>> use Time::HiRes qw/usleep tv_interval gettimeofday/;
>>>>>>>>>> use threads ('yield', 'stack_size' => 32*4096, 'exit' =>
>>>>>>>>>> 'threads_only', 'stringify');
>>>>>>>>>>
>>>>>>>>>> my $host = '192.168.160.179';
>>>>>>>>>> my $sid = 'HOMOLOG';
>>>>>>>>>> my $user = 'mac_user';
>>>>>>>>>> my $pass = 'mac_user';
>>>>>>>>>>
>>>>>>>>>> my $dbh = DBI->connect("dbi:Oracle:host=$host;sid=$sid", $user,
>>>>>>>>>> $pass,
>>>>>>>>>> { AutoCommit => 1 });
>>>>>>>>>>
>>>>>>>>>> my $sth = $dbh->prepare('SELECT CPE_ID, CLIENTE_ID, CPE_DESC,
>>>>>>>>>> CPE_IP_WAN, CPE_NODE, CPE_IF FROM MAC_CPE WHERE CPE_PING = ? AND
>>>>>>>>>> CPE_ATIVO = ?');
>>>>>>>>>> $sth->execute('1', 'A');
>>>>>>>>>>
>>>>>>>>>> my @cpes;
>>>>>>>>>>
>>>>>>>>>> while (my (@row) = $sth->fetchrow_array) {
>>>>>>>>>> push @cpes, [ @row ];
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> $dbh->disconnect();
>>>>>>>>>>
>>>>>>>>>> my $maxThreads = 250;
>>>>>>>>>> my $total = @cpes;
>>>>>>>>>>
>>>>>>>>>> my $cpePerThread = int($total / $maxThreads);
>>>>>>>>>>
>>>>>>>>>> my @toMonitor;
>>>>>>>>>> my $i = 0;
>>>>>>>>>> my $c = 0;
>>>>>>>>>>
>>>>>>>>>> foreach my $cpe (@cpes) {
>>>>>>>>>> push @{$toMonitor[$i]}, $cpe;
>>>>>>>>>> $c++;
>>>>>>>>>>
>>>>>>>>>> if ($c> $cpePerThread) {
>>>>>>>>>> $i++;
>>>>>>>>>> $c = 0;
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> print "Para $total cpes, e $maxThreads threads, tenho no final $i
>>>>>>>>>> grupos, com ", scalar @{$toMonitor[0]}, " elementos cada\n";
>>>>>>>>>>
>>>>>>>>>> sub processResult {
>>>>>>>>>> my $r = shift;
>>>>>>>>>> if (ref $r ne 'ARRAY') {
>>>>>>>>>> print STDERR "Erro na thread\n";
>>>>>>>>>> return 0;
>>>>>>>>>> }
>>>>>>>>>> my ($result, $host, $ifs) = @$r;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> sub testeClient {
>>>>>>>>>> my $threadNum = shift;
>>>>>>>>>> my $cpeGroup = shift;
>>>>>>>>>> foreach my $cpe (@{$cpeGroup}) {
>>>>>>>>>> my ($cpeId, $clientId, $cpeDesc, $cpeIp, $cpeNode, $cpeIf)
>>>>>>>>>> =
>>>>>>>>>> @$cpe;
>>>>>>>>>> my $pingOk =&checkIp($cpeIp);
>>>>>>>>>> my $status = $pingOk ? 'RESPONDE' : 'MORTO';
>>>>>>>>>> # print join(", ", $threadNum, $cpeId, $clientId, $cpeDesc,
>>>>>>>>>> $cpeIp, $cpeNode, $cpeIf, $status), "\n";
>>>>>>>>>> }
>>>>>>>>>> # print "Finishing thread $threadNum\n";
>>>>>>>>>> return([1, 0]);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> my $threadCount = 1;
>>>>>>>>>> foreach my $cpeGroup (@toMonitor) {
>>>>>>>>>> print "Trying to create thread";
>>>>>>>>>> my $t0 = [gettimeofday];
>>>>>>>>>>
>>>>>>>>>> print " in ", scalar localtime $t0->[0], ".", $t0->[1], "\n";
>>>>>>>>>>
>>>>>>>>>> my $thr = threads->create({scalar => '1'},
>>>>>>>>>> 'testeClient',
>>>>>>>>>> $threadCount, $cpeGroup);
>>>>>>>>>> my $elapsed = tv_interval ( $t0, [gettimeofday]);
>>>>>>>>>> print "Created thread $thr ($threadCount) - took $elapsed
>>>>>>>>>> seconds
>>>>>>>>>> to create\n";
>>>>>>>>>> $threadCount++;
>>>>>>>>>>
>>>>>>>>>> while (threads->list()>= $maxThreads) {
>>>>>>>>>> my @joinable = threads->list(threads::joinable);
>>>>>>>>>> for (@joinable) {
>>>>>>>>>> my $r = $_->join();
>>>>>>>>>> &processResult($r);
>>>>>>>>>> }
>>>>>>>>>> usleep(10000);
>>>>>>>>>> }
>>>>>>>>>> my $elapsed = tv_interval ( $t0, [gettimeofday]);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> while (threads->list()) {
>>>>>>>>>> my @joinable = threads->list(threads::joinable);
>>>>>>>>>> for (@joinable) {
>>>>>>>>>> my $r = $_->join();
>>>>>>>>>> &processResult($r);
>>>>>>>>>> }
>>>>>>>>>> usleep(10000);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> sub checkIp {
>>>>>>>>>> my $ip = shift;
>>>>>>>>>> my $pingOk = 0;
>>>>>>>>>>
>>>>>>>>>> eval {
>>>>>>>>>> for (1..3) {
>>>>>>>>>> my $ping = `/bin/ping -c 1 -w 1 $ip`;
>>>>>>>>>>
>>>>>>>>>> if (grep { / 0% packet loss/ } $ping) {
>>>>>>>>>> $pingOk = 1;
>>>>>>>>>> return 1;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> if ($pingOk == 0) {
>>>>>>>>>> my $p = Net::Ping->new('icmp');
>>>>>>>>>> $p->service_check(1);
>>>>>>>>>> if ($p->ping($ip, 1)) {
>>>>>>>>>> $pingOk = 1;
>>>>>>>>>> return 1;
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> if ($pingOk == 0) {
>>>>>>>>>> my $p = Net::Ping->new('udp');
>>>>>>>>>> $p->service_check(1);
>>>>>>>>>> if ($p->ping($ip, 1)) {
>>>>>>>>>> $pingOk = 1;
>>>>>>>>>> return 1;
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> sleep 3;
>>>>>>>>>> }
>>>>>>>>>> };
>>>>>>>>>>
>>>>>>>>>> return $pingOk;
>>>>>>>>>> }
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Rio-pm mailing list
>>>>>>>>>> Rio-pm em pm.org
>>>>>>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What is the sound of Perl? Is it not the sound of a wall that people
>>>>>>>>> have
>>>>>>>>> stopped banging their heads against?
>>>>>>>>> —Larry Wall
>>>>>>>>>
>>>>>>>>> Thiago Glauco Sanchez
>>>>>>>>> Intrutor Perl e Redes
>>>>>>>>> www.ticursos.net
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Rio-pm mailing list
>>>>>>>>> Rio-pm em pm.org
>>>>>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Rio-pm mailing list
>>>>>>>> Rio-pm em pm.org
>>>>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> What is the sound of Perl? Is it not the sound of a wall that people
>>>>>>> have
>>>>>>> stopped banging their heads against?
>>>>>>> —Larry Wall
>>>>>>>
>>>>>>> Thiago Glauco Sanchez
>>>>>>> Intrutor Perl e Redes
>>>>>>> www.ticursos.net
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Rio-pm mailing list
>>>>>>> Rio-pm em pm.org
>>>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Rio-pm mailing list
>>>>>> Rio-pm em pm.org
>>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Rio-pm mailing list
>>>>> Rio-pm em pm.org
>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> What is the sound of Perl? Is it not the sound of a wall that people have
>>>> stopped banging their heads against?
>>>> —Larry Wall
>>>>
>>>> Thiago Glauco Sanchez
>>>> Intrutor Perl e Redes
>>>> www.ticursos.net
>>>>
>>>> _______________________________________________
>>>> Rio-pm mailing list
>>>> Rio-pm em pm.org
>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Rio-pm mailing list
>>> Rio-pm em pm.org
>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>
>>>
>>>
>>
>> --
>> What is the sound of Perl? Is it not the sound of a wall that people have
>> stopped banging their heads against?
>> —Larry Wall
>>
>> Thiago Glauco Sanchez
>> Intrutor Perl e Redes
>> www.ticursos.net
>>
>> _______________________________________________
>> Rio-pm mailing list
>> Rio-pm em pm.org
>> http://mail.pm.org/mailman/listinfo/rio-pm
>>
>>
> _______________________________________________
> Rio-pm mailing list
> Rio-pm em pm.org
> http://mail.pm.org/mailman/listinfo/rio-pm
>
>
--
What is the sound of Perl? Is it not the sound of a wall that people have
stopped banging their heads against?
—Larry Wall
Thiago Glauco Sanchez
Intrutor Perl e Redes
www.ticursos.net
Mais detalhes sobre a lista de discussão Rio-pm