[Rio-pm] Ajuda com Threads - tempo de abertura

João André Simioni jasimioni em gmail.com
Segunda Julho 12 11:38:03 PDT 2010


O pessoal comentou - eu vou dar uma olhada nele com atenção.

Obrigado pela dica.

2010/7/12 Thiago Glauco Sanchez <thiagoglauco em ticursos.net>:
> O Perl tem um framework para aplicações em rede e Multitarefa chamada POE
> que
> "Provide a cooperative scheduling and multitasking environment rivalling
> threads and IPC"
> Não seria o caso de estudarmos a possibilidade de usar esse framework no seu
> sistema???
>
> Em 11/07/2010 23:57, João André Simioni escreveu:
>>
>> Foi um typo - erro de digitação - o Net::Ping testa TCP e UDP. O ping
>> externo ICMP.
>>
>> Mas assim, eu preciso resolver o problema com esse teste em Perl -
>> como eu disse, essa parte do código é só um pedaço de um sistema
>> maior. SNMP já é usado para várias outras coisas, mas dependem de
>> access lists em routers e eu não tenho como liberar.
>>
>> []'s
>>
>> 2010/7/11 Thiago Glauco Sanchez<thiagoglauco em ticursos.net>:
>>
>>>
>>> O que eu quero dizer: você testou com o ping externo que é um teste
>>> icmp...
>>> depois fez outro teste icmp com o Net::Ping...
>>>
>>> Não seria mais interessante testar de outra forma?
>>>
>>>    $p = Net::Ping->new("tcp", 2);
>>>    $p->port_number(getservbyname("http", "tcp"));
>>>
>>> Mais umas coisas: Suas máquinas/equipamentos em teste estão via internet
>>> ou
>>> rede local?
>>> Já pensou em usar SNMP? Já tive um problema parecido e iria utilizar um
>>> processo para cada equipamento por desempenho e acabai optando por
>>> utilizar
>>> o Net:SNMP
>>> que permite utilizar uma função de callback em vez de utilizar um
>>> processo
>>> para cada equipamento. Eu mando uma requisição para todos os equipamentos
>>> e
>>> conforme eles respondem
>>> a função de call-back trata a requisição.
>>>
>>> Porém, não conheço o seu ambiente. Apenas acho que criar 4000 threads não
>>> seja a melhor solução.
>>>
>>>
>>> Em 11/07/2010 23:18, João André Simioni escreveu:
>>>
>>>>
>>>> É typo - já está certo no código. ICMP vai via comando externo, UDP e
>>>> TCP via Net::Ping.
>>>>
>>>> Com relação aos testes, é normal um momento de sobrecarga causar a não
>>>> resposta de testes - essas sobrecargas não devem duram mais que 5
>>>> segundos. Por isso o sleep - assim eu faço o mesmo teste 3x, com
>>>> intervalos de 3 segundos.
>>>>
>>>> []'s
>>>>
>>>> On Sun, Jul 11, 2010 at 10:31 PM, Blabos de Blebe<blabos em gmail.com>
>>>>  wrote:
>>>>
>>>>
>>>>>
>>>>> Não seria um typo?
>>>>>
>>>>> icmp ->    ping;
>>>>> tcp e udp ->    Net::Ping
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Jul 11, 2010 at 6:42 PM, Thiago Glauco Sanchez
>>>>> <thiagoglauco em ticursos.net>    wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Sim sim, entendi sua preocupação.
>>>>>> Mas em vez de:
>>>>>>
>>>>>> for (1..3) {
>>>>>>           my $ping = `/bin/ping -c 1 -w 1 $ip`;
>>>>>>
>>>>>> não seria melhor já mandar pingar as 3 vezes no comando ou apenas 1?
>>>>>> Você
>>>>>> está gerando 3 processos externos sem necessidade real.
>>>>>>
>>>>>> outra... o comando ping já não é icmp?
>>>>>> my $p = Net::Ping->new('icmp')???
>>>>>>
>>>>>>
>>>>>> Em 11/07/2010 17:04, João André Simioni escreveu:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Eu quero fazer o ping de 3 formas - UDP Echo / TCP Echo e ICMP Echo.
>>>>>>> Alguns clientes tem filtros e assim eu evito falsos positivos.
>>>>>>>
>>>>>>> Net::Ping mesmo com root tem resultados incoerentes quando uso ICMP e
>>>>>>> Threads.
>>>>>>>
>>>>>>> []'s
>>>>>>>
>>>>>>> On Sun, Jul 11, 2010 at 4:56 PM, Thiago Glauco Sanchez
>>>>>>> <thiagoglauco em ticursos.net>      wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Endosso a solução do Lorn. Use o POE.
>>>>>>>>
>>>>>>>> E cara... pra que isso??
>>>>>>>>
>>>>>>>> sub checkIp {
>>>>>>>>    my $ip = shift;
>>>>>>>>    my $pingOk = 0;
>>>>>>>>
>>>>>>>>    eval {
>>>>>>>>        for (1..3) {
>>>>>>>>            my $ping = `/bin/ping -c 1 -w 1 $ip`;
>>>>>>>>
>>>>>>>>            if (grep { / 0% packet loss/ } $ping) {
>>>>>>>>                $pingOk = 1;
>>>>>>>>                return 1;
>>>>>>>>            }
>>>>>>>>
>>>>>>>>            if ($pingOk == 0) {
>>>>>>>>                my $p = Net::Ping->new('icmp');
>>>>>>>>                $p->service_check(1);
>>>>>>>>                if ($p->ping($ip, 1)) {
>>>>>>>>                    $pingOk = 1;
>>>>>>>>                    return 1;
>>>>>>>>                }
>>>>>>>>            }
>>>>>>>>
>>>>>>>>            if ($pingOk == 0) {
>>>>>>>>                my $p = Net::Ping->new('udp');
>>>>>>>>                $p->service_check(1);
>>>>>>>>                if ($p->ping($ip, 1)) {
>>>>>>>>                    $pingOk = 1;
>>>>>>>>                    return 1;
>>>>>>>>                }
>>>>>>>>            }
>>>>>>>>
>>>>>>>>            sleep 3;
>>>>>>>>        }
>>>>>>>>    };
>>>>>>>>
>>>>>>>> Para pingar com ping externo, depois com o Net::Ping via icmp e
>>>>>>>> depois
>>>>>>>> com
>>>>>>>> net::Ping udp? Ah... se não me engano, Net::Ping precisa de
>>>>>>>> permissão
>>>>>>>> root
>>>>>>>> para retornar um icmp de forma correta...
>>>>>>>>
>>>>>>>>
>>>>>>>> Em 09/07/2010 17:48, João André Simioni escreveu:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Caros,
>>>>>>>>>
>>>>>>>>> tenho um script para testar cerca de 4700 equipamentos, usando
>>>>>>>>> ping.
>>>>>>>>> Queria usar cerca 250 threads para isso, então o que eu fiz foi
>>>>>>>>> quebrar o grupo de 4300 em grupos 18 elementos e abri uma thread
>>>>>>>>> para
>>>>>>>>> cada.
>>>>>>>>>
>>>>>>>>> Não há nenhum problema com o código, o que acontece é que o tempo
>>>>>>>>> de
>>>>>>>>> abertura das threads é muito alto. E quanto mais threads eu abro,
>>>>>>>>> maior o tempo para abertura das próximas. As primeiras abrem em
>>>>>>>>> 30ms,
>>>>>>>>> mas perto da thread 70 esse tempo já vai para quase 1 segundo.
>>>>>>>>>
>>>>>>>>> Vocês sabem como otimizar esse tempo de abertura? Ou alguma outra
>>>>>>>>> solução?
>>>>>>>>>
>>>>>>>>> Segue o output do script como exemplo:
>>>>>>>>> Para 4677 cpes, e 250 threads, tenho no final 246 grupos, com 19
>>>>>>>>> elementos
>>>>>>>>> cada
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:50 2010.61858
>>>>>>>>> Created thread 1 (1) - took 0.030138 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:50 2010.92080
>>>>>>>>> Created thread 2 (2) - took 0.042668 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:50 2010.134968
>>>>>>>>> Created thread 3 (3) - took 0.064245 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:50 2010.199430
>>>>>>>>> Created thread 4 (4) - took 0.082642 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:50 2010.282278
>>>>>>>>> Created thread 5 (5) - took 0.128257 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:50 2010.410769
>>>>>>>>> Created thread 6 (6) - took 0.136458 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:50 2010.547383
>>>>>>>>> Created thread 7 (7) - took 0.208085 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:50 2010.755594
>>>>>>>>> Created thread 8 (8) - took 0.173014 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:50 2010.928756
>>>>>>>>> Created thread 9 (9) - took 0.132713 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:51 2010.61608
>>>>>>>>> Created thread 10 (10) - took 0.127795 seconds to create
>>>>>>>>> Trying to create thread in Fri Jul  9 17:46:51 2010.189536
>>>>>>>>> Created thread 11 (11) - took 0.301836 seconds to create
>>>>>>>>>
>>>>>>>>> Obrigado
>>>>>>>>>
>>>>>>>>> João André Simioni
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #!/usr/bin/perl
>>>>>>>>>
>>>>>>>>> use strict;
>>>>>>>>> use DBI;
>>>>>>>>> use Net::Ping;
>>>>>>>>> use Time::HiRes qw/usleep tv_interval gettimeofday/;
>>>>>>>>> use threads ('yield', 'stack_size' =>        32*4096, 'exit' =>
>>>>>>>>> 'threads_only', 'stringify');
>>>>>>>>>
>>>>>>>>> my $host = '192.168.160.179';
>>>>>>>>> my $sid  = 'HOMOLOG';
>>>>>>>>> my $user = 'mac_user';
>>>>>>>>> my $pass = 'mac_user';
>>>>>>>>>
>>>>>>>>> my $dbh = DBI->connect("dbi:Oracle:host=$host;sid=$sid", $user,
>>>>>>>>> $pass,
>>>>>>>>> { AutoCommit =>        1 });
>>>>>>>>>
>>>>>>>>> my $sth = $dbh->prepare('SELECT CPE_ID, CLIENTE_ID, CPE_DESC,
>>>>>>>>> CPE_IP_WAN, CPE_NODE, CPE_IF FROM MAC_CPE WHERE CPE_PING = ? AND
>>>>>>>>> CPE_ATIVO = ?');
>>>>>>>>> $sth->execute('1', 'A');
>>>>>>>>>
>>>>>>>>> my @cpes;
>>>>>>>>>
>>>>>>>>> while (my (@row) = $sth->fetchrow_array) {
>>>>>>>>>     push @cpes, [ @row ];
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> $dbh->disconnect();
>>>>>>>>>
>>>>>>>>> my $maxThreads = 250;
>>>>>>>>> my $total      = @cpes;
>>>>>>>>>
>>>>>>>>> my $cpePerThread = int($total / $maxThreads);
>>>>>>>>>
>>>>>>>>> my @toMonitor;
>>>>>>>>> my $i = 0;
>>>>>>>>> my $c = 0;
>>>>>>>>>
>>>>>>>>> foreach my $cpe (@cpes) {
>>>>>>>>>     push @{$toMonitor[$i]}, $cpe;
>>>>>>>>>     $c++;
>>>>>>>>>
>>>>>>>>>     if ($c>        $cpePerThread) {
>>>>>>>>>         $i++;
>>>>>>>>>         $c = 0;
>>>>>>>>>     }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> print "Para $total cpes, e $maxThreads threads, tenho no final $i
>>>>>>>>> grupos, com ", scalar @{$toMonitor[0]}, " elementos cada\n";
>>>>>>>>>
>>>>>>>>> sub processResult {
>>>>>>>>>     my $r = shift;
>>>>>>>>>     if (ref $r ne 'ARRAY') {
>>>>>>>>>         print STDERR "Erro na thread\n";
>>>>>>>>>         return 0;
>>>>>>>>>     }
>>>>>>>>>     my ($result, $host, $ifs) = @$r;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> sub testeClient {
>>>>>>>>>     my $threadNum = shift;
>>>>>>>>>     my $cpeGroup  = shift;
>>>>>>>>>     foreach my $cpe (@{$cpeGroup}) {
>>>>>>>>>         my ($cpeId, $clientId, $cpeDesc, $cpeIp, $cpeNode, $cpeIf)
>>>>>>>>> =
>>>>>>>>> @$cpe;
>>>>>>>>>         my $pingOk =&checkIp($cpeIp);
>>>>>>>>>         my $status = $pingOk ? 'RESPONDE' : 'MORTO';
>>>>>>>>>         # print join(", ", $threadNum, $cpeId, $clientId, $cpeDesc,
>>>>>>>>> $cpeIp, $cpeNode, $cpeIf, $status), "\n";
>>>>>>>>>     }
>>>>>>>>>     # print "Finishing thread $threadNum\n";
>>>>>>>>>     return([1, 0]);
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> my $threadCount = 1;
>>>>>>>>> foreach my $cpeGroup (@toMonitor) {
>>>>>>>>>     print "Trying to create thread";
>>>>>>>>>     my $t0 = [gettimeofday];
>>>>>>>>>
>>>>>>>>>     print " in ", scalar localtime $t0->[0], ".", $t0->[1], "\n";
>>>>>>>>>
>>>>>>>>>     my $thr = threads->create({scalar =>        '1'},
>>>>>>>>> 'testeClient',
>>>>>>>>> $threadCount, $cpeGroup);
>>>>>>>>>     my $elapsed = tv_interval ( $t0, [gettimeofday]);
>>>>>>>>>     print "Created thread $thr ($threadCount) - took $elapsed
>>>>>>>>> seconds
>>>>>>>>> to create\n";
>>>>>>>>>     $threadCount++;
>>>>>>>>>
>>>>>>>>>     while (threads->list()>= $maxThreads) {
>>>>>>>>>         my @joinable = threads->list(threads::joinable);
>>>>>>>>>         for (@joinable) {
>>>>>>>>>             my $r = $_->join();
>>>>>>>>>             &processResult($r);
>>>>>>>>>         }
>>>>>>>>>         usleep(10000);
>>>>>>>>>     }
>>>>>>>>>     my $elapsed = tv_interval ( $t0, [gettimeofday]);
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> while (threads->list()) {
>>>>>>>>>     my @joinable = threads->list(threads::joinable);
>>>>>>>>>     for (@joinable) {
>>>>>>>>>         my $r = $_->join();
>>>>>>>>>         &processResult($r);
>>>>>>>>>     }
>>>>>>>>>     usleep(10000);
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> sub checkIp {
>>>>>>>>>     my $ip = shift;
>>>>>>>>>     my $pingOk = 0;
>>>>>>>>>
>>>>>>>>>     eval {
>>>>>>>>>         for (1..3) {
>>>>>>>>>             my $ping = `/bin/ping -c 1 -w 1 $ip`;
>>>>>>>>>
>>>>>>>>>             if (grep { / 0% packet loss/ } $ping) {
>>>>>>>>>                 $pingOk = 1;
>>>>>>>>>                 return 1;
>>>>>>>>>             }
>>>>>>>>>
>>>>>>>>>             if ($pingOk == 0) {
>>>>>>>>>                 my $p = Net::Ping->new('icmp');
>>>>>>>>>                 $p->service_check(1);
>>>>>>>>>                 if ($p->ping($ip, 1)) {
>>>>>>>>>                     $pingOk = 1;
>>>>>>>>>                     return 1;
>>>>>>>>>                 }
>>>>>>>>>             }
>>>>>>>>>
>>>>>>>>>             if ($pingOk == 0) {
>>>>>>>>>                 my $p = Net::Ping->new('udp');
>>>>>>>>>                 $p->service_check(1);
>>>>>>>>>                 if ($p->ping($ip, 1)) {
>>>>>>>>>                     $pingOk = 1;
>>>>>>>>>                     return 1;
>>>>>>>>>                 }
>>>>>>>>>             }
>>>>>>>>>
>>>>>>>>>             sleep 3;
>>>>>>>>>         }
>>>>>>>>>     };
>>>>>>>>>
>>>>>>>>>     return $pingOk;
>>>>>>>>> }
>>>>>>>>> _______________________________________________
>>>>>>>>> Rio-pm mailing list
>>>>>>>>> Rio-pm em pm.org
>>>>>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What is the sound of Perl? Is it not the sound of a wall that people
>>>>>>>> have
>>>>>>>> stopped banging their heads against?
>>>>>>>> —Larry Wall
>>>>>>>>
>>>>>>>> Thiago Glauco Sanchez
>>>>>>>> Intrutor Perl e Redes
>>>>>>>> www.ticursos.net
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Rio-pm mailing list
>>>>>>>> Rio-pm em pm.org
>>>>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Rio-pm mailing list
>>>>>>> Rio-pm em pm.org
>>>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What is the sound of Perl? Is it not the sound of a wall that people
>>>>>> have
>>>>>> stopped banging their heads against?
>>>>>> —Larry Wall
>>>>>>
>>>>>> Thiago Glauco Sanchez
>>>>>> Intrutor Perl e Redes
>>>>>> www.ticursos.net
>>>>>>
>>>>>> _______________________________________________
>>>>>> Rio-pm mailing list
>>>>>> Rio-pm em pm.org
>>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Rio-pm mailing list
>>>>> Rio-pm em pm.org
>>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Rio-pm mailing list
>>>> Rio-pm em pm.org
>>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>>
>>>>
>>>>
>>>
>>> --
>>> What is the sound of Perl? Is it not the sound of a wall that people have
>>> stopped banging their heads against?
>>> —Larry Wall
>>>
>>> Thiago Glauco Sanchez
>>> Intrutor Perl e Redes
>>> www.ticursos.net
>>>
>>> _______________________________________________
>>> Rio-pm mailing list
>>> Rio-pm em pm.org
>>> http://mail.pm.org/mailman/listinfo/rio-pm
>>>
>>>
>>
>> _______________________________________________
>> Rio-pm mailing list
>> Rio-pm em pm.org
>> http://mail.pm.org/mailman/listinfo/rio-pm
>>
>>
>
>
> --
> What is the sound of Perl? Is it not the sound of a wall that people have
> stopped banging their heads against?
> —Larry Wall
>
> Thiago Glauco Sanchez
> Intrutor Perl e Redes
> www.ticursos.net
>
> _______________________________________________
> Rio-pm mailing list
> Rio-pm em pm.org
> http://mail.pm.org/mailman/listinfo/rio-pm
>


Mais detalhes sobre a lista de discussão Rio-pm