From fulko.hew at gmail.com Fri Oct 9 17:41:46 2020 From: fulko.hew at gmail.com (Fulko Hew) Date: Fri, 9 Oct 2020 20:41:46 -0400 Subject: [tpm] UDP broadcast/receive problems Message-ID: I'm trying to accomplish what I thought was easy, and yet I can't get it to work. I have a number of devices that listen on UDP port 9999. So I want to send a broadcast message to my network and see all of their specific responses, so I can collect the list of IP addresses (for subsequent messaging). For a single device, it looks like this: 192.168.1.149:port --> 255.255.255.255:9999 192.168.1.149:port <-- 192.168.1.130:9999 So I've tried a number of things starting from a single socket all the way to a send socket and a second receive socket, but I can never read the response that's sent. My last attempt is this code snippet... can anyone tell me what's wrong ? $out = IO::Socket::INET->new( PeerPort => 9999, PeerAddr => inet_ntoa(INADDR_BROADCAST), Proto => udp, ReuseAddr => 1, Broadcast => 1) or die "Can't bind : $@\n"; my $lport = $out->sockport(); # get the local port that was assigned print "sending from $lport\n"; $in = IO::Socket::INET->new( PeerPort => 9999, LocalPort => $lport, Proto => udp, ReuseAddr => 1) or die "Can't bind : $@\n"; my $s = IO::Select->new($in); $s->add($out); $out->send('hello') or die "send: $!"; while ($i++ < 100) { my @ready = $s->can_read(1); foreach (@ready) { die("readable but nothing read\n") unless defined($_->recv($rsp, 1024)); print $rsp; } } -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkeenan at pobox.com Fri Oct 9 18:05:49 2020 From: jkeenan at pobox.com (James E Keenan) Date: Fri, 9 Oct 2020 21:05:49 -0400 Subject: [tpm] UDP broadcast/receive problems In-Reply-To: References: Message-ID: <7c49514b-4973-91d3-4963-74227403e1f9@pobox.com> On 10/9/20 8:41 PM, Fulko Hew wrote: > I'm trying to accomplish what I thought was easy, and yet I can't get it > to work. > > I have a number of devices that listen on UDP port 9999. > So I want to send a broadcast message to my network and see all > of their specific responses, so I can collect the list of IP addresses > (for subsequent messaging). > For a single device, it looks like this: > > 192.168.1.149:port? --> 255.255.255.255:9999 > 192.168.1.149:port? <-- 192.168.1.130:9999 > > So I've tried a number of things starting from a single socket all the > way to > a send socket and a second receive socket, but I can never read the > response that's sent. > > My last attempt is this code snippet... can anyone tell me what's wrong ? > > > $out = IO::Socket::INET->new( > ? ? PeerPort ?=> 9999, > ? ? PeerAddr ?=> inet_ntoa(INADDR_BROADCAST), > ? ? Proto ? ? => udp, > ? ? ReuseAddr => 1, > ? ? Broadcast => 1) > ? ? ? ? or die "Can't bind : $@\n"; > > my $lport = $out->sockport(); ? ? ? ? ? # get the local port that was > assigned > print "sending from $lport\n"; > > $in = IO::Socket::INET->new( > ? ? PeerPort ?=> 9999, > ? ? LocalPort => $lport, > ? ? Proto ? ? => udp, > ? ? ReuseAddr => 1) > ? ? ? ? or die "Can't bind : $@\n"; > > my $s = IO::Select->new($in); > $s->add($out); > > $out->send('hello') or die "send: $!"; > > while ($i++ < 100) { > ? ? my @ready = $s->can_read(1); > ? ? foreach (@ready) { > ? ? ? ? die("readable but nothing read\n") unless > defined($_->recv($rsp, 1024)); > ? ? ? ? print $rsp; > ? ? } > } > First suggestion: use strict; In particular, I can't tell what you mean by $rsp From fulko.hew at gmail.com Fri Oct 9 18:42:19 2020 From: fulko.hew at gmail.com (Fulko Hew) Date: Fri, 9 Oct 2020 21:42:19 -0400 Subject: [tpm] UDP broadcast/receive problems In-Reply-To: <7c49514b-4973-91d3-4963-74227403e1f9@pobox.com> References: <7c49514b-4973-91d3-4963-74227403e1f9@pobox.com> Message-ID: On Fri, Oct 9, 2020 at 9:06 PM James E Keenan wrote: > On 10/9/20 8:41 PM, Fulko Hew wrote: > > I'm trying to accomplish what I thought was easy, and yet I can't get it > > to work. > > > > I have a number of devices that listen on UDP port 9999. > > So I want to send a broadcast message to my network and see all > > of their specific responses, so I can collect the list of IP addresses > > (for subsequent messaging). > > For a single device, it looks like this: > > > > 192.168.1.149:port --> 255.255.255.255:9999 < > http://255.255.255.255:9999> > > 192.168.1.149:port <-- 192.168.1.130:9999 > > > > So I've tried a number of things starting from a single socket all the > > way to > > a send socket and a second receive socket, but I can never read the > > response that's sent. > > > > My last attempt is this code snippet... can anyone tell me what's wrong ? > > > > > > $out = IO::Socket::INET->new( > > PeerPort => 9999, > > PeerAddr => inet_ntoa(INADDR_BROADCAST), > > Proto => udp, > > ReuseAddr => 1, > > Broadcast => 1) > > or die "Can't bind : $@\n"; > > > > my $lport = $out->sockport(); # get the local port that was > > assigned > > print "sending from $lport\n"; > > > > $in = IO::Socket::INET->new( > > PeerPort => 9999, > > LocalPort => $lport, > > Proto => udp, > > ReuseAddr => 1) > > or die "Can't bind : $@\n"; > > > > my $s = IO::Select->new($in); > > $s->add($out); > > > > $out->send('hello') or die "send: $!"; > > > > while ($i++ < 100) { > > my @ready = $s->can_read(1); > > foreach (@ready) { > > die("readable but nothing read\n") unless > > defined($_->recv($rsp, 1024)); > > print $rsp; > > } > > } > > > > First suggestion: use strict; > OK, I added strict. It didn't tell me anything I didn't already know. > In particular, I can't tell what you mean by $rsp > $rsp is the receive buffer that'll contain each response to my broadcast. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fulko.hew at gmail.com Fri Oct 9 19:22:04 2020 From: fulko.hew at gmail.com (Fulko Hew) Date: Fri, 9 Oct 2020 22:22:04 -0400 Subject: [tpm] UDP broadcast/receive problems In-Reply-To: References: Message-ID: On Fri, Oct 9, 2020 at 8:41 PM Fulko Hew wrote: > I'm trying to accomplish what I thought was easy, and yet I can't get it > to work. > ... snip ... As a followup to my own message... After having run my app a million times, and looking at the traffic on Wireshark... this last time (which I can honestly say I never saw before) I saw an ICMP message telling me my 'host was administratively unreachable'. So I turned off the firewall on Fedora, and I now see my inbound message. Yesterday I _was_ thinking it could have been a firewall issue, but I saw nothing in the transactions to lead me to believe that was my issue. Perhaps my earlier (same single socket) version of code had other issues and yet didn't cause ICMP to complain. Problem solved, for now. Next up would be 'how do I configure the firewall to (temporarily) accept incoming UDP to the same port as an outgoing UDP used? Thanks for listening. Fulko -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkeenan at pobox.com Fri Oct 9 19:26:03 2020 From: jkeenan at pobox.com (James E Keenan) Date: Fri, 9 Oct 2020 22:26:03 -0400 Subject: [tpm] UDP broadcast/receive problems In-Reply-To: References: <7c49514b-4973-91d3-4963-74227403e1f9@pobox.com> Message-ID: <9c8834dc-4f33-5933-0225-40cf8e04caa9@pobox.com> On 10/9/20 9:42 PM, Fulko Hew wrote: > > > On Fri, Oct 9, 2020 at 9:06 PM James E Keenan > wrote: > > On 10/9/20 8:41 PM, Fulko Hew wrote: > > I'm trying to accomplish what I thought was easy, and yet I can't > get it > > to work. > > > > I have a number of devices that listen on UDP port 9999. > > So I want to send a broadcast message to my network and see all > > of their specific responses, so I can collect the list of IP > addresses > > (for subsequent messaging). > > For a single device, it looks like this: > > > > 192.168.1.149:port? --> 255.255.255.255:9999 > > > 192.168.1.149:port? <-- 192.168.1.130:9999 > > > > > So I've tried a number of things starting from a single socket > all the > > way to > > a send socket and a second receive socket, but I can never read the > > response that's sent. > > > > My last attempt is this code snippet... can anyone tell me what's > wrong ? > > > > > > $out = IO::Socket::INET->new( > >? ? ? PeerPort ?=> 9999, > >? ? ? PeerAddr ?=> inet_ntoa(INADDR_BROADCAST), > >? ? ? Proto ? ? => udp, > >? ? ? ReuseAddr => 1, > >? ? ? Broadcast => 1) > >? ? ? ? ? or die "Can't bind : $@\n"; > > > > my $lport = $out->sockport(); ? ? ? ? ? # get the local port that > was > > assigned > > print "sending from $lport\n"; > > > > $in = IO::Socket::INET->new( > >? ? ? PeerPort ?=> 9999, > >? ? ? LocalPort => $lport, > >? ? ? Proto ? ? => udp, > >? ? ? ReuseAddr => 1) > >? ? ? ? ? or die "Can't bind : $@\n"; > > > > my $s = IO::Select->new($in); > > $s->add($out); > > > > $out->send('hello') or die "send: $!"; > > > > while ($i++ < 100) { In the line above, are you trying to say, "do this 100 times"? > >? ? ? my @ready = $s->can_read(1); > >? ? ? foreach (@ready) { > >? ? ? ? ? die("readable but nothing read\n") unless > > defined($_->recv($rsp, 1024)); What is $_ in the line above? > >? ? ? ? ? print $rsp; > >? ? ? } > > } > > > > First suggestion:? use strict; > > > OK, I added strict.? It didn't tell me anything I didn't already know. > > In particular, I can't tell what you mean by $rsp > > > $rsp is the receive buffer that'll contain each response to my broadcast. > > From fulko.hew at gmail.com Fri Oct 9 20:18:26 2020 From: fulko.hew at gmail.com (Fulko Hew) Date: Fri, 9 Oct 2020 23:18:26 -0400 Subject: [tpm] UDP broadcast/receive problems In-Reply-To: <9c8834dc-4f33-5933-0225-40cf8e04caa9@pobox.com> References: <7c49514b-4973-91d3-4963-74227403e1f9@pobox.com> <9c8834dc-4f33-5933-0225-40cf8e04caa9@pobox.com> Message-ID: On Fri, Oct 9, 2020 at 10:26 PM James E Keenan wrote: > On 10/9/20 9:42 PM, Fulko Hew wrote: > > > > > > On Fri, Oct 9, 2020 at 9:06 PM James E Keenan > > wrote: > > > > On 10/9/20 8:41 PM, Fulko Hew wrote: > > > I'm trying to accomplish what I thought was easy, and yet I can't > > get it > > > to work. > > > > > > I have a number of devices that listen on UDP port 9999. > > > So I want to send a broadcast message to my network and see all > > > of their specific responses, so I can collect the list of IP > > addresses > > > (for subsequent messaging). > > > For a single device, it looks like this: > > > > > > 192.168.1.149:port --> 255.255.255.255:9999 > > > > > 192.168.1.149:port <-- 192.168.1.130:9999 > > > > > > > > So I've tried a number of things starting from a single socket > > all the > > > way to > > > a send socket and a second receive socket, but I can never read > the > > > response that's sent. > > > > > > My last attempt is this code snippet... can anyone tell me what's > > wrong ? > > > > > > > > > $out = IO::Socket::INET->new( > > > PeerPort => 9999, > > > PeerAddr => inet_ntoa(INADDR_BROADCAST), > > > Proto => udp, > > > ReuseAddr => 1, > > > Broadcast => 1) > > > or die "Can't bind : $@\n"; > > > > > > my $lport = $out->sockport(); # get the local port that > > was > > > assigned > > > print "sending from $lport\n"; > > > > > > $in = IO::Socket::INET->new( > > > PeerPort => 9999, > > > LocalPort => $lport, > > > Proto => udp, > > > ReuseAddr => 1) > > > or die "Can't bind : $@\n"; > > > > > > my $s = IO::Select->new($in); > > > $s->add($out); > > > > > > $out->send('hello') or die "send: $!"; > > > > > > while ($i++ < 100) { > > In the line above, are you trying to say, "do this 100 times"? > Yes, although the foreach loop below allows me to scan and process all of the messages on each socket that 'currently' have messages to read... the while loop allows me to look for stuff a number of times, in case messages come in later. (the 1 in the can_read() signifies a one second timeout, so the 100 loops in the while() allows me to listen and process responses that can arrive up to 100 seconds later.) Don't obsess about the timeouts, it's just mainly for debugging purposes. Anyway, and my followup says, I now have to deal with the firewall config issue. > > > my @ready = $s->can_read(1); > > > foreach (@ready) { > > > die("readable but nothing read\n") unless > > > defined($_->recv($rsp, 1024)); > > What is $_ in the line above? > The next socket, from the list of sockets that have stuff awaiting to be read. > > > print $rsp; > > > } > > > } > > > > > > > First suggestion: use strict; > > > > > > OK, I added strict. It didn't tell me anything I didn't already know. > > > > In particular, I can't tell what you mean by $rsp > > > > > > $rsp is the receive buffer that'll contain each response to my broadcast. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From felipe at felipegasper.com Sun Oct 11 16:35:56 2020 From: felipe at felipegasper.com (Felipe Gasper) Date: Sun, 11 Oct 2020 19:35:56 -0400 Subject: [tpm] Scrape covid-19.ontario.ca? Message-ID: <3167EF00-13D9-45A8-92E2-829C4D60CC4F@felipegasper.com> Hi all, Has anyone tried to scrape https://covid19results.ehealthontario.ca:4443/agree? It?s not working for me w/ WWW::Mechanize, which I?m guessing is because the server does some sort of verification that the various line-noise-ish Google JS URLs embedded in the page actually get loaded. Thoughts, anyone? Thanks! cheers, -Felipe Gasper Mississauga From zoffix at zoffix.com Mon Oct 12 15:52:23 2020 From: zoffix at zoffix.com (zoffix at zoffix.com) Date: Mon, 12 Oct 2020 18:52:23 -0400 Subject: [tpm] Scrape covid-19.ontario.ca? In-Reply-To: <3167EF00-13D9-45A8-92E2-829C4D60CC4F@felipegasper.com> Message-ID: <20201012185223.Horde.AU23SiMKPWHz7r_Fe9vDepJ@echo.gendns.com> Hi, > Has anyone tried to scrape Well, it's against terms of use. So, hopefully no one :) https://covid19results.ehealthontario.ca:4443/terms It has a lot of JavaScript (see https://metacpan.org/pod/distribution/WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod#JavaScript ) You can pop open browser console (F12, usually), then go into network tab and see what kind of requests occur when you submit the form. Then you'd try to replicate that with Mech. I see a ReCaptcha thing at the bottom, so I'd assume there's more protection on that site. Cheers, ZZ Quoting Felipe Gasper : > Hi all, > > Has anyone tried to scrape > https://covid19results.ehealthontario.ca:4443/agree? > > It?s not working for me w/ WWW::Mechanize, which I?m guessing is > because the server does some sort of verification that the various > line-noise-ish Google JS URLs embedded in the page actually get > loaded. > > Thoughts, anyone? Thanks! > > cheers, > -Felipe Gasper > Mississauga > _______________________________________________ > toronto-pm mailing list > toronto-pm at pm.org > https://mail.pm.org/mailman/listinfo/toronto-pm From felipe at felipegasper.com Mon Oct 12 17:02:31 2020 From: felipe at felipegasper.com (Felipe Gasper) Date: Mon, 12 Oct 2020 20:02:31 -0400 Subject: [tpm] Scrape covid-19.ontario.ca? In-Reply-To: <20201012185223.Horde.AU23SiMKPWHz7r_Fe9vDepJ@echo.gendns.com> References: <20201012185223.Horde.AU23SiMKPWHz7r_Fe9vDepJ@echo.gendns.com> Message-ID: > On Oct 12, 2020, at 6:52 PM, zoffix at zoffix.com wrote: > > Hi, > >> Has anyone tried to scrape > > Well, it's against terms of use. So, hopefully no one :) https://covid19results.ehealthontario.ca:4443/terms Well I suppose there is that ? I was hoping to rig up something so I wouldn?t have to keep filling out that silly form every time I want to check. I get why they would only call if you?re positive, but at least an _email_ to let you know you?re negative would seem feasible. Eh well. At least testing is readily available. -F From olaf.alders at gmail.com Mon Oct 12 17:46:35 2020 From: olaf.alders at gmail.com (Olaf Alders) Date: Mon, 12 Oct 2020 20:46:35 -0400 Subject: [tpm] Scrape covid-19.ontario.ca? In-Reply-To: References: <20201012185223.Horde.AU23SiMKPWHz7r_Fe9vDepJ@echo.gendns.com> Message-ID: <1D8A937D-7BB5-437F-9967-97675E4A06C2@gmail.com> > On Oct 12, 2020, at 8:02 PM, Felipe Gasper wrote: > > > >> On Oct 12, 2020, at 6:52 PM, zoffix at zoffix.com wrote: >> >> Hi, >> >>> Has anyone tried to scrape >> >> Well, it's against terms of use. So, hopefully no one :) https://covid19results.ehealthontario.ca:4443/terms > > Well I suppose there is that ? > > I was hoping to rig up something so I wouldn?t have to keep filling out that silly form every time I want to check. I get why they would only call if you?re positive, but at least an _email_ to let you know you?re negative would seem feasible. > > Eh well. At least testing is readily available. ToS aside, you may have more luck with https://metacpan.org/pod/WWW::Mechanize::Firefox when scraping something that needs to deal with JavaScript. (I haven?t tried this one myself, though) Olaf -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.alders at gmail.com Thu Oct 29 15:56:35 2020 From: olaf.alders at gmail.com (Olaf Alders) Date: Thu, 29 Oct 2020 18:56:35 -0400 Subject: [tpm] Tonight's meeting Message-ID: <68282E85-5391-4547-A469-9FDA0C752E18@gmail.com> About to start at https://8x8.vc/acebxj/toperlmongers