[sf-perl] Testing a web crawler

[Neil Heller]

I have been asked to consider ways to test a web crawler (aka search
engine).

There are lots of "positive"-type tests I can think of that mostly deal with
"are the returned pages really relevant to the request".

How does one test for pages that are (or might be) relevant but were missed
by the web crawler?

What might be the best (is that a trick word?) or most optimized request
given someone's desire to find information?