Mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vickyk <>
Subject RE: Dynamic Crawling, URL with query parameters.
Date Thu, 05 Jan 2017 03:21:06 GMT
Hello Markus,

I can't disclose the exact URL but I have been testing it with the the
following sample urls on local machine 

And the list will go on.

Every new search by the user creates a new URL with the different query
parameter, the more the number of users more will be ingested URL's.

Assuming the system had got N number of users and each making UR number of
unique searches then there will be total N*UR urls ingested.

Every time the user is making search to the system the crawling should
trigger, my concern would be about the scale when there are large number of
users say 1000000 ;)


View this message in context:
Sent from the Nutch - User mailing list archive at

View raw message