Mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Nagel <wastl.na...@googlemail.com>
Subject Re: Fetcher "hung while processing"
Date Sun, 18 Dec 2016 22:37:56 GMT
Hi Michael,

> I have less than the default amount of memory allocated: mapreduce.map.memory.mb=500;
mapreduce.map.java.opts=-Xmx300m.

300 MB Java heap is probably not enough. A fetcher with 50 threads is expected keep in memory
- 50 open connections with 50 buffers to hold the fetched content
- the queues with URLs to fetch next
- parsed and cached robots.txt rules

Afaiks, also Hadoop needs a certain amount of memory to buffer the output before it's spilled
to
disk.

If a task runs low in memory, it naturally becomes slower because CPU is used for garbage
collecting. That could explain why the hang ups happen only sometimes somewhat unpredictable.

Best,
Sebastian


On 12/16/2016 08:17 PM, Michael Coffey wrote:
> Hello again, here is a lot more detail on my issue.
> 
> I have had the same problem with many different runs, but certainly not all of them.
Typically, it seems to happen on one of several nodes of the cluster (but not the same node
each time). One time, I used numThreads=200, instead of 50, and I got 200 hung threads at
the end of the fetching task.
> Nutch version is 1.12. It is running distributed on Hadoop with anywhere from 2 to 7
slave nodes. I have no explicit timeout settings in mapred-site.xml, nutch-site.xml, or the
crawl script. I have the following fetcher.* props: fetcher.threads.per.queue=1; fetcher.timelimit.mins=180;
fetcher.server.delay=1.0. I also have generate.max.count=25.
> 
> I have less than the default amount of memory allocated: mapreduce.map.memory.mb=500;
mapreduce.map.java.opts=-Xmx300m.
> 
> Looking at the logs, I don't see anything very obvious. In the 200-thread case, the word
"error" appeared only about 70 times. 
> 
> Here is an example describing 3 of the 50 hung threads of one run that attemps to fetch
about 300 resources.
> 2016-12-16 04:22:45,644 WARN [main] org.apache.nutch.fetcher.Fetcher: Aborting with 50
hung threads.
> 2016-12-16 04:22:45,645 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread #0 hung
while processing http://aktr.jp/news/x-girl-sports-x-aktr-02-%E3%83%97%E3%83%AD%E3%82%B8%E3%82%A7%E3%82%AF%E3%83%88%E3%82%B5%E3%82%A4%E3%83%88%E5%85%AC%E9%96%8B/
> 2016-12-16 04:22:45,647 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread #1 hung
while processing http://www.googletagmanager.com/ns.html?id=GTM-MF52KS
> 2016-12-16 04:22:45,648 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread #2 hung
while processing http://mashable.com/2016/12/14/teens-drinking-down/
> 
> Earlier in the log2016-12-16 04:12:38,315 INFO [FetcherThread] org.apache.nutch.fetcher.FetcherThread:
fetching http://aktr.jp/news/x-girl-sports-x-aktr-02-%E3%83%97%E3%83%AD%E3%82%B8%E3%82%A7%E3%82%AF%E3%83%88%E3%82%B5%E3%82%A4%E3%83%88%E5%85%AC%E9%96%8B/
(queue crawl delay=1000ms)
> 
> And there was no other mention of aktr.jp/news/x-girl-sports-x-aktr. However, there were
24 akter.jp urls that it tried to fetch, 4 of which ended up on hung threads.
> 
> 2016-12-16 04:12:33,427 INFO [FetcherThread] org.apache.nutch.fetcher.FetcherThread:
fetching http://www.googletagmanager.com/ns.html?id=GTM-MF52KS (queue crawl delay=1000ms)
> 
> And there was no other mention of googletagmanager.com.
> 2016-12-16 04:12:57,991 INFO [FetcherThread] org.apache.nutch.fetcher.FetcherThread:
fetching http://mashable.com/2016/12/14/teens-drinking-down/ (queue crawl delay=1000ms)
> 
> And there was no other mention of mashable.com/2016/12/14/teens. There were 17 mashable.com
urls thet it tried to fetch, 2 of which ended up on hung threads.
> 
> I get only about 9 mentions of the word "error" in the syslog for this run.  It includes
some failures to get robots.txt for various sites, some NoHttpResponseException, and some
ConnectTimeoutException. None of those errors relate to the URLs I mentioned above
> 
> 
>       From: Sebastian Nagel <wastl.nagel@googlemail.com>
>  To: user@nutch.apache.org 
>  Sent: Friday, December 9, 2016 12:42 PM
>  Subject: Re: Fetcher "hung while processing"
>    
> Hi Michael,
> 
>> What other post-fetch actions are there?
> 
> Well, the fetched content is spilled to disk which may also become slow in pathological
cases.
> 
> But I think it's more important to analyze what happened with the URLs before. The logs
> should contain a message "fetching ..." for every hanging URL. When does it happen?
> 
> If possible, let us know about
> - Nutch version
> - environment (local, distributed)
> - configuration, esp. if not the default:
>     mapreduce.task.timeout
>     fetcher.threads.tlimeout.divisor
>     http.timeout
>   and in doubt all other modified
>     fetcher.*
>   properties
> 
> Is the problem reproducible, or does it happen only sometimes?
> 
> Thanks,
> Sebastian
> 
> On 12/09/2016 04:58 PM, Michael Coffey wrote:
>> The property fetcher.parse is false and I pass -noParsing to the fetch command. What
other post-fetch actions are there?
>>
>>
>>       From: Sebastian Nagel <wastl.nagel@googlemail.com>
>>   To: user@nutch.apache.org 
>>   Sent: Friday, December 9, 2016 12:58 AM
>>   Subject: Re: Fetcher "hung while processing"
>>     
>> Hi Michael,
>>
>> what about the property fetcher.parse ?
>>
>> The queue is unblocked after a page has been fetched but before parsing.
>> If the parser is hanging or one of the post-fetch actions take too long
>> it may happen that there are multiple URLs from the same host still in
>> process.
>>
>> Sebastian
>>
>> On 12/09/2016 02:15 AM, Michael Coffey wrote:
>>> I sometimes get a bunch of warning messages that say Thread #x hung while processing
<url>
>>> Is this just a normal thing to see occasionally, or should I look to find some
resolution? I do have an example where the same host shows up on a multitude of these messages,
which puzzles me. I think there should be only one thread per host, due to me specifying fetcher.threads.per.queue=1
>>> Here is example log showing the first 20 of 50 hung threads. Note that http://shinystat.com
and http://fabulous.com show up more than once.
>>>
>>> 2016-12-09 00:47:29,559 WARN [main] org.apache.nutch.fetcher.Fetcher: Aborting
with 50 hung threads.
>>> 2016-12-09 00:47:29,560 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#0 hung while processing https://www.hugedomains.com/domain_search.cfm?catSearch=434
>>> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#1 hung while processing http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=820
>>> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#2 hung while processing http://shinystat.com/it/pro/info_pro.html
>>> 2016-12-09 00:47:29,561 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#3 hung while processing http://events.stanford.edu/byCategory/13/
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#4 hung while processing https://www.ladesk.com/pricing/hosted/terms-and-conditions/
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#5 hung while processing http://shinystat.com/en/opt-out_free.html
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#6 hung while processing http://shinystat.com/fr/biz/info_biz.html
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#7 hung while processing http://fabulous.com/informationcenter/index.htm?formcode%5Bobjective%5D=&formcode%5Bevent%5D=&formcode%5Bregistrytime%5D=1481233769&formcode%5Bcertificate%5D=dfd737bc4490a09d4786cb0e87a15ba6&formdata%5Bqid%5D=88
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#8 hung while processing http://www.youronlinechoices.com/sk/slovnik-pojmov
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#9 hung while processing https://twitter.com/sakura_ope
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#10 hung while processing http://europa.eu/european-union/topics/culture_en
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#11 hung while processing http://www.youronlinechoices.com/ee/opt-out-help
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#12 hung while processing https://www.hugedomains.com/domain_search.cfm?catSearch=437
>>> 2016-12-09 00:47:29,562 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#13 hung while processing http://hosted.ap.org/dynamic/stories/U/US_OBIT_JOHN_GLENN?SITE=AP&SECTION=HOME&TEMPLATE=DEFAULT
>>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#14 hung while processing http://static.fc2.com/sh_css/common/base.css?1200605
>>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#15 hung while processing https://www.hugedomains.com/terms.cfm
>>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#16 hung while processing https://www.ladesk.com/comparisons/
>>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#17 hung while processing http://hu.statcounter.com/features/
>>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#18 hung while processing http://europa.eu/european-union/about-eu/working_el
>>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#19 hung while processing http://www.atinternet.com/es/recursos/
>>> 2016-12-09 00:47:29,563 WARN [main] org.apache.nutch.fetcher.Fetcher: Thread
#20 hung while processing http://ietf.org/rfc/rfc2026.txt
>>>
>>>
>>
>>
>>
>>     
>>
> 
> 
> 
>    
> 


Mime
View raw message