Mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bikas Saha <bi...@hortonworks.com>
Subject RE: Task count
Date Thu, 24 Jul 2014 18:38:57 GMT
The patch should work for all types of vertices because it gives each task
the total number of tasks for its vertex. Do you have any other use case?

Bikas

-----Original Message-----
From: Johannes Zillmann [mailto:jzillmann@googlemail.com]
Sent: Thursday, July 24, 2014 6:58 AM
To: Gopal V
Cc: user@tez.apache.org
Subject: Re: Task count

Hey Gopal,

using the task count basically for 2 things (in mr for both the map stage
and the reduce stage):
- each task samples its output-data up to a certain number. This number is
the desired sample count divided by the number of tasks
- also we use the task count in some scenarios to let the last task (of a
stage or a vertex) do some extra logic. That plays in combination of the
task-index.

Looking at your patch it looks like it will do the job for kind of the
map-like vertex but not for the aggregation vertex, right ?
Also what jira issue is that ?

best
Johannes

On 24 Jul 2014, at 07:40, Gopal V <gopalv@apache.org> wrote:

> On 7/23/14, 6:07 PM, Johannes Zillmann wrote:
>> Hey Tez team,
>>
>> is there some way to get the task count within a vertex from within a
task ?
>> Some equivalent to mapred.map.tasks and mapred.reduce.tasks for
map-reduce ?
>
> Could you explain the use-case for this particular requirement?
>
> I intend to add the vertex parallelism to the task context as part of
one of my WIP branches.
>
> I uploaded my base patch-set as is (including the TODO markers).
>
>
https://issues.apache.org/jira/secure/attachment/12657536/TEZ-broadcast-sh
uffle%2Bvertex-parallelism.patch
>
> If you can explain what you are actually looking to do with this
information, perhaps I can roll the two feature reqs together.
>
> Cheers,
> Gopal
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message