Mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Seth <ss...@apache.org>
Subject Re: Re: why only need to configure "the client node hadoop classpath"?
Date Tue, 05 Apr 2016 17:38:58 GMT
Yes. It depends upon the YARN distributed cache. The relevant jars
(tez.lib.uris, tez.aux.uris) are localized on each node using YARN local
resources (distributed cache). Configs are constructed on the client node
and then written out to HDFS, and localized again via the distributed
cache. If you're looking for the implementation -
TezClientUtils.createApplicationSubmissionContext.
If additional jars need to be made available as part of a running DAG -
that is done by 1) setting tez.aux.uris, or 2) programmatically using APIs
on TezClient / during dag construction.

On Tue, Apr 5, 2016 at 12:47 AM, Maria <linanmengxiaoai@126.com> wrote:

>
> Much thanks for your quick reply. Siddharth~ :)
> OK, I got it. But another question arises:How can it propagate the
> libraries and configuration to other nodes? Does it depends on hadoop
> distributed cache? I still can not find the code logic. :(
>
> At 2016-04-05 10:27:55, "Siddharth Seth" <sseth@apache.org> wrote:
>
>
> Tez will run tasks throughout the cluster. However, it takes care of
> propagating the libraries and configuration to other nodes - which is why
> only the client needs to be configured.
>
>
> On Mon, Apr 4, 2016 at 6:51 PM, Maria <linanmengxiaoai@126.com> wrote:
>
>
>
> Hi,all.
>
>  Is it just because tez is a client-side plug-in, so we can only need to
> configure tez-libraries in the hadoop client node hadoop classpath,and do
> not need to send and configure them to any datanode?  Is there any other
> reasons?
>
>
>
> Thanks for any reply.
>
>
>
> Maria.
>
>
>
>

Mime
View raw message