Mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiaoyong Zhu <>
Subject RE: ATS Hook for Hive Query Text
Date Fri, 10 Apr 2015 08:57:40 GMT
+Hive users in case there are some experts there...


From: Joe Zhang (SDE)
Sent: Friday, April 10, 2015 4:57 PM
Cc: Xiaoyong Zhu
Subject: ATS Hook for Hive Query Text

Hi Hive&Tez experts:

I was using ws/v1/timeline/HIVE_QUERY_ID/ AST APIs to get tez query text for each DAG, but
I am not sure whether the code has been optimized or it is same with the previous user-submmited
hive code?

I has read hive<>/ql<>/src<>/java<>/org<>/apache<>/hadoop<>/hive<>/ql<>/hooks<>/
source code and confuse whether the QueryPlan has been optimized  QueryPlan plan = hookContext.getQueryPlan();


  public void run(final HookContext hookContext) throws Exception {

    final long currentTime = System.currentTimeMillis();

    final HiveConf conf = new HiveConf(hookContext.getConf());

    executor.submit(new Runnable() {


        public void run() {

          try {

            QueryPlan plan = hookContext.getQueryPlan();

            if (plan == null) {



            String queryId = plan.getQueryId();

            String opId = hookContext.getOperationId();

            long queryStartTime = plan.getQueryStartTime();

            String user = hookContext.getUgi().getUserName();

            String requestuser = hookContext.getUserName();

            if (hookContext.getUserName() == null ){

              requestuser = hookContext.getUgi().getUserName() ;


            int numMrJobs = Utilities.getMRTasks(plan.getRootTasks()).size();

            int numTezJobs = Utilities.getTezTasks(plan.getRootTasks()).size();

            if (numMrJobs + numTezJobs <= 0) {

              return; // ignore client only queries


            switch(hookContext.getHookType()) {

            case PRE_EXEC_HOOK:

              ExplainTask explain = new ExplainTask();

              explain.initialize(conf, plan, null);

              String query = plan.getQueryStr();

              List<Task<?>> rootTasks = plan.getRootTasks();

              JSONObject explainPlan = explain.getJSONPlan(null, null, rootTasks,

                   plan.getFetchTask(), true, false, false);

              fireAndForget(conf, createPreHookEvent(queryId, query,

                   explainPlan, queryStartTime, user, requestuser, numMrJobs, numTezJobs,


            case POST_EXEC_HOOK:

              fireAndForget(conf, createPostHookEvent(queryId, currentTime, user, requestuser,
true, opId));


            case ON_FAILURE_HOOK:

              fireAndForget(conf, createPostHookEvent(queryId, currentTime, user, requestuser
, false, opId));






          } catch (Exception e) {

  "Failed to submit plan to ATS: " + StringUtils.stringifyException(e));





Best wishes
Joe zhang

View raw message