Replies: 4 comments 14 replies
-
|
Hello @qianmoQ, |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Obviously, the |
Beta Was this translation helpful? Give feedback.
-
|
I tried setting 'mapred.job.name' and 'mapreduce.job.name' but it didn't work output ： 2023-09-19 18:24:09.781 INFO 53999 --- [gine-launch-log] o.a.kyuubi.jdbc.hive.KyuubiConnection : --conf
2023-09-19 18:24:09.781 INFO 53999 --- [gine-launch-log] o.a.kyuubi.jdbc.hive.KyuubiConnection : mapreduce.job.name=admin@dsj_onedata_instance_1695039251_18115342_go_0030_inke
2023-09-19 18:24:09.781 INFO 53999 --- [gine-launch-log] o.a.kyuubi.jdbc.hive.KyuubiConnection : --conf
2023-09-19 18:24:09.781 INFO 53999 --- [gine-launch-log] o.a.kyuubi.jdbc.hive.KyuubiConnection : spark.app.name=admin@dsj_onedata_instance_1695039251_18115342_go_0030_inke
2023-09-19 18:24:09.781 INFO 53999 --- [gine-launch-log] o.a.kyuubi.jdbc.hive.KyuubiConnection : --conf
2023-09-19 18:24:09.781 INFO 53999 --- [gine-launch-log] o.a.kyuubi.jdbc.hive.KyuubiConnection : spark.driver.memory=6g |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Unlike Spark, the Hive engine itself won't run as a YARN application, instead, when you submit some complex queries, the Hive engine will compile them to MR/Tez applications and then submit them to YARN. |
Beta Was this translation helpful? Give feedback.
-
|
The application can be found in the cluster by setting |
Beta Was this translation helpful? Give feedback.
-
|
I tried it, and it didn't work either by adding parameters from the uri or by setting them through properties.
|
Beta Was this translation helpful? Give feedback.
-
|
read my first reply message. PS: use text to provide information as much as possible, the picture is NOT searchable and is not friendly for future explorers |
Beta Was this translation helpful? Give feedback.
-
|
Example Code: public void started()
{
try {
Class.forName("org.apache.kyuubi.jdbc.KyuubiHiveDriver");
Properties properties = new Properties();
properties.put("user", source.getUsername());
if (StringUtils.isNotEmpty(source.getPassword())) {
properties.put("password", source.getPassword());
}
if (env == null) {
env = Maps.newConcurrentMap();
}
if (commonEnv != null) {
env.putAll(commonEnv);
}
String kyuubiEngineType = String.valueOf(env.get("kyuubi.engine.type"));
if (kyuubiEngineType.equals("HIVE_SQL") || kyuubiEngineType.equalsIgnoreCase("TRINO")) {
isHiveEngine = true;
}
env.put("spark.app.name", configure.getClusterTaskName());
String url;
if (ObjectUtils.isEmpty(source.getPort())) {
url = String.format("jdbc:%s://%s", "kyuubi", source.getHost());
}
else {
url = String.format("jdbc:%s://%s:%s", "kyuubi", source.getHost(), source.getPort());
}
if (env != null) {
StringBuilder urlBuilder = new StringBuilder(url).append("?");
for (Map.Entry<String, Object> entry : env.entrySet()) {
String key = entry.getKey();
String value = entry.getValue().toString();
urlBuilder.append(key)
.append("=")
.append(value)
.append(";");
}
url = urlBuilder.toString().substring(0, urlBuilder.length() - 1);
}
logger.info("Jdbc url: {}", url);
connection = DriverManager.getConnection(url, properties);
}
catch (Exception exception) {
logger.error("Start failed", exception);
configure.setState(TaskState.FAILURE);
configure.setMessage(exception.getMessage());
}
} |
Beta Was this translation helpful? Give feedback.
-
|
@pan3793 I use the following links to query Hive running. However, related tasks cannot be found in the YARN cluster. Do I need to perform any configuration independently? jdbc:kyuubi://localhost:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi_cluster170?kyuubi.engine.type=HIVE_SQL;timeout=43200;spark.yarn.queue=default;spark.app.name=app_xxxxxxReturn logs: 2023-10-08 18:04:09,113 �[34mINFO �[0;39m �[1;35m[Thread-677]�[0;39m �[36m[KyuubiLogInject.java:42]�[0;39m Scan log from statement [ 2023-10-08T18:04:09.113 ]
2023-10-08 18:04:13,813 �[34mINFO �[0;39m �[1;35m[Thread-677]�[0;39m �[36m[KyuubiLogInject.java:28]�[0;39m 2023-10-08 18:04:09.214 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[ffbe9407-f7ef-4206-88e7-42f53e4370d9] in RUNNING_STATE
2023-10-08 18:04:14,114 �[34mINFO �[0;39m �[1;35m[Thread-677]�[0;39m �[36m[KyuubiLogInject.java:42]�[0;39m Scan log from statement [ 2023-10-08T18:04:14.113 ]
2023-10-08 18:04:18,814 �[34mINFO �[0;39m �[1;35m[Thread-677]�[0;39m �[36m[KyuubiLogInject.java:28]�[0;39m 2023-10-08 18:04:14.215 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[ffbe9407-f7ef-4206-88e7-42f53e4370d9] in RUNNING_STATE |
Beta Was this translation helpful? Give feedback.
-
Code of Conduct
Search before asking
Describe the bug
java code:
jdbc url :
jdbc:kyuubi://localhost:11009/;kyuubi.engine.type=HIVE_SQL;timeout=43200;spark.yarn.queue=onlinelog output
kyuubi jdbc version:
kyuubi-hive-jdbc-shaded: 1.7.1The
spark.app.nameand queue configurations that I specified do not take effect on the yarn clusterAffects Version(s)
1.7.0
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
No response
Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?
Beta Was this translation helpful? Give feedback.
All reactions