Skip to content

Commit 2ab01da

Browse files
Java SDK: Read the Dag run conf into a class with typed fields (#75542)
* Java SDK: Read the Dag run conf into a class with typed fields Signed-off-by: PoAn Yang <payang@apache.org> * Document the clashing field names case of getConf Signed-off-by: PoAn Yang <payang@apache.org> * Reject nulls for primitives in getConf and document JsonProperty Signed-off-by: PoAn Yang <payang@apache.org> * Move getConf details from the Java SDK guide to the API reference Signed-off-by: PoAn Yang <payang@apache.org> --------- Signed-off-by: PoAn Yang <payang@apache.org>
1 parent 4495e92 commit 2ab01da

4 files changed

Lines changed: 588 additions & 2 deletions

File tree

‎airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -719,6 +719,35 @@ The ``Client``
719719
A value from a secrets backend, such as an ``AIRFLOW_VAR_*`` environment variable, still wins over the stored
720720
value when the Variable is read back. ``setVariable`` without a description clears the existing description.
721721

722+
The Dag run conf
723+
~~~~~~~~~~~~~~~~
724+
725+
``context.dagRun.conf`` is the conf the Dag run was triggered with. It is a ``Map``, and its values have the Java
726+
types listed in :ref:`the XCom type mapping <java-sdk/types>`. ``context.dagRun.getConf(type)`` reads the conf into
727+
an instance of a class with typed fields instead:
728+
729+
.. code-block:: java
730+
731+
public static class EtlConf {
732+
public String targetTable;
733+
public int limit = 100;
734+
public LocalDate since;
735+
}
736+
737+
public class Load implements Task {
738+
@Override
739+
public void execute(Context context, Client client) {
740+
EtlConf etl = context.dagRun.getConf(EtlConf.class);
741+
}
742+
}
743+
744+
For a run triggered with the conf ``{"target_table": "sales", "since": "2024-01-31"}``, ``etl.targetTable`` is
745+
``"sales"``, ``etl.since`` is ``LocalDate.of(2024, 1, 31)``, and ``etl.limit`` stays ``100``. Keys match field
746+
names when case and underscores are ignored, and keys that match no field are ignored. ``getConf`` does not add
747+
the defaults of the Dag's params, and the conf of a scheduled run is empty. For the classes that ``getConf``
748+
accepts and the cases where it throws, see ``DagRun.getConf`` in the `Java SDK API reference
749+
<https://airflow.apache.org/docs/java-sdk/stable/>`__.
750+
722751
.. _java-sdk/task-state-store:
723752

724753
Task state store

‎java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Context.kt‎

Lines changed: 64 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ package org.apache.airflow.sdk
2121

2222
import org.apache.airflow.sdk.execution.Logger
2323
import org.apache.airflow.sdk.execution.comm.StartupDetails
24+
import org.apache.airflow.sdk.internal.bindConf
2425
import java.time.OffsetDateTime
2526
import org.apache.airflow.sdk.execution.comm.DagRun.DagRunType as CommDagRunType
2627

@@ -81,7 +82,8 @@ enum class DagRunType {
8182
* @property dataIntervalEnd End of the data interval.
8283
* @property runAfter A date-time tells the scheduler when the Dag run can be scheduled.
8384
* @property runType How the run was created.
84-
* @property conf The configuration for this run.
85+
* @property conf The configuration for this run. [getConf] reads it into an
86+
* instance of a class with typed fields.
8587
*/
8688
data class DagRun(
8789
@JvmField val dagId: String,
@@ -92,7 +94,67 @@ data class DagRun(
9294
@JvmField val runAfter: OffsetDateTime?,
9395
@JvmField val runType: DagRunType?,
9496
@JvmField val conf: Map<String, Any?>,
95-
)
97+
) {
98+
/**
99+
* Reads [conf] into a new instance of [type]. For example, [type] can be a
100+
* class with one field per conf key.
101+
*
102+
* ```java
103+
* public static class EtlConf {
104+
* public String targetTable; // set from the "target_table" key
105+
* public int limit = 100; // stays 100 when the conf has no "limit" key
106+
* }
107+
*
108+
* EtlConf etl = context.dagRun.getConf(EtlConf.class);
109+
* ```
110+
*
111+
* Each key goes to the field, setter or record component whose name matches
112+
* the key when case and underscores are ignored. If one key matches a name
113+
* exactly, that key wins over keys that match the same name only when case
114+
* and underscores are ignored. Objects nested in the conf follow the same
115+
* rules, including objects in lists and objects that are map values. The keys
116+
* of a map field stay as written.
117+
*
118+
* Keys that match no field are ignored, because other tasks in the run may
119+
* read them. A field with no matching key keeps its default value. The conf
120+
* holds only the values that the run was triggered with. This method does not
121+
* add the defaults of the Dag's params, so in a scheduled run, where the conf
122+
* is empty, no field has a matching key.
123+
*
124+
* The class needs a no-argument constructor and public fields or setters, so
125+
* a nested class must be `static`. A record also works. A record component
126+
* with no matching key is null. If the component has a primitive type, this
127+
* method throws instead, so a task that reads the conf into a record with a
128+
* primitive component fails in a scheduled run. In Kotlin, declare the fields
129+
* with `@JvmField` so that a property such as `isEnabled` keeps its name.
130+
*
131+
* Annotating a field with [ArgName] has no effect on this method. To set a
132+
* field from a key that no Java name can match, such as `target-table`, add
133+
* `@JsonProperty("target-table")` to that field. `@JsonProperty` needs
134+
* `com.fasterxml.jackson.core:jackson-annotations` in your build, because the
135+
* SDK keeps Jackson off your compile classpath. You can also read the key
136+
* from [conf].
137+
*
138+
* This method throws [IllegalArgumentException] in these cases:
139+
*
140+
* * A value, such as an empty string for an `int` field, cannot be converted
141+
* to the type of its field.
142+
* * A key holds null for a primitive field, a list holds null for a primitive
143+
* array such as `int[]`, or a primitive record component has no matching
144+
* key. To let a field or a component receive null, declare it with a boxed
145+
* type, such as `Integer`.
146+
* * Two keys, such as `target_table` and `TARGET_TABLE`, match the same field
147+
* after case and underscores are ignored, but neither key matches the field
148+
* name exactly.
149+
* * The class has two fields whose names differ only in case or underscores,
150+
* such as `targetTable` and `target_table`. This method checks the class
151+
* even when the conf is empty, so it throws in a scheduled run too.
152+
*
153+
* @param type Class to read the conf into.
154+
* @throws IllegalArgumentException in the cases listed above.
155+
*/
156+
fun <T : Any> getConf(type: Class<T>): T = bindConf(conf, type)
157+
}
96158

97159
/**
98160
* Identifies the task instance that is currently executing.
Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.airflow.sdk.internal
21+
22+
import com.fasterxml.jackson.databind.DeserializationFeature
23+
import com.fasterxml.jackson.databind.JavaType
24+
import com.fasterxml.jackson.databind.ObjectMapper
25+
import com.fasterxml.jackson.databind.introspect.BeanPropertyDefinition
26+
import com.fasterxml.jackson.databind.json.JsonMapper
27+
import java.util.concurrent.ConcurrentHashMap
28+
29+
// Other tasks in the Dag run may read keys that the target class does not
30+
// declare, so this mapper must not fail on an unknown key. By default, Jackson
31+
// moves a date-time that carries an offset to UTC. For example,
32+
// 2024-01-31T02:00+08:00 would become 2024-01-30T18:00Z and fall on another
33+
// date. This mapper keeps the offset as written instead.
34+
//
35+
// FAIL_ON_NULL_FOR_PRIMITIVES stops Jackson from turning a null into 0, for
36+
// example in an int[]. It also rejects a record whose primitive component has
37+
// no key, because Jackson handles the missing value as null. matchFields still
38+
// checks a key that holds null, because its message names the key and the
39+
// matching field.
40+
private val confMapper: ObjectMapper by lazy {
41+
JsonMapper
42+
.builder()
43+
.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
44+
.disable(DeserializationFeature.ADJUST_DATES_TO_CONTEXT_TIME_ZONE)
45+
.enable(DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES)
46+
.build()
47+
.findAndRegisterModules()
48+
}
49+
50+
/**
51+
* Converts a Dag run conf into an instance of [type] for
52+
* [org.apache.airflow.sdk.DagRun.getConf]. That function documents how keys
53+
* match fields.
54+
*
55+
* Jackson matches a key to a field only when the key is exactly the field
56+
* name. So this function first uses [foldArgName] to find the field that each
57+
* key matches, then renames the key to the exact name of that field.
58+
*/
59+
internal fun <T : Any> bindConf(
60+
conf: Map<String, Any?>,
61+
type: Class<T>,
62+
): T {
63+
val javaType = confMapper.constructType(type)
64+
val matched = matchKeys(conf, javaType)
65+
return try {
66+
confMapper.convertValue(matched, javaType)
67+
} catch (e: IllegalArgumentException) {
68+
throw IllegalArgumentException("Cannot convert the Dag run conf to ${type.name}: ${e.message}", e)
69+
}
70+
}
71+
72+
/**
73+
* Renames the keys of every object in [value] that becomes an instance of a
74+
* class when [value] is converted to [type]. The renaming also reaches objects
75+
* in lists and arrays, objects that are map values, and the value inside a
76+
* reference type such as `Optional`. The keys of a map are data rather than
77+
* field names, so they stay as written.
78+
*/
79+
private fun matchKeys(
80+
value: Any?,
81+
type: JavaType,
82+
): Any? =
83+
when {
84+
type.isReferenceType -> matchKeys(value, type.contentType)
85+
value is Map<*, *> && type.isMapLikeType -> value.mapValues { (_, v) -> matchKeys(v, type.contentType) }
86+
value is List<*> && (type.isCollectionLikeType || type.isArrayType) -> value.map { matchKeys(it, type.contentType) }
87+
value is Map<*, *> -> matchFields(value, type)
88+
else -> value
89+
}
90+
91+
/**
92+
* Renames each key of [obj] to the name of the [type] field that the key
93+
* matches. If a field has exactly the key's name, the key matches that field.
94+
* Otherwise the key matches the field whose name folds to the same token as the
95+
* key, unless another key has exactly that field's name. A key that matches no
96+
* field keeps its name, and Jackson ignores that key.
97+
*
98+
* @throws IllegalArgumentException if the names of two fields fold to the same
99+
* token, if two keys match the same field only after folding, or if a key
100+
* holds null for a primitive field.
101+
*/
102+
private fun matchFields(
103+
obj: Map<*, *>,
104+
type: JavaType,
105+
): Map<String, Any?> {
106+
val fields = findFields(type)
107+
val className = type.rawClass.simpleName
108+
val keys = obj.keys.mapTo(mutableSetOf()) { it.toString() }
109+
val keyOfField = mutableMapOf<String, String>()
110+
val matched = LinkedHashMap<String, Any?>()
111+
obj.forEach { (rawKey, value) ->
112+
val key = rawKey.toString()
113+
val field = fields.byName[key] ?: fields.byFold[foldArgName(key)]?.takeUnless { it.name in keys }
114+
if (field == null) {
115+
matched[key] = value
116+
return@forEach
117+
}
118+
val other = keyOfField.put(field.name, key)
119+
require(other == null) {
120+
"Dag run conf keys '$other' and '$key' both match field $className.${field.name} when case and " +
121+
"underscores are ignored, so the SDK cannot tell which one to use. Name the field exactly as the key " +
122+
"it should take."
123+
}
124+
require(value != null || !field.primaryType.isPrimitive) {
125+
"Dag run conf key '$key' is null, but field $className.${field.name} has a primitive type that cannot " +
126+
"be null. Declare a boxed type (e.g. Integer instead of int) to receive null."
127+
}
128+
matched[field.name] = matchKeys(value, field.primaryType)
129+
}
130+
return matched
131+
}
132+
133+
private class ConfFields(
134+
val byName: Map<String, BeanPropertyDefinition>,
135+
val byFold: Map<String, BeanPropertyDefinition>,
136+
)
137+
138+
/**
139+
* Finds the fields of [type] so that each field can be looked up by its exact
140+
* name or by its folded token from [foldArgName].
141+
*
142+
* The result is cached per type. Jackson takes far longer to find the fields of
143+
* a class than to convert one object of that class. Without the cache,
144+
* converting a list of many objects would spend most of the time finding the
145+
* same fields again.
146+
*
147+
* @throws IllegalArgumentException if two fields fold to the same token.
148+
*/
149+
private fun findFields(type: JavaType): ConfFields = confFieldsByType.computeIfAbsent(type) { indexFields(it) }
150+
151+
private val confFieldsByType = ConcurrentHashMap<JavaType, ConfFields>()
152+
153+
private fun indexFields(type: JavaType): ConfFields {
154+
val fields =
155+
confMapper.deserializationConfig
156+
.introspect(type)
157+
.findProperties()
158+
.filter { it.isWritable() }
159+
val className = type.rawClass.simpleName
160+
val byFold = mutableMapOf<String, BeanPropertyDefinition>()
161+
fields.forEach { field ->
162+
val previous = byFold.put(foldArgName(field.name), field)
163+
require(previous == null) {
164+
"Fields $className.${previous?.name} and $className.${field.name} differ only in case or underscores, " +
165+
"so the SDK cannot tell which one a Dag run conf key should go to. Rename one of them."
166+
}
167+
}
168+
return ConfFields(fields.associateBy { it.name }, byFold)
169+
}
170+
171+
/**
172+
* Whether Jackson can set this property from a key. Jackson also lists
173+
* read-only properties. For example, a `getBaseURL()` getter next to a
174+
* `baseUrl` field gives a read-only property named `baseURL`. A read-only
175+
* property must not take a key, and it must not clash with a field, as
176+
* `baseURL` would clash with `baseUrl`. A collection or a map that has only a
177+
* getter is not read-only, because Jackson adds the elements to the object that
178+
* the getter returns.
179+
*/
180+
private fun BeanPropertyDefinition.isWritable(): Boolean =
181+
hasSetter() || hasField() || hasConstructorParameter() || getter?.rawType?.let(::isCollectionOrMap) == true
182+
183+
private fun isCollectionOrMap(type: Class<*>): Boolean =
184+
Collection::class.java.isAssignableFrom(type) || Map::class.java.isAssignableFrom(type)

0 commit comments

Comments
 (0)