You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<!-- You can either remove the following explanatory text or move it into this comment for later reference -->
17
17
18
-
Concept for a new API provided to UI5 Tooling build tasks, enabling easy use of Node.js [Worker Threads](https://nodejs.org/api/worker_threads.html) to execute CPU intensive operations outside of the main thread.
18
+
Concept for a new API provided to UI5 Tooling tasks, enabling easy use of Node.js [Worker Threads](https://nodejs.org/api/worker_threads.html) to execute CPU intensive operations outside of the main thread.
19
19
20
20
## Motivation
21
21
<!-- You can either remove the following explanatory text or move it into this comment for later reference -->
@@ -30,65 +30,128 @@ The pool should also be re-used when multiple projects are being built, either i
30
30
### Terminology
31
31
32
32
***`Worker`**: A Node.js [Worker thread](https://nodejs.org/api/worker_threads.html) instance
33
-
***`Build Task`**: A UI5 Tooling build task such as "minify" or "buildThemes" (standard tasks) or any [custom task](https://sap.github.io/ui5-tooling/stable/pages/extensibility/CustomTasks/)
34
-
***`Task Processor`**: A module associated with a UI5 Tooling Build Task (standard or custom) that can be executed in a `Worker`
35
-
***`Build Context`**: An already existing ui5-project module, coupled to the lifecycle of a Graph Build. It shall be extended to provide access to the `Work Dispatcher` by forwarding requests from `Build Tasks`
36
-
***`Thread Runner`**: A ui5-project module that will be loaded in a `Worker`. It handles communication with the main thread and executes a `Task Processor` on request
37
-
***`Work Dispatcher`**: A ui5-project singleton module which uses a library like [`workerpool`](https://github.com/josdejong/workerpool) to spawn and manage `Worker` instances in order to have them execute any `Task Processor` requested by the Build Task
38
-
- Handles the `Worker` lifecycle
33
+
***`Task`**: A UI5 Tooling task such as `minify` or `buildThemes` (both standard tasks) or any [custom task](https://sap.github.io/ui5-tooling/stable/pages/extensibility/CustomTasks/)
34
+
***`Task Processor`**: A module associated with a UI5 Tooling task (standard or custom) that can be executed in a worker
35
+
***`Build Context`**: An already existing ui5-project module, coupled to the lifecycle of a Graph Build. It shall be extended to provide access to the Work Dispatcher` by forwarding requests from tasks
36
+
***`Thread Runner`**: A `@ui5/project` module that will be loaded in a worker. It handles communication with the main thread and executes a task processor on request
37
+
***`Work Dispatcher`**: A `@ui5/project` singleton module which uses a library like [`workerpool`](https://github.com/josdejong/workerpool) to spawn and manage worker instances in order to have them execute any task processor requested by the task
38
+
- Handles the worker lifecycle
39
39
40
40
41
41
42
42
### Key Design Decisions
43
43
44
-
* Task Processors shall be called with a well defined signature as described [below](#task-processor)
45
-
* A Task Processor should not be exposed to Worker-specific API
46
-
- I.e. it can be executed on the main thread as well as in a Worker
47
-
- This allows users as well as UI5 Tooling logic to control whether Workers are used or not
48
-
- For example in CI environments where only one CPU core is available to the build, Workers are expected to produce overhead
49
-
- Users might want to disable Workers to easily debug issues in Processors
50
-
- The UI5 Tooling build itself might already be running in a Worker
44
+
* Task processors shall be called with a defined signature as described [below](#task-processor)
45
+
* A task processor should not be exposed to Worker-specific API
46
+
- i.e. it can be executed on the main thread as well as in a Worker
47
+
- This allows UI5 Tooling to dynamically decide whether to use Workers or not
48
+
+ For example in CI environments where only one CPU core is available to the build, Workers might cause unnecessary overhead
49
+
+ Users might want to disable Workers to easily debug issues in processors
50
+
+ The UI5 Tooling build itself might already be running in a Worker
51
51
* The Work Dispatcher and Thread Runner modules will handle all inter-process communication
52
52
- This includes serializing and de-serializing `@ui5/fs/Resource` instances
53
-
* Custom tasks can opt into this feature by defining one ore more "Task Processor" modules in its ui5.yaml
54
-
* A Task can only invoke its own Task Processor(s)
53
+
* Custom tasks can opt into this feature by defining one ore more task processor modules in their ui5.yaml
54
+
* A task can only invoke its own task processor(s)
55
+
* The work dispatcher or thread runners have no understanding of dependencies between the workloads
56
+
- Tasks are responsible for waiting on the completion of their processors
57
+
- Task processors should be executed in a first in, first out order
55
58
56
59
### Assumptions
57
60
58
-
* A Task Processor is assumed to utilize a CPU thread by 90-100%
61
+
* A task processor is assumed to utilize a single CPU thread by 90-100%
59
62
- Accordingly they are also assumed to execute little to no I/O operations
60
-
- This means one Worker should never execute more than one Task Processor at the same time
61
-
*A Task Processor is stateless
63
+
* A Worker should never execute more than one task processor at a time
64
+
* Task processors are generally stateless
62
65
63
66
### Task Processor
64
67
65
-
Similar to Tasks, Task Processors shall be invoked with a well defined signature:
68
+
[Processors](https://sap.github.io/ui5-tooling/stable/pages/Builder/#processors) are an established concept in UI5 Tooling but not yet exposed to custom tasks. The basic idea is that tasks act as the glue code that connects a more generic processor to UI5 Tooling. For example, UI5 Tooling processors make use of very little UI5 Tooling API, making them easily re-usable in different environments like plain Node.js scripts.
66
69
67
-
***`resources`**: An array of `@ui5/fs/Resource` provided by the Build Task
68
-
***`options`**: An object provided by the Build Task
69
-
***`fs`**: An optional fs-interface provided by the Build Task
70
-
**[To be discussed]**`workspace`**: An optional workspace __reader__ provided by the Build Task*
71
-
**[To be discussed]**`dependencies`**: An optional dependencies reader provided by the Build Task*
72
-
**[To be discussed]**`reader`**: An optional generic reader provided by the Build Task*
70
+
With this RFC, we extend this concept to custom tasks. A task can define one or more processors and execute them with a defined API. Their execution is managed by UI5 Tooling, which might execute them on the main thread or in a worker.
71
+
72
+
#### Input Parameters
73
+
74
+
***`resources`**: An array of `@ui5/fs/Resource` provided by the task
75
+
***`options`**: An object provided by the task
76
+
***`fs`**: An optional fs-interface provided by the task
77
+
***`resourceFactory`** Specification-version dependent object providing helper functions to create and manage resources.
78
+
- **`resourceFactory.createResource`** Creates a `@ui5/fs/Resource` (similar to [TaskUtil#resourceFactory.createResource](https://sap.github.io/ui5-tooling/stable/api/@ui5_project_build_helpers_TaskUtil.html#~resourceFactory))
79
+
- No other API for now and now general "ProcessorUtil" or similar, since processors should remain as UI5 Tooling independent as possible
80
+
81
+
**_Potential future additions:_**
82
+
*_**`workspace`**: An optional workspace __reader__ provided by the task_
83
+
*_**`dependencies`**: An optional dependencies reader provided by the task_
84
+
*_**`reader`**: An optional generic reader provided by the task_
85
+
86
+
#### Return Values
87
+
88
+
The allowed return values are rather generic. But since UI5 Tooling needs to serialize and de-serialize the values while transferring them back to the main thread, there are some limitations.
89
+
90
+
The thread runner shall validate the **return value must be either**:
91
+
1. A value that adheres to the requirements stated in [Serializing Data](#serializing-data)
92
+
2. A flat object (`[undefined, Object].includes(value.constructor)`, to detect `Object.create(null)` and `{}`) with property values adhering to the requirements stated in [Serializing Data](#serializing-data)
93
+
3. An array (`Array.isArray(value)`) with values adhering to the requirements stated in [Serializing Data](#serializing-data)
94
+
95
+
Note that nested objects or nested arrays must not be allowed until we become aware of any demand for that.
96
+
97
+
Processors should be able to return primitives and `@ui5/fs/Resource` instances directly:
98
+
```js
99
+
returncreateResource({
100
+
path:"resource/path"
101
+
string:"content"
102
+
});
103
+
````
104
+
105
+
It should also be possible to return simple objects with primitive values or `@ui5/fs/Resource` instances:
106
+
107
+
```js
108
+
return {
109
+
code: "string",
110
+
map: "string",
111
+
counter: 3,
112
+
someResource: createResource({
113
+
path: "resource/path"
114
+
string: "content"
115
+
}),
116
+
}
117
+
```
118
+
119
+
Alternatively, processors might also return a lists of primitives or `@ui5/fs/Resource` instances:
120
+
121
+
```js
122
+
return [
123
+
createResource({
124
+
path: "resource/path"
125
+
string: "content"
126
+
}),
127
+
createResource({
128
+
path: "resource/path"
129
+
string: "content"
130
+
}),
131
+
//...
132
+
]
133
+
```
134
+
135
+
#### Example
73
136
74
137
```js
75
138
/**
76
139
* Task Processor example
77
140
*
78
141
* @param {Object} parameters Parameters
79
-
* @param{@ui5/fs/Resource[]}parameters.resources Array of resources provided by the build task
142
+
* @param {@ui5/fs/Resource[]} parameters.resources Array of resources provided by the task
80
143
* @param {Object} parameters.options Options provided by the calling task
81
144
* @param {@ui5/fs/fsInterface} parameters.fs [fs interface]{@link module:@ui5/fs/fsInterface}-like class that internally handles communication with the main thread
82
-
* @returns{Promise<object|@ui5/fs/Resource[]>} Promise resolving with either a flat object containing Resource instances as values, or an array of Resources
145
+
* @param {@ui5/project/ProcessorResourceFactory} parameters.resourceFactory Helper object providing functions for creating and managing resources
146
+
* @returns {Promise<object|Array|@ui5/fs/Resource|@ui5/fs/Resource[]>} Promise resolving with either a flat object containing Resource instances as values, or an array of Resources
Tasks defining processors in their `ui5.yaml` configuration shall be provided with a new `processors` object, allowing them to trigger execution of the configured processors.
170
+
171
+
The `processors.execute` function shall accept the following parameters:
172
+
*`resources`_(optional)_: Array of `@ui5/fs/Resource` instances if required by the processor
173
+
*`options`_(optional)_: An object with configuration for the processor.
174
+
*`reader`_(optional)_: An instance of `@ui5/fs/AbstractReader` which will be used to read resources requested by the task processor. If supplied, the task processor will be provided with a `fs` parameter to read those resources
175
+
176
+
177
+
The `execute` function shall validate that `resources` only contains `@ui5/fs/Resource` instances and that `options` adheres to the requirements stated in [Serializing Data](#serializing-data).
fs: fsInterface(workspace) // To allow reading additional files if necessary
198
+
reader:workspace // To allow the processor to read additional files if necessary
127
199
});
128
200
awaitworkspace.write(res);
129
201
// [...]
130
202
};
131
203
````
132
204
205
+
### Serializing Data
206
+
207
+
In order to ensure all data supplied to- and returned from- a processor can be serialized correctly, the following checks must be implemented:
208
+
209
+
In case of an object, all property values and in case of an array, all values must be either [**primitives**](https://developer.mozilla.org/en-US/docs/Glossary/Primitive) (except `symbol`?) or **`@ui5/fs/Resource`** instances (do not use `instanceof` checks since Resource instances might differ depending on the specification version).
210
+
211
+
Note: Instances of`@ui5/fs/Resource` might loose their original `stat` value since it is not fully serializable. Any serializable information will be preserved however.
133
212
134
213
## How we teach this
135
214
<!-- You can either remove the following explanatory text or move it into this comment for later reference -->
136
215
137
-
**TODO**
216
+
* Documentation for custom task developers on how to decide whether a task should use processors or not. For instance depending on their CPU demand
138
217
139
218
## Drawbacks
140
219
<!-- You can either remove the following explanatory text or move it into this comment for later reference -->
0 commit comments