Skip to content

Commit a5318b6

Browse files
Fix (#63)
* fix * fix * update readme * up node * . * . * fix * remove dev
1 parent e6ae872 commit a5318b6

File tree

13 files changed

+821
-1766
lines changed

13 files changed

+821
-1766
lines changed

.circleci/config.yml

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ version: 2.1
22
parameters:
33
node-version:
44
type: string
5-
default: "16.13.2"
5+
default: "24.10.0"
66
orbs:
77
node: circleci/[email protected]
88
slack: circleci/[email protected]
@@ -72,30 +72,30 @@ commands:
7272
jobs:
7373
test:
7474
docker:
75-
- image: circleci/node:16-stretch
75+
- image: cimg/node:24.10.0
7676
steps:
7777
- checkout
7878
- node/install:
7979
node-version: << pipeline.parameters.node-version >>
80-
- run:
81-
name: Audit Dependencies
82-
command: npm audit --production --audit-level=high
8380
- node/install-packages:
8481
cache-path: ./node_modules
8582
override-ci-command: npm install
8683
- run:
87-
name: Running Mocha Tests
84+
name: Audit Dependencies
85+
command: npm audit --production --audit-level=high
86+
- run:
87+
name: Running Tests
8888
command: npm test
8989
build:
9090
docker:
91-
- image: circleci/node:16-stretch
91+
- image: cimg/node:24.10.0
9292
user: root
9393
steps:
9494
- checkout
9595
- node/install:
9696
node-version: << pipeline.parameters.node-version >>
9797
- setup_remote_docker:
98-
version: 19.03.13
98+
version: default
9999
docker_layer_caching: true
100100
# build and push Docker image
101101
- run:

CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
## 1.4.1 (November 14, 2025)
2+
* Fix: support maester attachments in `XML Attachment to JSON` action
3+
* Updated `Sailor` version to 2.7.6
4+
* Updated `@elastic.io/component-commons-library` version to 4.0.0
5+
* Removed `elasticio-node` dependency
6+
17
## 1.4.0 (June 09, 2023)
28
* Implemented support `attachments` inside message body for `XML Attachment to JSON` action
39
* Updated Sailor version to 2.7.1

README.md

Lines changed: 63 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,20 @@
11
# XML Component [![CircleCI](https://circleci.com/gh/elasticio/xml-component.svg?style=svg)](https://circleci.com/gh/elasticio/xml-component)
22

33
## Description
4-
iPaaS component to convert between XML and JSON data.
4+
An iPaaS component that converts data between XML and JSON formats.
55

66
### Purpose
7-
Allows users to convert XML attachments and strings to and from JSON.
8-
This component has 3 actions allowing users to pass in either generic but well formatted XML/JSON strings or XML attachments
9-
and produces a generic string or attachment of the other file type. The output then can be mapped and used in other components.
7+
This component converts XML attachments or strings to and from JSON. It exposes three actions that accept either well‑formed XML/JSON strings or XML attachments and returns the converted payload as a string or attachment. The result can be consumed by downstream components.
108

119
### Requirements and Conversion Behavior
12-
Provided XML document (for `XML to JSON`) should be [well-formed](https://en.wikipedia.org/wiki/Well-formed_document)
13-
in order to be parsed correctly. You will get an error otherwise.
10+
- XML content supplied to the `XML to JSON` action must be [well-formed](https://en.wikipedia.org/wiki/Well-formed_document); invalid XML results in an error.
11+
- JSON inputs must be objects with exactly one field, matching the single root element requirement for XML documents.
12+
- [JSON inputs cannot contain field names that violate XML tag naming rules](https://www.w3schools.com/xml/xml_elements.asp):
13+
- They must start with a letter or underscore.
14+
- They cannot start with the letters `xml` (in any casing).
15+
- They may only contain letters, digits, hyphens, underscores, and periods.
1416

15-
JSON inputs must be objects with exactly one field as XML documents must be contained in a single 'root' tag.
16-
[JSON inputs can not have any field names which are not valid as XML tag names:](https://www.w3schools.com/xml/xml_elements.asp)
17-
* They must start with a letter or underscore
18-
* They cannot start with the letters xml (or XML, or Xml, etc)
19-
* They must only contain letters, digits, hyphens, underscores, and periods
20-
21-
XML attributes on a tag can be read and set by setting an `_attr` sub-object in the JSON.
22-
The inner-text of an XML element can also be controlled with `#` sub-object.
17+
XML attributes on a tag can be represented with an `_attr` object, and element text content can be set with the `_` key.
2318

2419
For example:
2520
```json
@@ -32,28 +27,22 @@ For example:
3227
}
3328
}
3429
```
35-
is equivalent to
30+
31+
is equivalent to:
3632
```xml
3733
<someTag id="my id">my inner text</someTag>
3834
```
3935

4036
#### Environment variables
41-
* `MAX_FILE_SIZE`: *optional* - Controls the maximum size of an attachment to be read or written in MB.
42-
43-
Defaults to 10 MB where 1 MB = 1024 * 1024 bytes.
44-
* `EIO_REQUIRED_RAM_MB`: *optional* - You can increase memory usage limit for component if you going to work with big files
45-
46-
Defaults to 256 MB where 1 MB = 1024 * 1024 bytes.
37+
- `MAX_FILE_SIZE` (optional): Maximum attachment size, in bytes, that can be read or written. Defaults to 10 MB (`10 * 1024 * 1024` bytes).
38+
- `EIO_REQUIRED_RAM_MB` (optional): Memory usage limit for the component. Defaults to 256 MB.
4739

4840
## Actions
4941

5042
### XML to JSON
51-
Takes XML string and converts it to generic JSON object.
52-
53-
**Limitation:**
54-
Value inside xml tags will be converting into string only, e.g.:
43+
Converts an XML string into a generic JSON object.
5544

56-
given xml
45+
**Limitation:** Values inside XML tags are converted to strings. For example, given:
5746
```xml
5847
<note>
5948
<date>2015-09-01</date>
@@ -64,7 +53,7 @@ given xml
6453
</note>
6554
```
6655

67-
will be converted into:
56+
the output is:
6857
```json
6958
{
7059
"note": {
@@ -79,73 +68,68 @@ will be converted into:
7968

8069
### XML Attachment to JSON
8170
#### Configuration Fields
82-
83-
* **Pattern** - (string, optional): RegEx for filtering files by name provided via old attachment mechanism (outside message body)
84-
* **Upload single file** - (checkbox, optional): Use this option if you want to upload a single file
71+
- **Pattern** (string, optional): Regular expression used to filter attachments provided via the legacy attachment mechanism.
72+
- **Upload single file** (checkbox, optional): Enable when the message contains a single attachment object.
8573

8674
#### Input Metadata
87-
If `Upload single file` checked, there will be 2 fields:
88-
* **URL** - (string, required): link to file on Internet or platform
89-
90-
If `Upload single file` unchecked:
91-
* **Attachments** - (array, required): Collection of files to upload, each record contains object with two keys:
92-
* **URL** - (string, required): link to file on Internet or platform
93-
94-
If you going to use this option with static data, you need to switch to Developer mode
95-
<details><summary>Sample</summary>
96-
<p>
97-
98-
```json
99-
{
100-
"attachments": [
101-
{
102-
"url": "https://example.com/files/file1.xml"
103-
},
104-
{
105-
"url": "https://example.com/files/file2.xml"
106-
}
107-
]
108-
}
109-
```
110-
</p>
111-
</details>
75+
When **Upload single file** is enabled:
76+
- **URL** (string, required): Link to the file, either public or internal (including steward/maester storage).
11277

113-
#### Output Metadata
78+
When **Upload single file** is disabled:
79+
- **Attachments** (array, required): Collection of attachment objects.
80+
- **URL** (string, required): Link to the file on the internet or the platform.
11481

115-
Resulting JSON object
82+
If you plan to use this option with static data, switch to Developer mode.
83+
<details><summary>Sample</summary>
84+
<p>
85+
86+
```json
87+
{
88+
"attachments": [
89+
{
90+
"url": "https://example.com/files/file1.xml"
91+
},
92+
{
93+
"url": "https://example.com/files/file2.xml"
94+
}
95+
]
96+
}
97+
```
98+
</p>
99+
</details>
100+
101+
#### Output Metadata
102+
A JSON object built from the parsed XML.
116103

117104
### JSON to XML
118-
Provides an input where a user provides a JSONata expression that should evaluate to an object to convert to JSON.
119-
See [Requirements & Conversion Behavior](#requirements-and-conversion-behavior) for details on conversion logic.
120-
The following options are supported:
121-
* **Upload XML as file to attachments**: When checked, the resulting XML will be placed directly into an attachment.
122-
The attachment information will be provided in both the message's attachments section as well as `attachmentUrl` and `attachmentSize`
123-
will be populated. The attachment size will be described in bytes.
124-
When this box is not checked, the resulting XML will be provided in the `xmlString` field.
125-
* **Exclude XML Header/Description**: When checked, no XML header of the form `<?xml version="1.0" encoding="UTF-8" standalone="no"?>` will be prepended to the XML output.
126-
* **Is the XML file standalone**: When checked, the xml header/description will have a value of `yes` for standalone. Otherwise, the value will be `no`. Has no effect when XML header/description is excluded.
127-
128-
The incoming message should have a single field `input`. When using integrator mode, this appears as the input **JSON to convert** When building mappings in developper mode, one must set the `input` property. E.g.:
105+
Accepts a JSONata expression that must resolve to an object, then converts it to XML. See [Requirements and Conversion Behavior](#requirements-and-conversion-behavior) for more detail.
106+
107+
Options:
108+
- **Upload XML as file to attachments**: When enabled, the resulting XML is stored as an attachment. `attachmentUrl` and `attachmentSize` are provided in the body and the message attachments.
109+
- **Exclude XML Header/Description**: When enabled, the XML declaration (`<?xml version="1.0" encoding="UTF-8" standalone="no"?>`) is omitted.
110+
- **Is the XML file standalone**: Controls the `standalone` attribute in the XML declaration (`yes` or `no`). Ignored when the header is excluded.
111+
112+
Incoming messages must contain a single `input` field. In integrator mode this appears as **JSON to convert**. In developer mode, set the `input` property manually. Example:
129113
```
130114
{
131115
"input": {
132-
"someTag": {
133-
"_attr": {
134-
"id": "my id"
135-
},
136-
"_": "my inner text"
137-
}
138-
}
116+
"someTag": {
117+
"_attr": {
118+
"id": "my id"
119+
},
120+
"_": "my inner text"
121+
}
122+
}
139123
}
140124
```
141125

142126
## Known limitations
143-
- All actions involving attachments are not supported on local agents due to current platform limitations.
144-
- When creating XML files with invalid XML tags, the name of the potentially invalid tag will not be reported.
145-
- When you try to retrieve sample in `XML Attachment to JSON` action and it's size is more then 500Kb, there will be generated new smaller sample with same structure as original
127+
- Actions working with attachments are not supported on local agents due to current platform constraints.
128+
- When creating XML files with invalid XML tags, the invalid tag name is not reported.
129+
- If an attachment used for sampling in `XML Attachment to JSON` exceeds 500 KB, a smaller sample with the same structure is generated.
146130

147131
## Additional Info
148-
Icon made by Freepik from www.flaticon.com
132+
Icon made by Freepik from www.flaticon.com.
149133

150134
## License
151135

component.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"title": "XML",
3-
"version": "1.4.0",
3+
"version": "1.4.1",
44
"description": "Component to convert between XML and JSON data",
55
"actions": {
66
"xmlToJson": {

lib/actions/attachmentToJson.js

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ const { AttachmentProcessor } = require('@elastic.io/component-commons-library')
33
const { createSchema } = require('genson-js');
44
const jsf = require('json-schema-faker');
55
const sizeof = require('object-sizeof');
6-
const { newMessageWithBody } = require('elasticio-node/lib/messages');
6+
const { messages } = require('../utils');
77
const { readFile, writeFile, stat } = require('fs/promises');
88
const {
99
getUserAgent,
@@ -38,6 +38,33 @@ function checkFileName(self, fileName, pattern) {
3838
const tooLargeErrMsg = (fileName, fileSize) => `Attachment ${fileName} is too large to be processed by XML component. `
3939
+ `File limit is: ${MAX_FILE_SIZE} byte, file given was: ${fileSize} byte.`;
4040

41+
const DEFAULT_ATTACHMENT_ENCODING = 'utf-8';
42+
43+
function resolveEncoding(headers = {}) {
44+
const contentTypeHeader = headers['content-type'] || headers['Content-Type'];
45+
if (!contentTypeHeader) return DEFAULT_ATTACHMENT_ENCODING;
46+
const charsetMatch = /charset=([^;]+)/i.exec(contentTypeHeader);
47+
return charsetMatch ? charsetMatch[1].trim().toLowerCase() : DEFAULT_ATTACHMENT_ENCODING;
48+
}
49+
50+
async function attachmentDataToBuffer(data) {
51+
if (!data) return Buffer.alloc(0);
52+
if (Buffer.isBuffer(data)) return data;
53+
if (typeof data === 'string') return Buffer.from(data);
54+
if (data instanceof ArrayBuffer) return Buffer.from(data);
55+
if (ArrayBuffer.isView(data)) {
56+
return Buffer.from(data.buffer, data.byteOffset, data.byteLength);
57+
}
58+
if (typeof data[Symbol.asyncIterator] === 'function') {
59+
const chunks = [];
60+
for await (const chunk of data) {
61+
chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
62+
}
63+
return Buffer.concat(chunks);
64+
}
65+
throw new Error('Unsupported attachment response payload type');
66+
}
67+
4168
module.exports.process = async function processAction(msg, cfg) {
4269
const self = this;
4370
const { attachments, body = {} } = msg;
@@ -61,20 +88,23 @@ module.exports.process = async function processAction(msg, cfg) {
6188
const attachmentProcessor = new AttachmentProcessor(getUserAgent(), msg.id);
6289
for (const file of files) {
6390
self.logger.info('Processing attachment');
64-
let response = await attachmentProcessor.getAttachment(file.url, 'text');
91+
let response = await attachmentProcessor.getAttachment(file.url, 'stream');
6592
this.logger.debug(`For provided filename response status: ${response.status}`);
66-
let responseBodyString = response.data;
93+
let attachmentBuffer = await attachmentDataToBuffer(response.data);
94+
let responseBodyString = attachmentBuffer.toString(resolveEncoding(response.headers));
6795
if (response.status >= 400) {
6896
throw new Error(`Error in making request to ${file.url} Status code: ${response.status}, Body: ${responseBodyString}`);
6997
}
7098
if (!responseBodyString) {
7199
throw new Error(`Empty attachment received for file ${file.fileName || ''}`);
72100
}
73-
const fileSize = response.headers['content-length'];
101+
const fileSizeHeader = response.headers['content-length'];
102+
const fileSize = Number(fileSizeHeader) || attachmentBuffer.length;
74103
response = null;
75104
if (Number(fileSize) > MAX_FILE_SIZE) throw new Error(tooLargeErrMsg(file.fileName || '', fileSize));
76105
let { body: json } = await xml2Json.process(this, responseBodyString);
77106
responseBodyString = null;
107+
attachmentBuffer = null;
78108
await writeFile(tempFile, JSON.stringify(json));
79109
if ((await stat(tempFile)).size > MAX_FILE_SIZE_FOR_SAMPLE && isDebugFlow) {
80110
this.logger.warn('The message size exceeded the sample size limit. To match the limitation we will generate a smaller sample using the structure/schema from the original file.');
@@ -86,7 +116,7 @@ module.exports.process = async function processAction(msg, cfg) {
86116
json = jsf.generate(schema);
87117
}
88118
this.logger.debug(`Attachment to XML finished, emitting message. ${memUsage()}`);
89-
await self.emit('data', newMessageWithBody(json));
119+
await self.emit('data', messages.newMessageWithBody(json));
90120
}
91121
};
92122

lib/actions/jsonToXml.js

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,9 @@
11
/* eslint-disable no-param-reassign */
22
const { AttachmentProcessor } = require('@elastic.io/component-commons-library');
3-
const { messages } = require('elasticio-node');
43
const xml2js = require('xml2js');
54
const _ = require('lodash');
65
const { Readable } = require('stream');
7-
const { getUserAgent, MAX_FILE_SIZE } = require('../utils');
6+
const { getUserAgent, MAX_FILE_SIZE, messages } = require('../utils');
87

98
module.exports.process = async function process(msg, cfg) {
109
const { input } = msg.body;

lib/actions/jsonToXmlOld.js

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
const eioUtils = require('elasticio-node').messages;
1+
const { messages } = require('../utils');
22
const xml2js = require('xml2js');
33
const _ = require('lodash');
44

@@ -61,7 +61,7 @@ function processAction(msg, cfg) {
6161

6262
const result = builder.buildObject(jsonToTransform);
6363
this.logger.debug('Successfully converted body to XML');
64-
return eioUtils.newMessageWithBody({
64+
return messages.newMessageWithBody({
6565
xmlString: result,
6666
});
6767
}

lib/utils.js

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,8 @@ exports.memUsage = () => {
1818
// eslint-disable-next-line max-len
1919
return (`Memory usage - rss: ${f(usage.rss)}, heapTotal: ${f(usage.heapTotal)}, heapUsed: ${f(usage.heapUsed)}, external: ${f(usage.external)}, arrayBuffers: ${f(usage.arrayBuffers)}`);
2020
};
21+
22+
exports.messages = {
23+
newMessageWithBody: (body) => ({ body, headers: {} }),
24+
newEmptyMessage: () => ({ body: {}, headers: {} }),
25+
};

lib/xml2Json.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/* eslint-disable no-param-reassign */
2-
const { messages } = require('elasticio-node');
2+
const { messages } = require('../lib/utils');
33
const xml2js = require('xml2js');
44

55
module.exports.process = async function xml2Json(self, xmlString) {

0 commit comments

Comments
 (0)