Skip to content

Commit 0a07079

Browse files
heremaps-botheremaps-bot
heremaps-bot
authored and
heremaps-bot
committed
HERE Data SDK for Java & Scala Release 2.20.0
1 parent 8027a77 commit 0a07079

File tree

88 files changed

+1531
-760
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

88 files changed

+1531
-760
lines changed

‎README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,12 @@ For more information, see a use case illustration of [keeping a client map up to
2828

2929
| Name | Description | Source | Labels / Topics |
3030
| ---- | ----------- | ------ | --------------- |
31-
| Geometry Lifter | An application that takes level 14 partitions of road topology and geometry and aggregates them to higher-level (i.e. bigger) partitions. | [Java](data-processing/java/geometry-lifter) / [Scala](data-processing/scala/geometry-lifter) | Data Processing Library, Spark, Batch, Protobuf, HERE Map Content |
31+
| Geometry Lifter | An application that takes level 12 partitions of road topology and geometry and aggregates them to higher-level (i.e. bigger) partitions. | [Java](data-processing/java/geometry-lifter) / [Scala](data-processing/scala/geometry-lifter) | Data Processing Library, Spark, Batch, Protobuf, HERE Map Content |
3232
| Pedestrian Topologies Extraction to GeoJSON | Topologies, accessible by pedestrians, are selected based on the segment attributes and then are transformed into GeoJSON file format and stored in a new catalog. | [Java](data-processing/java/pedestrian-topologies-extraction-geojson) / [Scala](data-processing/scala/pedestrian-topologies-extraction-geojson) | Data Processing Library, Spark, Batch, GeoJSON, HERE Map Content |
3333
| Pedestrian Topologies Extraction to Protobuf | Topologies, accessible by pedestrians, are selected based on the segment attributes and then are transformed to a newly created proto schema format and stored in a new catalog layer that follows that schema. | [Java](data-processing/java/pedestrian-topologies-extraction-protobuf) / [Scala](data-processing/scala/pedestrian-topologies-extraction-protobuf) | Data Processing Library, Spark, Batch, Protobuf, HERE Map Content |
3434
| Statistics creation across multiple processing runs with stateful processing | The application counts how often the node cardinality of the topology changes in each partition. | [Java](data-processing/java/stateful-nodecardinality-extraction) / [Scala](data-processing/scala/stateful-nodecardinality-extraction) | Data Processing Library, Spark, Batch, JSON, HERE Map Content |
3535
| Here Map Content Diff-Tool | An application to compare the content of two different versions of an input catalog. | [Java](data-processing/java/heremapcontent-difftool) / [Scala](data-processing/scala/heremapcontent-difftool) | Data Processing Library, Spark, Batch, JSON, HERE Map Content |
36+
| Here Map Content Validation | An application to validate road topology and geometry content against a set of acceptance criteria using [scalatest](www.scalatest.org). | [Scala](data-processing/scala/heremapcontent-validation) | Data Processing Library, Spark, Batch, JSON, HERE Map Content |
3637
| Data Validation and Testing Part 1: Content Testing | The test considers input partitions containing an octagon as "PASS" tiles and partitions containing a straight line as "FAIL". It records the test result in a new output catalog. | [Java](data-validation/java/quick-start/testing) / [Scala](data-validation/scala/quick-start/testing) | Data Validation Library, Spark, Batch, Protobuf |
3738
| Data Validation and Testing Part 2: Metrics | Reads the output of the testing component, assigns an error severity of CRITICAL for the "FAIL" tiles in the test results, aggregates the failed tile IDs and records the metrics in a new output catalog. | [Java](data-validation/java/quick-start/metrics) / [Scala](data-validation/scala/quick-start/metrics) | Data Validation Library, Spark, Batch, Protobuf |
3839
| Data Validation and Testing Part 3: Assessment | Reads the output of the metrics component, gives a final value of FAIL, if more than 10% of the candidate tiles have CRITICAL errors, and writes the result to a new catalog. | [Java](data-validation/java/quick-start/assessment) / [Scala](data-validation/scala/quick-start/assessment) | Data Validation Library, Spark, Batch, Protobuf |

‎data-archive/java/avro-example/pom.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
<groupId>com.here.platform.data.archive</groupId>
55
<artifactId>data-archive-avro-example</artifactId>
6-
<version>0.0.565</version>
6+
<version>0.0.582</version>
77
<packaging>jar</packaging>
88

99
<!-- Meta information section start -->
@@ -23,7 +23,7 @@
2323
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
2424
<java.version>1.8</java.version>
2525

26-
<sdk-bom.version>2.19.10</sdk-bom.version>
26+
<sdk-bom.version>2.20.11</sdk-bom.version>
2727
<scala.compat.version>2.11</scala.compat.version>
2828

2929
<avro.version>1.10.0</avro.version>

‎data-archive/java/parquet-example/pom.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
<groupId>com.here.platform.data.archive</groupId>
55
<artifactId>data-archive-parquet-example</artifactId>
6-
<version>0.0.565</version>
6+
<version>0.0.582</version>
77
<packaging>jar</packaging>
88

99
<!-- Meta information section start -->
@@ -23,7 +23,7 @@
2323
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
2424
<java.version>1.8</java.version>
2525

26-
<sdk-bom.version>2.19.10</sdk-bom.version>
26+
<sdk-bom.version>2.20.11</sdk-bom.version>
2727
<scala.compat.version>2.11</scala.compat.version>
2828

2929
<hadoop-client.version>2.7.3</hadoop-client.version>

‎data-archive/java/protobuf-example/pom.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
<groupId>com.here.platform.data.archive</groupId>
55
<artifactId>data-archive-protobuf-example</artifactId>
6-
<version>0.0.565</version>
6+
<version>0.0.582</version>
77
<packaging>jar</packaging>
88

99
<!-- Meta information section start -->
@@ -23,7 +23,7 @@
2323
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
2424
<java.version>1.8</java.version>
2525

26-
<sdk-bom.version>2.19.10</sdk-bom.version>
26+
<sdk-bom.version>2.20.11</sdk-bom.version>
2727
<scala.compat.version>2.11</scala.compat.version>
2828

2929
<commons-logging.version>1.2</commons-logging.version>

‎data-archive/java/sensoris-parquet-example/pom.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
<groupId>com.here.platform.examples</groupId>
55
<artifactId>data-archive-sensoris-parquet-example</artifactId>
6-
<version>0.0.565</version>
6+
<version>0.0.582</version>
77
<packaging>jar</packaging>
88

99
<!-- Meta information section -->
@@ -23,7 +23,7 @@
2323
<java.version>1.8</java.version>
2424
<artifact.wagon.version>1.6.0</artifact.wagon.version>
2525

26-
<sdk-bom.version>2.19.10</sdk-bom.version>
26+
<sdk-bom.version>2.20.11</sdk-bom.version>
2727
<scala.compat.version>2.11</scala.compat.version>
2828
<hadoop-client.version>2.7.3</hadoop-client.version>
2929
<parquet-protobuf.version>1.11.0</parquet-protobuf.version>

‎data-archive/java/sensoris-protobuf-example/pom.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
<groupId>com.here.platform.examples</groupId>
55
<artifactId>data-archive-sensoris-protobuf-example</artifactId>
6-
<version>0.0.565</version>
6+
<version>0.0.582</version>
77
<packaging>jar</packaging>
88

99
<!-- Meta information section -->
@@ -25,7 +25,7 @@
2525
<java.version>1.8</java.version>
2626
<artifact.wagon.version>1.6.0</artifact.wagon.version>
2727

28-
<sdk-bom.version>2.19.10</sdk-bom.version>
28+
<sdk-bom.version>2.20.11</sdk-bom.version>
2929
<scala.compat.version>2.11</scala.compat.version>
3030

3131
<parquet-protobuf.version>1.10.1</parquet-protobuf.version>

‎data-processing/java/geometry-lifter/README.md

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ To run the compiler locally, you will need to run the entry point to the compile
148148

149149
- `com.here.platform.data.processing.example.java.geometry.lifter.Main`
150150

151-
As _arguments_, you must provide the `--master` _parameter_ with the address of the Spark server
151+
As _arguments_, you must provide the `-Dspark.master` _parameter_ with the address of the Spark server
152152
master to connect to, and any configuration parameters you want to override. Alternatively, you
153153
can add those parameters to the `application.conf` file.
154154

@@ -165,6 +165,9 @@ partition of HERE Map Content. Make sure you update the layer coverage to reflec
165165
geographical region. In order to use this configuration file, you need to use the `-Dconfig.file`
166166
parameter.
167167

168+
Setup the environment variable `$PATH_TO_CONFIG_FOLDER` to `./config/here`,
169+
for the HERE platform China environment use the files in the `./config/here-china` directory.
170+
168171
### Run the Compiler from the Command Line
169172

170173
Combine all of this together and run the following command line in the `geometry-lifter`
@@ -175,22 +178,10 @@ For the HERE platform environment:
175178
```bash
176179
mvn exec:java \
177180
-Dexec.mainClass=com.here.platform.data.processing.example.java.geometry.lifter.Main \
178-
-Dpipeline-config.file=./config/here/pipeline-config.conf \
181+
-Dpipeline-config.file=$PATH_TO_CONFIG_FOLDER/pipeline-config.conf \
179182
-Dpipeline-job.file=./config/here/pipeline-job.conf \
180183
-Dconfig.file=./config/here/local-application.conf \
181-
-Dexec.args="--master local[*]" \
182-
-Dhere.platform.data-client.request-signer.credentials.here-account.here-token-scope=$PROJECT_HRN
183-
```
184-
185-
For the HERE platform China environment:
186-
187-
```bash
188-
mvn exec:java \
189-
-Dexec.mainClass=com.here.platform.data.processing.example.java.geometry.lifter.Main \
190-
-Dpipeline-config.file=./config/here-china/pipeline-config.conf \
191-
-Dpipeline-job.file=./config/here-china/pipeline-job.conf \
192-
-Dconfig.file=./config/here-china/local-application.conf \
193-
-Dexec.args="--master local[*]" \
184+
-Dspark.master=local[*] \
194185
-Dhere.platform.data-client.request-signer.credentials.here-account.here-token-scope=$PROJECT_HRN
195186
```
196187

‎data-processing/java/geometry-lifter/pom.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
<groupId>com.here.platform.data.processing.example.java.geometry.lifter</groupId>
55
<artifactId>geometry-lifter</artifactId>
6-
<version>0.0.565</version>
6+
<version>0.0.582</version>
77
<packaging>jar</packaging>
88

99
<name>Batch Processor compiler for Geometry Lifter in Java</name>
@@ -22,7 +22,7 @@
2222
<java.version>1.8</java.version>
2323
<maven.compiler.source>${java.version}</maven.compiler.source>
2424
<maven.compiler.target>${java.version}</maven.compiler.target>
25-
<sdk-bom.version>2.19.10</sdk-bom.version>
25+
<sdk-bom.version>2.20.11</sdk-bom.version>
2626
<exec-maven-plugin.version>1.6.0</exec-maven-plugin.version>
2727
<maven-shade-plugin.version>3.0.0</maven-shade-plugin.version>
2828
<scala-maven-plugin.version>3.3.1</scala-maven-plugin.version>

‎data-processing/java/heremapcontent-difftool/README.md

Lines changed: 8 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,7 @@ To run the compiler locally, you will need to run the entry point to the compile
173173

174174
- `com.here.platform.data.processing.example.java.difftool.processor.Main`
175175

176-
As _arguments_, you must provide the `--master` _parameter_ with the address of the Spark server
176+
As _arguments_, you must provide the `-Dspark.master` _parameter_ with the address of the Spark server
177177
master to connect to, and any configuration parameters you want to override. Alternatively, you can
178178
add those parameters to the `application.conf` file.
179179

@@ -190,6 +190,9 @@ partition of HERE Map Content. Make sure you update the layer coverage to reflec
190190
geographical region. In order to use this configuration file, you need to use the `-Dconfig.file`
191191
parameter.
192192

193+
Setup the environment variable `$PATH_TO_CONFIG_FOLDER` to `./config/here`,
194+
for the HERE platform China environment use the files in the `./config/here-china` directory.
195+
193196
### Run the Compiler from the Command Line
194197

195198
The first run of the pipeline will use the job configuration `pipeline-job-first.conf`. As
@@ -203,10 +206,10 @@ For the HERE platform environment:
203206
```bash
204207
mvn exec:java \
205208
-Dexec.mainClass=com.here.platform.data.processing.example.java.difftool.processor.Main \
206-
-Dpipeline-config.file=./config/here/pipeline-config.conf \
209+
-Dpipeline-config.file=$PATH_TO_CONFIG_FOLDER/pipeline-config.conf \
207210
-Dpipeline-job.file=./config/here/pipeline-job-first.conf \
208211
-Dconfig.file=./config/here/local-application.conf \
209-
-Dexec.args="--master local[*]" \
212+
-Dspark.master=local[*] \
210213
-Dhere.platform.data-client.request-signer.credentials.here-account.here-token-scope=$PROJECT_HRN
211214
```
212215

@@ -217,32 +220,10 @@ the version specified in `pipeline-job-second.conf`. Run the following command l
217220
```bash
218221
mvn exec:java \
219222
-Dexec.mainClass=com.here.platform.data.processing.example.java.difftool.processor.Main \
220-
-Dpipeline-config.file=./config/here/pipeline-config.conf \
223+
-Dpipeline-config.file=$PATH_TO_CONFIG_FOLDER/pipeline-config.conf \
221224
-Dpipeline-job.file=./config/here/pipeline-job-second.conf \
222225
-Dconfig.file=./config/here/local-application.conf \
223-
-Dexec.args="--master local[*]" \
224-
-Dhere.platform.data-client.request-signer.credentials.here-account.here-token-scope=$PROJECT_HRN
225-
```
226-
227-
For the HERE platform China environment, instead, use the files in the `config/here-chine` directory:
228-
229-
```bash
230-
mvn exec:java \
231-
-Dexec.mainClass=com.here.platform.data.processing.example.java.difftool.processor.Main \
232-
-Dpipeline-config.file=./config/here-china/pipeline-config.conf \
233-
-Dpipeline-job.file=./config/here-china/pipeline-job-first.conf \
234-
-Dconfig.file=./config/here-china/local-application.conf \
235-
-Dexec.args="--master local[*]" \
236-
-Dhere.platform.data-client.request-signer.credentials.here-account.here-token-scope=$PROJECT_HRN
237-
```
238-
239-
```bash
240-
mvn exec:java \
241-
-Dexec.mainClass=com.here.platform.data.processing.example.java.difftool.processor.Main \
242-
-Dpipeline-config.file=./config/here-china/pipeline-config.conf \
243-
-Dpipeline-job.file=./config/here-china/pipeline-job-second.conf \
244-
-Dconfig.file=./config/here-china/local-application.conf \
245-
-Dexec.args="--master local[*]" \
226+
-Dspark.master=local[*] \
246227
-Dhere.platform.data-client.request-signer.credentials.here-account.here-token-scope=$PROJECT_HRN
247228
```
248229

‎data-processing/java/heremapcontent-difftool/pom.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
<groupId>com.here.platform.data.processing.example.java</groupId>
55
<artifactId>heremapcontent-difftool</artifactId>
6-
<version>0.0.565</version>
6+
<version>0.0.582</version>
77
<packaging>jar</packaging>
88

99
<name>HERE Map Content Differences example for Java</name>
@@ -22,7 +22,7 @@
2222
<java.version>1.8</java.version>
2323
<maven.compiler.source>${java.version}</maven.compiler.source>
2424
<maven.compiler.target>${java.version}</maven.compiler.target>
25-
<sdk-bom.version>2.19.10</sdk-bom.version>
25+
<sdk-bom.version>2.20.11</sdk-bom.version>
2626
<exec-maven-plugin.version>1.6.0</exec-maven-plugin.version>
2727
<maven-shade-plugin.version>3.0.0</maven-shade-plugin.version>
2828
<scala-maven-plugin.version>3.3.1</scala-maven-plugin.version>

0 commit comments

Comments
 (0)