Merge branch 'marklogic-community_dev' into marklogic-community_master

Author: Scott Stafford

README.md

@@ -1,61 +1,85 @@
-![GitHub version](https://img.shields.io/github/tag/marklogic-community/marklogic-spring-batch.svg)
+# MarkLogic Spring Batch
 
-| Branch | Status |
-| ------------- | ------------- |
-| master | ![master](https://circleci.com/gh/sastafford/marklogic-spring-batch/tree/master.png?circle-token=e1b8b3198d3416fcb535509f2e7d600444ef153e)  |
-| dev  | ![dev](https://circleci.com/gh/sastafford/marklogic-spring-batch/tree/dev.png?circle-token=e1b8b3198d3416fcb535509f2e7d600444ef153e)  |
-| ml-8  | ![ml-8](https://circleci.com/gh/sastafford/marklogic-spring-batch/tree/ml-8.png?circle-token=e1b8b3198d3416fcb535509f2e7d600444ef153e)  |
+The MarkLogic Spring Batch project is an extension of the CORE and INFRASTRUCTURE components of Spring Batch to make it easier to write batch processing programs using MarkLogic. 
 
+Start with the [project home page](https://github.com/marklogic-community/marklogic-spring-batch/wiki) to get started. 
 
-# What is Spring Batch?
+## Prerequisites
 
-[Spring Batch](http://docs.spring.io/spring-batch/trunk/reference/html/) is an open source framework for batch processing based on the [Spring Framework](http://projects.spring.io/spring-framework/).
+ * MarkLogic 8+
+ * JDK 1.8+
 
-Figure 1 depicts a high-level diagram of a Spring Batch program.  The green APPLICATION box represents the batch processing program that gets executed by a user.  This is usually a command line based application but it could also be triggered by an external source like a user interface.  Spring Batch provides two components, CORE and INFRASTRUCTURE, that enables the boilerplate code for creating an APPLICATION.  
-  
-  * CORE - contains the core runtime classes necessary to launch and control a batch job
-  * INFRASTRUCTURE - contains common readers and writers, and services which are used by APPLICATION and CORE
-  
-![Spring Batch Architecture](http://docs.spring.io/spring-batch/trunk/reference/html/images/spring-batch-layers.png.pagespeed.ce.sMqaNr3V1Z.png)
+## Installing the Test Environment
 
-*Figure 1*
+Open $PROJECT_ROOT/gradle.properties.  Review the mlHost, mlRestPort, mlJobRepoPort properties to confirm there are no conflicts
 
-Figure 2 shows the key concepts that make up the domain language of Spring Batch.  The boxes in blue represent concepts from the CORE component and yellow from the INFRASTRUCTURE component.  A Job has one to many steps, which has exactly one ItemReader, ItemProcessor, and ItemWriter. A job needs to be launched (JobLauncher), and meta data about the currently running process needs to be stored (JobRepository).
+    gradlew mlDeploy
 
-![Spring Batch Stereotypes](http://docs.spring.io/spring-batch/trunk/reference/html/images/spring-batch-reference-model.png.pagespeed.ce.TrtTC751hI.png)
+This command will set up your test database and application server.
 
-*Figure 2*
+## Running the tests
 
-# Why use Spring Batch?
+Review the host/port/user credentials properties for the following property files.
 
- * If you need to migrate data from a relational database, mainframe, or other external source to MarkLogic.
- * If you have a batch processing job that is outside the scope of other batch processing tools like MLCP or CORB 
- * If you find that you need 'boilerplate' code to provide failover capability,  performance, and extensibility for a custom batch processing job. 
- * If you are a Java shop running MarkLogic 8+
+ * ./core/src/test/resources/job.properties
+ * ./infrastructure/src/test/resources/job.properties
+ * ./rdf/src/test/resources/job.properties
+ * ./samples/src/test/resources/job.properties
+ 
+ Run the following command to execute all project tests.  All tests should pass. 
 
-# What is MarkLogic Spring Batch?
+     gradlew test
 
-MarkLogic Spring Batch (MSB) extends the CORE and INFRASTRUCTURE components of Spring Batch to make it easier to work with data and MarkLogic.  Sample code templates are included for developers getting started writing a batch processing APPLICATION.
+## Coding Style Tests
 
-## What are the main features of MarkLogic Spring Batch?
+This project uses both checkstyle and PMD.
 
- * Extends the [INFRASTRUCTURE]() classes to facilitate reading, writing, and processing documents for MarkLogic.  
- * A sample [APPLICATION]() that provides a template program to create your own Spring Batch program
- * (Beta) MarkLogic implementation of a JobRepository
+     gradlew check
 
-# How can I get started using MarkLogic Spring Batch?
+## Deployment
 
-Check out the [Getting Started Wiki](), review the sample application, and check out other batch processing applications using Spring Batch
+The product of this project are jar files that are published to [bintray](https://dl.bintray.com/sastafford/maven/).  The following libraries are created. 
 
- * [ml-migration-starter]() - Migrate data from a relational database into MarkLogic
- * [Hector]() - Ingest CSV files into MarkLogic
- * [Penny]() - Use Apache Natural Language Processing library to perform named entity recognition over documents
+| Group | Artifact |
+| ------|----------|
+| com.marklogic | marklogic-spring-batch-core |
+| com.marklogic | marklogic-spring-batch-infrastructure |
+| com.marklogic | marklogic-spring-batch-test |
+| com.marklogic | spring-batch-http |
+| com.marklogic | spring-batch-rdbms |
 
-# What are the goals for the MarkLogic Spring Batch project?
+The gradle bintray plugin is used to publish to bintray.
+
+     gradlew :core:bintrayUpload
+     gradlew :infrastructure:bintrayUpload
+     gradlew :test:bintrayUpload
+     gradlew :http:bintrayUpload
+     gradlew :rdbms:bintrayUpload
+
+## How do I use these libraries?
+
+### Gradle
+
+```groovy
+dependencies {
+    compile 'com.marklogic:marklogic-spring-batch-core:1.4.0'
+    testCompile 'com.marklogic:marklogic-spring-batch-test:1.4.0'
+}
+```
+
+
+### Maven
+
+```xml
+<dependencies>
+    <dependency>
+        <groupId>com.marklogic</groupId>
+        <artifactId>marklogic-spring-batch-core</artifactId>
+        <version>1.4.2</version>
+    </dependency>
+</dependencies>
+```
 
-  * Write batch processing solutions that are reliable, robust, and high performing
-  * Reduce the amount of time it takes to build, operate, and maintain batch processing jobs with MarkLogic
-  * Minimize complexity of batch processing applications
 
 # How can I contribute to the project?
 
@@ -64,6 +88,3 @@ Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to contribute
 # What license does MarkLogic Spring Batch use?
 
 See the [LICENSE.md](LICENSE.md) file for details
-
-----
-1. Spring Batch Introduction: http://docs.spring.io/spring-batch/trunk/reference/html/spring-batch-intro.html

build.gradle

@@ -12,13 +12,16 @@ buildscript {
 }
 
 plugins {
-    id "com.marklogic.ml-gradle" version "3.2.0"
+    id "com.marklogic.ml-gradle" version "3.3.0"
 }
 
 mlPostDeploy.dependsOn ":mlJobRepo:deployMarkLogicJobRepository"
 mlUndeploy.dependsOn "mlJobRepo:undeployMarkLogicJobRepository"
 
 allprojects {
+    apply plugin: "checkstyle"
+    apply plugin: "pmd"
+
     repositories {
         jcenter()
         mavenCentral()
@@ -27,20 +30,29 @@ allprojects {
     }
 
     task wrapper(type: Wrapper) {
-        gradleVersion = '4.3'
+        gradleVersion = '4.4.1'
+    }
+
+    pmd {
+        ruleSets = ["java-basic", "java-braces", "java-unusedcode"]
     }
+
+    checkstyle {
+        configFile = file("${project.rootDir}/config/checkstyle/checkstyle.xml")
+        toolVersion = '7.1.2'
+    }
+
 }
 
 subprojects {
 
     apply plugin: "java"
     apply plugin: "maven-publish"
     apply plugin: "idea"
-    apply plugin: "checkstyle"
 
     ext {
-        springBatchVersion = '3.0.8.RELEASE'
-        springVersion = '4.3.9.RELEASE'
+        springBatchVersion = '4.0.0.RELEASE'
+        springVersion = '5.0.2.RELEASE'
     }
 
     dependencies {
@@ -54,8 +66,9 @@ subprojects {
     sourceCompatibility = 1.8
     targetCompatibility = 1.8
 
-    checkstyle {
-        configFile = file("${project.rootDir}/dev-tools/checkstyle/checkstyle.xml")
-        toolVersion = '7.1.2'
+    task cleanJavaDocs(type: Delete) {
+        delete fileTree("../sastafford.github.io/marklogic-spring-batch/javadoc")
     }
+
+    javadoc.dependsOn cleanJavaDocs
 }

dev-tools/checkstyle/checkstyle.xml renamed to config/checkstyle/checkstyle.xml

@@ -1,15 +1,13 @@
 apply plugin: "com.jfrog.bintray"
 
 dependencies {
-    //core doesn't depend on infrastructure but want to grab both jars when importing MSB
-    compile project(":infrastructure")
-
     // For the CommandLineInterface main program
     compile "net.sf.jopt-simple:jopt-simple:5.0.1"
 
     compile "org.springframework.batch:spring-batch-core:$springBatchVersion"
     compile "com.marklogic:marklogic-client-api:4.0.3"
     compile "com.marklogic:ml-javaclient-util:3.2.0"
+    compile "org.apache.commons:commons-lang3:3.5"
 
     testCompile "org.springframework.batch:spring-batch-test:$springBatchVersion"
     testCompile "com.marklogic:ml-junit:3.0.0"
@@ -26,6 +24,10 @@ ext {
     artifact = "marklogic-spring-batch-core"
 }
 
+javadoc {
+    destinationDir = new File("../sastafford.github.io/marklogic-spring-batch/javadoc/core")
+}
+
 task sourcesJar(type: Jar, dependsOn: classes) {
     baseName = artifactId
     version = project.version

core/README.md

@@ -2,4 +2,4 @@ distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-4.3-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-4.4.1-bin.zip