Merge branch 'dev'

Author: Scott Stafford

CONTRIBUTING.md

@@ -14,23 +14,18 @@ Each contributor should be knowledgeable and proficient at the following project
 * [MarkLogic App Deployer](https://github.com/rjrudin/ml-app-deployer)
 * [Git](http://git-scm.com/doc)
 
-# How should I set up my development environment? 
-
-## What software do I need?
+# What software do I need?
 
 * [MarkLogic 8+](http://developer.marklogic.com/products)
 * [Java Development Kit 1.8+](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)
-* Recommended: [Gradle 2.+](http://gradle.org/gradle-download/)
+* Recommended: [Gradle 3.+](http://gradle.org/gradle-download/)
 
 You will need to first install MarkLogic if you haven't already.  It is recommended to use a virtual machine or remote server to run MarkLogic to partition our resources (MarkLogic likes to use a lot of memory).  
 
-## What code baseline should I use?
-We use a [fork-and-Branch Git workflow](http://blog.scottlowe.org/2015/01/27/using-fork-branch-git-workflow/).  The dev branch contains the latest and greatest code, while master represents the latest published version.  **Always branch from DEV**.
-  
-Review the [jobs/gradle.properties](https://github.com/sastafford/marklogic-spring-batch/blob/master/core/gradle.properties) file to make sure that your host, port, username, and password are correct for your environment.  Do not check your local version of gradle.properties.  
-
-## Should I use an IDE? 
+# What code baseline should I use?
+We use a [fork-and-Branch Git workflow](http://blog.scottlowe.org/2015/01/27/using-fork-branch-git-workflow/).  The dev branch contains the latest and greatest code, while master represents the latest published version.  **Always branch from DEV**.  If you issue a pull request, make sure to compare against the DEV branch.  
 
+# Should I use an IDE? 
 Yes, it is recommended to use a Java friendly IDE to make one's life easier.  Of course, you can use a simple text editor if that is your preference.  The lead authors have a recent preference for [IntelliJ](https://www.jetbrains.com/idea/).  We also use Gradle for our swiss army knife of building, testing, deploying, etc.  One of the cool features of Gradle is the ability to create the project files for two popular IDE's, IntelliJ and Eclipse.    
 
 ```
@@ -43,39 +38,47 @@ Yes, it is recommended to use a Java friendly IDE to make one's life easier.  Of
 
 The gradlew (or gradlew.bat on Windows) command is a Java program called the [Gradle Wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html) which allows you to run a gradle command without installing Gradle on your machine.  If you plan to contribute (or do any type of Java development) it would be worth while to go ahead and install Gradle on your dev machine.  
 
-## How do I verify things are set up correctly?  
-Once you have cloned the project locally execute the test cases.  Gradle will handle downloading any dependencies that you need for this effort.  
-
-```
-gradlew testAll
-```
+# What is the structure for the project?
 
-It is our policy that the dev branch tests always are 100% passing.  If all tests run to completion and 100% passing then you are ready to start coding.    
+* [core](https://github.com/sastafford/marklogic-spring-batch/tree/master/core) - contains enhancements and helper classes to the Spring Batch framework.  This includes the following features
+    * A job repository that uses MarkLogic to persist job metadata
+    * Custom ItemReaders/Processors/Writers and tasklets that work with MarkLogic
+    * Configuration annotations for using MarkLogic and Spring Batch together 
+* [test](https://github.com/sastafford/marklogic-spring-batch/tree/master/test) - helper classes to assist in the testing of Spring Batch jobs written for MarkLogic.  
+* [examples](https://github.com/sastafford/marklogic-spring-batch/tree/master/examples) - A variety of recipes and examples to guide in the creation of your own MarkLogic batch processing jobs (i.e. migrating from a RDBMS) 
 
-# What should I work on?
+# How do I set up my environment?  
 
-If you are new to the project the best way to contribute is to add a Job to the job library.  Choose of the [job issues](https://github.com/sastafford/marklogic-spring-batch/labels/job) to get started on an assignment.  If you have an example of a unique batch processing job then it would be a good idea to add it to the examples project.    
+## Deploy your MarkLogic test database
 
-# What is the structure for the project?
+Check the following properties file to see if there is any conflicts with your existing system.
 
-There are three sub-projects for MarkLogic Spring Batch.  Most contributions will be in the examples and jobs projects where the core work would be for those intimately familiar with the Spring Batch code baseline.  
+    $PROJET_ROOT/core/gradle.properties
 
-* [examples](https://github.com/sastafford/marklogic-spring-batch/tree/master/examples) are the recipes for creating your own batch processing jobs (i.e. migrating from a RDBMS).  Developers should use these as templates and guidelines for writing their own MarkLogic batch processing jobs. 
-* [jobs](https://github.com/sastafford/marklogic-spring-batch/tree/master/jobs) are a library of common MarkLogic batch processing jobs to import, export, and transform.  
-* [core](https://github.com/sastafford/marklogic-spring-batch/tree/master/core) contains all enhancements on the Spring Batch framework.  This includes the jobs utility, the MarkLogic Job Repository, and any custom ItemReaders/Processors/Writers and tasklets.  
+From the project root, run the following command.  This will create a test database that is configured as a MarkLogic Job Repository.  
 
-# How do I test? 
+    gradlew :core:mlDeploy
 
-The quality of this project is based on the tests that exercise the code.  Every issue worked should be accompanied by one or more tests that test the primary use case and several edge cases.  Code that is submitted without any accompanied tests will not be accepted.  Before submitting a pull request, run the test suite and confirm 100% test passing with the following gradle command.  
+Run the automated unit tests from the project root
 
 ```
-./gradlew test
+gradlew test
 ```
 
+The policy is the dev branch tests are 100% passing.  If all tests run to completion and 100% passing then you are ready to start commiting code.    
+
+# What should I work on?
+
+If you are new to the project the best way to contribute is to add or improve an example to the directory of examlpes.  Choose of the [job issues](https://github.com/sastafford/marklogic-spring-batch/labels/example) to get started on an assignment.  
+
+# How do I test? 
+
+The quality of this project is based on the tests that exercise the code.  Every issue worked should be accompanied by one or more tests that test the primary use case and several edge cases.  Code that is submitted without any accompanied tests will not be accepted.  Before submitting a pull request, run the test suite and confirm 100% test passing with the following gradle command.  
+
 # How do I deploy the MarkLogic Spring Batch artifacts?
 
 ## MarkLogic Spring Batch Libraries
-To deploy the marklogic-spring-batch core and jobs library to your local maven repository, first, increment the relevant version number in the gradle.properties file.
+To deploy the marklogic-spring-batch core and test library to your local maven repository, first, increment the relevant version number in the gradle.properties file.
 
 It is advised to first publish artifacts to the local maven repository and run local tests before publishing out to [bintray](https://bintray.com/)
 
@@ -89,15 +92,6 @@ Once you have verified your version then run the following command to publish to
 gradle bintrayUpload
 ```
 
-## MSB Command Line Utility
-To deploy the MarkLogic Spring Batch command line program, execute the following gradle command
-
-```
-gradle distZip
-```
-
-This will create the distribution archive file under ./jobs/build/distribution/jobs.zip.  This can then be uploaded to the distribution website.  
-
 # What is the best way to ask a question?  
 Open an issue and label it as a Question.  Please don't use email.  This is the best way to ask once and answer for all.  If you are a MarkLogic employee, I recommend that you subscribe to the java-sig email newsgroup.  
 

README.md

@@ -38,8 +38,8 @@ repositories {
 }
 
 dependencies {
-    compile "com.marklogic:marklogic-spring-batch-core:0.6.0"
-    testComplile "com.marklogic:marklogic-spring-batch-test:0.6.0"
+    compile "com.marklogic:marklogic-spring-batch-core:0.7.0"
+    testComplile "com.marklogic:marklogic-spring-batch-test:0.7.0"
 }
 ```
 

build.gradle

@@ -1,9 +1,3 @@
-allprojects {
-    task wrapper(type: Wrapper) {
-        gradleVersion = '2.14'
-    }
-}
-
 subprojects {
     /*
      * Not applying the bintray plugin. Couldn't get it working properly when declared
@@ -16,6 +10,7 @@ subprojects {
     apply plugin: "eclipse"
     apply plugin: "idea"
     apply plugin: "checkstyle"
+    apply plugin: "maven-publish"
 
     repositories {
         jcenter()
@@ -24,6 +19,15 @@ subprojects {
         mavenLocal()
     }
 
+    dependencies {
+        compile "org.springframework.batch:spring-batch-core:3.0.7.RELEASE"
+        compile "com.marklogic:ml-javaclient-util:2.10.0-alpha"
+    }
+
+    task wrapper(type: Wrapper) {
+        gradleVersion = '3.1'
+    }
+
     sourceCompatibility = 1.8
     targetCompatibility = 1.8
 
@@ -33,10 +37,6 @@ subprojects {
         configFile = file("${project.rootDir}/dev-tools/checkstyle/checkstyle.xml")
         toolVersion = '6.19'
     }
-}
-
-configure(subprojects.findAll {it.name != 'examples'}) {
-    apply plugin: "maven-publish"
 
     task sourcesJar(type: Jar, dependsOn: classes) {
         baseName = artifactId
@@ -66,45 +66,19 @@ configure(subprojects.findAll {it.name != 'examples'}) {
     }
 }
 
-/**
- * Include every project that you want to be part of the msb application.
- */
 project(':core') {
-    dependencies {
-        runtime "org.springframework.batch:spring-batch-core:3.0.7.RELEASE"
-
-        runtime fileTree(include: ['*.jar'], dir: 'lib')
-
-        //com.marklogic.spring.batch.Main extends com.marklogic.client.helper.LoggingObject
-        runtime "com.marklogic:ml-javaclient-util:2.9.1"
-
-        //Main depends on joptsimple.OptionParser;
-        runtime "net.sf.jopt-simple:jopt-simple:5.0.1"
-
-        runtime "org.springframework:spring-jdbc:4.2.6.RELEASE"
+    task testExamples(type: GradleBuild) {
+        buildFile = '../examples/build.gradle'
+        tasks = [ 'test' ]
     }
-}
 
-project(':test') {
-    dependencies {
-        compile project(':core')
-    }
+    test.dependsOn testExamples
 }
 
-configure(subprojects.findAll {it.name != 'core' && it.name != "test"}) {
-
-    apply plugin: "application"
-    mainClassName = "com.marklogic.spring.batch.Main"
-
+project(':test') {
     dependencies {
         compile project(':core')
-        testCompile project(':test')
     }
 }
 
-task testExamples(type: GradleBuild) {
-    buildFile = 'examples/build.gradle'
-    tasks = [ 'test' ]
-}
 
-task testAll(dependsOn:[':core:test', 'examples:test', 'testExamples'] )

core/build.gradle

@@ -1,54 +1,49 @@
 plugins {
-    id "com.marklogic.ml-gradle" version "2.3.3"
-    id "com.jfrog.bintray" version "1.6"
-}
-
-if (System.getProperty("user.dir").endsWith("jobs")) {
-    project.ext.smartModulePaths = ["../marklogic-spring-batch-core/src/test/ml-modules"]
-} else if (System.getProperty("user.dir").endsWith("marklogic-spring-batch")) {
-    project.ext.smartModulePaths = ["marklogic-spring-batch-core/src/test/ml-modules"]
-} else {
-    project.ext.smartModulePaths = ["src/test/ml-modules"]
-}
-
-ext {
-    mlAppConfig {
-        modulePaths = smartModulePaths
-        createTriggersDatabase = false
-    }
+    id "com.jfrog.bintray" version "1.7.1"
+    id "com.marklogic.ml-gradle" version "2.3.4"
 }
 
 dependencies {
-    compile fileTree(include: ['*.jar'], dir: 'lib')
-    compile "org.springframework.batch:spring-batch-core:3.0.7.RELEASE"
-
+    compile "com.marklogic:java-client-api:4.0.0-EA3"
     // Used by ML Job Repo implementation; depends on DataFieldMaxValueIncrementer
     compile "org.springframework:spring-jdbc:4.2.6.RELEASE"
 
     // Used by MarkLogicBatchConfiguration, which depends on Jaxb2Marshaller
     compile "org.springframework:spring-oxm:4.2.6.RELEASE"
 
-    compile("com.marklogic:ml-javaclient-util:2.9.1") {
-  	  exclude group: 'com.marklogic.java-client', module: 'client-api-java'
-    }
-
     // For deploying the ML Job Repo
-    compile "com.marklogic:ml-app-deployer:2.2.0"
+    compile "com.marklogic:ml-app-deployer:2.3.0"
 
     // For the CLI
     compile "net.sf.jopt-simple:jopt-simple:5.0.1"
 
     testCompile "com.marklogic:ml-junit:2.6.0"
+
+    runtime "org.springframework.batch:spring-batch-core:3.0.7.RELEASE"
+
+    //com.marklogic.spring.batch.Main extends com.marklogic.client.helper.LoggingObject
+    runtime "com.marklogic:ml-javaclient-util:2.10.0-alpha"
+
+    //Main depends on joptsimple.OptionParser;
+    runtime "net.sf.jopt-simple:jopt-simple:5.0.1"
+
+    runtime "org.springframework:spring-jdbc:4.2.6.RELEASE"
 }
 
 jar {
     manifest {
         attributes(
-            'Main-Class': 'com.marklogic.spring.batch.Main'
+                'Main-Class': 'com.marklogic.spring.batch.Main'
         )
     }
 }
 
+ext {
+    mlAppConfig {
+        createTriggersDatabase = false
+    }
+}
+
 task deployMarkLogicJobRepository(type: JavaExec) {
     classpath = sourceSets.main.runtimeClasspath
     main = 'com.marklogic.spring.batch.Main'

core/gradle/wrapper/gradle-wrapper.jar

@@ -1,6 +1,6 @@
-#Wed Jun 01 23:09:17 EDT 2016
+#Fri Sep 30 20:57:34 EDT 2016
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-2.13-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-3.1-bin.zip

core/gradle/wrapper/gradle-wrapper.properties

@@ -161,4 +161,9 @@ function splitJvmOpts() {
 eval splitJvmOpts $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS
 JVM_OPTS[${#JVM_OPTS[*]}]="-Dorg.gradle.appname=$APP_BASE_NAME"
 
+# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
+if [[ "$(uname)" == "Darwin" ]] && [[ "$HOME" == "$PWD" ]]; then
+  cd "$(dirname "$0")"
+fi
+
 exec "$JAVACMD" "${JVM_OPTS[@]}" -classpath "$CLASSPATH" org.gradle.wrapper.GradleWrapperMain "$@"

core/gradlew

@@ -49,7 +49,6 @@ goto fail
 @rem Get command-line arguments, handling Windows variants
 
 if not "%OS%" == "Windows_NT" goto win9xME_args
-if "%@eval[2+2]" == "4" goto 4NT_args
 
 :win9xME_args
 @rem Slurp the command line arguments.
@@ -60,11 +59,6 @@ set _SKIP=2
 if "x%~1" == "x" goto execute
 
 set CMD_LINE_ARGS=%*
-goto execute
-
-:4NT_args
-@rem Get arguments from the 4NT Shell from JP Software
-set CMD_LINE_ARGS=%$
 
 :execute
 @rem Setup the command line

core/gradlew.bat

@@ -53,4 +53,9 @@ public DocumentMetadataWriteHandle getMetadata() {
     public AbstractWriteHandle getContent() {
         return handle;
     }
+    
+    @Override
+    public String getTemporalDocumentURI() {
+        return null;
+    }
 }

core/src/main/java/com/marklogic/client/io/MarkLogicWriteHandle.java

@@ -21,6 +21,7 @@
  * The assumption is most MarkLogic Spring Batch config classes will need at least a JobBuilderFactory,
  * a StepBuilderFactory, and a DatabaseClient.
  */
+@Deprecated
 @EnableBatchProcessing
 public abstract class AbstractMarkLogicBatchConfig extends LoggingObject implements ApplicationContextAware {