Merge branch 'dev'

Author: Scott Stafford

README.md

@@ -16,13 +16,13 @@ Have you ever written a long running batch processing job, like migrating data f
 
 # What are the main features of MarkLogic Spring Batch?
 * Provide custom ItemReaders, ItemProcessors, ItemWriters, and tasklets for writing custom MarkLogic batch processing jobs.
+* Mitigate the risk of the Spring Batch learning curve by providing examples of creating your own custom batch processing job
 * Persist the metadata of any JobExecution with a MarkLogic [JobRepository](http://docs.spring.io/spring-batch/trunk/reference/html/domain.html#domainJobRepository)
 * Execute one of many generic MarkLogic batch processing jobs for importing, transforming, and exporting data in a MarkLogic database
-* Mitigate the risk of the Spring Batch learning curve by providing examples of creating your own custom batch processing job
 
 # How do I build a custom MarkLogic batch processing job? 
 
-The [examples directory](https://github.com/sastafford/marklogic-spring-batch/tree/master/examples) provides examples of MarkLogic batch processing jobs.  Each example is setup to be deployed as a command line utility.  The [base job](https://github.com/sastafford/marklogic-spring-batch/tree/master/examples/base) provides a template for creating a custom import batch processing job.    
+The [examples directory](https://github.com/sastafford/marklogic-spring-batch/tree/master/examples) provides different examples of MarkLogic batch processing jobs.  Each example is setup to be deployed as a command line utility.  The [base job example](https://github.com/sastafford/marklogic-spring-batch/tree/master/examples/base) is a bare bones Spring configuration template for creating a custom ingest batch processing job.
 
 ## Import the MSB jars
 Create a build.gradle file to import the MarkLogic Spring Batch (MSB) jars.  Use the [latest version](https://github.com/sastafford/marklogic-spring-batch/releases) of the MSB jars.  
@@ -38,8 +38,8 @@ repositories {
 }
 
 dependencies {
-    compile "com.marklogic:marklogic-spring-batch-core:0.5.0"
-    testComplile "com.marklogic:marklogic-spring-batch-test:0.5.0"
+    compile "com.marklogic:marklogic-spring-batch-core:0.6.0"
+    testComplile "com.marklogic:marklogic-spring-batch-test:0.6.0"
 }
 ```
 
@@ -52,10 +52,10 @@ Create a test class by subclassing the [AbstractJobTest](https://github.com/sast
 # How do I execute my job? 
 An easy way to execute your job is to execute it as a command line utility similar to MarkLogic Content Pump or CORB.  An easy way to deploy the job is via the [Gradle Application Plugin](https://docs.gradle.org/current/userguide/application_plugin.html).  Once you have your JobConfiguration created and have verified that it works, then execute the following gradle command to build an executable.
 
-    gradle distZip
-    
-This will create a zip file under ../install/distributions that can be transferred and executed on any host.  
+    gradle installDist
 
+This command will create two start scripts, one for Windows and one for Unix, and will package up all the libraries.  The installation package can be found under the ./build/install/<artifactId>.   directory where artifactId is a variable of the same name defined in your build.gradle or gradle.properties file.  Once this exists, you can then execute the start script and pass the required and any optional parameters defined by your job.  See either the [importInvoices task from the rdbms_2 build.gradle file](https://github.com/sastafford/marklogic-spring-batch/blob/master/examples/rdbms_2/build.gradle) or the manually created [runInovices.bat](https://github.com/sastafford/marklogic-spring-batch/blob/master/examples/rdbms_2/runInvoices.bat) file.  The runInvoices.bat assumes that the gradle installDist task has already been executed.   
+    
 Each MarkLogic Spring Batch execution application (based on the Main class) expects a few command line parameters.  Custom parameters can be defined in the JobConfiguration. 
 
   * config - The class name of your Job configuration
@@ -69,9 +69,15 @@ Each MarkLogic Spring Batch execution application (based on the Main class) expe
   * jrPassword (optional) - the Job Repository MarkLogic password
 
 ```
-./jobExec --config com.marklogic.spring.batch.job.JobNameConfig.class --host localhost --port 8010 --username admin --password admin --customParam1 xyz --customParamX abc
+./<artifactId_script_name> --config com.marklogic.spring.batch.job.JobNameConfig --host localhost --port 8010 --username admin --password admin --customParam1 xyz --customParamX abc
 ```
 
+# How do I distribute my command line job utility?
+
+The following gradle command will create a zip file under ./build/distributions that can be transferred and executed on any host.  
+ 
+    gradle distZip
+
 ## Create a Gradle JavaExec task
 
 The second option is to create a gradle JavaExec task that would execute your job via a gradle command.  Here is an example gradle task taken from the [mysql-sakila example](https://github.com/sastafford/marklogic-spring-batch/blob/master/examples/mysql-sakila/build.gradle).  
@@ -105,19 +111,19 @@ gradle migrateActors
 If you want to use a MarkLogic Job Repository then you first need to install a new application onto your MarkLogic database.  A MarkLogic Job Repository can be installed via the executable created with your Gradle application plugin.  
 
 ```
-./jobs deployMarkLogicJobRepository --jr_host localhost --jr_port 8011 --jr_username admin --jr_password admin
+./<jobArtifactScriptName> deployMarkLogicJobRepository --jr_host localhost --jr_port 8011 --jr_username admin --jr_password admin
 ```
 
 If you ever need to undeploy the JobRepository then you can issue the following command.
 
 ```
-./jobs undeployMarkLogicJobRepository --jr_host localhost --jr_port 8011 --jr_username admin --jr_password admin
+./<jobArtifactScriptName> undeployMarkLogicJobRepository --jr_host localhost --jr_port 8011 --jr_username admin --jr_password admin
 ```
 
 Now when you execute your job, then add the additional parameters for the MarkLogic Job Repository.  All JobExecution metadata is now logged to the MarkLogic JobRepository.  
 
 ```
-./jobs --config com.marklogic.spring.batch.job.JobNameConfig.class --host localhost --port 8010 --username admin --password admin --custom_param1 xyz --customParamX abc --jr_host localhost --jr_port 8011 --jr_username admin --jr_password admin
+./<jobArtifactScriptName> --config com.marklogic.spring.batch.job.JobNameConfig --host localhost --port 8010 --username admin --password admin --custom_param1 xyz --customParamX abc --jr_host localhost --jr_port 8011 --jr_username admin --jr_password admin
 ```
 
 # How can I contribute to the project?

build.gradle

@@ -35,7 +35,7 @@ subprojects {
     }
 }
 
-configure(subprojects.findAll {it.name != 'jobs'}) {
+configure(subprojects.findAll {it.name != 'examples'}) {
     apply plugin: "maven-publish"
 
     task sourcesJar(type: Jar, dependsOn: classes) {
@@ -107,4 +107,4 @@ task testExamples(type: GradleBuild) {
     tasks = [ 'test' ]
 }
 
-task testAll(dependsOn:[':core:test', 'jobs:test', 'testExamples'] )
+task testAll(dependsOn:[':core:test', 'examples:test', 'testExamples'] )

core/build.gradle

@@ -1,5 +1,5 @@
 plugins {
-    id "com.marklogic.ml-gradle" version "2.2.0"
+    id "com.marklogic.ml-gradle" version "2.3.3"
     id "com.jfrog.bintray" version "1.6"
 }
 

core/src/main/java/com/marklogic/spring/batch/config/ConfigTypeFilter.java

@@ -2,7 +2,6 @@
 
 import com.marklogic.client.spring.BasicConfig;
 import com.marklogic.spring.batch.MainConfig;
-import com.marklogic.spring.batch.core.repository.dao.MarkLogicDaoConfig;
 import org.springframework.core.type.ClassMetadata;
 import org.springframework.core.type.filter.AbstractClassTestingTypeFilter;
 
@@ -17,7 +16,6 @@ protected boolean match(ClassMetadata metadata) {
 		String name = metadata.getClassName();
 		return name.startsWith("org.springframework")
 			|| name.equals(MainConfig.class.getName())
-			|| name.equals(MarkLogicDaoConfig.class.getName())
 			|| name.equals(BasicConfig.class.getName());
 	}
 

core/src/main/java/com/marklogic/spring/batch/config/MarkLogicBatchConfigurer.java

@@ -0,0 +1,106 @@
+package com.marklogic.spring.batch.config;
+
+import com.marklogic.client.DatabaseClient;
+import com.marklogic.client.helper.DatabaseClientProvider;
+import com.marklogic.spring.batch.core.explore.support.MarkLogicJobExplorerFactoryBean;
+import com.marklogic.spring.batch.core.repository.support.MarkLogicJobRepositoryFactoryBean;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.batch.core.configuration.annotation.BatchConfigurer;
+import org.springframework.batch.core.explore.JobExplorer;
+import org.springframework.batch.core.explore.support.MapJobExplorerFactoryBean;
+import org.springframework.batch.core.launch.JobLauncher;
+import org.springframework.batch.core.launch.support.SimpleJobLauncher;
+import org.springframework.batch.core.repository.JobRepository;
+import org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean;
+import org.springframework.batch.support.transaction.ResourcelessTransactionManager;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+import org.springframework.transaction.PlatformTransactionManager;
+
+import javax.annotation.PostConstruct;
+
+@Component
+public class MarkLogicBatchConfigurer implements BatchConfigurer {
+    
+    private final static Logger logger = LoggerFactory.getLogger(MarkLogicBatchConfigurer.class);
+    private DatabaseClient databaseClient;
+    private JobRepository jobRepository;
+    private JobExplorer jobExplorer;
+    private JobLauncher jobLauncher;
+    private PlatformTransactionManager transactionManager;
+    
+    protected MarkLogicBatchConfigurer() {}
+    
+    @Autowired
+    public MarkLogicBatchConfigurer(DatabaseClientProvider databaseClientProvider) {
+        this.databaseClient = databaseClientProvider.getDatabaseClient();
+    }
+    
+    @PostConstruct
+    public void initialize() throws Exception {
+        if(databaseClient == null) {
+            logger.warn("No DatabaseClient was provided...using a Map based JobRepository");
+    
+            if(this.transactionManager == null) {
+                this.transactionManager = new ResourcelessTransactionManager();
+            }
+    
+            MapJobRepositoryFactoryBean jobRepositoryFactory = new MapJobRepositoryFactoryBean(this.transactionManager);
+            jobRepositoryFactory.afterPropertiesSet();
+            this.jobRepository = jobRepositoryFactory.getObject();
+    
+            MapJobExplorerFactoryBean jobExplorerFactory = new MapJobExplorerFactoryBean(jobRepositoryFactory);
+            jobExplorerFactory.afterPropertiesSet();
+            this.jobExplorer = jobExplorerFactory.getObject();
+        } else {
+            this.transactionManager = new ResourcelessTransactionManager();
+            this.jobRepository = createJobRepository();
+            this.jobExplorer = createJobExplorer();
+        }
+        this.jobLauncher = createJobLauncher();
+    }
+    
+    protected JobRepository createJobRepository() throws Exception {
+        MarkLogicJobRepositoryFactoryBean factory = new MarkLogicJobRepositoryFactoryBean();
+        factory.setDatabaseClient(databaseClient);
+        factory.setTransactionManager(transactionManager);
+        factory.afterPropertiesSet();
+        return factory.getObject();
+    }
+    
+    protected JobExplorer createJobExplorer() throws Exception {
+        MarkLogicJobExplorerFactoryBean factory = new MarkLogicJobExplorerFactoryBean();
+        factory.setDatabaseClient(databaseClient);
+        factory.afterPropertiesSet();
+        return factory.getObject();
+    }
+    
+    protected JobLauncher createJobLauncher() throws Exception {
+        SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
+        jobLauncher.setJobRepository(jobRepository);
+        jobLauncher.afterPropertiesSet();
+        return jobLauncher;
+    }
+
+    @Override
+    public JobRepository getJobRepository() throws Exception {
+        return jobRepository;
+    }
+    
+    @Override
+    public PlatformTransactionManager getTransactionManager() throws Exception {
+        return transactionManager;
+    }
+    
+    @Override
+    public JobLauncher getJobLauncher() throws Exception {
+        return jobLauncher;
+    }
+    
+    @Override
+    public JobExplorer getJobExplorer() throws Exception {
+        return jobExplorer;
+    }
+    
+}

core/src/main/java/com/marklogic/spring/batch/config/support/JobRepositoryConfigurer.java

@@ -61,7 +61,7 @@ public void initialize() {
 
             JobInstanceDao jobInstanceDao = new MarkLogicJobInstanceDao(client);
             JobExecutionDao jobExecutionDao = new MarkLogicJobExecutionDao(client);
-            MarkLogicStepExecutionDao stepExecutionDao = new MarkLogicStepExecutionDao(client);
+            MarkLogicStepExecutionDao stepExecutionDao = new MarkLogicStepExecutionDao(client, jobExecutionDao);
             stepExecutionDao.setJobExecutionDao(jobExecutionDao);
             ExecutionContextDao executionContextDao = new MarkLogicExecutionContextDao(jobExecutionDao, stepExecutionDao);
 

core/src/main/java/com/marklogic/spring/batch/core/explore/support/MarkLogicJobExplorerFactoryBean.java

@@ -0,0 +1,69 @@
+package com.marklogic.spring.batch.core.explore.support;
+
+import com.marklogic.client.DatabaseClient;
+import com.marklogic.spring.batch.core.repository.dao.MarkLogicExecutionContextDao;
+import com.marklogic.spring.batch.core.repository.dao.MarkLogicJobExecutionDao;
+import com.marklogic.spring.batch.core.repository.dao.MarkLogicJobInstanceDao;
+import com.marklogic.spring.batch.core.repository.dao.MarkLogicStepExecutionDao;
+import org.springframework.batch.core.explore.JobExplorer;
+import org.springframework.batch.core.explore.support.AbstractJobExplorerFactoryBean;
+import org.springframework.batch.core.explore.support.SimpleJobExplorer;
+import org.springframework.batch.core.repository.dao.ExecutionContextDao;
+import org.springframework.batch.core.repository.dao.JobExecutionDao;
+import org.springframework.batch.core.repository.dao.JobInstanceDao;
+import org.springframework.batch.core.repository.dao.StepExecutionDao;
+import org.springframework.beans.factory.InitializingBean;
+import org.springframework.util.Assert;
+
+public class MarkLogicJobExplorerFactoryBean extends AbstractJobExplorerFactoryBean implements InitializingBean {
+    
+    private DatabaseClient databaseClient;
+    private JobInstanceDao jobInstanceDao;
+    private JobExecutionDao jobExecutionDao;
+    private StepExecutionDao stepExecutionDao;
+    private ExecutionContextDao executionContextDao;
+    
+    /**
+     * Public setter for the {@link DatabaseClient}.
+     * @param databaseClient a {@link DatabaseClient}
+     */
+    public void setDatabaseClient(DatabaseClient databaseClient) {
+        this.databaseClient = databaseClient;
+    }
+
+    @Override
+    protected JobInstanceDao createJobInstanceDao() throws Exception {
+        return jobInstanceDao;
+    }
+    
+    @Override
+    protected JobExecutionDao createJobExecutionDao() throws Exception {
+        return jobExecutionDao;
+    }
+    
+    @Override
+    protected StepExecutionDao createStepExecutionDao() throws Exception {
+        return stepExecutionDao;
+    }
+    
+    @Override
+    protected ExecutionContextDao createExecutionContextDao() throws Exception {
+        return executionContextDao;
+    }
+    
+    @Override
+    public JobExplorer getObject() throws Exception {
+        return new SimpleJobExplorer(createJobInstanceDao(),
+                createJobExecutionDao(), createStepExecutionDao(),
+                createExecutionContextDao());
+    }
+    
+    @Override
+    public void afterPropertiesSet() throws Exception {
+        Assert.notNull(databaseClient, "DatabaseClient must not be null.");
+        jobInstanceDao = new MarkLogicJobInstanceDao(databaseClient);
+        jobExecutionDao = new MarkLogicJobExecutionDao(databaseClient);
+        stepExecutionDao = new MarkLogicStepExecutionDao(databaseClient, jobExecutionDao);
+        executionContextDao = new MarkLogicExecutionContextDao(jobExecutionDao, stepExecutionDao);
+    }
+}

core/src/main/java/com/marklogic/spring/batch/core/repository/MarkLogicSimpleJobRepository.java

@@ -11,8 +11,10 @@
 import org.springframework.batch.core.repository.dao.JobInstanceDao;
 import org.springframework.batch.core.repository.dao.StepExecutionDao;
 import org.springframework.batch.core.repository.support.SimpleJobRepository;
+import org.springframework.stereotype.Component;
 import org.springframework.util.Assert;
 
+@Component
 public class MarkLogicSimpleJobRepository extends SimpleJobRepository {
 
 	private StepExecutionDao stepExecutionDao;

core/src/main/java/com/marklogic/spring/batch/core/repository/dao/AbstractMarkLogicBatchMetadataDao.java

@@ -1,17 +1,12 @@
 package com.marklogic.spring.batch.core.repository.dao;
 
 import com.marklogic.client.helper.LoggingObject;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.context.ApplicationContext;
 import org.springframework.jdbc.support.incrementer.DataFieldMaxValueIncrementer;
 
 import com.marklogic.client.DatabaseClient;
 
 public abstract class AbstractMarkLogicBatchMetadataDao extends LoggingObject {
 
-	@Autowired
-	protected ApplicationContext ctx;
-	
 	protected DatabaseClient databaseClient;
 
 	protected DataFieldMaxValueIncrementer incrementer;
@@ -22,24 +17,13 @@ public abstract class AbstractMarkLogicBatchMetadataDao extends LoggingObject {
     public final String SEARCH_OPTIONS_NAME = "spring-batch";
 
 	public final String SPRING_BATCH_DIR = "/projects.spring.io/spring-batch/";
-	public final String SPRING_BATCH_INSTANCE_DIR = SPRING_BATCH_DIR + "instance/";
 
-	public final String COLLECTION_JOB_EXECUTION = "http://marklogic.com/spring-batch/job-execution";
 	public final String COLLECTION_JOB_INSTANCE = "http://marklogic.com/spring-batch/job-instance";
-	public final String COLLECTION_STEP_EXECUTION = "http://marklogic.com/spring-batch/step-execution";
 
 	public DatabaseClient getDatabaseClient() {
 		return databaseClient;
 	}
-
-	public void setDatabaseClient(DatabaseClient databaseClient) {
-		this.databaseClient = databaseClient;
-	}
 
-	public DataFieldMaxValueIncrementer getIncrementer() {
-		return incrementer;
-	}
-
 	public void setIncrementer(DataFieldMaxValueIncrementer incrementer) {
 		this.incrementer = incrementer;
 	}