JavaEE Sample::Batch::Batch Chunk Checkpoint

Batch Chunk Checkpoint

The source code for this sample can be found in the javaee7-samples GitHub repository. The first thing we need to do is to get the source by downloading the repository and then go into the samples folder:

git clone git://github.com/javaee-samples/javaee7-samples.git
cd javaee7-samples/batch/chunk-checkpoint/

Now we are ready to start testing. You can run all the tests in this sample by executing:

mvn test

Or you can run individual tests by executing one of the following:

mvn test -Dtest=BatchChunkCheckpointTest

Chunk Checkpoint - Custom Checkpoint Policy

Specifications in use:
Batch
CDI

BatchChunkCheckpointTest

The Batch specification provides a Chunk Oriented processing style. This style is defined by enclosing into a transaction a set of reads, process and write operations via ItemReader, ItemProcessor and ItemWriter. Items are read one at a time, processed and aggregated. The transaction is then committed when the defined checkpoint-policy is triggered.

The checkpoint-policy can be defined as item or custom. The item policy means the chunk is checkpointed after a specified number of items are processed. The custom policy means the chunk is checkpointed according to a checkpoint algorithm implementation. To use the custom policy you also need to define a checkpoint-algorithm element.

<?xml version="1.0" encoding="UTF-8"?>
<job id="myJob" xmlns="http://xmlns.jcp.org/xml/ns/javaee" version="1.0">
    <step id="myStep" >
        <chunk checkpoint-policy="custom">
            <reader ref="myItemReader"/>
            <processor ref="myItemProcessor"/>
            <writer ref="myItemWriter"/>
            <checkpoint-algorithm ref="myCheckpointAlgorithm"/>
        </chunk>
    </step>
</job>

A very simple job is defined in the myJob.xml file. Just a single step with a reader, a processor and a writer. For this sample, a custom checkpoint policy is going to be used. The custom policy needs to implement CheckpointAlgorithm or in alternative extend AbstractCheckpointAlgorithm that already provides empty implementations for all methods.

@Named()
public class MyCheckpointAlgorithm extends AbstractCheckpointAlgorithm {

    public MyCheckpointAlgorithm();
    public static CountDownLatch checkpointCountDownLatch;

    @Override()
    public boolean isReadyToCheckpoint() throws Exception;
}

Note that the behaviour of this custom checkpoint algorithm could also be achieved by using the item policy and defining the item-count element at the chunk level.

We’re just going to deploy the application as a web archive. Note the inclusion of the following files:

/META-INF/batch-jobs/myJob.xml

The myJob.xml file is needed for running the batch definition.

@Deployment
public static WebArchive createDeployment() {
    WebArchive war = ShrinkWrap.create(WebArchive.class)
            .addClass(BatchTestHelper.class)
            .addPackage("org.javaee7.batch.chunk.checkpoint")
            .addAsWebInfResource(EmptyAsset.INSTANCE, ArchivePaths.create("beans.xml"))
            .addAsResource("META-INF/batch-jobs/myJob.xml");
    System.out.println(war.toString(true));
    return war;
}

In the test, we’re just going to invoke the batch execution and wait for completion. To validate the test expected behaviour we need to query the Metric object available in the step execution.

The batch process itself will read and process 10 elements from numbers 1 to 10, but only write the odd elements. Commits are executed after 5 elements are read by the custom checkpoint algorithm.

@Test
public void testBatchChunkCheckpoint() throws Exception {
    JobOperator jobOperator = BatchRuntime.getJobOperator();
    Long executionId = jobOperator.start("myJob", new Properties());
    JobExecution jobExecution = jobOperator.getJobExecution(executionId);

    jobExecution = BatchTestHelper.keepTestAlive(jobExecution);

    List<StepExecution> stepExecutions = jobOperator.getStepExecutions(executionId);
    for (StepExecution stepExecution : stepExecutions) {
        if (stepExecution.getStepName().equals("myStep")) {
            Map<Metric.MetricType, Long> metricsMap = BatchTestHelper.getMetricsMap(stepExecution.getMetrics());

            (1)
            assertEquals(10L, metricsMap.get(Metric.MetricType.READ_COUNT).longValue());
            (2)
            assertEquals(10L / 2L, metricsMap.get(Metric.MetricType.WRITE_COUNT).longValue());
            (3)
            assertEquals(10L / 5L + 1, metricsMap.get(Metric.MetricType.COMMIT_COUNT).longValue());
        }
    }

    (4)
    assertTrue(MyCheckpointAlgorithm.checkpointCountDownLatch.await(0, TimeUnit.SECONDS));
    (5)
    assertEquals(jobExecution.getBatchStatus(), BatchStatus.COMPLETED);
}

The read count should be 10 elements. Check MyItemReader.
The write count should be 5. Only half of the elements read are processed to be written.
The commit count should be 3. Checkpoint is on every 5th read, plus one final read-commit.
The checkpoint algorithm should be checked 10 times. One for each element read.
Job should be completed.

Share the Knowledge

Find this sample useful? Share on

There's a lot more about JavaEE to cover. If you're ready to learn more, check out the other available samples.

Help Improve

Find a bug in the sample? Something missing? You can fix it by editing the source, making the correction and sending a pull request. Or report the problem to the issue tracker

Recent Changelog

Dec 14, 2014: Switch from polling on jobexecution (for job completion) to polling with joboperator and executionid by Scott Kurz
Jul 05, 2014: Removed header license for batch xml files by Roberto Cortez
Jun 22, 2014: Removed header license. the licensing is now referenced in the license file in the root of the project by Roberto Cortez
Jun 20, 2014: Added fqn to java ee api references to generate direct links to javadocs by radcortez
Jun 19, 2014: Documentation clarifications and typos by radcortez
Jun 18, 2014: Fixed documentation of chunk-checkpoint project by radcortez
Jun 18, 2014: Added documentation to chunk-checkpoint project by radcortez
Dec 31, 2013: Code style issues by Roberto Cortez
Dec 31, 2013: Removed servlets and jsp's by Roberto Cortez
Dec 04, 2013: Changed asserts for countdownlatch by Roberto Cortez

The source code for this sample can be found in the javaee7-samples GitHub repository. The first thing you need to do is to get the source by downloading the repository and then go into the samples folder:

git clone git://github.com/javaee-samples/javaee7-samples.git
cd javaee7-samples/batch/chunk-checkpoint/

Do the changes as you see fit and send a pull request!

Good Luck!