Spring Batch Example

In this post, we will see spring batch example using MySql and Spring Boot. we are going to cover below points.

  1. How to upload CSV file to Oracle/MySql database using Spring Batch.
  2. How to read data from the database and write it to CSV files.
  3. Spring batch scheduler example.

Spring Batch is one of the Spring modules and open-source frameworks used to build a lightweight and robust application. Spring Batch is mainly used for batch processing.

Batch processing allows the execution of jobs automatically(i.e without human interaction). Let’s see how Spring Batch is useful.

Consider we have to read a large number of records from a database or some excel file or from the queue. One possible solution, we will have some rest endpoints that will read all data, modify the data and save data back to the database.

Another possible solution we want to happen these all processes automatically. It should read the data, modify them and save the data back into the database. Here Spring Batch can perform these tasks for us.

Note – On the basis of input resources, Spring Batch applications can be categorized as below.

  • Database driven applications – Intract with database(e.g MySql, Postgresql) for read & write operation.
  • File driven applications – Intract with File.
  • Message driven – Intract with Messaging Queue. See AWS queue example, Kafka messaging and Active MQ example.

Benefits of Spring Batch

  • Spring Batch can be used to process a large amount of data for example logging/tracing, transaction management, job processing, and resource management.
  • It allows us to perform a particular task at a scheduled time.

Spring Batch example CSV file upload to Oracle/MySql database

Consider we have few records in excel as below.

We will read this record and insert to MySQL database using Spring Batch.

High level overview of example.

Spring Batch Example
Spring Batch example tp read data from excel and write to database

Spring batch maven dependency

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.netsurfingzone.com</groupId>
    <artifactId>springbatch</artifactId>
    <version>1.0-SNAPSHOT</version>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.1.RELEASE</version>
    </parent>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.batch</groupId>
            <artifactId>spring-batch-core</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-batch</artifactId>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
        </dependency>
    </dependencies>
    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
    </properties>
</project>

Directory structure

Define model class i.e Student.java

package com.springbatchexample.entity;

public class Student {

    private Long id;
    private String name;
    private String rollNumber;

    public Student() {

    }
    public Student(Long id, String name, String rollNumber) {
        this.id = id;
        this.name = name;
        this.rollNumber = rollNumber;
    }
    //gettet & setter
}

Define custom StudentItemProcessor class implementing ItemProcessor interface and override process() method.

Note – The ItemProcessor is an optional component, used to transform, validate and filter the data.

package com.springbatchexample.config;

import com.springbatchexample.entity.Student;
import org.springframework.batch.item.ItemProcessor;

public class StudentItemProcessor implements ItemProcessor<Student, Student> {

    @Override
    public Student process(Student student) throws Exception {
        return student;
    }
}

The ItemProcessorinterface is the top-level interface in the hierarchy.

Define springBatchConfig.java class

package com.springbatchexample.config;

import com.springbatchexample.entity.Student;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.job.builder.FlowJobBuilder;
import org.springframework.batch.core.job.builder.JobBuilder;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.core.step.builder.SimpleStepBuilder;
import org.springframework.batch.core.step.builder.StepBuilder;
import org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider;
import org.springframework.batch.item.database.JdbcBatchItemWriter;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.LineMapper;
import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;

import javax.sql.DataSource;

@EnableBatchProcessing
@Configuration
public class SpringBatchConfig {
    @Autowired
    private DataSource dataSource;

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    public FlatFileItemReader<Student> reader() {
        FlatFileItemReader<Student> itemReader = new FlatFileItemReader<>();
        itemReader.setResource(new ClassPathResource("data.csv"));
        itemReader.setLineMapper(getLineMapper());
        itemReader.setLinesToSkip(1);
        return itemReader;
    }

    private LineMapper<Student> getLineMapper() {
        DefaultLineMapper<Student> lineMapper = new DefaultLineMapper<>();
        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();

        String[] columnsToBeInserted = new String[]{"id", "roll_number", "name"};
        int[] fields = new int[]{0, 1, 2};
        tokenizer.setNames(columnsToBeInserted);
        tokenizer.setIncludedFields(fields);
        BeanWrapperFieldSetMapper<Student> fieldSetMapper = new BeanWrapperFieldSetMapper<>();
        fieldSetMapper.setTargetType(Student.class);
        lineMapper.setLineTokenizer(tokenizer);
        lineMapper.setFieldSetMapper(fieldSetMapper);
        return lineMapper;
    }

    @Bean
    public StudentItemProcessor processor() {
        return new StudentItemProcessor();
    }

    @Bean
    public JdbcBatchItemWriter<Student> writer() {
        JdbcBatchItemWriter<Student> writer = new JdbcBatchItemWriter<>();
        writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>());
        writer.setSql("insert into student(id,roll_number,name) values (:id,:rollNumber,:name)");
        writer.setDataSource(dataSource);
        return writer;
    }

    @Bean
    public Job writeStudentDataIntoSqlDb() {
        JobBuilder jobBuilder = jobBuilderFactory.get("STUDENT_JOB");
        jobBuilder.incrementer(new RunIdIncrementer());
        FlowJobBuilder flowJobBuilder = jobBuilder.flow(getFirstStep()).end();
        Job job = flowJobBuilder.build();
        return job;
    }

    @Bean
    public Step getFirstStep() {
        StepBuilder stepBuilder = stepBuilderFactory.get("getFirstStep");
        SimpleStepBuilder<Student, Student> simpleStepBuilder = stepBuilder.chunk(1);
        return simpleStepBuilder.reader(reader()).processor(processor()).writer(writer()).build();
    }
}

Define SpringMain.java

package com.springbatchexample.main;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.domain.EntityScan;
import org.springframework.context.annotation.ComponentScan;

@SpringBootApplication
@ComponentScan(basePackages = "com.springbatchexample.*")
public class SpringMain {
    public static void main(String[] args) {
        SpringApplication.run(SpringMain.class, args);
    }
}

application.properties file for mysqldb

spring.datasource.url=jdbc:mysql://localhost:3306/springbootcrudexample
spring.datasource.username=root
spring.datasource.password=root
spring.jpa.hibernate.ddl-auto=create
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.format_sql=true
server.port = 9091
spring.batch.initialize-schema=always	

application.properties for oracle database

spring.datasource.url=jdbc:oracle:thin:@localhost:1521:XE
spring.datasource.username=SYSTEM
spring.datasource.password=oracle
spring.datasource.driver-class-name=oracle.jdbc.driver.OracleDriver
# Show or not log for each sql query
spring.jpa.show-sql = true
spring.jpa.properties.hibernate.format_sql=true 
 
spring.jpa.hibernate.ddl-auto =create
spring.jpa.properties.hibern

data.csv

id,roll_number,name
1,0126,Daenerys
2,0127,Bran
3,0128,Sam
4,0129,Eddard
5,0130,Jon
6,0131,Cersei
7,0132,Tyrion
8,0133,Sansa
9,0134,Arya
10,0135,Jaime

Run SpringMain class and Verify the MySqlDataBase. We should able to see record in Database.
Spring Batch

Spring Batch read from database and write to CSV

In this example we will read data from database instead of excel and write it to excel. For reading the data from database we will use JdbcCursorItemReader.

High level overview of example

Spring Batch example CSV file upload to Oracle
Spring Batch example tp read data from database and write to excel

Define RowMapper i.e StudentResultRowMapper

@Component
public class StudentResultRowMapper implements RowMapper<Student> {
    @Override
    public Student mapRow(ResultSet rs, int i) throws SQLException {
        Student student = new Student();
        student.setId(rs.getLong("id"));
        student.setRollNumber(rs.getString("roll_number"));
        student.setName(rs.getString("name"));
        return student;
    }
}

Changes need to be made in the SpringBatchConfig file as below.

Configure JdbcCursorItemReader.

    @Bean
    public JdbcCursorItemReader<Student> reader(){
        JdbcCursorItemReader<Student> reader = new JdbcCursorItemReader<>();
        reader.setDataSource(dataSource);
        reader.setSql("select id, roll_number, name from student");
        reader.setRowMapper(new StudentResultRowMapper());
        return reader;
    }

Configure FlatFileItemReader

    @Bean
    public FlatFileItemWriter<Student> writer(){
        FlatFileItemWriter<Student> writer =  new FlatFileItemWriter<>();
        writer.setResource(new FileSystemResource("C://data/batch/data.csv"));
        DelimitedLineAggregator<Student> aggregator = new DelimitedLineAggregator<>();
        writer.setLineAggregator(getDelimitedLineAggregator());
        return writer;
    }
    private DelimitedLineAggregator<Student> getDelimitedLineAggregator() {
        BeanWrapperFieldExtractor<Student> beanWrapperFieldExtractor = new BeanWrapperFieldExtractor<Student>();
        beanWrapperFieldExtractor.setNames(new String[] {"id", "rollNumber", "name"});
        DelimitedLineAggregator<Student > aggregator = new DelimitedLineAggregator<Student>();
        aggregator.setDelimiter(",");
        aggregator.setFieldExtractor(beanWrapperFieldExtractor);
        return aggregator;
    }

Configure step

    @Bean
    public Step getDbToCsvStep() {
        StepBuilder stepBuilder = stepBuilderFactory.get("getDbToCsvStep");
        SimpleStepBuilder<Student, Student> simpleStepBuilder = stepBuilder.chunk(1);
        return simpleStepBuilder.reader(reader()).writer(writer()).build();
    }

Configure Job

    @Bean
    public Job dbToCsvJob() {
        JobBuilder jobBuilder = jobBuilderFactory.get("dbToCsvJob");
        jobBuilder.incrementer(new RunIdIncrementer());
        FlowJobBuilder flowJobBuilder = jobBuilder.flow(getDbToCsvStep()).end();
        Job job = flowJobBuilder.build();
        return job;
    }

Let’s put all code together in SpringBatchConfig.java file

package com.springbatchexample.config;

import com.springbatchexample.entity.Student;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.job.builder.FlowJobBuilder;
import org.springframework.batch.core.job.builder.JobBuilder;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.core.step.builder.SimpleStepBuilder;
import org.springframework.batch.core.step.builder.StepBuilder;
import org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider;
import org.springframework.batch.item.database.JdbcBatchItemWriter;
import org.springframework.batch.item.database.JdbcCursorItemReader;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.batch.item.file.LineMapper;
import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor;
import org.springframework.batch.item.file.transform.DelimitedLineAggregator;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.batch.item.validator.ValidatingItemProcessor;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.FileSystemResource;
import javax.sql.DataSource;

@EnableBatchProcessing
@Configuration
public class SpringBatchConfig {

    @Autowired
    private DataSource dataSource;

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Bean
    public JdbcCursorItemReader<Student> reader(){
        JdbcCursorItemReader<Student> reader = new JdbcCursorItemReader<>();
        reader.setDataSource(dataSource);
        reader.setSql("select id, roll_number, name from student");
        reader.setRowMapper(new StudentResultRowMapper());
        return reader;
    }

    @Bean
    public FlatFileItemWriter<Student> writer(){
        FlatFileItemWriter<Student> writer =  new FlatFileItemWriter<>();
        writer.setResource(new FileSystemResource("C://data/batch/data.csv"));
        DelimitedLineAggregator<Student> aggregator = new DelimitedLineAggregator<>();
        writer.setLineAggregator(getDelimitedLineAggregator());
        return writer;
    }
    private DelimitedLineAggregator<Student> getDelimitedLineAggregator() {
        BeanWrapperFieldExtractor<Student> beanWrapperFieldExtractor = new BeanWrapperFieldExtractor<Student>();
        beanWrapperFieldExtractor.setNames(new String[] {"id", "rollNumber", "name"});

        DelimitedLineAggregator<Student > aggregator = new DelimitedLineAggregator<Student>();
        aggregator.setDelimiter(",");
        aggregator.setFieldExtractor(beanWrapperFieldExtractor);
        return aggregator;
    }

    @Bean
    public Step getDbToCsvStep() {
        StepBuilder stepBuilder = stepBuilderFactory.get("getDbToCsvStep");
        SimpleStepBuilder<Student, Student> simpleStepBuilder = stepBuilder.chunk(1);
        return simpleStepBuilder.reader(reader()).writer(writer()).build();
    }

    @Bean
    public Job dbToCsvJob() {
        JobBuilder jobBuilder = jobBuilderFactory.get("dbToCsvJob");
        jobBuilder.incrementer(new RunIdIncrementer());
        FlowJobBuilder flowJobBuilder = jobBuilder.flow(getDbToCsvStep()).end();
        Job job = flowJobBuilder.build();
        return job;
    }
}

Rest of code would be same. After running the main class we should able to see records in excel file.

Spring Batch scheduler example

Create SpringMain.java and BatchTaskScheduler .java.

Create SpringMain.java

package com.springbatchexample.main;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.scheduling.annotation.EnableScheduling;

@SpringBootApplication
@EnableScheduling
@ComponentScan(basePackages = "com.springbatchexample.*")
public class SpringMain {
    public static void main(final String[] args) {
        SpringApplication.run(SpringMain.class, args);
    }
}




TaskScheduler.java

package com.springbatchexample.scheduler;

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
public class BatchTaskScheduler {
    @Scheduled(fixedRate = 2000)
    public void performTask() {
        System.out.println("This is simple example using spring batch");
    }
}




Run the main class SpringMain.java. Tomcat will deploy and performTask() method will execute automatically at every 2 seconds.

Sample output:-

Spring Batch scheduler example

Let’s modify BatchTaskScheduler.java logic. The rest of the things should be the same.


@Component
public class BatchTaskScheduler {
    @Scheduled(fixedRate = 4000)
    public void exampleOfFixedrate() {
        Date date = new Date();
        String stringDateFormat = "hh:mm:ss a";
        DateFormat dateFormat = new SimpleDateFormat(stringDateFormat);
        //get current time
        String formattedDateAsString = dateFormat.format(date);
        System.out.println("This method will execute after every 4 sec " + formattedDateAsString);
    }
}




Output is –

This method will execute after every 4 sec 12:03:39 PM
This method will execute after every 4 sec 12:03:43 PM

That’s all about Spring batch task scheduling example using spring boot

Annotations used for Spring Batch scheduler.

@EnabledScheduling

  • This annotation enables Spring batch task scheduling feature.
  • This is available in org.springframework.scheduling.annotation package. See here documentation.
  • Mainly used with configuration class or Spring boot main class.

@Scheduled

  • This annotation is used to make any method scheduled.
  • Available in org.springframework.scheduling.annotation package. See here documentation.
  • We need to define attributes fixedDelay or fixedRate.
  • Attributes of @Scheduled annotation
    •  cron – An expression, triggers on the second, minute, hour, day of month, month and day of week.
    • fixedRate – if we give value 2000 for this, the method will get invoked after each 2 sec.
    • fixedRateString – behave similarly as the fixedRate only difference we can pass the value as String.
    • fixedDelay – if we give value 2000 for this, the method will get invoked after each 2 sec + time to execute that method. Suppose we have given the value for fixedDelay is 2000 i.e 2 sec and time taking to execute some method(with we are using fixedDelay) is 2 sec then total time will be 4 sec.
    • fixedDelayString –  behave similarly as the fixedRate only difference we can pass the value as String.
    • initialDelay – Number of milliseconds to delay before the first execution of a fixedRate() or fixedDelay() task.
    • initialDelayString – Similar to initialDelay except we can pass the parameter as String.
    • zone – A time zone for which the cron expression will be resolved.

Dowload example from github.

Spring Batch tutorials.

That’s all about the Spring Batch example.

Spring Transaction Management tutorial.

Spring Data JPA Examples.

Spring Batch Docs.