Skip to content

Spring Batch: Handling a non-standard file format

October 16, 2013

In most of the Spring Batch jobs I have written that consume files, they are typically in some type of standard delimited or fixed width format. But what happens if you have something where multiple lines make up a record and you have some kind of marker that indicates the end of a record. Here is some example data where the end of a record is marked with an eor tag:


Mike
Smith
555-1234
eor
Sam
Johnson
555-2311
eor
Dave
Williams
555-9999
eor

One way to handle this is to define a custom RecordSeparatorPolicy. Spring already provides SuffixRecordSeparatorPolicy that allows you to look for a specific String that terminates a record. Now that we have identified how to indicate the end of the record, what we want to do is treat the file like delimited file. To do this, we extend the SuffixRecordSeparatorPolicy and we can override the preProcess method to add a delimiter to the line. We create a CustomRecordSeparatorPolicy to do this.

 package org.reil.example;  
   
 import org.springframework.batch.item.file.separator.SuffixRecordSeparatorPolicy;  
   
 public class CustomRecordSeparatorPolicy extends SuffixRecordSeparatorPolicy {  
        
      private char delimiter = ',';  
        
      @Override  
      public String preProcess(String line) {  
           line = line.trim() + delimiter;  
           return super.preProcess(line);  
      }  
   
      public char getDelimiter() {  
           return delimiter;  
      }  
   
      public void setDelimiter(char delimiter) {  
           this.delimiter = delimiter;  
      }  
        
 }  

Now you can you set the policy on your FlatFileItemReader and process like a normal delimited file. Example:

       FlatFileItemReader<Person> reader = new FlatFileItemReader<Person>();  
       reader.setResource(new ClassPathResource("testFile.txt"));  
       DefaultLineMapper<Person> lineMapper = new DefaultLineMapper<Person>();  
       lineMapper.setFieldSetMapper(new PersonFieldSetMapper());  
       lineMapper.setLineTokenizer(new DelimitedLineTokenizer(','));  
       reader.setLineMapper(lineMapper);  
       //set the policy  
       CustomRecordSeparatorPolicy policy = new CustomRecordSeparatorPolicy();  
       policy.setSuffix("eor");  
       reader.setRecordSeparatorPolicy(policy);  

For the full runnable example you can download it from GitHub here. The project is spring-batch-file. Run the JobConfigurationTests JUnit to run the job.

Check out the Spring Batch reference documentation for more on handling of flat files. Also, for handling multiple record types within a file, take a look at the PatternMatchingCompositeLineMapper.

As a side note, it is really easy to create Spring Batch jobs using the STS Spring Project wizard. Also, there is quick start here if you are interested in creating a batch job that runs using Spring Boot.

Advertisements

From → Spring

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: