# Batch Framework 

## Basic Concept

Batch is a basic module that allows certain sets of system programs to run at the scheduled time.
It aims to automate the operation of some system programs and to manage system performance loads asynchronously.
Batch trigger can be further divided into ad-hoc batch, which will run when triggered manually, and scheduled batch, which will run at pre-defined points in time.

## User Scenario

For anybody who wants to use our platform to develop a batch and schedule the trigger time of the batch.

## Composition of Batch

Batch has two basic aspects: 

*  ***Batch job*** is the system job to achieve the specific system operation.
*  ***Batch trigger*** is the trigger point that allows the batch to run at a specific time.

Our platform allows users to define both aspects. From a business perspective, there are two business scenarios depending on when the batch is triggered.

*  ***Pre-scheduled batch***: The trigger is defined in advance to run at a fixed time.
   For the pre-scheduled batch, we can define jobs and triggers in the MC environment and then deploy them together to other runtime environments.

*  ***Ad-hoc batch***: The trigger is not defined in advance and it is mainly triggered manually in production at a random time.
   For the ad-hoc batch, we do not need to define the trigger in advance, and we can submit the job to the corresponding runtime environment at a suitable time.

Batch configuration UI supports the user to define the batch job and the corresponding trigger, and run ad-hoc batch job. 

In most scenarios, we can monitor and check the execution status and history in case any abnormal behavior occurs.

Technically, we are using the springbatch framework to run the batch. Any pre-knowledge of the springbatch framework will be highly beneficial to comprehending the practical batch use.


## Basic UI Operation

<div class="docs-note"><span class="docs-admonitions-text">note</span>

**- Batch and trigger definitions need to be performed within either the runtime or portal MC environment, and are not allowed in other environments unless for troubleshooting purposes.**

**- Please make sure you possess read and write authorities to make the operation beforehand.**

</div>

### Job Definition

1. Find the menu and click **New**.

![BatchJob01](./image/batch/batch_job_create_01.PNG)

2. Create the job and save it.

![BatchJob02](./image/batch/batch_job_create_02.PNG)

##### Field Description

* **Job Name**: Each tenant must have a unique job name.  
* **Job Type**: Options include manual task, weekly task, monthly task, daily task, and timing task.  
* **Micro Service**: Micro service name. Users can select the microservice name or enter the microservice name manually (the microservice name must be in lowercase).  
* **Estimated Duration (Minute)**: Estimate the time required for job completion.  
* **Trigger**: Select from the list of triggers with the predefined run time.  
* **Spring Batch Name**: It is a bean name from the background code.
* **Allow Multiple Instances**: If set to Yes, the system will allow users to start the job while the same job is executed.
* **Callback URL**: [optional]Callback after the job is successfully executed.   
* **Job Description**: [optional]Describe the trigger's function.  
* **Job Parameter**: [optional]Ignored if the job has no parameters. If the job has parameters, fill in the parameters according to the format.

### Job Submission

1. Find the menu and job and click the submit icon.

![BatchJobSubmit01](./image/batch/batch_job_submit_01.PNG)

2. Click **Submit**.

![BatchJobSubmit02](./image/batch/batch_job_submit_02.PNG)

A dialog box will pop up after the job is successfully submitted.  

<div class="docs-note"><span class="docs-admonitions-text">note</span>

When one of the Run IDs and Job IDs is null or empty, an exception may occur when the job is executed.  

</div>

![BatchJobSubmit03](./image/batch/batch_job_submit_03.PNG)

### Trigger Definition

1. Find the menu and click **New**.

![BatchTrigger01](./image/batch/batch_trigger_create_01.PNG)

2. Create the trigger and save it.

![BatchTrigger02](./image/batch/batch_trigger_create_02.PNG)

3. Start and Stop

<div class="docs-note"><span class="docs-admonitions-text">note</span>

- The **Start** and **Stop** operations are contingent upon the presence of triggers and jobs.  
- Start the trigger first and then the job.  
- Stop the job first before the trigger. 

</div>

#### Field Description  
* **Trigger Name**: Each tenant must have a unique trigger name.  
* **Trigger Interval(s) and Corn Expression**: Fill in one of the two text boxes.  
* **Status**: If the value is set to true, the trigger will work; otherwise, it will fail to work.  
* **Trigger Description**: Describe the function of the trigger.  

##### Start Trigger  

![BatchTrigger03](./image/batch/batch_trigger_start_01.PNG)

##### Start Job  

![BatchTrigger04](./image/batch/batch_job_start_01.PNG)

![BatchTrigger05](./image/batch/batch_job_start_02.PNG)

##### Stop Job  

![BatchTrigger06](./image/batch/batch_job_stop_01.PNG)

![BatchTrigger07](./image/batch/batch_job_stop_02.PNG)

##### Stop Trigger  

![BatchTrigger08](./image/batch/batch_trigger_stop_01.PNG)


### Job Monitoring

1. Find the menu and then query by condition.

![BatchJobMonitoring01](./image/batch/batch_job_monitoring_01.PNG)


2. Click **Run Id** and then view the details.

![BatchJobMonitoring02](./image/batch/batch_job_monitoring_02.PNG)


### Batch Report

1. Find the menu and then query by condition.

![BatchBatchReport01](./image/batch/batch_batch_report_01.PNG)

2. Download the result.

![BatchBatchReport02](./image/batch/batch_batch_report_02.PNG)


### Publish Business Data

Batch jobs and triggers need to be defined within the MC environment first, and then be exported and deployed to run within the runtime environment.   

Therefore, there are two scenarios for publishing business data. One is to publish the trigger, and the other is to publish the job. If the job has the trigger, the corresponding trigger should be published alongside the job.

#### Publish trigger

1. Download trigger.

![BatchDownloadTrigger01](./image/batch/batch_trigger_export_01.PNG)

2. Upload trigger and publish
Upload the data to the specified business data path and publish the business data.  

#### Publish job (Including the corresponding trigger)

1. Download job.

![BatchDownloadJob02](./image/batch/batch_job_export_01.PNG)

2. Upload job and publish.
Upload the data to the specified business data path and publish the business data.  


## Batch Development Step

### Understand Functional Requirement

First, prior to the development, we need to clearly identify what type of system requirements we want to implement. For instance, this may involve importing policy issuing data into Excel and sending it to salesmen on a daily basis, and specifying the timing to trigger such a job.

### Develop Batch Code

#### Understand the overall architecture of Spring Batch

![image-springbatch-overview](./image/batch/image-springbatch-overview.png)

#### Understand the flow diagram of typical batch job steps 

![image-springbach-steps](./image/batch/image-springbach-steps.png)

#### Understand develop your code with reference to the sample

1. URL of the Sample

   You can ask the InsureMO team for the source code of the developed batch samples for reference.

2. Main Points for Development

   Make sure you have added the GIMO common dependency.

   - Add batch-client Dependency

     ```
     <dependency>
     			<groupId>com.ebao.vela</groupId>
     			<artifactId>vela-batch-client</artifactId>
     			<version>${vela.version}</version>
     		</dependency>
     ```


   - The main configuration class is PAFileUploadConfigApp.java

   - Annotation for Class

     ```
     @Configuration
     @EnableBatchProcessing
     public class PAFileUploadConfigApp {
     ..............
     }
     ```

     

   - Inject Reader Bean

     ```
     @Bean
         public PAFileUploadReader paFileUploadReader() {
             return new PAFileUploadReader();
         }
     ```

   - Inject Processor Bean

     ```
      @Bean
         public PAFileUploadProcessor paFileUploadProcessor() {
             return new PAFileUploadProcessor();
         }
     ```

   - Inject Writer Bean

     ```
     @Bean
         public PAFileUploadWriter paFileUploadWriter() {
           return new PAFileUploadWriter();
         }
     ```

   - Inject Step Bean

     - single thread mode

     ```
      @Autowired
         public StepBuilderFactory stepBuilderFactory;
     
         @Bean
         public Step paFileUploadStep() {
             return stepBuilderFactory.get("paFileUploadStep")
                     .<BigInteger, PAUploadFileEntity>chunk(1000)
                     .reader(paFileUploadReader())
                     .processor(paFileUploadProcessor())
                     .writer(paFileUploadWriter())
                     .build();
         }
     ```

     - multi-threaded mode 

       ```
       @Autowired
           public StepBuilderFactory stepBuilderFactory;
           
        @Resource(name="batchTaskExecutor")
         ThreadPoolTaskExecutor batchTaskExecutor;
       
           @Bean
           public Step paFileUploadStep() {
               return stepBuilderFactory.get("paFileUploadStep")
                    .<BigInteger, PAUploadFileEntity>chunk(1000)
                       .reader(paFileUploadReader())
                    .processor(paFileUploadProcesser())
                       .writer(paFileUploadWriter())
                       .taskExecutor(batchTaskExecutor)
                       .build();
           }
       ```

       

   - Inject Job Bean

     ```
     @Autowired
         public JobBuilderFactory jobBuilderFactory;
     
         @Bean
         public Job paFileUploadJob(@Qualifier("paFileUploadStep") Step paFileUploadStep) {
             return jobBuilderFactory.get("paFileUploadJob")
                     .start(paFileUploadStep)
                     .build();
         }
     ```

     

   - Multi-threaded Mode for Batch Step

     By default, the springbatch processes framework uses a single thread to execute tasks;

     The springbatch provides support for thread pools with the chunk mode being the main method to implement multi-threading. Using the taskexecutor attribute of tasklet, you can easily convert ordinary steps. When converting to a multi-threaded step, thread safety needs to be considered in multi-threaded mode. The throttlelimit specifies the maximum number of thread pools to be used.

     To enable multi-threading in batch, the user needs to inject taskexecutor in batchstep. This enables multi-threading and ensures the thread safety of code running.

     There are two types of injected taskexecutors:

     Mode 1: Spring Native

     ```
     @Autowired
     
     TaskExecutor taskExecutor;
     ```

     Mode 2: Platform Overloaded

     ```
     @Resource(name="batchTaskExecutor")
     
     TaskExecutor taskExecutor;
     ```

     For mode 1, spring native is used. Due to the lack of some customized initialization of the platform, there may be some defects in use, such as missing logs, incomplete links, and potential failure in remote calls due to the lack of necessary context information;

     In mode 2, the platform is overloaded. The platform initializes it and sets the necessary context information to address the shortcomings present in mode 1;

     Therefore, the platform recommends that users use mode 2 to enable multi-threading.

   - In the file *spring.factories*, specify the main configuration class as follows:

     ```
     org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
     com.ebao.gimo.integration.batch.sample.pafileupload.PAFileUploadConfigApp
     ```

   - Two configuration parameters in *application.properties*  

     ```
     spring.batch.job.enabled=false
     batch.client.enabled=true
     ```

### Configure the batch and its triggers 

Follow the above guidelines to configure the batch and its triggers in the MC environment.

### Deploy the batch and its triggers

After configuration, deploy the batch and its triggers to the runtime environment, then run the batch and monitor its status.