# Upload

## Basic Concept

Business operation often require the bulk upload a large amount of data, especially risk information or schedule information in group policies.

The upload framework is designed to cater for such a case by parsing microsoft excel data to database. It supports different excel formats like `xls`, `xlsx` and `csv` with one standard API only. You can control how the framework parses by configuring upload structure (struct) and node.


## User Scenario

For anybody who wants to use our platform to upload a large amount of data.


## Upload Framework Composition

### 1. Excel Template

When gathering end requirement, usually it comes to a physical excel template which a business user wants to operate with. 

Usually, such a business-oriented template will contain things like date format, dropdown list, help message, business guideline and even VBA validation. Although it is not related to key policy data capture, it will greatly reduce the mistakes business users might make.

Now our upload framework supports maintenance of such excel template and deploys the template into runtime environment.

### 2. Struct and Node

Struct and Node are two major categories of upload framework program:

* Struct represents abstract template of excel parser. It defines the upload parser's behavior and contains nodes. One struct usually means one physical excel template.

* Node represents microsoft excel file and data struct, which helps parser to understand the microsoft excel file. One node usually means one physical column in an excel template.

Such struct/node configuration can be viewed as a mapping layer between an excel and a policy object. Such a mapping can be:

* Dynamic mapping based on DD configuration - Node name strictly follows data dictionary definition and can be dynamically extended according to data dictionary configuration.

* Coding mapping based on BFF program - Node name is physically mapped in the upstream BFF invoke source code and node change must come along with code change.


### 3. Workflow

![upload workflow](./image/upload/upload_workflow.png)

### 4. Restriction

   * Upload framework supports microsoft excel with suffix xls, xlsx, csv only. For compressed file, zip only.
   * Upload framework does not support file which is too big.
   * These rows above the row represents file header will be ignored.


## UI Operation Guide

### 1. Struct

![upload struct](./image/upload/upload_struct.png)

- **Name**: struct name, required
- **Description**: struct description
- **Header Index**: define which row represents header
- **Batch Name**: define which batch job will be triggered after parsing
- **Rule Event Code**: If setting event, the event mapped rules will be fired when you upload data.

### 2. Node

![upload node](./image/upload/upload_node.png)

- Node is a part of a cascaded tree which represents data JSON.
- When struct is created, a node with the same name as that of the struct will be created too. It is the father of all customized node.
- Node Fields:
    + **Name**: node name, required, representing JSON key as well
    + **Whether Child Node is Array**: define if the children node of this node will be parsed as JSON array rather than JSON object
    + **Header**: define which column in header row the node represents 
    + **Index**: The column SEQ number of the node. It is not recommended to set, unless you need strict validation.
    + **Is Unique Index Node**: Set the node as Primary Key. The primary key can be set for multiple nodes.
    + **Nullable**: define if a node is nullable and it will be validated.
    + **Standalone**: not used yet  
    + **Default Value**: The value will be used if the data in excel is null.
    + **Code Table**: where the options to pick up from the code tables
    + **Data Type**: data type
    + **Date String Format**: Date Format if the data type is set to be Date
    + **Max Length**: Max length of data

- Function
    + Click the node in left side, then use the function buttons;
    + Click **Edit** to edit fields;
    + Click **Add Child Nodes** to add child nodes for this node;
    + Click **Delete** to delete a node, and it is cascaded.

### 3. Template

  System supports to upload/download the template `.csv` and `.xlsx` files. You can directly upload/download the template on UI.
  
  Only after the template is uploaded can you download the template.

  ![template upload download](./image/upload/template_upload.png)


## Quick Start

###   Configure Upload Template

After you understand all the above concepts, you can configure your uploaded template with your business. The menu path is `Public Setting > Upload Template`.

We can take a group policy schedule upload batch - (template **RLOCKGroupPolicyScheduleNB**) as an example.  


###   Call Upload File RestAPI   

Please see the following image. You can invoke this RestAPI at your UI directly, or you can wrapper additional logic at Backend for Frontend service, e.g. check logic.   

`/platform-pub/v1/file`

Once the upload API is invoked, upload framework will start to process the excel line by line and transform it into your configured target format and save it into database.


![uploadRestAPI](./image/upload/uploadfileRestAPI.png)


###   Callback Business Batch   

Once all lines in excel have been processed, upload framework will callback and start your business batch. It will pass the batch parameter "taskId" which you need to capture in your batch implementation.

Then in your batch implementation, with taskId, you can either:

* Call API to get successful records to further process:

`/upload/content/v1/slice/faster/perfect`

* Call API to get failure records to further process:
 
`/upload/content/v1/errors/task`