# File Storage and Default S3 Bucket

## Basic Concept

### Standard Bucket

Our standard approach at insureMO for document storage during tenant setup is as follows:

* Each tenant is provided with a standard tenant bucket. For example, for the UAT tenant, the bucket format will be `tenant-sg-uat-bucket`. Each tenant bucket will have a unique physical endpoint address. 

* The standard tenant bucket functions when the BFF (Backend for Frontend) calls the standard platform API. For example, the BFF may call the DMS API for document management or when the project team needs to deploy configuration data and temporarily store the configuration file.

* If your project team requires detailed bucket information, please ask the insureMO SiteOps team for help.

### Storage Method

The project team has three options for storing content in the BFF:

* For technical purposes, call the Container CloudDisk API. This allows you to configure the bucket and store documents in any location. It is more suitable for technical scenarios such as file integration. For guidance, see [CloudDisk-Service](https://docs.insuremo.com/common/doc_00000316).

* For business purposes, utilize the ICS DMS API (not the filesupport JAR mentioned in this document). This API facilitates saving files to the tenant’s standard bucket. It is commonly used for business documents, such as policy document, as it offers business ID indexing. 

* For SAAS purposes, adopt the insureMO iDocs offering, which provides the entire UI journey for document management.

| Characteristics | DMS                                                                 | CloudDisk                                                                            | iDocs                                                                            |
|---------------|---------------------------------------------------------------------|--------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|
| Position      | PAAS                                                                | PAAS                                                                                 | SAAS                                                                                |
| Support Team  | Appframework                                                        | Container                                                                            | iDocs                                                                            |
| Purpose       | Business doc management                                             | Technical file management                                                            | Business doc management                                                             |
| With UI       | No                                                                  | No                                                                                   | Yes                                                                                 |
| Subscription  | insureMO                                                            | insureMO                                                                             | Separate                                                                            |
| Reference     | [Document Management](https://docs.insuremo.com/common/doc_00000018) | [CloudDisk-Service](https://docs.insuremo.com/common/doc_00000319) | [iDocs Introduction](https://docs.insuremo.com/common/doc_00000348) |


## Bucket Naming Rule

For each environment, a common bucket will be set up for each tenant. The bucket format is `{tenant code}-{env bucket prefix}bucket`, e.g., `TenantCode-vela-smk-bucket`, `freshman-ptdev-bucket`.  

Within each bucket, separate folders are created underneath, typically categorized as follows:


|            Folder Name            | Level  | Creator | Usage                      | Sample |
| ------------------------------- | ---- | -----  | ------------------------- |--------------- |
| `{tenant code}-{env bucket prefix}bucket-template`   | tenant |   TS    | Batch, product, plan, pp  |`TenantCode-vela-smk-bucket-template`|
| `{tenant code}-{env bucket prefix}bucket-instance`   | tenant |   TS    | Ratetable upload, upload  |`TenantCode-vela-smk-bucket-instance`|
| `{tenant code}-{env bucket prefix}bucket-dms-thumb`   | tenant |   TS    |  DMS thumbnail images              |`TenantCode-vela-smk-bucket-dms-thumb`|
| `{tenant code}-{env bucket prefix}bucket-dms-document` | tenant |   TS    |  DMS document storage                 |`TenantCode-vela-smk-bucket-dms-document`|
| `{tenant code}-{env bucket prefix}bucket-platform`   | tenant |   TS    | Data I/O,  lib product copy |`TenantCode-vela-smk-bucket-platform`|


## Share Bucket

If the project team has specific usage requirements and wants to share the bucket with the client's IT team, for instance, exposing the bucket to the client's team to fetch data like batch printing PDFs, we recommend setting up a separate bucket. This approach ensures:

* A clean new bucket, free from any temporary files in the platform bucket.
* Easy control over both read-only and editable authorities.

Depending on the volume of business transactions and security considerations, buckets can be set up under either a client’s account or an insureMO account.

It is preferable to set buckets up under a client’s account to reduce S3 costs (storage and transmission fees) and address security concerns. However, if data security and regulatory compliance are the primary concerns, it is recommended that the bucket be set up under an insureMO account. 


## Environment Parameter (for Private Cloud)

### Global

|         Parameter Name                  |      Usage      |       Sample Value                  | Remark |
| ------------------------------------- | ------------ | --------------------------------- | ---- |
|           `file.bucket.prefix`            |   Append bucket   | vela-dev-, vela-smk-              | Necessary |
|        `file.supportType.default`         |    Storage type    | S3, Azure, Aliyun, Sftp, SystemFile, Google, Custom | Necessary |
| `platform.foundation.file.type.whiteList` | File extension whitelist | .csv, .xls, .xlsx    | Necessary |


### For S3

|   Parameter Name          | Usage |   Sample Value    |          Remark           |
| ------------------------------ | -- | ------------ | --------------------- |
| `{tenant}.file.s3client.accessKey` | Connection |      -         |        Necessary for tenants         |
| `{tenant}.file.s3client.secretKey` | Connection |      -         |        Necessary for tenants        |
|  `{tenant}.file.s3client.region`   | Connection | us-east-1     |        Necessary for tenants         |
| `{tenant}.file.s3client.endpoint`  | Connection |    -           | Unnecessary（default as an AWS server） |

### For Azure

|                Parameter Name         | Usage |              Default Value              |           Remark            |
| ----------------------------------- | ---- | ---------------------------------- | ------------------------- |
| `{tenant}.file.azureClient.accountName` | Connection |                        -          |          Necessary for tenants          |
| `{tenant}.file.azureClient.accountKey`  | Connection |                    -              |          Necessary for tenants          |
| `{tenant}.file.azureClient.endpoint`       | Connection | `https://%s.blob.core.windows.net` | Not Necessary（Default Azure server） |

### For Aliyun

|              Parameter Name       | Usage |               Sample               |   Remark   |
|  -------------------------------  | ----- | ---------------------------------- | ------ |
| `{tenant}.file.ossClient.accessKey` | Connection |          -                            | Necessary for tenants |
| `{tenant}.file.ossClient.secretKey` | Connection |         -                             | Necessary for tenants |
| `{tenant}.file.ossClient.endpoint`  | Connection | `https://oss***aliyuncs.com` | Necessary for tenants |
| `{tenant}.file.ossClient.internalEndpoint`  | Upload connection (save/copy) | `https://oss***aliyuncs.com` (internal) | Optional; if configured, upload operations use this endpoint. Falls back to `endpoint` if not set |
| `{tenant}.file.ossClient.externalEndpoint`  | Download connection (get/share) | `https://oss***aliyuncs.com` (external) | Optional; if configured, download and share operations use this endpoint. Falls back to `endpoint` if not set |

> **Internal/External Endpoint Separation**: For Aliyun OSS, upload operations (save, copy) and download operations (get, share) can use different endpoints. This is useful when the application server is inside Aliyun VPC and can access OSS via the internal endpoint (faster, no public network cost), while clients downloading files need the external endpoint. If `internalEndpoint` or `externalEndpoint` is not configured, the default `endpoint` is used for all operations, maintaining backward compatibility.

### For SFTP

|           Parameter Name    | Usage | Default Value |   Remark   |
| -------------------------- | ---- | ---- | ------ |
|   `{tenant}.file.sftp.host`   | Connection |  -      | Necessary for Tenant |
|   `{tenant}.file.sftp.port`   | Connection |   22   | Necessary for Tenant |
| `{tenant}.file.sftp.username` | Connection |   -     | Necessary for Tenant |
| `{tenant}.file.sftp.password` | Connection |    -    | Necessary for Tenant |
| {tenant}.sftp.mount.path | Storge |  | The custom root folder path |

### For Google

| Parameter Name                          | Usage      | Default Value | Remark               |
| --------------------------------------- | ---------- | ------------- | -------------------- |
| {tenant}.file.googleClient.clientId     | Connection |               | Necessary for Tenant |
| {tenant}.file.googleClient.clientEmail  | Connection |               | Necessary for Tenant |
| {tenant}.file.googleClient.privateKey   | Connection |               | Necessary for Tenant |
| {tenant}.file.googleClient.privateKeyId | Connection |               | Necessary for Tenant |

### For Custom

Custom storage, You can store files and operate it in customized storage media through API

To enable this function, here is the steps

1. by passing field `customAppName` in `AbstractFileRequest`
2. implement the interface `CustomFileSupportRest`

**note: It is recommended to use this function through  DMS **

**note: `CustomFileSupportRest` also supports large file operations (multipart upload, range download). See the [Large File Upload and Download](#large-file-upload-and-download) section for the full endpoint list.**

#### POM Dependency

```xml
 			<dependency>
                <groupId>com.insuremo.vela</groupId>
                <artifactId>vela-file-support</artifactId>
                <version>${project.version}</version>
            </dependency>
```



#### Method Summary

| Modifier And Type     | Method                                | Return             | Description                                                  |
| --------------------- | ------------------------------------- | ------------------ | ------------------------------------------------------------ |
| public FileSaveResult | save(FileSaveRequest request)         | FileSaveResult     | adds an object                                               |
| public byte[]         | get(FileGetRequest request)           | byte[] of object   | retrieves an object                                          |
| public void           | delete(FileDeleteRequest request)     |                    | remove  an object from a bucket                              |
| public Object         | listVersion(FileGetRequest request)   | List ObjectVersion | get all versions of the object in a bucket                   |
| public Long           | getObjectSize(FileGetRequest request) | the object size    | get the object size from an object without returning the object itself |
| public void           | copy(FileCopyRequest request)         |                    | creates a copy of an object that is already stored in a bucket |
| public MultipartInitResult | initiateMultipart(FileMultipartInitRequest request) | MultipartInitResult | initiates a multipart upload session and returns storeKey + uploadId |
| public PartETag       | uploadPart(String storeKey, String uploadId, int partNumber, InputStream partStream, long partSize) | PartETag | uploads a single part of a multipart upload |
| public FileSaveResult | completeMultipart(String storeKey, String uploadId, List\<PartETag\> parts) | FileSaveResult | completes a multipart upload by merging all parts into the final object |
| public void           | abortMultipart(String storeKey, String uploadId) |                    | aborts a multipart upload and discards uploaded parts |
| public List\<PartStatus\> | listParts(String storeKey, String uploadId) | List of PartStatus | lists parts that have been uploaded for a multipart session |
| public InputStream    | getRange(FileGetRequest request)      | InputStream        | retrieves a byte range of an object (for partial download / online preview) |
| public FileSaveResult | uploadLargeFile(FileSaveRequest request, InputStream largeStream) | FileSaveResult | one-call large file upload; auto-splits into parts internally |

##### FileSaveRequest 

Used in save() method

| Field     | Description           | Required |
| --------- | --------------------- | -------- |
| objectKey | The key of the object | Y        |
| bytes[]   | bytes of  the object  | Y        |

##### FileSaveResult

Return of save() method

| Field     | Description            | Type | Required |
| --------- | ---------------------- | ---- | -------- |
| versionId | version id of anobject | Long | N        |

##### FileGetRequest

Used in get(), listVersion(), getObjectSize() method

| Field     | Description             | Required |
| --------- | ----------------------- | -------- |
| objectKey | The key of the object   | Y        |
| versionId | version id of an object | N        |

##### FileDeleteRequest

Used in delete() method

| Field     | Description           | Required |
| --------- | --------------------- | -------- |
| objectKey | The key of the object | Y        |

##### FileCopyRequest

Use in copy() method

| Field                | Description                       | Required |
| -------------------- | --------------------------------- | -------- |
| sourceObjectKey      | The key of the object to copy     | Y        |
| destinationObjectKey | The key of the destination object | Y        |

##### ObjectVersion

Return of listVersion() method

| Field        | Description                                                  | Type    | Requried |
| ------------ | ------------------------------------------------------------ | ------- | -------- |
| key          | the key of the object                                        | String  | N        |
| eTag         | The entity tag is an MD5 hash of that version of the object. | String  | N        |
| size         | Size in bytes of the object.                                 | Long    | N        |
| versionId    | version id of an object                                      | String  | N        |
| isLatest     | Specifies whether the object is (true) or is not (false) the latest version of an object. | Boolean | N        |
| lastModified | Date and time when the object was last modified.             | Instant | N        |


##### FileMultipartInitRequest

Used in initiateMultipart() method

| Field        | Description                          | Type                | Required |
| ------------ | ------------------------------------ | ------------------- | -------- |
| bucketName   | Bucket name prefix                   | String              | Y        |
| targetDir    | Target directory (auto-generated if empty) | String         | N        |
| category     | Business category                    | String              | Y        |
| fileName     | File name with extension             | String              | N        |
| module       | Module name                          | String              | N        |
| userMetaData | User-defined metadata                | Map\<String,String\> | N        |
| uniqueId     | Pre-defined storeKey (auto-generated if empty) | String       | N        |
| customAppName| Custom app name (for Custom storage mode) | String          | N        |

##### MultipartInitResult

Return of initiateMultipart() method

| Field      | Description                                              | Type   |
| ---------- | -------------------------------------------------------- | ------ |
| storeKey   | The unique object key (uniqueId), used in subsequent calls | String |
| uploadId   | The upload session ID from the storage provider           | String |
| bucketName | The resolved bucket name prefix                           | String |

##### PartETag

Return of uploadPart() method. Also used as input in completeMultipart().

| Field       | Description                                    | Type   |
| ----------- | ---------------------------------------------- | ------ |
| partNumber  | The part number (1-based)                      | int    |
| eTag        | The ETag returned by the storage provider      | String |

##### PartStatus

Return item of listParts() method

| Field       | Description                                    | Type   |
| ----------- | ---------------------------------------------- | ------ |
| partNumber  | The part number                                | int    |
| size        | Size of the part in bytes                      | long   |
| eTag        | The ETag of the part                           | String |


## Large File Upload and Download

For large files (e.g. videos, large datasets), the standard `save` / `get` methods may cause memory pressure or timeout. Two layers of API are provided:

### Two-Layer API Design

| Layer | Audience | Methods | Use case |
|---|---|---|---|
| **Low-level multipart primitives** | Clients needing control (e.g. weak-network upload with progress bar, resumable upload) | `initiateMultipart` → `uploadPart` (loop) → `completeMultipart` / `abortMultipart`; `listParts` for resume; `getRange` for partial download | Claim site survey video upload, online preview |
| **High-level one-call** | Projects wanting simplicity (server-to-server, stable network) | `uploadLargeFile(request, stream)` — auto-splits, uploads parts, merges; aborts on failure | Batch data sync, backend processing |

### Multipart Upload Flow

```
Client                          file-support                     Storage (S3/OSS/Azure/GCS)
  │
  │ 1. initiateMultipart ──────▶ │ createMultipartUpload ───────▶ │
  │ ◀── storeKey, uploadId ───── │ ◀── uploadId ───────────────── │
  │
  │ 2. uploadPart(1) ───────────▶ │ uploadPart ──────────────────▶ │
  │    uploadPart(2) ───────────▶ │ uploadPart ──────────────────▶ │
  │    ...                         │
  │
  │ 3. completeMultipart ───────▶ │ completeMultipartUpload ─────▶ │
  │ ◀── FileSaveResult ────────── │ ◀── success ────────────────── │
  │
  │ (on failure) abortMultipart ▶ │ abortMultipartUpload ────────▶ │ cleanup
```

### Range Download

Pass `rangeStart` and `rangeEnd` in `FileGetRequest` to retrieve a byte range. The result is returned as a streaming `InputStream` (memory usage is O(1), independent of file size). If both are null, the entire object is read (equivalent to `get`).

### Per-Adapter Notes

| Storage | Multipart model | Notes |
|---|---|---|
| **S3 / OSS** | Native multipart (uploadId) | Full support including listParts and resumable upload |
| **Azure** | BlockBlob (blockId) | initiateMultipart returns a placeholder uploadId; uploadPart generates base64 blockId as pseudo-ETag; completeMultipart calls commitBlockList |
| **GCS** | Resumable upload (WriteChannel) | **Multi-instance limitation**: WriteChannel is cached in instance memory; uploadPart and completeMultipart must route to the same instance that called initiateMultipart (sticky session required). listParts is not supported (returns empty). Use resumable session URL for stateless GCS if needed |
| **Custom** | REST proxy | Forwards to `/public/file/v1/multipart/*` endpoints on the host service. uploadPart sends file part via multipart/form-data |

### Custom REST Endpoints (Large File)

The `CustomFileSupportRest` interface includes the following endpoints in addition to the existing save/get/delete/copy endpoints:

| Endpoint | Method | Description |
|---|---|---|
| `POST /public/file/v1/multipart/init` | `initiateMultipart` | Initiate multipart upload session |
| `POST /public/file/v1/multipart/uploadPart` | `uploadPart` | Upload a single part (multipart/form-data) |
| `POST /public/file/v1/multipart/complete` | `completeMultipart` | Complete multipart upload |
| `POST /public/file/v1/multipart/abort` | `abortMultipart` | Abort multipart upload |
| `POST /public/file/v1/multipart/listParts` | `listParts` | List uploaded parts |
| `POST /public/file/v1/multipart/uploadLargeFile` | `uploadLargeFile` | One-call large file upload (multipart/form-data) |
| `POST /public/file/v1/getRange` | `getRange` | Range download (streaming Resource) |

### Memory Safety

- **S3/OSS `uploadPart`**: Uses `RequestBody.fromInputStream` — data streams directly from InputStream to the network, only a small buffer (~8KB) in memory.
- **`getRange` (S3/OSS)**: Returns `ResponseInputStream` — streaming, no full-file read into memory.
- **`uploadLargeFile`**: Allocates one `byte[partSize]` buffer (default 8MB), reused across parts. Memory = O(1) regardless of total file size.
- **Azure `getRange`**: Uses `ByteArrayOutputStream` — memory depends on the Range size requested. Large ranges may cause memory pressure.
- **Custom `uploadPart`**: Reads the part into `byte[]` for REST forwarding — memory = one part size per call.