# Index Management
## Basic Concept
Search Index is a module that offers users the capabilities to perform data related actions such as adding, updating, deleting, and querying data on top of our InsureMO search engine. By use of search engine, not only DB pressure can be greatly reduced, but also the user will be able to perform all kinds of query method which cannot be well supported via DB script.
Now most platform domains now feature some search API that interacts with the search engine. Also, with the help of index management UI, the user will be able to dynamically expand index fields for more search criteria.
This document will provide comprehensive introduction as to how to search data via search engine and how to maintain the data in the search engine.
## User Scenario
This feature is designed for anyone who wishes to use our platform to expand or customize search criteria.
## Composition of Index Management
Index Management primarily consists of two main components:
* ***Index*** represents a search domain itself, such as policy, claim, or sales channel.
* ***Index Field*** refers to individual search fields within the index.
It differs from the underlying platform's ***Search*** microservice, which maintains the index and implements index-based search functionality.
On the backend, we utilize ***Elastic Search (ES) Engine***, the industry's most widely used search engine.
### Index Management
Upon tenant creation, a series of pre-embedded indexes and APIs become available on the platform for domain-based business queries. These include:
* Policy
* Endorsement
* Claim
* Accounts Receivables/Payable (AR/AP)
* Collection
* Payment
* Sales Channel
* Plan
If the provided fields are insufficient, you can dynamically extend index fields using a data dictionary-based index field configuration.
For special cases where new field data needs to be indexed and searched, we can assist in creating a new index. For more information, contact the platform team.
For large amounts of configuration data, we also offer the **ESCodeTable** mechanism, which functions as a dynamic index allowing users to write and retrieve data.
### Performance Considerations
From a business perspective, ES is ideal for transactions involving large datasets and fuzzy search requirements. However, we do not recommend using ES for all scenarios, particularly for low-frequency transactions such as commission or agent statements, which are only generated monthly. In these cases, DB search is a more suitable approach.
To ensure performance, each index is query-limited at the platform level to return only the maximum number of records, enhancing search safety and stability.
The Search API is currently limited to returning a maximum of 1,000 records. Here's how it works:
If there are 134 records:
1. Searching with **PageNo=1 and PageSize=100** returns the first 100 records.
2. Searching with **PageNo=2 and PageSize=100** returns the next 34 records.
3. Searching with **PageNo=3 and PageSize=100** returns no records.
If there are 1340 records:
1. Searching with **PageNo=1 and PageSize=500** returns the first 500 records.
2. Searching with **PageNo=2 and PageSize=500** returns the second 500 records.
3. Searching with **PageNo=3 and PageSize=500** exceeds 1,000-record limit and prompts an error message **Current search reaches max result count: 1000**,**message**:**Current search reaches max result count: 1000**.
This 1,000-record limit is sufficient for UI display and selection, as users cannot browse through such a large number of records and must specify clearer criteria for operations.
In cases where users need to generate reports with more than 1,000 records using search engine-powered APIs, iteration logic must be implemented to process records in batches. For example, to search for the latest 340 policy records, users can use policy ID as a search criterion, sort by policy ID first. Then, add a criterion requiring the policy ID to be greater than the maximum policy ID from the first search result in subsequent searches.
Please note that similar limitations are imposed in our Rainbow UI framework. Users will not be able see records more than 1,000. For instance, if there are 1,020 records and 30 rows per page, users will only see a table with a maximum of 33 pages.
## UI Operation Guide
caution
All operations outlined in this guide must be executed within the Master Configuration (MC) portal environment. They are prohibited in other environments, except for troubleshooting purposes.
### Maintaining Index Field
In most scenarios, you'll only interact with the platform's default indexes and extended index fields. Once you've located the index and click **Edit**, you will encounter the following interface.
Field Overview:
* **Field Name**: The name of the index field.
* **Is Overridden**: Automatically set by the system. If index field has already been set by the platform, then when the tenant adds the same field, it will be set as **Yes**.
* **Type**: The index field data type.
* **Add to_all**: Select **Y** if this field needs to be added to a common field (the common field is used to search by one common field).
* **DD Model Name**: Map to the data dictionary, or leave blank if not required.
* **DD Object Code**: Map to the data dictionary, or keep blank if not required.
* **DD Field Name**: Map to the data dictionary, or keep blank if not required.
* **Only Exact Search**: If **Yes**, then the field only supports exact search. If the field is queried by fuzzy conditions, no result will be returned.
* **Code Table**: If **Yes**, the platform's PAAS Query will display it as as a dropdown list.
* **Need Display**: If **Yes**, the platform's PAAS Query will display it as an automatic display column.
* **Description**: If set to a non-null value, the platform's PAAS Query will display it as a UI label.
* **Custom Script**: If the field can not be gotten by *Model + Field*, then you can try to code a script here.
* **Operation**: Delete the field (only used when adding an index or a new field).
tip
For **Custom Script**, there's a requirement to set the PolicyHolderIdNo as the PolicyCustomer with "IsPolicyHolder" set to 'Y'.
Here's the script:
```groovy
def policyHolders = binding.model.lookupChildObjectListCascadeByExpression("PolicyCustomer", "IsPolicyHolder == 'Y'")
def policyHolderIdNo = null
if (policyHolders.size > 0) {
policyHolderIdNo = policyHolders[0].lookupVariableDirectly("IdNo")
}
return policyHolderIdNo
```
### Adding A New Index Schema and Index
caution
Before adding a new index, please contact the InsureMO operations team for guidance, as it requires advanced technical skills.
An index schema is the same as tables in a Database (DB), and only related indexes can be created in the same schema. Typically, a new index schema is necessary before adding a new index.
If you need a new index schema, create it first.
To add a new index, only the *Add Index* step is needed. For rebuilding an index, other steps are needed.
Before selecting an index schema, you must fill in the index name, model name (sourced from the data dictionary or use arbitrary numbers if not needed), and main analyzer (select **edge** for now, and it will be removed in the future).
The index data provider, index rebuilding provider, and index rebuilding job are specific to Policy indexes; they are not required for new indexes.
The rebuild status field does not need to be filled in.
Select **N** for the Sign field (this option will be removed in the future).
### Rebuild Index Data
The index data of Policy/Proposal is generated from the index fields that exist at that time, so the index data cannot be updated after a new index field is added to Policy/Proposal.
There is only one way to update the index data of a Policy/Proposal, that is, to rebuild the procedure about the index batch, named *policyRebuildIndexBatch* or *policyAdminRebuildIndexBatch* for policies.(policyRebuildIndexBatch is renamed to policyAdminRebuildIndexBatch from Version V23.04, it can rebuild policy, endorsement, and group policy risk/schedule index.)
For how to rebuild the index data, take policy as an example:
1. The label of the batch job that triggers *policyRebuildIndexBatch*.
The index related batch as below,
* Policy: policyRebuildIndexBatch or policyAdminRebuildIndexBatch
* Endorsement: endorsementRebuildIndexBatch or policyAdminRebuildIndexBatch
* SalesChanel: salesChannelIndexRebuildJob
2. Monitor job status and view job logs.
tip
1. Confirm the name of the batch job. You cannot rebuild index data if the batch job is not defined.
1. Confirm job successfully by job time and job status, and confirm that the policy index data is renewed.
### Rebuild Index Structure (Deprecate)
If you need to change the field type or delete the field, you need to rebuild the index. Check through **Copy**.
As you can see, you have permission to change the existing fields.
When you have dealt with the changes, save them, and then check the migration.
Source index schema and source index name are the indexes you need to change before. Then the data will be copied to the new index. Finally, the indexes will be swapped.
This step only changes the index name to the new one and finishes rebuilding the index structure.
### Configuration Data Import and Export
Once you make all the changes, you will need to **Export Index Domain** first to export index definition data, then **Export Index Field** to export index mapped fields.
## Comparison Between Base Query and Advanced Query
### Comparison of searchRuntimeService
| Base Query | Advance Query |
| ------ | ------ |
| searchRuntimeService.queryBaseResult(searchCondition) | searchRuntimeService.advancedQuery(searchConditionRequest) |
### Comparison of Structure
| Base Query |
Advance Query |
Comments |
| Conditions |
conditions |
- |
| fuzzyConditions |
fuzzyConditions |
- |
| inConditions |
inConditions |
- |
| notInConditions |
notInConditions |
- |
| ExistFields |
ExistFieldsConditions |
- |
| NotExistFields |
NotExistFieldsConditions |
- |
| fromRangeConditions |
gteRangeConditions |
Greater than or equal. |
| toRangeConditions |
lteRangeCondition |
Less than or equal. |
| OrConditions |
OrSearchConditionsList |
OrSearchConditionsList in advanced Query can implement all "OR" situations in base Query. |
| OrConditionsList |
| OrFuzzyConditions |
| OrFuzzyConditionsList |
| OrInConditions |
| SortFieldAndTypeList |
SortFieldAndTypeList |
- |
| AggConditions |
AggConditions |
- |
| GroupField |
GroupField |
- |
| IncludeFields |
IncludeFields |
- |
|
GtRangeConditions |
Greater than. |
|
LtRangeConditions |
Less than. |
|
InFuzzyConditions |
- |
|
NotInFuzzyConditions |
- |
|
NotSearchConditionsList |
- |
## Backend Development & API Usage Guide
### Dependency
Please add this dependency to your source code project first:
```java
com.ebao.vela
vela-search-runtime
${project.version}
```
### Base Query Search Criteria Illustration
```java
searchRuntimeService.queryBaseResult(searchCondition)
```
RESTful API: `{domain}/search/public/query/v1/result` --the RESTful removed at 20.0.12
#### SearchCondition Structure
```java
public class SearchCondition
private Map conditions;
private Map fuzzyConditions;
private Map orConditions;
private List> orConditionsList;
private Map orFuzzyConditions;
private Map> inConditions;
private Map> notInConditions;
private Map fromRangeConditions;
private Map toRangeConditions;
private List> orFuzzyConditionsList;
private String module;
private int pageNo;
private int pageSize;
private String sortField;
private String sortType;
private List> sortFieldAndTypeList;
```
#### Conditions
Exact query, like the term query for ES. For example, query the policy from the policy Index where the policy number is "PCE201809223587".
##### JSON Example
```json
{
"Conditions": {
"PolicyNo": "PCE201809223587"
}
}
```
##### SQL Example
```sql
select * from Policy where PolicyNo="PCE201809223587";
```
##### Groovy Example
```Groovy
setConditions(["PolicyNo": "PCE201809223587"])
```
#### fuzzyConditions
Fuzzy query, like ES wildcard query. Please note that it can only be used for string type (text types in ES) and fuzzy search flag should be put as "Yes". For example, query the policies from the policy index where the policy number contains 'CE20' and the insured ID number is '233455210075'
##### JSON Example
```json
{
"Conditions": {
"InsuredIdNo": 233455210075
},
"FuzzyConditions": {
"PolicyNo": "CE20"
}
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and PolicyNo like "%CE20%"
```
##### Groovy Example
```groovy
setConditions(["InsuredIdNo": 233455210075])
setFuzzyConditions(["PolicyNo": "CE20"])
```
On pre-fuzzing, since we use an edge analyzer, it does not lose much query performance, but requires more space to store.
#### orConditions
Or query, like bool should be in ES query. Please note that these orConditions use internally or externally. For example, query the policies from the policy index where the policy number contains 'CE20' and the insured ID number is '233455210075' while also policy types needs to be 1 or print status needs to be 2.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"FuzzyConditions": {
"PolicyNo": "CE20"
},
"OrConditions": {
"PolicyType": 1,
"PrintStatus": 2
}
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and PolicyNo like "%CE20%" and (PolicyType=1 or PrintStatus=2)
```
##### Groovy Example
```groovy
setConditions(["InsuredId": 233455210075])
setFuzzyConditions(["PolicyNo": "CE20"])
setOrConditions(["PolicyType": 1,"PrintStatus": 2])
```
#### orConditionsList
Or query (support field merge or query), like bool should be in ES query. It is like List < orConditions > which supports multiple orConditions working together.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"OrConditionsList": [
{
"PolicyType": 1,
"PrintStatus": 2
},
{
"PolicyType": 2,
"ProjectName": "Wildlife"
},
]
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and (PolicyType=1 or PrintStatus=2) and (PolicyType=1 or ProjectName="Wildlife")
```
##### Groovy Example
```groovy
setConditions(["InsuredId": 233455210075])
setOrConditionsList([
[
"PolicyType": 1,
"PrintStatus": 2
],
[
"PolicyType": 2,
"ProjectName": "Wildlife"
]
])
```
#### orFuzzyConditions
Or fuzzy query, like bool should be an ES wildcard query.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"orFuzzyConditions": {
"ProductName": "dbc",
"ProjectAddress": "AddressSample2"
}
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and (ProductName like "%dbc%" or ProjectAddress like "%AddressSample2%")
```
##### Groovy Example
```groovy
setConditions({
"InsuredId": 233455210075
})
setOrFuzzyConditions({
"ProductName": "dbc",
"ProjectAddress": "AddressSample2"
})
```
#### inConditions
In query, like ES query is a key-value mapping, and the values need to be a list. If there are multiple conditions defined, it will be treated as "and" operation.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"InConditions": {
"ProductName": ["dbc","CSQ"],
"ProjectAddress": ["AddressSample1","AddressSample2","AddressSample3"]
}
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and ProductName in ("dbc","CSQ") and ProjectAddress in ("AddressSample1","AddressSample2","AddressSample3")
```
##### Groovy Example
```groovy
setConditions([
"InsuredId": 233455210075
])
setInConditions([
"ProductName": ["dbc", "CSQ"],
"ProjectAddress": ["AddressSample1", "AddressSample2", "AddressSample3"]
])
```
#### orInConditions
The difference is linked by or than inConditions. If there are multiple conditions defined, it will be treated as "or" operation.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"OrInConditions": {
"ProductName": ["dbc","CSQ"],
"ProjectAddress": ["AddressSample1", "AddressSample2", "AddressSample3"]
}
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and (ProductName in ("dbc","CSQ") or ProjectAddress in ("AddressSample1", "AddressSample2", "AddressSample3"))
```
##### Groovy Example
```groovy
setConditions({
"InsuredId": 233455210075
})
setOrInConditions({
"ProductName": ["dbc", "CSQ"],
"ProjectAddress": ["AddressSample1", "AddressSample2", "AddressSample3"]
})
```
#### notInConditions
It is not in query, such as not in the ES query.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"NotInConditions": {
"ProductName": ["dbc","CSQ"],
"ProjectAddress": ["AddressSample1", "AddressSample2", "AddressSample3"]
}
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and ProductName not in ("dbc","CSQ") and ProjectAddress not in ("AddressSample1", "AddressSample2", "AddressSample3")
```
##### Groovy Example
```groovy
setConditions([
"InsuredId": 233455210075
])
setNotInConditions([
"ProductName": ["dbc", "CSQ"],
"ProjectAddress": ["AddressSample1", "AddressSample2", "AddressSample3"]
])
```
#### FromRangeConditions
It forms a range query, which represents greater than or equal to the value just like the range query of ES. If user want to form a range query for date or number, it’s better to adopt it together with ToRangeConditions.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"FromRangeConditions": {
"CommonPolicyStatus": 2,
"EffectiveDate": "2024-01-01T00:00:00"
}
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and CommonPolicyStatus >= 2 and EffectiveDate >= '2024-01-01T00:00:00'
```
##### Groovy Example
```
setConditions({
"InsuredId" = 233455210075
})
setFromRangeConditions({
"CommonPolicyStatus" = 2,
"EffectiveDate" = "2024-01-01T00:00:00"
})
```
#### ToRangeConditions
Range query, like ES range query, means less than or equal to the value. If user want to form a range query for date or number, it’s better to adopt it together with FromRangeConditions.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"ToRangeConditions": {
"CommonPolicyStatus": 2,
"EffectiveDate": "2023-12-31T23:59:59"
}
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and CommonPolicyStatus <= 2 and EffectiveDate <= '2024-12-31T23:59:59'
```
##### Groovy Example
```groovy
setConditions({
"InsuredId" = 233455210075
})
setToRangeConditions({
"CommonPolicyStatus" = 2,
"EffectiveDate" = "2023-12-31T23:59:59"
})
```
#### orFuzzyConditionsList
Or fuzzy query list, like bool should be an ES wildcard query. It is like List < orFuzzyConditions > which supports multiple FuzzyConditions working together.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"orFuzzyConditionsList": [
{
"ProductName": "dbc",
"ProjectAddress": "AddressSample2"
},
{
"ProductName": "ASQ",
"ProjectAddress": "AddressSample1"
}
]
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and (ProductName like "%dbc%" or ProjectAddress like "%AddressSample2%") and (ProductName like "%ASQ%" or ProjectAddress like "%AddressSample1%")
```
##### Groovy Example
```groovy
setConditions([
"InsuredId": 233455210075
])
setOrFuzzyConditionsList([
[
"ProductName": "dbc",
"ProjectAddress": "AddressSample2"
],
[
"ProductName": "ASQ",
"ProjectAddress": "AddressSample1"
]
])
```
#### existFields
It only searches for existing fields, like bool must exist for query in ES.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"ExistFields":["DuePremium","ProductName"]
}
```
##### SQL Example
```sql
select * from Policy where InsureId=-233455210075 and DuePremium is not null and ProductName is not null
```
##### Groovy Example
```groovy
setExistFields(["DuePremium", "ProductName"])
```
#### notExistFields
It only searches for fields that do not exist, like bool cannot exist ES query.
##### JSON Example
```Json
{
"Conditions": {
"InsuredId": 233455210075
},
"NotExistFields":["ProposalDate","ProposalNo"]
}
```
##### SQL Example
```sql
select * from Policy where InsuredId =-233455210075 and ProposalDate is null and ProposalNo is null
```
##### Groovy Example
```groovy
setNotExistFields(["ProposalDate", "ProposalNo"])
```
#### AggConditions
Aggregate queries can accept aggregation conditions to perform operations such as summation or averaging. The example below calculates the sum and average of DuePremium.
##### JSON Example 1
```Json
{
"AggConditions":{
"SumFields":["DuePremium"],
"AvgFields":["DuePremium"]
}
```
##### Postman Example 1
* url
POST /api/platform/proposal-v2/v1/query
* request body
```Json
{
"InConditions":{
},
"Conditions": {},
"FuzzyConditions": {},
"OrFuzzyConditions": {},
"FromRangeConditions": {"IssueDate": "2025-01-01T00:00:00"},
"ToRangeConditions": {"IssueDate": "2025-01-31T23:59:59"},
"PageNo": 1,
"PageSize": 10,
"SortField": "index_time",
"SortType": "desc",
"Module": "Policy",
"AggConditions":{
"AvgFields": [
"DuePremium"
],
"SumFields": [
"DuePremium"
]
}
}
```
* response body
```Json
{
"AggConditions": {
"AvgFields": [
"DuePremium"
],
"SumFields": [
"DuePremium"
]
},
"Aggs": {
"Avg_DuePremium": 31297.569047619047,
"Sum_DuePremium": 657248.95
},
"GroupField": "entity_type",
"PageNo": 1,
"PageSize": 10,
"Results": [
{
"EsDocs": [
],
"GroupTotalNum": 21,
"GroupValue": "Policy"
}
],
"Total": 21
}
```
The above query can be converted into the following SQL statement:
```sql
select sum(DuePremium), avg(DuePremium) from Policy where IssueDate >= '2025-01-01T00:00:00' and IssueDate <= '2025-01-31T23:59:59';
```
##### Groovy Example 1
```groovy
setAggConditions([
"SumFields": ["DuePremium"],
"AvgFields": ["DuePremium"]
])
```
We also support scenarios where grouping is performed before aggregation. The example below uses each product as the grouping condition and calculates the sum and average of DuePremium after grouping.
##### JSON Example 2
```Json
"AggConditions":{
"GroupFields":["ProductCode"],
"AvgFields": [
"DuePremium"
],
"SumFields": [
"DuePremium"
]
}
```
##### Postman Example 2
* url
POST /api/platform/proposal-v2/v1/query
* request body
```Json
{
"InConditions":{
},
"Conditions": {},
"FuzzyConditions": {},
"OrFuzzyConditions": {},
"FromRangeConditions": {"IssueDate": "2025-01-01T00:00:00"},
"ToRangeConditions": {"IssueDate": "2025-01-31T23:59:59"},
"PageNo": 1,
"PageSize": 10,
"SortField": "index_time",
"SortType": "desc",
"Module": "Policy",
"AggConditions":{
"GroupFields":["ProductCode"],
"AvgFields": [
"DuePremium"
],
"SumFields": [
"DuePremium"
]
}
}
```
* response body
```Json
{
"AggConditions": {
"AvgFields": [
"DuePremium"
],
"GroupFields": [
"ProductCode"
],
"SumFields": [
"DuePremium"
]
},
"Aggs": {
"GroupResult": [
{
"Avg_DuePremium": 5222.67,
"ProductCode": "RG001",
"Sum_DuePremium": 47004.03,
"docCount": 9
},
{
"Avg_DuePremium": 556.59,
"ProductCode": "TBTI",
"Sum_DuePremium": 3339.54,
"docCount": 6
},
{
"Avg_DuePremium": 18.19,
"ProductCode": "PLANTEST01",
"Sum_DuePremium": 36.38,
"docCount": 2
},
{
"Avg_DuePremium": 0,
"ProductCode": "CGHP",
"Sum_DuePremium": 0,
"docCount": 1
},
{
"Avg_DuePremium": 6750,
"ProductCode": "PAPP",
"Sum_DuePremium": 6750,
"docCount": 1
},
{
"Avg_DuePremium": 119,
"ProductCode": "PRPCON",
"Sum_DuePremium": 119,
"docCount": 1
},
{
"Avg_DuePremium": 600000,
"ProductCode": "RLOCK",
"Sum_DuePremium": 600000,
"docCount": 1
}
]
},
"GroupField": "entity_type",
"PageNo": 1,
"PageSize": 10,
"Results": [
{
"EsDocs": [
],
"GroupTotalNum": 21,
"GroupValue": "Policy"
}
],
"Total": 21
}
```
The above query can be converted into the following SQL statement:
```sql
select sum(DuePremium), avg(DuePremium) from Policy where IssueDate >= '2025-01-01T00:00:00' and IssueDate <= '2025-01-31T23:59:59' group by ProductCode;
```
##### JSON Example 3
```Json
{
"AggConditions":{
"SumFields":[
{
"Field": "SumActiveTime",
"Script": "(doc['ExpiryDate'].value.millis - doc['EffectiveDate'].value.millis)/86400000"
} //will return a new field named SumActiveTime
],
"AvgFields":[
{
"Field": "AvgActiveTime",
"OperationFields":["ExpiryDate","EffectiveDate"],
"Operation":"SUB"
} //will return a new field named AvgActiveTime, Operation support ADD(+), SUB(-), MUL(*), DIV(/)
],
"GroupFields":[
{
"Field":"IsRenewalPolicy",
"Missing":"undefined"
} //missing will return a undefined group when lose the field IsRenewalPolicy
]
}
}
```
##### SQL Example 3
```sql
select sum(ExpiryDate - EffectiveDate) as SumActiveTime, avg(ExpiryDate - EffectiveDate) as AvgActiveTime, IsRenewalPolicy, count(1) from Policy group by IsRenewalPolicy
```
##### Groovy Example 3
```groovy
setAggConditions([
"SumFields": [
[
"Field": "SumActiveTime",
"Script": "(doc['ExpiryDate'].value.millis - doc['EffectiveDate'].value.millis)/86400000"
]
],
"AvgFields": [
[
"Field": "AvgActiveTime",
"OperationFields": ["ExpiryDate", "EffectiveDate"],
"Operation": "SUB"
]
],
"GroupFields": [
[
"Field": "IsRenewalPolicy",
"Missing": "undefined"
]
]
])
```
#### Module
The definition of index name at Search UI, like the table name at DB.
#### includeFields
It only returns the fields including these.
#### pageNo
PageNo defaults to be 1 and cannot be more than 100.
#### pageSize
PageSize defaults to be 10 and pageNo * pageSize cannot be more than 1000.
#### sortField
This field is used to sort.
#### sortType
DESC or ASC.
#### sortFieldAndTypeList
Multi-sort fields are sequential.
##### JSON Example
```Json
{
"FuzzyConditions": {
"PolicyNo": "YEI",
},
"Module": "Policy",
"PageNo": 1,
"PageSize": 10,
"SortField": "index_time",
"SortType": "desc",
"SortFieldAndTypeList":[
{
"SortField": "BusinessType",
"SortType": "desc"
},
{
"SortField": "ExpiryDate",
"SortType": "desc"
}
]
}
```
##### SQL Example
```sql
select * from Policy where PolicyNo like "%YEI%" order by index_time desc, BusinessType desc, ExpiryDate desc
```
##### Groovy Example
```Groovy
searchCondition.setModule("Policy")
searchCondition.setPageNo(1)
searchCondition.setPageSize(10)
searchCondition.setSortField("index_time")
searchCondition.setSortType("desc")
List> sortFieldAndTypeList = []
sortFieldAndTypeList.add(["SortField": "BusinessType", "SortType": "desc"])
sortFieldAndTypeList.add(["SortField": "ExpiryDate", "SortType": "desc"])
searchCondition.setSortFieldAndTypeList(sortFieldAndTypeList)
Map fuzzyConditions = ["PolicyNo": "YEI"]
searchCondition.setFuzzyConditions(fuzzyConditions)
```
### Advance Query Search Criteria Illustration
```java
searchRuntimeService.advancedQuery(searchConditionRequest)
```
#### SearchConditionRequest structure:
```java
public class SearchConditionRequest
private QueryCondition queryCondition;
private String index;
private String module;
private int pageNo;
private int pageSize;
private String sortField;
private String sortType;
private LinkedHashMap sortFieldsAndTypes;
private String[] includeFields;
private String groupField;
private String groupMethod;
```
```java
public class QueryCondition
private Map conditions;
private Map fuzzyConditions;
private Map> inConditions;
private Map> inFuzzyConditions;
private Map> notInConditions;
private Map> notInFuzzyConditions;
private Map gtRangeConditions; // Greater than
private Map ltRangeConditions; // Less than
private Map gteRangeConditions; // Greater than or equal
private Map lteRangeConditions; // Less than or equal
private List existFieldsConditions;
private List notExistFieldsConditions;
private List andSearchConditionsList;
private List orSearchConditionsList;
private List notSearchConditionsList;
```
#### QueryCondition
The queryCondition is the same as the base query method, but it supports nested querycondition, which is linked to *and*, *or* or *not*.
Example:
```json
{
"QueryCondition": {
"Conditions": {
"PolicyHolder": "Customer"
},
"InConditions": {
"PolicyType": ["1","2","3"],
"PolicyStatus": [2,3,4]
},
"LteRangeConditions": {
"IssueDate": "2020-06-14"
},
"AndSearchConditionsList": [
{
"ExistFieldsConditions": ["DuePremium"]
}
],
"OrSearchConditionsList": [
{
"ExistFieldsConditions": ["ProductName"]
}
],
"NotSearchConditionsList": [
{
"ExistFieldsConditions": [
"DuePremium"
],
"NotSearchConditionsList": [
{
"Conditions": {
"ProposalNo": "PTBTI0000000806"
}
}
]
}
]
},
"PageNo": 1,
"PageSize": 10,
"SortField": "index_time",
"SortType": "desc",
"Module": "Policy"
}
```
###### Convert to SQL
```sql
select * from Policy where PolicyHolder = 'Customer' and PolicyType in ('1','2','3') and PolicyStatus in (2,3,4) and IssueDate <= '2020-06-14' and DuePremium is not null and (ProductName is not null) and not (DuePremium is not null and not (ProposalNo = 'PTBTI0000000806')) order by index_time desc, BusinessType desc, ExpiryDate desc
```
#### OrSearchConditionsList
Support various "OR" queries. Various types of "OR" queries in base query, such as OrCodintions, OrFuzzyConditions, you can use OrSearchConditionestList. Additionally, basic query methods such as conditions, fuzzyConditions, and GteRangeConditions can be nested within OrSearchConditionsList.
##### JSON Example
```json
{
"QueryCondition": {
"fuzzyConditions": {
"ReceiptNo": "2024"
},
"orSearchConditionsList": [
{
"Conditions": {
"CollectionStatus": "1"
}
},
{
"gteRangeConditions": {
"ReceiveTime": "2023-04-30"
},
"lteRangeConditions": {
"ReceiveTime": "2024-06-09"
}
},
{
"gteRangeConditions": {
"Amount": "1000"
},
"lteRangeConditions": {
"Amount": "2000"
}
}
]
},
"PageNo": 1,
"PageSize": 10,
"SortField": "index_time",
"SortType": "desc",
"Module": "Collection"
}
```
##### SQL Example
```sql
SELECT *
FROM t_bcp_collection a
WHERE a.RECEIPT_NO LIKE '%2024%' AND
(a.COLLECTION_STATUS = '1' OR a.RECEIVE_TIME BETWEEN DATE'2023-04-30' AND DATE'2024-06-09' OR a.AMOUNT BETWEEN 1000 AND 2000)
```
### ES CodeTable Search Criteria Illustration
#### Method
```java
searchRuntimeService.queryGetCodeTable(CodeTableName, keyWord, queryMap, inKeys, notInKeys, pageNo, pageSize)
```
#### Parameter
| Parameters | Type | Required | Description |
| ------------- | ------ | -------- | ----------------------------------------------------- |
| codeTableName | String | True. | Codetable name. |
| keyWord | String | False. | Query condition. |
| queryMap | String | False. | JSON map query condition. |
| inKeys | String | False. | Split by comma. |
| notInKeys | String | False. | Split by comma. |
| pageNo | String | False. | When it is not filled, it is 0. Do not fill more than 100. |
| pageSize | String | False. | When it is not filled, it is 20. Do not fill more than 1000. |
### Add, Update and Delete Business Data via Search API
It is about extending a new query API based on the search service, which will happen sacredly as the query API related to the platform domain is provided for developers to use, with regard to most business domains like policy and claim.
#### Add, Update and Use this Class and Method
```java
searchDocRuntimeService.doIndexWithMap(doIndexDataVo)
```
DoIndexDataVo structure is:
```java
public class DoIndexDataVo implements Serializable
private String schema;
private Map dataEntity;
```
Schema: The index name about data definition.
DataEntity: The data which needs to be saved or updated. Please note that it must include `entity_id` field, which is the primary key of unicorn-search and on which additions or updates depend.
Example:
```javascript
{
"DataEntity": {
"AerospaceInsuredName": null,
"Age": 28,
"BenefitModeCode": "1",
"CheckInDate": null,
"EffectiveDate": "2018-11-29",
"EndoFlag": "-1",
"EndoId": "2717381589000",
"ExpiryDate": "2019-11-28T23:59:59",
"FlightNo": null,
"GlobalSort": "129382405254000",
"IdNo": "3453453",
"IdType": "113",
"IndiGenderCode": "2",
"InsuredId": "271738203",
"OccupationCode": "0001001",
"OccupationType": "01",
"PolHolderInsuredRelaCode": "10",
"PolicyId": "2717140635318",
"PolicyStatus": "1",
"SequenceNumber": "2",
"entity_id": "2717381589000"
},
"Schema": "InsuredEndo"
}
```
#### Delete Method
```java
searchDocRuntimeService.deleteIndexDataById(doIndexDataVo)
```
DoIndexDataVo structure is:
```java
public class DoIndexDataVo implements Serializable
private String schema;
private String delId;
```
Schema: the index name about data definition.
DelId: the `entity_id` of the data to be deleted.
Example:
```javascript
{
"Schema":"InsuredEndo",
"DelId":"2717381589000"
}
```
### Add and Update ES Code Table Data via Search API
#### Method
```java
ClientUtilRestfulService.upsertCodeTable(CodeTableEntity codeTableEntity)
```
#### Parameter
| Parameters | Type | Required | Description |
| ------------- | ------ | -------- | -------------------------- |
| id | String | True | Codetable name. |
| codeTableName | String | True | Query condition. |
| code | String | False | Like codetable code. |
| description | String | False | Like codetable description. |
| esId | String | False | CodeTableName + "_" + id |
| otherParams | String | False | Other parameters if need. |
Example:
```java
{
"CodeTableName":"SalesChannel",
"Id":"PTY37001960111",
"Code":"PTY37001960111",
"Description":"PTY37001960111",
"OtherParams":{
"verificationFlag":"Y"
}
}
```
## Synonyms Query
From Version 25.02, system supports synonym-based queries. For example, you may want search results to include both US and UK spellings, or both lowercase and uppercase variants.
**Setup**
Go to **iTables > Global Parameter** and open the **TextSynonymForIndex** table.
- **Key**: Enter the policy name (e.g., "Policy").
- **Value**: Enter a JSON object where keys are the "standard" terms, and values are arrays of synonyms (including case, regional spelling, or phonetic variants).
**Example value for English scenarios**:
```json
{
"color": ["color", "colour"],
"center": ["center", "centre"],
"analyze": ["analyze", "analyse"],
"i": ["i", "I"],
"theater": ["theater", "theatre"]
}
```
**How it works**
- A search for **"color"** will also return results containing **"colour"**.
- A search for **"theater"** will include **"theatre"** in results.
#### initEsScript
After the table related info configuration done, please call the API initEsScript to make the configuration work.
{{server}}/{{apigw-platform-url}}/search/index/management/v1/initEsScript
Afer initEsScript, the index related synonyms query can work. You can create the data with synonyms words or rebuild the existed data.
Then you can see the result with synonyms word.
## Specific Scenario Study
### Business Query UI Auto Generation
There are some of the attributes during index field configuration which are not used by index module but for query module to have a configurable query experience
If the user is using the index management along with GI business API, the user can refer to [Policy PAAS Query](https://docs.insuremo.com/gi_insurance_service/statistics#configuration-for-search) to understand its potential usage.
### Query Condition Without Case Sensitive
By default, we query data match condition case sensitive.
If we want query data match condition without case sensitive, we can use "**.caseInsensitive**".
#### Case Sensitive
Take `RiskName": "riskname"` for example, if case sensitive, there is no result returned.
###### Request
```
{
"Conditions": {
"ProductCode": "TBTI",
"RiskName": "riskname"
},
"PageNo": 1,
"PageSize": 100,
"SortField": "index_time",
"SortType": "desc",
"Module": "Policy"
}
```
###### Response
```
{
"GroupField": "entity_type",
"PageNo": 1,
"PageSize": 100,
"Total": 0
}
```
#### Without Case Sensitive
If you want to set **RiskName** to be not case sensiticve, you can use the condition `RiskName.caseInsensitive": "riskname"`, and the result will be returned.
###### Request
```
{
"Conditions": {
"ProductCode": "TBTI",
"RiskName.caseInsensitive": "riskname"
},
"PageNo": 1,
"PageSize": 100,
"SortField": "index_time",
"SortType": "desc",
"Module": "Policy"
}
```
###### Response
```
{
"GroupField": "entity_type",
"PageNo": 1,
"PageSize": 5,
"Results": [
{
"EsDocs": [
{
"DuePremium": 38.52,
"EffectiveDate": "2024-05-10",
"ExpiryDate": "2024-05-24",
"FirstDataEntryDate": "2024-05-17T10:12:39.097",
"FullOrgCode": "10002\n10001",
"IdNo": "IdNo",
"IdType": "1",
"InsuredCustomerNo": "",
"InsuredIdNo": "IdNo",
"InsuredIdType": "1",
"InsuredName": "RiskName",
"IssueDate": "2024-05-17T10:12:39",
"MaxUwLevelCode": "0",
"OrgCode": "10002",
"PolicyHolder": "Customer",
"PolicyHolderIDNo": "IdNo",
"PolicyId": "24291282877,09AC20AAF75525834F0C8FDE4B1E577F",
"PolicyNo": "POTBTI00024978",
"PolicyStatus": 2,
"PolicyType": "1",
"ProductCode": "TBTI",
"ProductId": 351925022,
"ProductName": "Oversea Travel (PBU for Auto Test)",
"ProposalDate": "2019-04-22",
"ProposalNo": "PTBTI-202405-0000000425",
"ProposalStatus": 3,
"RiskName": "RiskName",
"SumInsured": 900000,
"entity_id": "24291282877,09AC20AAF75525834F0C8FDE4B1E577F",
"entity_type": "Policy",
"id": "Policy_24291282877",
"index_time": "2024-05-17T02:12:40.246Z",
"index_timestamp": 1715911959751,
"owned_org_code": "10002",
"tenant_code": "xxxxxx",
"TempData": {
"Mask-IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-IdNo": "IdNo",
"Mask-InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-InsuredIdNo": "IdNo"
}
},
{
"DuePremium": 38.52,
"EffectiveDate": "2024-05-06",
"ExpiryDate": "2024-05-20",
"FirstDataEntryDate": "2024-05-13T09:32:18.212",
"FullOrgCode": "10002\n10001",
"IdNo": "IdNo",
"IdType": "1",
"InsuredCustomerNo": "",
"InsuredIdNo": "IdNo",
"InsuredIdType": "1",
"InsuredName": "RiskName",
"IssueDate": "2024-05-13T09:32:19",
"OrgCode": "10002",
"PolicyHolder": "Customer",
"PolicyHolderIDNo": "IdNo",
"PolicyId": "24289552863,4FDFE2972D51A53C11DE167F7E3BD88A",
"PolicyNo": "POTBTI00024870",
"PolicyStatus": 2,
"PolicyType": "1",
"ProductCode": "TBTI",
"ProductId": 351925022,
"ProductName": "Oversea Travel (PBU for Auto Test)",
"ProposalDate": "2019-04-22",
"ProposalNo": "PTBTI-202405-0000000209",
"ProposalStatus": 3,
"RiskName": "RiskName",
"SumInsured": 900000,
"entity_id": "24289552863,4FDFE2972D51A53C11DE167F7E3BD88A",
"entity_type": "Policy",
"id": "Policy_24289552863",
"index_time": "2024-05-13T01:32:19.828Z",
"index_timestamp": 1715563939273,
"owned_org_code": "10002",
"tenant_code": "xxxxxx",
"TempData": {
"Mask-IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-IdNo": "IdNo",
"Mask-InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-InsuredIdNo": "IdNo"
}
},
{
"DuePremium": 38.52,
"EffectiveDate": "2024-05-04",
"ExpiryDate": "2024-05-18",
"FirstDataEntryDate": "2024-05-11T10:34:10.068",
"FullOrgCode": "10002\n10001",
"IdNo": "IdNo",
"IdType": "1",
"InsuredCustomerNo": "",
"InsuredIdNo": "IdNo",
"InsuredIdType": "1",
"InsuredName": "RiskName",
"IssueDate": "2024-05-11T10:34:10",
"OrgCode": "10002",
"PolicyHolder": "Customer",
"PolicyHolderIDNo": "IdNo",
"PolicyId": "24288992876,4A6A2ED3B3FCEF73CF6B359845CD2D36",
"PolicyNo": "POTBTI00024826",
"PolicyStatus": 2,
"PolicyType": "1",
"ProductCode": "TBTI",
"ProductId": 351925022,
"ProductName": "Oversea Travel (PBU for Auto Test)",
"ProposalDate": "2019-04-22",
"ProposalNo": "PTBTI-202405-0000000177",
"ProposalStatus": 3,
"RiskName": "RiskName",
"SumInsured": 900000,
"entity_id": "24288992876,4A6A2ED3B3FCEF73CF6B359845CD2D36",
"entity_type": "Policy",
"id": "Policy_24288992876",
"index_time": "2024-05-11T02:34:11.036Z",
"index_timestamp": 1715394850592,
"owned_org_code": "10002",
"tenant_code": "xxxxxx",
"TempData": {
"Mask-IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-IdNo": "IdNo",
"Mask-InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-InsuredIdNo": "IdNo"
}
},
{
"DuePremium": 38.52,
"EffectiveDate": "2024-05-03",
"ExpiryDate": "2024-05-17",
"FirstDataEntryDate": "2024-05-10T10:08:21.899",
"FullOrgCode": "10002\n10001",
"IdNo": "IdNo",
"IdType": "1",
"InsuredCustomerNo": "",
"InsuredIdNo": "IdNo",
"InsuredIdType": "1",
"InsuredName": "RiskName",
"IssueDate": "2024-05-10T10:08:22",
"OrgCode": "10002",
"PolicyHolder": "Customer",
"PolicyHolderIDNo": "IdNo",
"PolicyId": "24288482890,F0C6C4E4CF907A5F72456F855B1892ED",
"PolicyNo": "POTBTI00024789",
"PolicyStatus": 2,
"PolicyType": "1",
"ProductCode": "TBTI",
"ProductId": 351925022,
"ProductName": "Oversea Travel (PBU for Auto Test)",
"ProposalDate": "2019-04-22",
"ProposalNo": "PTBTI-202405-0000000142",
"ProposalStatus": 3,
"RiskName": "RiskName",
"SumInsured": 900000,
"entity_id": "24288482890,F0C6C4E4CF907A5F72456F855B1892ED",
"entity_type": "Policy",
"id": "Policy_24288482890",
"index_time": "2024-05-10T02:08:22.807Z",
"index_timestamp": 1715306902426,
"owned_org_code": "10002",
"tenant_code": "xxxxxx",
"TempData": {
"Mask-IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-IdNo": "IdNo",
"Mask-InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-InsuredIdNo": "IdNo"
}
},
{
"DuePremium": 38.52,
"EffectiveDate": "2024-05-02",
"ExpiryDate": "2024-05-16",
"FirstDataEntryDate": "2024-05-09T17:55:40.751",
"FullOrgCode": "10002\n10001",
"IdNo": "IdNo",
"IdType": "1",
"InsuredCustomerNo": "",
"InsuredIdNo": "IdNo",
"InsuredIdType": "1",
"InsuredName": "RiskName",
"IssueDate": "2024-05-09T17:55:41",
"OrgCode": "10002",
"PolicyHolder": "Customer",
"PolicyHolderIDNo": "IdNo",
"PolicyId": "24287852889,C88A5F63FADA085E0792D613DF8315A7",
"PolicyNo": "POTBTI00024751",
"PolicyStatus": 2,
"PolicyType": "1",
"ProductCode": "TBTI",
"ProductId": 351925022,
"ProductName": "Oversea Travel (PBU for Auto Test)",
"ProposalDate": "2019-04-22",
"ProposalNo": "PTBTI-202405-0000000090",
"ProposalStatus": 3,
"RiskName": "RiskName",
"SumInsured": 900000,
"entity_id": "24287852889,C88A5F63FADA085E0792D613DF8315A7",
"entity_type": "Policy",
"id": "Policy_24287852889",
"index_time": "2024-05-09T09:55:41.928Z",
"index_timestamp": 1715248541434,
"owned_org_code": "10002",
"tenant_code": "xxxxxx",
"TempData": {
"Mask-IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"IdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-IdNo": "IdNo",
"Mask-InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"InsuredIdNo": "Jmf1vghdrScrTdpdeyKu7w==",
"MaskAfter-InsuredIdNo": "IdNo"
}
}
],
"GroupTotalNum": 6,
"GroupValue": "Policy"
}
],
"Total": 6
}
```
### Whether to Allow Fuzzy Search by Default
By default, we will not allow fuzzy search to save the ES resource for any newly created fields. Of course, there can be three scenarios:
1. Field Type is not String--generically fuzzy search is not applicable.
2. Field Type is String and it's code value--fuzzy search is not required for this scenario either like status or branch code which only requires exact search based on exact code.
3. Field Type is String and it's a business code or name--fuzzy search is required for this scenario for example customer name or policy number.
If the scenario is 3, users can choose to change "Only Exact Search" in index field management UI from "Yes" to "No" to enable the fuzzy search.
1. If it's changed during index field creation, then it's perfectly fine.
2. If it's changed after index field created and business data is already created, then users need to conduct a index rebuild to rebuild the business data.
3. If the user needs to change from No back to Yes, then it will not take effect unless entire index is removed and re-created again.
### How to Save and Search Multiple Values for one Index Field
Normally, let's say we have one index field CustomerName, it will be a single value inside.
There can be scenarios where one policy can have multiple customers and users want to put multiple customer names into one single index field.
Once that happens, during index field creation, users can point the index field to a policycustomerlist then during policy saving, if the policycustomerlist contains multiple values, they will be saved together in the index with format like `"CustomerName": "Jason\nJohn\nMary"`.
Then if users want to query them, there can be two options:
1. If user to use exact search criteria like "Conditions" or "InConditions", then users must add "multiLineExact" at the end to search, e.g., `"Conditions": {"CustomerName.multiLineExact": "John"}` or `"InConditions": {"RiskIdNo.multiLineExact": ["Z7126117","C915562A","V600163A"]}`.
2. Otherwise, users can use fuzzy search criteria to support it. (Please double check to make sure the allow fuzzy search for the field is switched on.)
### ES Update Timeliness
The index data in the Elasticsearch server is updated asynchronously. The time it takes for updates to be reflected depends on the system load—under low load, updates are reflected quickly, whereas under high load, it may take longer.
If refresh=wait_for is used, index operations on the application side can experience significant delays, and under heavy load, this may lead to issues.
Therefore, queries on ES indexes are recommended only for UI purposes and are not advised for direct access by backend processes, especially when dealing with newly generated data.
For reporting-related functionality, it is recommended to access the data directly from the database or to utilize the DataMO module.
### Search Business Data Deploy Check (Private Cloud Only)
When deploying Business Data in the wrong step, it may make the deployed ES mapping wrong, so it is better to ask TS to check the ES mapping after deployment.
Call
```sql
GET http://{ESIP}:{ESHTTPPort}/_cat/indices
```
Then, if you see some indexes that include the word *alias*, they are the wrong ones. If you need to keep the data in it, clone the index and delete it, and restart the search server. If you do not need to keep, delete and restart it.
## MC index Maintain Issue Handle
When maintaining an index field in MC, if you delete some index fields, there is always an error that pops as below when adding an index field or query.
```
{
"timestamp": "2024-02-07T12:38:32",
"timestampServer": "2024-02-07T07:08:32",
"status": 500,
"error": "Internal Server Error",
"path": "/customer/org",
"traceId": "33b0e3615a069253",
"traceIdContainer": null,
"rawMessage": "Commit global transaction failed, error:500 ,call [http://platform-pub/seata-rm/commitBranch/TCC/searchEngineAddDoc] failed, detail:commit TCC resource error: [ElasticsearchStatusException: Elasticsearch exception [type=strict_dynamic_mapping_exception, reason=mapping set to strict, dynamic introduction of [PANNumberr] within [_doc] is not allowed]], xid:894381652, branchId:894378688, resourceId:searchEngineAddDoc",
"exception": "java.lang.RuntimeException",
"code": "MO-PLATFORM-COMMON-E9999",
"message": "Exception has been thrown, check it in ELK with trace id: 33b0e3615a069253, message content is: Commit global transaction failed, error:500 ,call [http://platform-pub/seata-rm/commitBranch/TCC/searchEngineAddDoc] failed, detail:commit TCC resource error: [ElasticsearchStatusException: Elasticsearch exception [type=strict_dynamic_mapping_exception, reason=mapping set to strict, dynamic introduction of [PANNumberr] within [_doc] is not allowed]], xid:894381652, branchId:894378688, resourceId:searchEngineAddDoc"
}
```
Please copy the below curl script to postman to run the API-initEsScript .
`GET https://xx-mc.insuremo.com/api/platform/platform-pub/index/management/v1/initEsScript`
```
curl --location 'https://xx-mc.insuremo.com/api/platform/platform-pub/index/management/v1/initEsScript' \
--header 'Accept-Encoding: gzip, deflate' \
--header 'Connection: keep-alive' \
--header 'Postman-Token: f7433f7c-a7df-40af-9c37-1436a4a2d621,d0897366-0cbd-479d-a62f-7a4805a432a8' \
--header 'accept: application/json' \
--header 'accept-language: en-US,en;q=0.9' \
--header 'authorization: Bearer LzJ98sJLQ6e-ZijGnjXjfQ' \
--header 'cache-control: no-cache,no-cache' \
--header 'content-type: application/json; charset=UTF-8' \
--header 'pragma: no-cache' \
--header 'sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "Windows"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: same-origin' \
--header 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36' \
--header 'x-mo-env: portal' \
--header 'x-mo-module-permission-code: CONF-SYYQ' \
--header 'x-mo-tenant-id: xx' \
--header 'x-mo-user-identity: xxxx@insuremo.com' \
--header 'x-mo-user-name: xxxx@insuremo.com'
```
## Q&A
### If I want to get a data which will be much more than 50K, maybe 300K and keep growing, how can I do it? Any performance concern?
* There is no specific limit on the response size for a single API call.
* For query APIs, the total number of records returned for a paginated query with the same filter conditions cannot exceed 1000 (i.e., `page_size * page_number <= 1000`). This limit is in place to prevent abuse and reduce system load.
* To retrieve more than 1000 records, you must modify the query conditions for subsequent requests. For example, if sorting by `claimId`, use the last retrieved `claimId` as a starting point for the next query (e.g., `claimId >= last_retrieved_id`). This technique is known as keyset pagination.
* While you can make multiple queries and merge the results before returning them to the client, be aware of the iComposer API timeout limit (2 minutes, enforced by the load balancer and gateway).
* For long-running processes, an asynchronous approach is recommended:
1. Immediately return a `referenceId` to the caller.
2. Process the data in the background, merge the results, and upload the final dataset to cloud storage (cloud disk API).
3. Provide a separate API endpoint for the client to poll the status of the job using the `referenceId`. Once complete, this endpoint should return a download link for the results.
### During endorsement carrier removed one field from customer object, but policy search still gives OLD record, why?
Now we are using OpenSearch as the ES engine which does not support remove a field from document.
If you want to clear it, you have to update it to a blank string like a space "", or some string you have agreement with like "NA".
Please note that since we are using upsert mode when using opensearch, and support update several fields while leave the un-mentioned fields untouched. So it is impossible to get a field with empty value in the response payload.
So either not to have field in payload or consider NA as empty and you can update address to NA to mean it is empty.