---
Title: Redis Data Integration configuration file
linkTitle: RDI configuration file
description: Redis Data Integration configuration file reference
weight: 10
alwaysopen: false
categories: ["redis-di"]
aliases:
---
Configuration file for Redis Data Integration (RDI) source collectors and target connections
**Properties**
| Name | Type | Description | Required |
| ----------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| [**sources**](#sources)
(Source collectors) | `object` | Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector
| |
| [**processors**](#processors)
(Data processing configuration) | `object`, `null` | Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning
| |
| [**targets**](#targets)
(Target connections) | `object` | Configuration for target Redis databases where processed data will be written
| |
| [**secret\-providers**](#secret-providers)
(Secret providers) | `object` | | |
## sources: Source collectors
Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector
**Properties (Pattern)**
| Name | Type | Description | Required |
| -------- | ---- | ----------- | -------- |
| **\.\*** | | | |
## processors: Data processing configuration
Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning
**Properties**
| Name | Type | Description | Required |
| -------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| **on_failed_retry_interval**
(Retry interval on failure) | `integer`, `string` | Number of seconds to wait before retrying a failed operation
Default: `5`
Pattern: `^\${.*}$`
Minimum: `1`
| |
| **read_batch_size** | `integer`, `string` | Maximum number of records to process in a single batch
Default: `2000`
Pattern: `^\${.*}$`
Minimum: `1`
| |
| **debezium_lob_encoded_placeholder**
(Debezium LOB placeholder) | `string` | Placeholder value for LOB fields in Debezium
Default: `"__debezium_unavailable_value"`
| |
| **dedup**
(Enable deduplication) | `boolean` | Enable the deduplication mechanism to handle duplicate records
Default: `false`
| |
| **dedup_max_size**
(Deduplication set size) | `integer` | Maximum number of entries to store in the deduplication set
Default: `1024`
Minimum: `1`
| |
| **dedup_strategy**
(Deduplication strategy) | `string` | (DEPRECATED)
Property 'dedup_strategy' is now deprecated. The only supported strategy is 'ignore'. Please remove from the configuration.
Default: `"ignore"`
Enum: `"reject"`, `"ignore"`
| |
| **duration**
(Batch duration limit) | `integer`, `string` | Maximum time in milliseconds to wait for a batch to fill before processing
Default: `100`
Pattern: `^\${.*}$`
Minimum: `1`
| |
| **write_batch_size** | `integer`, `string` | Maximum number of records to write to target Redis database in a single batch
Default: `200`
Pattern: `^\${.*}$`
Minimum: `1`
| |
| **error_handling**
(Error handling strategy) | `string` | Strategy for handling errors: ignore to skip errors, dlq to store rejected messages in dead letter queue
Default: `"dlq"`
Pattern: `^\${.*}$\|ignore\|dlq`
| |
| **dlq_max_messages**
(DLQ message limit) | `integer`, `string` | Maximum number of messages to store in dead letter queue per stream
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
| |
| **target_data_type**
(Target Redis data type) | `string` | Data type to use in Redis: hash for Redis Hash, json for RedisJSON (requires RedisJSON module)
Default: `"hash"`
Pattern: `^\${.*}$\|hash\|json`
| |
| **json_update_strategy** | `string` | (DEPRECATED)
Property 'json_update_strategy' will be deprecated in future releases. Use 'on_update' job-level property to define the json update strategy.
Default: `"replace"`
Pattern: `^\${.*}$\|replace\|merge`
| |
| **use_native_json_merge**
(Use native JSON merge) | `boolean` | Controls whether to use the native `JSON.MERGE` command (when `true`) or Lua scripts (when `false`) for JSON merge operations. Introduced in RDI 1.15.0. The native command provides 2x performance improvement but handles null values differently:
**Previous behavior (Lua merge)**: When merging `{"field1": "value1", "field2": "value2"}` with `{"field2": null, "field3": "value3"}`, the result was `{"field1": "value1", "field2": null, "field3": "value3"}` (null value is preserved)
**New behavior (JSON.MERGE)**: The same merge produces `{"field1": "value1", "field3": "value3"}` (null value removes the field, following [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396))
**Note**: The native `JSON.MERGE` command requires RedisJSON 2.6.0 or higher. If the target database has an older version of RedisJSON, RDI will automatically fall back to using Lua-based merge operations regardless of this setting.
**Impact**: If your application logic distinguishes between a field with a `null` value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation. Set to `false` to revert to the previous Lua-based merge behavior if needed.
Default: `true`
| |
| **initial_sync_processes** | `integer`, `string` | Number of parallel processes for performing initial data synchronization
Default: `4`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `32`
| |
| **idle_sleep_time_ms**
(Idle sleep interval) | `integer`, `string` | Time in milliseconds to sleep between processing batches when idle
Default: `200`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
| |
| **idle_streams_check_interval_ms**
(Idle streams check interval) | `integer`, `string` | Time in milliseconds between checking for new streams when processor is idle
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
| |
| **busy_streams_check_interval_ms**
(Busy streams check interval) | `integer`, `string` | Time in milliseconds between checking for new streams when processor is busy
Default: `5000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
| |
| **wait_enabled**
(Enable replica wait) | `boolean` | Enable verification that data has been written to replica shards of the target database
Default: `false`
| |
| **wait_timeout**
(Replica wait timeout) | `integer`, `string` | Maximum time in milliseconds to wait for replica write verification of the target database
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
| |
| **retry_on_replica_failure** | `boolean` | Continue retrying writes until successful replication to replica shards is confirmed
Default: `true`
| |
**Additional Properties:** not allowed
## targets: Target connections
Configuration for target Redis databases where processed data will be written
**Properties (Pattern)**
| Name | Type | Description | Required |
| -------- | ---- | ----------- | -------- |
| **\.\*** | | | |
## secret\-providers: Secret providers
**Properties (Pattern)**
| Name | Type | Description | Required |
| --------------------------------------------------------- | -------- | ----------- | -------- |
| [**\.\***](#secret-providers)
(Secret provider entry) | `object` | | yes |
#### secret\-providers\.\.\*: Secret provider entry
**Properties**
| Name | Type | Description | Required |
| ----------------------------------------------------------------------- | -------- | ----------------------------- | -------- |
| **type**
(Provider type) | `string` | Enum: `"aws"`, `"vault"`
| yes |
| [**parameters**](#secret-providersparameters)
(Provider parameters) | `object` | | yes |
**Additional Properties:** not allowed
**Example**
```yaml
parameters:
objects:
- {}
```
##### secret\-providers\.\.\*\.parameters: Provider parameters
**Properties**
| Name | Type | Description | Required |
| ----------------------------------------------------------------------------- | ---------- | ----------- | -------- |
| [**objects**](#secret-providersparametersobjects)
(Secrets objects array) | `object[]` | | yes |
**Example**
```yaml
objects:
- {}
```
###### secret\-providers\.\.\*\.parameters\.objects\[\]: Secrets objects array
**Items: Secret object**
**No properties.**
**Example**
```yaml
- {}
```