Add Object Storage Panel
Overview
The Add Object Storage panel allows administrators and data engineers to securely connect S3-compatible object storage systems (such as MinIO, AWS S3, Ceph, etc.) to the DataFloem platform.
Connections established through this interface serve as operational storage endpoints, enabling jobs to ingest, transform, or output unstructured and semi-structured file payloads (CSV, Parquet, JSON, Images, etc.) into target buckets.
Access Requirements
To access this page, the user must either have Administrator privileges or the CREATE permission for the Source module.
The CREATE permission allows users to create new groups. Users without administrator privileges or the required permission cannot access this page.
If you do not have the required permission, ask your administrator to grant it from one of the authorization pages below.
Authorization
Users without the required authorization will receive the following error:
Access Denied
You don't have access to this resource.
Prerequisites
Before establishing an Object Storage connection, ensure that:
- Network line connectivity and firewall pathways are cleared between DataFloem and the object storage Endpoint/Port.
-
You possess valid
Access KeyandSecret Keycredentials provisioned with sufficient read/write (GetBucket,GetObject,PutObject) permissions on the target bucket.
Form Elements
The fields that need to be configured within the panel are as follows:
| Field | Required? | Description |
|---|---|---|
| Connection Name | Yes | A unique alias identifying this object storage connection across the platform. |
| Database Type | Yes (Dropdown) | Selects the provider engine architecture (e.g., MinIO, AWS S3). |
| Database Version | Yes (Dropdown) | Selects the precise version mapping to ensure correct client driver compatibility. |
| Is Active | Yes (Toggle) |
Defines whether the connection is active immediately upon
deployment. Enabled (on) by default.
|
| Endpoint | Yes |
The IP address or domain URL of the storage cluster (e.g.,
s3.amazonaws.com or custom IP).
|
| Port | Yes |
The service communication port (e.g.,
9000 for default MinIO or
443 for HTTPS).
|
| Access Key | Yes | The account identification string used to access the object storage service. |
| Secret Key | Yes | The secret key corresponding to the Access Key (Visibility can be toggled using the eye icon). |
| Bucket Name | Yes | The exact target storage bucket name to read from or write to. |
| Root Directory Path | Yes |
The base folder path where file operations will be scoped.
Defaults to / for bucket root.
|
| Path Style Access | Yes (Toggle) |
Controls URL syntax (endpoint/bucket
when active vs bucket.endpoint when
disabled; typically required for MinIO).
|
| Optional Parameters (Name / Value) | No |
Dynamic entries used to pass fine-grained parameters (e.g.,
region,
ssl).
|
| Is Secret (Optional Parameters) | Yes (Toggle) | Encrypts and masks the specific parameter value within internal stores. |
Path Style Access Behavior
-
When Path Style Access is enabled (
on), requests are structured using the path pattern:https://endpoint:port/bucket-name. -
When disabled (
off), requests revert to virtual-hosted style syntax:https://bucket-name.endpoint:port. - Enabling Path Style Access is highly recommended for self-hosted or On-Premises deployments (e.g., MinIO, Ceph) to prevent DNS resolution failures.
Object Storage Connection Deployment Process
- Click the Add Object Storage action button on the workspace dashboard to trigger the side panel.
- Enter a unique Connection Name, then designate the Database Type and Database Version.
- Fill in the server parameters including Endpoint, Port, Access Key, Secret Key, and Bucket Name.
- Define the Root Directory Path and set the Path Style Access toggle according to your storage architecture.
- If required, add custom key-value arguments under Optional Parameters and toggle Is Secret appropriately.
- Review all entered values and click Save to commit the asset to the platform, or click Cancel to abort the operation.
Validation Rules
- Connection Name, Endpoint, Port, Access Key, Secret Key, and Bucket Name fields cannot be left empty.
- The Port field must contain a valid numerical integer within the port range (1-65535).
-
The Root Directory Path field must follow a valid path
syntax (e.g.,
/or/data/input/).