Add Elasticsearch Panel
Overview
The Add Elasticsearch panel allows administrators and data engineers to securely connect Elasticsearch search and analytics clusters to the DataFloem platform.
Connections established through this interface serve as operational nodes within data integration pipelines, enabling jobs to query, search, and extract document indices or write processed payloads into target Elasticsearch clusters.
Access Requirements
To access this page, the user must either have Administrator privileges or the CREATE permission for the Source module.
The CREATE permission allows users to create new groups. Users without administrator privileges or the required permission cannot access this page.
If you do not have the required permission, ask your administrator to grant it from one of the authorization pages below.
Authorization
Users without the required authorization will receive the following error:
Access Denied
You don't have access to this resource.
Prerequisites
Before establishing an Elasticsearch connection, ensure that:
- Network line connectivity and firewall policies allow traffic between DataFloem and the Elasticsearch Endpoint/Port.
- You possess valid user credentials provisioned with proper read/write privileges on target indices (if authentication is enforced).
- You have verified the SSL/TLS certificate requirements of your cluster deployment.
Form Elements
The fields that need to be configured within the panel are as follows:
| Field | Required? | Description |
|---|---|---|
| Connection Name | Yes | A unique alias identifying this Elasticsearch connection across the platform. |
| Database Version | Yes (Dropdown) | Selects the precise version mapping to invoke correct client drivers (e.g., v7.x, v8.x). |
| Is Active | Yes (Toggle) |
Defines whether the connection is active immediately upon
deployment. Enabled (on) by default.
|
| Endpoint | Yes | The IP address or domain URL of the target Elasticsearch node or load balancer. |
| Port | Yes |
The service communication port (e.g., default REST HTTP port
9200).
|
| Is Auth | Yes (Toggle) | Toggle switch enabling user credential challenges. Reveals Username and Password fields when active. |
| Username | No (Yes if Is Auth is on) | The authorized Elasticsearch username. |
| Password | No (Yes if Is Auth is on) | The credential password (Visibility can be toggled using the eye icon). |
| SSL/TLS | Yes (Toggle) | Toggle switch forcing connections to route through HTTPS encrypted layers. |
| Skip SSL Validation | Yes (Toggle) | Toggle switch allowing the client to bypass certificate validation (useful for self-signed SSL certificates). |
| Test Connection | Button | Simulates an immediate end-to-end handshake with the target Elasticsearch cluster. |
| Optional Parameters (Name / Value) | No |
Dynamic entries used to pass fine-grained cluster parameters
(e.g., indexPrefix,
timeout).
|
| Is Secret (Optional Parameters) | Yes (Toggle) | Encrypts and masks the specific parameter value within internal stores. |
Test Connection Behavior
-
Clicking the Test Connection button triggers a
real-time verification handshake with the target Elasticsearch cluster
using the provided
Endpoint,Port, authentication credentials (if enabled), and SSL configurations. - If the handshake succeeds, a green success banner is displayed confirming network connectivity.
- If the handshake fails, a red error prompt appears detailing network timeout, authentication, or SSL certificate issues. Validating the connection prior to saving is highly recommended.
Elasticsearch Connection Deployment Process
- Click the Add Elasticsearch action button on the workspace dashboard to trigger the side panel.
- Enter a unique Connection Name and designate the target Database Version.
- Input your cluster parameters into the Endpoint and Port fields.
- If cluster security is active, enable Is Auth and fill in your Username and Password.
- Configure the SSL/TLS and Skip SSL Validation switches based on your network infrastructure.
- If required, add custom key-value arguments under Optional Parameters and toggle Is Secret appropriately.
- Click Test Connection to confirm connectivity to the target cluster.
- Once validated, click Save to commit the asset to the platform, or click Cancel to abort the operation.
Validation Rules
- Connection Name, Endpoint, and Port fields cannot be left empty.
- When Is Auth is enabled, Username and Password fields become mandatory.
- The Port field must contain a valid numerical integer within the port range (1-65535).