Google Cloud Storage Panel

Overview

The Google Cloud Storage panel enables administrators and data engineers to securely integrate Google Cloud Platform (GCP) object storage buckets into the DataFloem platform.

Connections established through this interface serve as operational cloud storage endpoints, allowing pipelines to read, process, and write unstructured or semi-structured file formats (Parquet, CSV, JSON, etc.) to target GCS bucket locations.

Access Requirements

To access this page, the user must either have Administrator privileges or the CREATE permission for the Source module.

The CREATE permission allows users to create new groups. Users without administrator privileges or the required permission cannot access this page.

If you do not have the required permission, ask your administrator to grant it from one of the authorization pages below.

Authorization

Users without the required authorization will receive the following error:

Access Denied

You don't have access to this resource.

Prerequisites

Before establishing a Google Cloud Storage connection, ensure that:

A dedicated Service Account has been provisioned via the GCP Console with access to the target storage bucket.
Necessary IAM roles (such as Storage Object Admin or Storage Object Viewer/Creator) are granted to the service account.
You have retrieved the required service account credential details (Project Id, Service Account Email, Service Account Private Id, and Service Account Private Key).

Form Elements

The fields that need to be configured within the panel are as follows:

Field	Required?	Description
Connection Name	Yes	A unique alias identifying this Google Cloud Storage connection across the platform.
Database Version	Yes (Dropdown)	Selects the precise version mapping to ensure correct client driver compatibility.
Is Active	Yes (Toggle)	Defines whether the connection is active immediately upon deployment. Enabled (`on`) by default.
Project Id	Yes	The master project identifier hosted on Google Cloud Platform (e.g., `my-gcp-project-id`).
Service Account Email	Yes	The authorized service account email address used for IAM authentication.
Service Account Private Id	Yes	The unique Key ID string corresponding to the service account private key (Visibility can be toggled using the eye icon).
Service Account Private Key	Yes	The RSA/PEM formatted private key string belonging to the service account.
Bucket Name	Yes	The exact GCS target storage bucket name to read from or write to.
Root Directory Path	Yes	The base folder path where file operations will be scoped. Defaults to `/` for bucket root.
Optional Parameters (Name / Value)	No	Dynamic entries used to pass fine-grained parameters (e.g., `location`, `kmsKeyName`).
Is Secret (Optional Parameters)	Yes (Toggle)	Encrypts and masks the specific parameter value within internal stores.

Google Cloud Storage Connection Deployment Process

Select Google Cloud Storage from the cloud storage wizard to open this panel.
Enter a unique Connection Name and designate the target Database Version.
Fill in the GCP Service Account details including Project Id, Service Account Email, Service Account Private Id, and Service Account Private Key.
Specify the target Bucket Name and set the appropriate Root Directory Path.
If required, add custom key-value arguments under Optional Parameters and toggle Is Secret appropriately.
Review all entered values and click Save to commit the asset to the platform, or click Cancel to abort the operation.

Validation Rules

Connection Name, Project Id, Service Account Email, Service Account Private Id, Service Account Private Key, and Bucket Name fields cannot be left empty.
Service Account Email must strictly adhere to valid email address syntax (e.g., account@project.iam.gserviceaccount.com).
The Root Directory Path field must follow a valid path syntax (e.g., / or /input/data/).