Modern analytics platforms are often built using managed cloud services. They reduce operational overhead, but they also increase dependency on a specific cloud ecosystem and can become expensive as workloads grow.
For this project, I wanted to explore a different approach:
How would I design a production-oriented lakehouse on Google Cloud while keeping the data platform itself mostly based on open-source technologies?
The result is an open lakehouse architecture built around:
- Google Cloud Storage for durable object storage
- Apache Iceberg as the open table format
- Apache Polaris as the Iceberg REST catalog
- StarRocks as the analytical query engine
- Airbyte for batch and CDC ingestion
- Apache Superset for BI and visualization
- PostgreSQL for service metadata
- Terraform for infrastructure provisioning
- Google Compute Engine for running the open-source services
The Problem
A typical data platform starts simply.
Applications write data into operational databases such as:
- PostgreSQL
- MySQL
- MongoDB
- SaaS platforms and APIs
Eventually, analytical requirements grow.
Teams need to:
- combine data from different systems,
- retain historical data,
- process CDC events,
- create reusable datasets,
- support BI workloads,
- isolate analytical workloads from transactional databases,
- and scale storage independently from compute.
Running analytical queries directly against production databases quickly becomes problematic.
At that point, a dedicated analytical platform becomes necessary.
The challenge is that I didn't want the architecture to be tightly coupled to a proprietary warehouse.
Instead, I wanted the data itself to remain stored in an open format inside object storage, while allowing different engines to potentially access it.
That requirement naturally led to a lakehouse architecture.
Architecture
The platform follows this high-level architecture:
flowchart LR
subgraph Sources["Data Sources"]
MYSQL[(MySQL)]
PG[(PostgreSQL)]
MONGO[(MongoDB)]
API[SaaS / APIs]
end
subgraph Ingestion["Ingestion Layer"]
AIRBYTE[Airbyte]
end
subgraph Lakehouse["Lakehouse"]
POLARIS[Apache Polaris]
GCS[(Google Cloud Storage)]
ICEBERG[Apache Iceberg]
end
subgraph Compute["Compute Layer"]
SR[StarRocks]
end
subgraph Serving["Serving Layer"]
SUPERSET[Apache Superset]
SQL[SQL Clients]
end
MYSQL --> AIRBYTE
PG --> AIRBYTE
MONGO --> AIRBYTE
API --> AIRBYTE
AIRBYTE --> POLARIS
POLARIS --> ICEBERG
ICEBERG --> GCS
SR --> POLARIS
SR --> ICEBERG
SUPERSET --> SR
SQL --> SR
The simplified data flow is:
Operational Sources
↓
Airbyte
↓
Apache Iceberg + Polaris
↓
GCS
↓
StarRocks
↓
Bronze → Silver → Gold
↓
Apache Superset / SQL
One important design principle here is the separation between storage, metadata, and compute.
Why Apache Iceberg?
Object storage such as GCS is excellent for storing large amounts of data cheaply.
But storing a collection of Parquet files is not enough to create a reliable analytical platform.
For example, consider a directory containing:
transactions/
├── part-00001.parquet
├── part-00002.parquet
├── part-00003.parquet
└── part-00004.parquet
These files contain data, but there is no strong table abstraction around them.
Questions quickly appear:
- Which files currently belong to the table?
- How do I safely update records?
- How do I evolve the schema?
- How do I maintain snapshots?
- How do multiple engines understand the same table state?
Apache Iceberg adds this table-management layer.
Conceptually:
Iceberg Table
│
├── Metadata
│
├── Snapshots
│
├── Manifest Lists
│
├── Manifests
│
└── Parquet Data Files
The actual data remains stored as files in object storage, while Iceberg manages the metadata required to treat those files as reliable database-like tables.
This gives the architecture an important property:
The data is not owned by the query engine.
StarRocks is responsible for querying and processing the data, but the underlying datasets remain Iceberg tables in GCS.
Why Apache Polaris?
Once Iceberg is introduced, another problem appears:
How do query engines discover and coordinate Iceberg tables?
This is the responsibility of the catalog.
For this project I use Apache Polaris, which exposes an Iceberg REST Catalog.
Instead of StarRocks managing table metadata itself, it communicates with Polaris.
Conceptually:
StarRocks
│
│ Iceberg REST API
▼
Apache Polaris
│
├── Catalog metadata
│
└── Table locations
│
▼
Google Cloud Storage
I created separate Polaris catalogs for:
bronze
silver
gold
Each catalog maps to its corresponding location in GCS.
For example:
gs://data-lake/bronze/
gs://data-lake/silver/
gs://data-lake/gold/
This creates a clean separation between data processing stages while keeping the underlying format consistent.
Why StarRocks?
I wanted the serving layer to provide interactive analytical performance without moving data into another proprietary warehouse.
StarRocks works well for this architecture because it can operate as the analytical compute layer on top of Iceberg.
The deployment uses StarRocks in a separated compute/storage architecture.
Its main components are:
Frontend (FE)
↓
Query planning
Metadata
Cluster coordination
Compute Node (CN)
↓
Query execution
Caching
Aggregation
Joins
Durable analytical data remains in GCS.
This means compute nodes can remain relatively stateless.
The architecture becomes:
┌───────────────┐
│ StarRocks FE │
└───────┬───────┘
│
┌─────────────┴─────────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Compute Node │ │ Compute Node │
│ CN-0 │ │ CN-1 │
└──────┬───────┘ └──────┬───────┘
│ │
└─────────────┬──────────────┘
▼
Apache Iceberg
▼
GCS
Additional compute nodes can therefore be added when query demand increases without redistributing the entire durable dataset between nodes.
Medallion Architecture
The lakehouse follows the familiar:
Bronze → Silver → Gold
But each layer has a different responsibility.
Bronze: Preserve Source Data
Bronze is the ingestion layer.
Its primary responsibility is to preserve data coming from upstream systems with as little transformation as practical.
For example:
MySQL
│
│ CDC
▼
Airbyte
│
▼
bronze.orders
The important principle is:
Bronze represents what the source system sent, not what the business eventually wants to query.
Keeping this separation makes debugging significantly easier.
If something goes wrong downstream, the original source representation is still available for reprocessing.
Silver: Standardize and Integrate
Silver represents cleaned and standardized datasets.
Typical transformations include:
- data type normalization,
- timestamp normalization,
- deduplication,
- identifier standardization,
- joining related datasets,
- data quality rules,
- and business-key resolution.
For example:
INSERT INTO silver.demo_clean.events
SELECT
e.event_id,
e.event_time,
e.user_id,
e.event_type,
u.country,
e.payload
FROM bronze.demo.events e
LEFT JOIN bronze.demo.users u
ON e.user_id = u.user_id;
At this point, consumers no longer need to understand every inconsistency from the original operational systems.
Silver becomes the standardized analytical representation.
Gold: Business-Ready Data
Gold contains datasets designed for specific analytical use cases.
Instead of exposing raw transactions directly to dashboards, the Gold layer can contain metrics such as:
daily_sales
daily_active_users
payment_success_rate
transactions_by_channel
revenue_by_product
For example:
INSERT INTO gold.demo_metrics.daily_user_clicks
SELECT
DATE(event_time) AS event_date,
event_type,
country,
COUNT(*) AS event_count,
COUNT(DISTINCT user_id) AS unique_users
FROM silver.demo_clean.events
GROUP BY
DATE(event_time),
event_type,
country;
Apache Superset can then query Gold through StarRocks.
The serving path becomes:
Superset
↓
StarRocks
↓
Gold Iceberg Tables
↓
GCS
This prevents BI consumers from repeatedly rebuilding complex transformation logic.
CDC and Data Ingestion
Airbyte handles ingestion into the platform.
Depending on the source, ingestion can use either:
Full / Incremental Batch
or:
Change Data Capture
CDC is particularly important for operational databases.
Instead of repeatedly extracting an entire table:
SELECT * FROM transactions
the pipeline processes changes:
INSERT
UPDATE
DELETE
Conceptually:
PostgreSQL WAL
↓
Airbyte
↓
Iceberg Bronze
This reduces unnecessary data movement and makes lower-latency ingestion possible.
The same ingestion layer can also support multiple source types, including PostgreSQL, MySQL, MongoDB, and external SaaS systems.
Separating Compute from Storage
One of the most important architectural decisions was separating compute from persistent data storage.
In a traditional distributed database, storage and compute often live on the same nodes.
Scaling can therefore mean scaling both simultaneously.
In this architecture:
Compute
────────────
StarRocks CN
StarRocks CN
StarRocks CN
↓
Storage
────────────
Google Cloud Storage
If query demand increases, additional compute nodes can be introduced.
If stored data increases, GCS grows independently.
This provides more flexibility than tightly coupling both dimensions.
Using Local NVMe as Cache
Although durable data lives in GCS, repeatedly reading remote objects introduces latency.
The StarRocks compute nodes therefore use local SSD storage as cache.
The resulting hierarchy becomes:
CPU / Memory
↓
Local NVMe Cache
↓
Google Cloud Storage
Frequently accessed data can remain closer to compute, while GCS remains the durable system of record.
This gives the platform a useful combination:
- object-storage durability,
- independent compute scaling,
- and faster access for frequently queried data.
Cost-Aware Compute Scaling
Another design consideration was avoiding expensive always-on compute capacity when it was unnecessary.
The first StarRocks compute node runs on a normal on-demand VM.
Additional compute nodes can run as GCP Spot VMs.
CN-0 → On-demand
CN-1 → Spot
CN-2 → Spot
CN-3 → Spot
This works because compute nodes do not hold the authoritative copy of analytical data.
If a Spot VM is terminated, the underlying datasets remain safely stored in Iceberg on GCS.
Once the VM returns, it can register itself with the StarRocks frontend again.
This is an example of how architectural decisions can create opportunities for infrastructure cost optimization.
Infrastructure as Code
The entire infrastructure is defined using Terraform.
Instead of manually creating:
- VPCs,
- subnets,
- firewall rules,
- GCS buckets,
- service accounts,
- VM instances,
- NAT configuration,
- and application infrastructure,
the desired environment is represented as code.
The deployment starts with:
terraform init
terraform plan
terraform apply
This makes the architecture reproducible.
More importantly, it means infrastructure configuration can be reviewed and version-controlled similarly to application code.
Private-by-Default Networking
The platform services run inside a private subnet.
The VMs do not require public IPv4 addresses for normal operation.
Administrative access uses Google Cloud Identity-Aware Proxy (IAP).
Instead of:
Internet
↓
Public VM IP
↓
SSH
the access path becomes:
Engineer
↓
Google Identity
↓
IAP Tunnel
↓
Private VM
The same mechanism can expose internal service ports temporarily to an administrator.
For example:
gcloud compute start-iap-tunnel data-lake-fe-0 9030 \
--local-host-port=localhost:9030
A local SQL client can then connect to:
localhost:9030
without exposing the StarRocks endpoint publicly.
Network Egress Through Cloud NAT
Although the services themselves remain private, ingestion workers still need to communicate with external sources and APIs.
Cloud NAT provides outbound internet access without assigning public IPv4 addresses to each VM.
Private Airbyte VM
↓
Cloud NAT
↓
External Database / API
This keeps inbound exposure minimal while still allowing ingestion jobs to reach external systems.
Access Control Between Data Layers
Not every user should have access to every layer.
For example, analysts may only require Gold:
GRANT USAGE ON CATALOG gold TO 'analyst'@'%';
GRANT SELECT ON gold.demo_metrics.* TO 'analyst'@'%';
Data engineers can receive broader access:
Bronze
Silver
Gold
This naturally supports a model where:
Data Engineer
├── Bronze
├── Silver
└── Gold
Analyst
└── Gold
The architecture therefore treats the Medallion layers not only as transformation stages, but also as useful access boundaries.
High Availability and Horizontal Scaling
StarRocks compute capacity can be increased by adding CN instances:
cn_instance_count = 3
Frontend nodes can also be increased for high availability:
fe_instance_count = 3
Frontend nodes use quorum-based coordination, so a production deployment can use an odd number of FE nodes.
Conceptually:
FE-0
/ \
FE-1 FE-2
↓
CN-0 CN-1 CN-2
The two scaling dimensions serve different purposes.
More FE nodes primarily improve control-plane availability.
More CN nodes increase analytical compute capacity.
Operational Design
Deploying the architecture was only one part of the problem.
A production-oriented platform also needs to be maintainable.
The installation scripts were therefore designed to be idempotent.
Running the same installation again should not blindly reinstall everything.
Instead, the process checks:
- installed versions,
- configuration state,
- service state,
- and build fingerprints.
Conceptually:
Run installer
↓
Is expected version installed?
↙ ↘
Yes No
↓ ↓
Check service Install / upgrade
↓ ↓
Exit Restart service
This becomes particularly useful during upgrades.
Upgrading Components Without Rebuilding the Platform
Versions are controlled from Terraform configuration.
For example, changing a StarRocks version updates the instance metadata containing its installation configuration.
The VM itself does not necessarily need to be replaced.
The operational workflow becomes:
Update version
↓
terraform apply
↓
Refresh startup metadata
↓
Run idempotent installer
↓
Service upgrade
For a StarRocks cluster, frontend followers should be upgraded before the leader.
Compute nodes are easier to replace because the authoritative data remains in GCS.
This again demonstrates an advantage of separated storage and compute.
BI with Apache Superset
Apache Superset provides the visualization layer.
It connects to StarRocks using its MySQL-compatible endpoint.
The BI architecture stays intentionally simple:
Dashboard
↓
Superset
↓
StarRocks
↓
Gold
↓
Iceberg / GCS
Superset does not need to understand Iceberg, Polaris, or the internal storage layout.
It only needs a SQL interface.
This separation keeps responsibilities clean:
| Component | Responsibility |
|---|---|
| Airbyte | Data ingestion |
| GCS | Durable object storage |
| Iceberg | Table format |
| Polaris | Catalog |
| StarRocks | SQL analytics and compute |
| Superset | BI and visualization |
| Terraform | Infrastructure provisioning |
Why Not Just Use BigQuery?
BigQuery would obviously be a much simpler choice on GCP.
A managed architecture could look like:
Sources
↓
Managed ingestion
↓
BigQuery
↓
Looker / BI
That architecture removes a significant amount of operational work.
There is nothing inherently wrong with it.
The goal of this project, however, was different.
I wanted to explore an architecture where:
- data remains in object storage,
- tables use an open format,
- the catalog follows an open API,
- compute can be changed independently,
- and the analytical engine does not own the underlying data.
The trade-off is operational complexity.
With managed BigQuery, Google operates much of the infrastructure.
With this architecture, the engineering team is responsible for components such as:
StarRocks
Polaris
Airbyte
Superset
PostgreSQL
VM lifecycle
Service upgrades
Monitoring
Capacity planning
So I would not argue that this architecture should replace BigQuery everywhere.
Instead:
Architecture should follow organizational requirements rather than technology preference.
If simplicity and low operational overhead are the priority, a managed warehouse may be the better choice.
If open table formats, engine independence, infrastructure control, or specific cost characteristics are important, an open lakehouse becomes more interesting.
Challenges and Lessons Learned
Building the platform exposed several practical issues that don't usually appear in high-level architecture diagrams.
1. A Lakehouse Is More Than Object Storage
Putting Parquet files in GCS does not automatically create a lakehouse.
A usable platform also needs:
Storage + Table Format + Catalog + Compute Engine + Ingestion + Governance + Operations
Iceberg and Polaris were therefore just as important as the storage layer itself.
2. Open Source Reduces Vendor Dependency, Not Operational Work
An open-source stack provides flexibility, but somebody still needs to operate it.
Every additional component introduces:
- configuration,
- upgrades,
- monitoring,
- failure modes,
- compatibility concerns,
- and security responsibilities.
Managed services hide much of this complexity.
Open-source infrastructure exposes it.
That is not necessarily bad, but it must be an intentional trade-off.
3. Stateless Compute Changes Infrastructure Strategy
Once persistent data moves out of compute nodes and into object storage, infrastructure becomes much more disposable.
Compute nodes can potentially:
- use Spot instances,
- be resized,
- be replaced,
- scale horizontally,
- or recover after failure
without moving the authoritative dataset.
That architectural property is more important than any individual VM configuration.
4. The Catalog Is a Critical Part of the Platform
Before building this project, it is easy to think of the data lake primarily as:
Query Engine + Object Storage
In practice, the catalog is a first-class infrastructure component.
Without reliable metadata coordination, multiple engines cannot safely reason about the same Iceberg tables.
The real architecture is closer to:
Compute → Catalog → Table Metadata → Object Storage
5. Production Readiness Is Mostly About Operations
Getting a query to successfully return data is relatively easy.
Operating the system repeatedly is harder.
Questions such as these matter much more over time:
- What happens when a VM is replaced?
- How is a new compute node registered?
- What happens when a Spot instance disappears?
- How are services upgraded?
- How are credentials managed?
- How are data files cleaned up?
- How are failed jobs detected?
- How is Iceberg maintenance performed?
- How is user access separated?
- How can the infrastructure be recreated?
Those questions often separate a demo architecture from a production-oriented one.
What I Would Improve Next
The current architecture establishes the core lakehouse infrastructure, but there are several areas I would expand for a more mature production environment.
Data Quality
Introduce explicit checks between Bronze and Silver:
Bronze → Validation → Valid (Silver) / Invalid (DLQ)
Useful metrics would include:
- rejected record count,
- null-rate changes,
- schema changes,
- duplicate rates,
- data freshness,
- and row-count anomalies.
Observability
Infrastructure health is only one part of observability.
The platform should also measure:
Pipeline freshness / failures, Record counts, Processing latency, Iceberg snapshot growth, Small-file counts, Query latency, Compute utilization
Iceberg Maintenance
Streaming and frequent CDC ingestion can produce many small files.
Over time, maintenance jobs should handle tasks such as:
Data file compaction, Snapshot expiration, Orphan file cleanup, Metadata cleanup
Transformation Orchestration
A complete production implementation should add an orchestration layer (e.g. Dagster or Airflow) responsible for transformations (Bronze → Silver → Gold), providing scheduling, dependencies, retries, and backfills.
Final Architecture
Putting everything together:
flowchart TB
subgraph Sources
PG[(PostgreSQL)]
MYSQL[(MySQL)]
MONGO[(MongoDB)]
SAAS[SaaS / APIs]
end
subgraph PrivateGCP["GCP Private Network"]
AIRBYTE[Airbyte]
subgraph Lakehouse
POLARIS[Apache Polaris]
GCS[(Google Cloud Storage)]
end
subgraph StarRocksCluster["StarRocks"]
FE[Frontend]
CN1[Compute Node]
CN2[Compute Node]
end
SUPERSET[Apache Superset]
end
PG --> AIRBYTE
MYSQL --> AIRBYTE
MONGO --> AIRBYTE
SAAS --> AIRBYTE
AIRBYTE --> POLARIS
POLARIS --> GCS
FE --> POLARIS
CN1 --> GCS
CN2 --> GCS
GCS -->|Bronze| GCS
GCS -->|Silver| GCS
GCS -->|Gold| GCS
SUPERSET --> FE
The final platform combines:
Open storage + Open table format + Open catalog + Independent compute + CDC ingestion + Medallion architecture + Infrastructure as Code
The most valuable takeaway from this project was not learning how to install each individual component.
It was understanding how those components divide responsibilities inside a modern data platform.
GCS stores the bytes.
Iceberg defines the tables.
Polaris coordinates their metadata.
StarRocks provides analytical compute.
Airbyte moves data into the platform.
Superset serves analytical users.
Terraform makes the infrastructure reproducible.
That separation is what makes the architecture interesting: each layer can evolve independently, while the underlying data remains stored in an open format.