When Does a Data Engineering Take-Home Become Overengineering?

Data Engineering Take-Home Architecture

There is a common temptation when working on a Data Engineering take-home assessment:

If I add more infrastructure, maybe the solution will look more impressive.

A simple ETL assignment can quickly turn into:

Airflow
Kafka
Debezium
Spark
dbt
ClickHouse
Iceberg
Kubernetes
Observability
Data Quality
CI/CD

At some point, the architecture diagram becomes larger than the actual problem.

But there is another mistake on the opposite side.

Sometimes the minimum implementation technically satisfies the assignment, but it misses an opportunity to demonstrate experience that is directly relevant to the role.

I encountered both situations while working on two Data Engineering take-home projects.

The first had a mostly prescribed technology stack.

The second was much more open-ended, with Airflow as one of the main required technologies.

The two assessments taught me something useful:

The question is not whether a take-home solution is complex. The question is whether every piece of complexity is there for a reason.

Sometimes going beyond the written requirement is justified.

Sometimes deliberately not building something is the better engineering decision.

There Are Actually Two Specifications

When working on a take-home assignment, it is easy to treat the assessment document as the complete specification.

In practice, I think there are two specifications:

Written Assessment
        +
Interview Context

The written assessment tells us what must be delivered.

The interviews tell us something different:

what the team actually operates,
what problems they are currently solving,
which technologies matter to them,
what kind of architecture they work with,
and which parts of our previous experience they are probably trying to validate.

I started treating both as inputs to the architecture.

This became especially important in the first assessment.

Case Study 1: A Constrained Data Engineering Stack

The first assessment was a Python/Data Engineering case study for a fintech-style system.

The required architecture was already fairly specific:

MySQL
MongoDB
Python
Dagster
ClickHouse

The pipeline needed to demonstrate:

data modeling,
batch ingestion,
incremental loading,
checkpointing,
CDC understanding,
basic streaming concepts,
modular Python code,
logging,
documentation,
and maintainability.

The expected data flow was roughly:

flowchart LR
    MYSQL[(MySQL)] --> ETL[Python ETL]
    MONGO[(MongoDB)] --> ETL
    ETL --> CH[(ClickHouse)]
    DAGSTER[Dagster] --> ETL

The assessment also explicitly allowed simpler implementations.

For example, ClickHouse could be simulated using CSV if necessary.

CDC did not need to be implemented using a real change-data-capture system either.

The requirement allowed something similar to:

JSON Lines
    ↓
Python Consumer
    ↓
ClickHouse

or a dummy Kafka stream.

From a purely requirement-driven perspective, that would have been enough.

The Minimum-Compliant Solution

A reasonable minimal implementation could have looked like this:

flowchart LR
    MYSQL[(MySQL)] --> PY[Python ETL]
    MONGO[(MongoDB)] --> PY
    PY --> CH[(ClickHouse)]

    CP[(Checkpoint Table)] --> PY

    JSONL[Simulated CDC JSONL] --> CDC[Python CDC Consumer]
    CDC --> CH

The incremental pipeline could track something like:

last_successful_timestamp

and subsequent runs could query:

WHERE updated_at > :last_checkpoint

That would demonstrate:

ETL,
incremental processing,
checkpointing,
basic duplicate handling,
and conceptual CDC understanding.

Technically, that would satisfy the assessment.

But I deliberately went further.

Why I Implemented Real CDC

During the user interview before the take-home assessment, the team explained that their actual CDC architecture used:

Debezium
   +
Kafka

That changed how I interpreted the optional CDC part of the assignment.

I already had Kafka and Debezium experience on my CV.

But there is a difference between writing:

Implemented CDC pipelines using Kafka and Debezium

on a résumé and showing:

Here is a working implementation.

So instead of simulating CDC with JSON files, I implemented an actual CDC path.

The final architecture became closer to:

flowchart LR

    subgraph Batch["Batch / Incremental Path"]
        MYSQL[(MySQL)]
        DAGSTER[Dagster]
        MYSQL --> DAGSTER
    end

    subgraph CDC["CDC Path"]
        MONGO[(MongoDB)]
        DBZ[Debezium]
        KAFKA[(Kafka)]
        SINK[ClickHouse Sink Connector]

        MONGO --> DBZ
        DBZ --> KAFKA
        KAFKA --> SINK
    end

    subgraph Warehouse["ClickHouse"]
        BRONZE[(Bronze)]
        SILVER[(Silver)]
    end

    DAGSTER --> BRONZE
    SINK --> BRONZE
    BRONZE --> SILVER

The MySQL path uses Dagster for batch and incremental loading.

The MongoDB path uses:

MongoDB
   ↓
Debezium
   ↓
Kafka
   ↓
ClickHouse Sink Connector
   ↓
ClickHouse Bronze

Dagster then handles downstream transformation and orchestration.

This was significantly more infrastructure than the minimum assessment required.

So was it overengineering?

Was This Overengineering?

Relative to the written requirement:

Yes, it was more than necessary.

The assessment explicitly allowed simulated CDC.

I could have implemented the feature with much less infrastructure.

But the more useful question is:

What did the additional complexity demonstrate?

In this case, it demonstrated something directly related to the target environment.

The additional components were not arbitrary.

Debezium
    ↓
demonstrates log/change-stream based CDC

Kafka
    ↓
demonstrates event transport and decoupling

ClickHouse Sink Connector
    ↓
demonstrates sink integration without a custom consumer

Dagster
    ↓
demonstrates orchestration outside the continuous CDC path

The implementation also made an architectural boundary much clearer.

CDC Is Not Just ETL Running Very Frequently

One reason I wanted to implement the CDC path properly was to demonstrate the difference between orchestration and continuous ingestion.

A scheduled ETL job has a bounded lifecycle:

Start
  ↓
Extract
  ↓
Transform
  ↓
Load
  ↓
Finish

For example:

flowchart LR
    START([Start]) --> EXTRACT[Extract]
    EXTRACT --> TRANSFORM[Transform]
    TRANSFORM --> LOAD[Load]
    LOAD --> END([Finish])

CDC behaves differently.

Conceptually:

Change
 ↓
Change
 ↓
Change
 ↓
Change
 ↓
...

There is no natural "job completed" state.

A CDC service continuously listens for changes.

That is why I separated:

Continuous ingestion

from:

Scheduled orchestration

Debezium and Kafka handle the continuous change stream.

Dagster handles bounded jobs such as transformations, incremental batch processing, monitoring, and dependencies.

The distinction is more important than the specific tools.

Complexity Should Buy Something

One rule I started using after these projects is:

Every additional component must buy a capability, guarantee, or useful signal.

For the first project:

Component	What It Buys
Dagster	Scheduling, orchestration, dependencies, retries
Checkpoint table	Durable incremental state
Bronze layer	Raw ingestion boundary
Silver layer	Cleaned and deduplicated analytical representation
Debezium	Actual CDC semantics
Kafka	Durable and decoupled change-event transport
ClickHouse Sink Connector	Direct Kafka-to-ClickHouse ingestion
Docker Compose	Reproducible local infrastructure

A useful test is:

If I remove this component, what capability disappears?

If the answer is:

Nothing important, but the diagram becomes less impressive.

then that component probably should not exist.

Architecture as Communication

A take-home project is unusual because its architecture has two jobs.

It has a technical function:

Does the system work?

But it also has a communication function:

What does this implementation tell the interviewer?

For example:

Checkpoint Table

communicates:

I understand that incremental pipelines need durable state.

Debezium + Kafka

communicates:

I understand the CDC architecture discussed during the interview and can implement it.

Bronze → Silver

communicates:

I separate ingestion concerns from transformation concerns.

That does not mean every take-home should become a production platform.

It means that extra work should send a useful signal.

Going Beyond the Requirement Should Be Intentional

I would not automatically make the same CDC decision for every assessment.

If the assignment had a four-hour time limit, I would probably choose something much simpler.

For example:

changes.jsonl
      ↓
Python Consumer
      ↓
ClickHouse

If Kafka knowledge itself were being tested:

Producer
   ↓
Kafka
   ↓
Python Consumer
   ↓
ClickHouse

If CDC were central to the role and the team explicitly used Debezium:

Database
   ↓
Debezium
   ↓
Kafka
   ↓
Sink
   ↓
Warehouse

The architecture should depend on:

flowchart TD
    REQ[Assessment Requirements] --> DECISION[Architecture Decision]
    INTERVIEW[Interview Context] --> DECISION
    TIME[Available Time] --> DECISION
    SIGNAL[Skills Worth Demonstrating] --> DECISION

    DECISION --> IMPL[Minimum Useful Implementation]

Not:

Technologies I know
        ↓
Technologies I want to show
        ↓
Massive docker-compose.yml

Case Study 2: The Opposite Problem

The second take-home assessment had almost the opposite constraint.

Airflow was required.

But most architectural choices were left open.

That sounds easier.

In some ways, it is more dangerous.

When the architecture is open-ended, there is almost no limit to what can be added:

Airflow
Spark
Kafka
dbt
ClickHouse
BigQuery
Iceberg
Trino
Data Quality
Streaming
Lakehouse
Feature Store

Almost every tool can be justified if we try hard enough.

The real challenge becomes:

What should I deliberately not build?

The Interview Context Was Different

The user interview for this role focused much more on a migration-style workload.

The environment discussed was closer to:

Legacy Hadoop / Cloudera Platform
              ↓
         Cloud Migration
              ↓
              GCP

with technologies and concepts around:

Google Cloud Storage
BigQuery
Spark ETL
Batch Processing
Data Migration

CDC was not the center of the discussion.

Real-time processing was not the primary problem either.

That heavily influenced how I approached the take-home.

Instead of building a streaming-first platform, I focused the core architecture on:

batch ETL,
data organization,
dimensional modeling,
analytical storage,
orchestration,
and transformations.

Designing Around the Workload

The main pipeline became:

flowchart TD

    MYSQL[(MySQL Source)]
        --> AIRFLOW[Apache Airflow]

    AIRFLOW --> BRONZE[(ClickHouse Bronze)]

    BRONZE --> SILVER[Silver Transformations]

    SILVER --> DIM_CUSTOMER[(gold.dim_customer)]
    SILVER --> DIM_PRODUCT[(gold.dim_product)]
    SILVER --> DIM_DATE[(gold.dim_date)]
    SILVER --> DIM_CAMPAIGN[(gold.dim_campaign)]

    DIM_CUSTOMER --> FACT[(gold.fact_sales)]
    DIM_PRODUCT --> FACT
    DIM_DATE --> FACT
    DIM_CAMPAIGN --> FACT

The Gold layer uses a star schema.

Conceptually:

erDiagram
    DIM_CUSTOMER ||--o{ FACT_SALES : customer_key
    DIM_PRODUCT ||--o{ FACT_SALES : product_key
    DIM_DATE ||--o{ FACT_SALES : date_key
    DIM_CAMPAIGN ||--o{ FACT_SALES : campaign_key

This architecture is very different from the CDC-oriented first project.

And that was intentional.

Why I Used Medallion Architecture

I used:

Bronze → Silver → Gold

because the role discussion suggested a data-lake/data-platform style environment.

The interview focused on moving away from a legacy Hadoop/Cloudera environment toward a GCP architecture using GCS, BigQuery, and Spark.

I did not try to reproduce the production environment exactly.

Instead, I tried to reflect the same kind of architectural concerns.

A layered model maps naturally to those concerns.

Bronze: Raw Landing

Bronze represents the ingested source data.

Source
  ↓
Bronze

The purpose is to preserve a representation close to the upstream system before analytical transformations are applied.

This creates a useful boundary:

Bronze represents what arrived from the source.

Silver: Clean and Conformed

Silver handles tasks such as:

type normalization,
data cleansing,
standardization,
deduplication,
mapping,
and reusable transformation logic.

Bronze
  ↓
Cleaning
Standardization
Deduplication
  ↓
Silver

At this point, downstream consumers no longer need to repeatedly understand inconsistencies from the operational source.

Gold: Analytics Model

Gold serves a different purpose.

Instead of treating Gold as something that must exist simply because the pattern says:

Bronze → Silver → Gold

I used it because the project had an actual analytical serving requirement.

The Gold layer contains dimensions and facts:

dim_customer
dim_product
dim_campaign
dim_date
fact_sales

The flow becomes:

flowchart LR
    RAW[Raw Data] --> BRONZE[Bronze]
    BRONZE --> SILVER[Silver]
    SILVER --> GOLD[Gold Star Schema]
    GOLD --> ANALYTICS[Analytics Queries]

Medallion architecture is useful here because each layer has a clear responsibility.

But I do not consider Bronze, Silver, and Gold mandatory for every pipeline.

Patterns Become Dangerous When They Become Cargo Cults

A pattern is useful when it solves a problem.

It becomes dangerous when it becomes a checklist.

For example:

Data Lake = Bronze + Silver + Gold

is too simplistic.

Some systems may only need:

Raw → Clean

Others may need:

Raw → Standardized → Consumer-Specific Datasets

The labels matter less than the responsibilities.

The same applies to Kafka.

Real-Time = Kafka

is not a useful architectural rule.

Kafka becomes valuable when the system actually benefits from properties such as:

buffering,
replay,
durable event streams,
multiple independent consumers,
or decoupled producers and consumers.

Otherwise, adding Kafka can simply create another distributed system that someone needs to operate.

Why Real-Time Stayed Optional

The second project does contain an optional streaming path.

It looks roughly like:

flowchart LR
    PROD[Event Producer]
        --> KAFKA[(Kafka)]

    KAFKA --> CONSUMER[Python Streaming Consumer]

    CONSUMER -->|"1-Minute Window"| RT[(ClickHouse Real-Time Metrics)]

The consumer aggregates events into one-minute windows.

But this pipeline is intentionally separate from the primary batch architecture.

The main analytical flow does not depend on it.

That distinction matters.

I could have redesigned the entire solution around Kafka:

Everything
   ↓
Kafka
   ↓
Streaming
   ↓
Warehouse

But that would not have matched the strongest signals I received about the role.

The interview focused much more on migration, batch transformation, GCS, BigQuery, Spark, and analytical data platforms.

So real-time remained what it was in the assessment:

an optional extension.

This was a different form of engineering judgment from the first project.

In the first assessment, I deliberately expanded the CDC requirement because it mapped directly to the environment described during the interview.

In the second assessment, I deliberately did not make streaming the center of the architecture because the role did not appear to be centered around CDC.

The Same Technology Can Represent Different Problems

One interesting comparison between the two projects is Kafka.

Kafka appears in both architectures.

But it does not have the same responsibility.

In the First Project: Kafka as CDC Infrastructure

flowchart LR
    MONGO[(MongoDB)]
        --> DBZ[Debezium]

    DBZ --> KAFKA[(Kafka)]

    KAFKA --> SINK[ClickHouse Sink]

    SINK --> BRONZE[(Bronze)]

Kafka is primarily part of a change-data-capture pipeline.

Its responsibility is transporting database change events between the source connector and the sink.

In the Second Project: Kafka as Event Streaming Infrastructure

flowchart LR
    EVENT[Transaction Event]
        --> KAFKA[(Kafka)]

    KAFKA --> CONSUMER[Streaming Consumer]

    CONSUMER --> AGG[Window Aggregation]

    AGG --> CH[(ClickHouse)]

Here Kafka is supporting event processing.

The pipeline performs a one-minute tumbling-window aggregation before writing analytical results.

Same technology.

Different architectural responsibility.

This is another reason I try not to make architecture decisions by starting from tools.

Start With Responsibilities, Not Products

Instead of asking:

Should I use Kafka?

I prefer asking:

Do I need durable asynchronous event transport?

Instead of:

Should I use Airflow or Dagster?

I ask:

Do I need scheduling, dependencies, retries, backfills, and observable job execution?

Instead of:

Should I build Bronze, Silver, and Gold?

I ask:

Do raw ingestion, conformed transformation, and consumer-facing models need separate contracts?

The products come later.

A simplified decision process looks like:

flowchart TD
    PROBLEM[Problem] --> RESPONSIBILITY[Required Responsibility]
    RESPONSIBILITY --> GUARANTEE[Required Guarantee]
    GUARANTEE --> PATTERN[Architecture Pattern]
    PATTERN --> TOOL[Technology Choice]

Starting from the tool reverses that reasoning.

Production-Like Does Not Mean Production-Complete

Another trap in take-home assignments is trying to make everything "production-grade."

A real production platform may require:

High Availability
Secrets Management
Monitoring
Alerting
Schema Registry
Infrastructure as Code
CI/CD
Disaster Recovery
Load Testing
Autoscaling
Data Quality Platform
Lineage
Access Governance

Trying to build all of that in a take-home project would usually be unreasonable.

Instead, I prefer the idea of a:

production-inspired implementation

That means the important architectural boundaries are realistic, while operational completeness is intentionally limited.

For example, the first project demonstrates a real CDC topology.

It does not mean I need to build a multi-region Kafka cluster.

The second project demonstrates layered analytical modeling.

It does not mean I need to reproduce an entire enterprise cloud migration environment locally.

The goal is to demonstrate the engineering idea without pretending that a take-home project is a production platform.

What I Would Prioritize Before Adding More Infrastructure

If I had additional time on either assessment, I would prioritize correctness before adding more tools.

For incremental processing, I would care about questions such as:

What happens if the job fails after data is written
but before the checkpoint is updated?

or:

Can rerunning the same batch create duplicates?

or:

How do updates to existing records get captured?

These questions are more important than whether the pipeline has ten different infrastructure components.

Similarly, for modeling:

What is the grain of fact_sales?

is more important than:

Should I add another query engine?

A complex architecture with incorrect data semantics is still an incorrect architecture.

Architecture Theatre

The failure mode I try to avoid is what I would call architecture theatre.

For example:

flowchart LR
    SRC[Source]
        --> KAFKA[Kafka]
        --> SPARK[Spark]
        --> AIRFLOW[Airflow]
        --> DBT[dbt]
        --> ICEBERG[Iceberg]
        --> TRINO[Trino]
        --> WH[Warehouse]

Then someone asks:

Why Spark?

And the answer is:

Because this is a big-data architecture.

How much data?

500 rows.

Or:

Why Kafka?

Because it is real-time.

Does the workload require real-time data?

No.

That is where complexity stops being architecture and starts becoming decoration.

A Framework I Use for Extra Complexity

Before adding something beyond the minimum requirement, I now ask several questions.

1. Is It Explicitly Required?

If the assessment says:

Use Airflow

then Airflow is not optional.

If it says:

Use Dagster

then replacing Dagster with another orchestrator does not demonstrate better architecture.

It demonstrates that I ignored the specification.

2. Does It Solve a Real Technical Problem?

For example:

Checkpoint table

solves durable incremental-state tracking.

Debezium

solves continuous capture of source database changes.

Gold star schema

solves analytical usability.

If I cannot clearly state the problem, I probably do not need the component.

3. Does It Demonstrate Something Relevant to the Role?

This is where the interview context becomes useful.

For the first assessment:

The team uses Debezium + Kafka

therefore implementing:

Debezium + Kafka

provides relevant evidence.

For the second:

The team discusses:
legacy Hadoop migration
GCS
BigQuery
Spark

therefore emphasizing:

batch processing
layered data organization
analytical modeling
orchestration

provides a stronger signal than turning the whole solution into a streaming platform.

4. What Happens If I Remove It?

This is probably my favorite question.

If I remove Kafka from a CDC architecture:

What capability disappears?

Potentially:

buffering,
replay,
connector decoupling.

That is meaningful.

If I remove some random technology and nothing changes except the architecture diagram becomes smaller:

Remove it.

5. Can I Explain Its Failure Behavior?

Using a technology means inheriting its failure modes.

If I add Kafka, I should at least understand questions such as:

what happens when a consumer restarts?
where are offsets stored?
can events be delivered more than once?
how do duplicates affect the destination?
what happens when the sink is unavailable?

If I cannot explain those behaviors, adding the technology may actually weaken the solution.

6. Is It Taking Time Away From the Fundamentals?

Imagine spending hours building Kafka infrastructure while the incremental logic is:

WHERE created_at > last_sync

even though existing records can be updated.

That would be a bad trade.

I would rather have:

Correct incremental semantics
Simple architecture

than:

Impressive infrastructure
Incorrect data

Spend Complexity Where the Assessment Is Testing Understanding

A useful rule is:

Spend complexity where the assessment is testing understanding.

If the assessment focuses on incremental processing, spend effort on:

Watermarks
Checkpointing
Retries
Idempotency
Updates
Late-arriving data

If it focuses on dimensional modeling, spend effort on:

Fact grain
Dimensions
Business keys
Surrogate keys
Metrics
Analytical queries

If it focuses heavily on CDC, then a real Debezium pipeline may actually be worth the effort.

If the assessment never mentions container orchestration, deploying Kubernetes probably does not provide much additional signal.

What the Two Projects Taught Me

The projects ended up teaching almost opposite lessons.

Project One: Working Inside Constraints

The technology stack was mostly predetermined.

The interesting decisions were hidden inside the implementation.

Questions included:

How should incremental state be stored?

How should batch and CDC responsibilities be separated?

Should CDC only be simulated?

Where should transformations happen?

How should duplicates be handled?

How should Bronze and Silver behave?

Going beyond the CDC simulation requirement was intentional because it aligned directly with what the team described during the interview.

It turned a résumé claim into something observable.

Project Two: Creating My Own Constraints

The second assessment offered much more architectural freedom.

That meant I had to define my own boundaries.

Instead of asking:

What else can I add?

I needed to ask:

What architecture best demonstrates the type of work discussed during the interview?

The role appeared much closer to:

Legacy Data Platform
        ↓
Cloud Migration
        ↓
Batch Data Platform
        ↓
Analytical Warehouse

So the core solution emphasized:

Airflow
Batch ETL
Medallion-style layers
Dimensional modeling
ClickHouse analytics

Streaming remained optional.

That was deliberate.

The Interview Should Influence the Architecture

One lesson I would carry into future assessments is that interview conversations are architectural input.

The written requirement may say:

Simulate CDC.

The interview may reveal:

We use Debezium and Kafka heavily in production.

Those two pieces of information together can justify a more complete CDC implementation.

Likewise, a technically open assignment may offer dozens of possible architectures.

But if the team spends most of the interview discussing:

Hadoop migration
GCS
BigQuery
Spark
batch ETL

that is probably a stronger signal than whatever architecture is currently popular on social media.

I was not trying to reproduce either production environment.

Instead:

I was trying to demonstrate that I understood the kind of engineering problems their environment creates.

The Best Take-Home Is Not the Biggest One

I no longer think the right goal is:

Build the most production-like platform possible.

Nor is it:

Do only the absolute minimum.

The better target is somewhere between them.

Minimum Requirement
        ↓
Correct Solution
        ↓
Relevant Extension
        ↓
Role-Specific Demonstration
        ↓
Stop

Past that point lies:

Architecture Theatre
        ↓
Overengineering

The exact boundary depends on:

the assignment,
the available time,
the interview context,
the role,
and what skills are actually worth demonstrating.

Final Takeaway

These two take-home projects used different architectures, but the most useful lesson was not about Dagster versus Airflow or batch versus streaming.

It was about deciding where complexity belongs.

For the first project, implementing real Debezium and Kafka CDC went beyond the minimum written requirement.

But the additional complexity had a specific job:

Demonstrate a production pattern
the team explicitly said they used.

For the second project, the open-ended architecture gave me enough freedom to build a much larger streaming platform.

I deliberately did not.

The interview context suggested that migration, batch ETL, data-lake-style organization, and analytical modeling were more relevant to the actual work.

That led me to a simple rule:

Every box in an architecture diagram should have a reason to exist.

That reason might be:

A requirement

A correctness guarantee

A role-specific skill worth demonstrating

A real architectural concern discussed during the interview

If I cannot explain which one it is, the box probably does not belong there.

The goal of a Data Engineering take-home is not to prove that I can deploy every technology in the modern data stack.

It is to demonstrate that I can understand a problem, identify what matters, choose appropriate boundaries, and spend complexity where it produces value.

Or, more simply:

Good engineering is not knowing how to add more architecture. It is knowing when to stop.