Microsoft Data Engineer Interview Questions

I would validate schema, row counts, freshness, null rates, and business-rule consistency at key checkpoints. Data quality should be monitored continuously so issues are detected before they reach dashboards or downstream models.

Idempotency means rerunning the same operation produces the same final result. It matters because retries, reprocessing, and backfills are normal in production systems, and non-idempotent logic can create duplicate or inconsistent results.

I would inspect the physical plan, stage metrics, shuffle volume, and task durations to see where the time is going. A data growth spike often exposes skew, memory pressure, or a bad join strategy, so I would tune partitioning or reduce shuffle cost based on the evidence.

I would first confirm skew with task-level metrics, then decide whether to broadcast the small table, salt the hot keys, or pre-aggregate before the join. Skew is dangerous because a few partitions can dominate runtime while the rest finish quickly.

I would isolate each source into a consistent landing zone, validate the data before transformation, and capture source metadata for traceability. Since upstream systems fail in different ways, each source should be independently observable and recoverable.