Microsoft Data Engineer Interview Questions

I would look for missing filters, poor partitioning, unnecessary columns, and join patterns that block pruning. Often the biggest win comes from rewriting the query so the engine can eliminate irrelevant data earlier.

I would partition on columns that match real query filters and support maintenance operations, usually time-based fields for event data. Good partitioning reflects the workload, not just the shape of the source system.

I would design a clear lateness window, allow backfills or reprocessing for affected partitions, and avoid finalizing metrics too early. Late-arriving data is normal, so the pipeline should correct outputs gracefully instead of ignoring it.

I would make writes atomic or merge-based, use deterministic transformations, and store processing state so reruns do not duplicate work. Safe reruns matter because production jobs fail for reasons that are often outside the code itself.

I would track freshness, success rate, volume changes, latency, failure counts, and business-level anomalies. Monitoring should tell me not just that a job ran, but whether the data looks healthy and useful.