Databricks Data Engineer Interview Questions

I would enforce access at the governance layer rather than relying on every downstream job to filter correctly. That usually means using catalog permissions, views, masking policies, or row filters so users only see what they are allowed to see. The important part is making the control systematic, auditable, and hard to bypass.

I would check whether the issue comes from too much data in a partition, a wide shuffle, large joins, or cached data that does not fit. Then I would look at the executor memory configuration, partition sizes, and whether the job is forcing too much state into one task. The fix may involve repartitioning, reducing shuffle pressure, increasing executor memory, or changing the algorithm.

I would use watermarking, change tracking, or a merge-based pattern depending on the source system. The goal is to process only new or changed records while keeping the target table consistent and replayable. Incremental loading is usually the right choice when the dataset is large and freshness matters.

Checkpointing stores progress metadata so a streaming job can resume safely after a failure. It preserves offsets, state, and execution progress, which is essential for exactly-once or effectively-once processing. Without checkpointing, a stream is much harder to recover reliably.

I would compare input rate, processing rate, and state growth to see whether the bottleneck is ingestion, computation, or output. Then I would inspect partitioning, shuffle load, state store size, and sink latency. Often the fix is to scale compute, reduce per-batch work, or simplify the stream so each micro-batch finishes faster.