Databricks Data Engineer Interview Questions

I would start by checking whether the slowdown comes from shuffle, skew, small files, or memory pressure rather than assuming the whole job is bad. Then I would inspect the physical plan, stage metrics, executor logs, and input data distribution to find the actual bottleneck. In practice, the fix is usually to reduce shuffle cost, repartition more intelligently, or change the data layout so the job scales with the larger volume instead of fighting it.

repartition reshuffles data across the cluster and can increase or decrease the number of partitions, while coalesce usually reduces partitions with less data movement. I would use repartition when I need a better distribution, especially before wide transformations or writes, and coalesce when I only want to reduce output file count cheaply. The trade-off is that repartition is more expensive but gives better balance.

I would first confirm skew with task duration and shuffle size metrics, then identify which keys are oversized. If possible, I would broadcast the small side, salt the skewed keys, or pre-aggregate before the join to reduce the imbalance. The main idea is to stop a few partitions from doing almost all the work while the rest sit idle.

I would use a broadcast join when one side is small enough to fit comfortably in executor memory and broadcasting it avoids a large shuffle. The risk is that broadcasting a table that is too large can cause memory pressure or even executor failures. So I would only use it when the size is predictable and I have verified that the optimizer's choice is safe for production.

Delta Lake adds a transaction log, which gives ACID guarantees, schema enforcement, and time travel on top of object storage. That makes reads and writes more reliable because jobs can reason about consistent table versions instead of raw files that may be partially written or concurrently modified. It also makes recovery and auditing much easier in production.