🔁 Four renames you need to know
High Concurrency cluster (Q9)Retired — access modes: Standard / Dedicated
ZORDER (Q18)Liquid clustering (recommended)
Delta Live Tables / DLT (Q22)Lakeflow Declarative Pipelines
APPLY CHANGES INTO (Q22)AUTO CDC (same syntax)
📌 Sections: Platform & Architecture · Clusters & Compute · Spark Core · Delta Lake · Governance & Advanced
Section 1 · Platform Fundamentals & Architecture
1. Migration Justification
Scenario. Your team is migrating a critical ETL pipeline from open-source Spark on EC2 to Databricks. The CTO asks, "Why are we paying for Databricks when we already have Spark?"
What specific architectural and performance advantages would you highlight to justify the cost and migration effort?
- Runtime is not stock Spark — Databricks Runtime is a forked, optimised Spark — Photon is a C++ vectorised engine, and AQE, dynamic file pruning and the Delta cache have no open-source equivalent at that maturity.
- Delta Lake plus governance — You stop paying engineers to run Spark: cluster provisioning, autoscaling, runtime upgrades, Delta Lake ACID and Unity Catalog governance arrive as product, not as your backlog.
- The cost argument — Frame it as total cost, not licence cost — DBUs on top of EC2 versus the salaried time your team currently spends on cluster babysitting and failed-job forensics.
🎯 Interview tip. Never answer this with a feature list. Put a number on the engineering time you get back, and concede the honest case: if you run one nightly job and have a strong platform team, staying on EMR can be the right call.
2. New Hire Onboarding
Scenario. A new data engineer joins and asks, "Where do I write code, how do I run it, and where is my actual data stored?"
Walk them through the Databricks workspace layout, explaining how Notebooks, Clusters, and DBFS interact with each other.
- Where you write — The notebook is only an editor — it holds code and results, and lives in the workspace file tree or a Git folder, never on the cluster.
- Where it runs — The cluster is the compute you attach the notebook to: a driver that plans and workers that execute. Detach and reattach and your variables are gone, because state lives in the cluster's JVM.
- Where data lives — Your data is in cloud object storage — S3, ADLS or GCS — registered through Unity Catalog as tables and volumes. DBFS is the legacy mount layer, still there but no longer where new work should point.
🎯 Interview tip. The sentence that makes it click for a new starter: notebook is the keyboard, cluster is the engine, object storage is the disk. Three separate things people assume are one.
3. Collaborative Workspace Chaos
Scenario. Your team of 10 keeps accidentally overwriting each other's production notebooks, leading to frequent broken pipelines.
How would you restructure the workspace (folder hierarchy, Git integration, and permissions) to enable safe, collaborative development with proper code reviews?
- Git is the fix — The real problem is that production code is editable in place. Put every pipeline in a Git folder so the repository, not the workspace, is the source of truth.
- Folder and permission model — Developers work in their own user folder on a feature branch; production lives in a folder where the team has run-only permission and nobody has CAN EDIT.
- Deploy, don't edit — Promote with Databricks Asset Bundles from CI, so a merged pull request deploys the job — the same artifact to each environment, reviewed before it moves.
🎯 Interview tip. Say the words pull request and no edit rights in production. Interviewers are checking whether you have worked somewhere that had an outage caused by exactly this.
4. Rotating Cloud Credentials
Scenario. Your company rotates AWS IAM keys every 30 days, and your current ETL scripts have hardcoded S3 paths that break whenever keys change.
How would you configure DBFS mounts or Unity Catalog external locations so that pipeline code never needs to be updated when keys rotate?
- Stop using keys — The fix is to stop using keys at all. A Unity Catalog storage credential wraps an IAM role, and the role is assumed at runtime, so there is no secret to rotate.
- External locations — Define an external location over the bucket path, grant on it, and register tables and volumes above it — governance and the path live in one place.
- Code references names — Pipeline code then references catalog.schema.table or a volume path, never a bucket URL, so rotating anything underneath is invisible to the code.
🎯 Interview tip. Name DBFS mounts as the legacy answer and move past them quickly. Mounts store the credential in the workspace, which is precisely the thing the question is complaining about.
5. Underwhelming Performance Gains
Scenario. You migrate a 2-hour Spark job to Databricks Runtime. With the exact same instance types, it now runs in 1.5 hours, but leadership expected a much bigger boost.
How would you leverage Databricks-specific features (like Photon, Adaptive Query Execution, or optimized Shuffle) to push this job under 30 minutes?
- Measure before tuning — A lift and shift only buys you the runtime. Open the Spark UI first and find whether the time is in scan, shuffle or write — tuning before measuring is guessing.
- Turn the engines on — Enable Photon for the scan, filter, join and aggregate work, and let AQE coalesce partitions, switch to broadcast joins and split skewed partitions at runtime.
- Fix the layout — The biggest single win is usually format, not compute: land the source in Delta with liquid clustering on the filter columns so the job reads a fraction of the files.
🎯 Interview tip. Set the expectation honestly. Photon accelerates SQL and DataFrame operations, not Python UDFs — if the job is a wall of UDFs, rewriting them is the optimisation, not a config flag.
Section 2 · Clusters & Compute
6. Driver Out-of-Memory Failure
Scenario. Your streaming job crashes daily with a java.lang.OutOfMemoryError: GC overhead limit exceeded on the Driver node.
Explain the Driver's specific responsibilities and walk through how you would reconfigure the cluster or rewrite the code to fix this permanently.
- What the driver does — The driver holds the SparkSession, builds the DAG, schedules tasks and collects results — it is a coordinator, and it is a single point of failure.
- What fills it — It fills up because code pulls data back to it: collect, toPandas, a large broadcast, or thousands of tiny streaming batches accumulating query state.
- The real fix — Rewrite so results are written from executors rather than collected, cap the broadcast threshold, and for streaming set a sensible maxFilesPerTrigger so batches stay bounded.
🎯 Interview tip. Raising driver memory is the answer that gets you a follow-up. Say it buys time, then name the collect. On a streaming job, a daily crash usually means something grows per batch and is never released.
7. Cost vs Latency Trade-off
Scenario. Your nightly ETL costs $100 per run on a large interactive cluster. Your data scientists also run heavy ad-hoc queries on this same cluster during the day.
How would you provision separate Interactive vs. Job clusters to cut infrastructure costs by 60% while keeping data scientists productive?
- Split the workloads — One cluster serving both is the whole problem: it is sized for the night's peak and sits idle-but-billing all day.
- Job clusters are cheaper — Move the ETL to a job cluster that is created for the run and terminated after it — it is billed at the lower jobs rate and cannot be left running by accident.
- Keep scientists fast — Give the scientists a smaller all-purpose cluster with aggressive auto-termination, or serverless, so they get a warm environment without paying for the ETL's shape.
🎯 Interview tip. Quote the two levers that actually produce the 60%: the jobs DBU rate is materially cheaper than all-purpose, and auto-termination removes the idle hours nobody notices they are paying for.
8. Unpredictable Daily Data Volume
Scenario. Some days your batch job processes 1 TB, other days it processes 10 TB. If you set a fixed cluster size, you either waste money or the job times out.
How would you configure Auto Scaling (min/max workers and spot instance usage) to handle this unpredictable volume dynamically?
- Min, max and why — Set the minimum to what a 1 TB night needs so small runs start immediately, and the maximum to what a 10 TB night needs — you pay for what scales up, not for the ceiling.
- Spot with a safe driver — Run workers on spot with a fallback to on-demand, but keep the driver on-demand always: losing the driver kills the job, losing a worker only costs a retry.
- Let AQE size the work — Pair it with AQE so partition counts adapt to the actual data volume rather than a number tuned for one particular night.
🎯 Interview tip. Driver on-demand, workers on spot is the sentence to have ready. It shows you have actually been burned by a spot reclamation rather than just read about the discount.
9. Mixed Workloads on One Cluster
Scenario. Your business analysts run complex SQL queries simultaneously on a shared cluster, while your ML engineer trains a heavy Spark ML model on the exact same data.
Why would you assign the analysts to a High Concurrency cluster and the ML training to a Standard or Single Node cluster? What specific risks do you avoid?
- rename The naming has moved — High Concurrency is retired. Access modes today are Standard (multi-user with isolation) and Dedicated (single user or group).
- Why separate them — Analysts want many small concurrent queries with fair scheduling, so a Standard cluster — or better, a serverless SQL warehouse — fits their shape.
- What you avoid — Training is one long memory-hungry job, so it belongs on a Dedicated cluster where it can consume the machine without starving anyone.
🎯 Interview tip. The risk to name is noisy neighbours: one training job takes the executors and every analyst query queues behind it, and a driver OOM in shared mode takes everybody down at once.
Section 3 · Spark Core Concepts
10. Debugging a Hung Job
Scenario. You chain 10 complex DataFrame transformations and finally call display(df). The job hangs for 5 minutes before showing any results. A junior asks, "Why didn't it fail or show progress earlier?"
Explain Lazy Evaluation and outline how you would use .cache() combined with an Action (like .count()) to pinpoint exactly which transformation is the bottleneck.
- Nothing ran yet — Transformations only build a plan. Nothing executed until display, which is why all ten steps appear to cost nothing and then the last line costs five minutes.
- Why it is a feature — It is deliberate: seeing the whole plan lets Catalyst reorder filters, prune columns and collapse steps in ways a step-by-step engine could not.
- Bisecting the chain — To bisect, cache an intermediate DataFrame and call count on it — that forces execution up to that point, so you can time each half of the chain separately.
🎯 Interview tip. Add the caveat that makes it correct: unpersist what you cache, and remember cache is itself lazy, so the timing only means something after the action has run.
11. Diagnosing a Slow Shuffle
Scenario. Your join operation is extremely slow. The Spark UI shows massive data spill to disk for a specific stage.
How do you differentiate between a wide dependency (shuffle) and a narrow dependency in the DAG? How would you check if this shuffle is caused by the join itself or a preceding repartition?
- Narrow vs wide — Narrow means each output partition reads one input partition — filter, select, union — and stays inside the stage. Wide means data crosses partitions, and that boundary is a new stage.
- Reading the DAG — In the DAG every stage boundary is a shuffle. Count the stages and you have counted the shuffles; the Exchange nodes in the physical plan name them.
- Which one caused it — Read the plan with explain: an Exchange hashpartitioning on the join key is the join's own shuffle, while a RoundRobinPartitioning is your repartition call.
🎯 Interview tip. Spill is the symptom worth explaining: partitions too big for executor memory. Say you would check spill alongside the max-versus-median task time, because spill plus skew is a different fix from spill alone.
12. Failed Stage Debugging
Scenario. Your job has 5 stages. Stage 3 fails with a task serialization error, but Stages 1 and 2 completed successfully.
How would you use the DAG visualization in the Spark UI to trace the lineage back to the exact line of code causing the failure?
- Start at the stage — Open the failed stage and read its DAG box — each stage shows the operators it contains, and the details pane carries the call site back to your notebook line.
- Map back to code — A task serialization error is almost always a closure problem: the function references something outside itself that cannot be pickled or sent to executors.
- What causes it — The usual culprits are a database connection, a file handle, or a Spark object referenced inside a UDF or foreach.
🎯 Interview tip. Name the fix, not just the cause: create the non-serialisable object inside the executor function, or broadcast the value if it is data. That distinction is the whole answer.
13. Post-Filter Repartitioning
Scenario. You read 200 partitions, apply a heavy filter that removes 99% of the data, leaving only 10 MB of actual data spread across all 200 partitions. For the next join, you only need 20 partitions.
Would you use repartition(20) or coalesce(20)? Justify your answer specifically regarding shuffle cost and execution speed.
- coalesce here — coalesce, because you are only reducing the count. It merges neighbouring partitions on the same executor with no shuffle at all.
- Why not repartition — repartition would do a full shuffle to get even sizes — pointless when the total is 10 MB, and it writes the whole set over the network to achieve nothing.
- The catch — The catch is that coalesce can pin the upstream work to fewer tasks, so if the filter itself is expensive you may lose more in parallelism than you save in shuffle.
🎯 Interview tip. Say you would let AQE handle it first. coalescePartitions does this automatically after a shuffle, and knowing that beats reciting the coalesce-versus-repartition table.
14. Forcing a Shuffle-Free Join
Scenario. You are joining a 1 TB fact table with a 10 MB country-code dimension table. The current job takes 30 minutes due to massive shuffling.
How would you force a Broadcast Join using a SQL hint (/*+ BROADCAST */), and how would you verify in the Spark UI that the shuffle was completely eliminated?
- How to force it — Put the hint on the small side — a SQL comment hint naming the dimension, or broadcast(df) in the DataFrame API.
- Why it is faster — The 10 MB table is sent whole to every executor, so the 1 TB side never moves. You trade one small copy for not shuffling a terabyte.
- How to verify — Verify in the physical plan: BroadcastHashJoin with a BroadcastExchange rather than SortMergeJoin, and the shuffle read on that stage drops to nothing.
🎯 Interview tip. Know why it was not automatic. The default autoBroadcastJoinThreshold is 10 MB, so a table right at the boundary — or one whose size stats are stale or missing — silently falls back to a sort-merge join.
Section 4 · Delta Lake
15. Corrupted Data Lake
Scenario. A failed ETL job wrote partial CSV files to your data lake yesterday, breaking all downstream reports. Your manager demands a solution to guarantee data reliability.
How would you pitch Delta Lake to solve this specific problem (focusing on ACID atomic writes and atomic replace/overwrite)?
- Why CSV failed — A plain file write has no commit. The job died halfway, the half-written files were already visible, and every reader picked them up as real data.
- What Delta changes — Delta adds a transaction log: files are written first, then a single atomic commit makes them visible. A failed job leaves no commit, so it leaves no data.
- Readers never see it — Readers get snapshot isolation — a query that starts during a write sees the previous consistent version, never a half-finished one.
🎯 Interview tip. Make it concrete rather than theoretical: with Delta, yesterday's failure would have been a job alert and nothing else, because the reports would have carried on reading the last good version.
16. Concurrent Write Conflicts
Scenario. Two production jobs try to write to the same Delta table at exactly 2:00 AM. One succeeds, but the other fails with a ConcurrentAppendException.
Explain optimistic concurrency control in Delta Lake. How would you modify the failing job's code to retry the write gracefully without manual intervention?
- Optimistic control — Delta assumes writers will not collide: each reads a snapshot, does its work, then checks at commit whether the files it depended on still hold. If they changed, it fails rather than corrupt.
- Why it conflicted — The exception means both jobs touched an overlapping set of files. It usually happens when a merge or update has no predicate narrow enough to separate them.
- How to fix it — The durable fix is to make the writes disjoint — partition or cluster so each job owns its slice, and give merges a predicate that names that slice explicitly.
🎯 Interview tip. Reach for retry second, not first. A blind retry loop on two jobs updating the same rows just moves the collision later; separating what each job writes is what actually removes it.
17. Bug Recovery Using Time Travel
Scenario. A bug in Wednesday morning's update corrupted the sales table. You need to restore Tuesday's correct state without losing valid changes that happened later on Wednesday afternoon.
Walk through how you would use Time Travel (DESCRIBE HISTORY, RESTORE with version/timestamp) to surgically recover the data.
- Find the version — Start with DESCRIBE HISTORY to list every version with its operation, timestamp and predicates, and identify the exact version before the bad write.
- Why not RESTORE — A blanket RESTORE is wrong here — it would roll the table back wholesale and discard Wednesday afternoon's valid changes along with the corruption.
- Surgical repair — Instead read the good version as a source, then MERGE just the affected rows back into the live table, so later legitimate changes survive.
🎯 Interview tip. Say you would take a copy of the current state before touching anything. Recovery goes wrong far more often than the original bug did, and RESTORE itself is just another commit you may need to undo.
18. Query Degradation Over Time
Scenario. Your Delta table queries used to take 2 seconds but now take 20 seconds. Upon inspection, you notice 10,000 small Parquet files in the directory.
Design a weekly maintenance strategy: when would you run OPTIMIZE, which columns would you apply Z-Ordering on, and how would you schedule VACUUM to safely clean up storage costs without breaking recent Time Travel queries?
- Compact the files — The cost is per-file overhead: 10,000 files means 10,000 open and read operations. OPTIMIZE compacts them into large files and the scan collapses.
- rename Clustering, not ZORDER — For layout, Databricks now recommends liquid clustering over both partitioning and ZORDER — CLUSTER BY on the columns you filter on, and you can change the keys later without rewriting.
- VACUUM safely — VACUUM removes files no longer referenced, but only past the retention window — the default seven days is also your time travel window, so shortening it shortens your recovery options.
🎯 Interview tip. Say liquid clustering and you sound current; say ZORDER only and you sound like 2022. Also mention predictive optimization, which runs this maintenance for you on managed tables.
19. Upstream Schema Change
Scenario. Your upstream source adds a new column discount_rate to the JSON files. Your existing auto-loader pipeline fails immediately due to a schema mismatch.
How would you use mergeSchema vs. overwriteSchema for an append-only Delta table? What are the specific risks associated with each approach?
- mergeSchema — mergeSchema adds the new column and backfills existing rows as null. It is additive and safe, and it is what an append-only table wants.
- overwriteSchema — overwriteSchema replaces the schema outright — it will drop columns and change types, so on an append table it is a data-loss button, not a migration tool.
- Auto Loader's own answer — In Auto Loader the cleaner answer is schema evolution: set a schema location, and with the default mode the stream fails once on the new column then restarts having learned it.
🎯 Interview tip. Name the risk in mergeSchema that people miss — it will happily accept a typo'd column name as a brand new column, so schema drift becomes silent column sprawl unless something is checking.
Section 5 · Governance, Security & Advanced Features
20. Cross-Workspace Access Control
Scenario. Your company has dev, test, and prod workspaces. You need to ensure that only HR users can view the salary column in the employees table, strictly enforced across all three workspaces.
How does Unity Catalog centralize this governance (metastore setup, privilege assignments, and lineage tracking) to fulfill this requirement?
- One metastore — One metastore per region, attached to all three workspaces, so identity and grants are defined once rather than three times in three places.
- Column masks — For a single column, a column mask on salary returns the real value when the user is in the HR group and a masked value otherwise — the rule lives on the table, not in each query.
- Lineage comes free — Because every query resolves through the catalog, you also get lineage and audit for free: who read salary, from which notebook or dashboard, and when.
🎯 Interview tip. The detail that shows real use: enforcement is at the table, so it holds no matter how the data is reached — SQL, a notebook, or a BI tool. Views you have to remember to point people at do not.
21. Designing a Lakehouse from Scratch
Scenario. You have messy, nested JSON logs arriving hourly from IoT devices. You are tasked with building a new Lakehouse architecture.
Design the Medallion (Bronze/Silver/Gold) pipeline. Specifically, how do you handle data quality checks (e.g., NULL foreign keys) in Silver, and what aggregation strategies do you apply in Gold?
- Bronze keeps everything — Bronze is the raw landing: append the JSON as it arrived, with ingest time and source file, and change nothing. It is your replay log when Silver logic turns out to be wrong.
- Silver is where quality lives — Silver is where you flatten the nesting, cast types, deduplicate on device and event time, and enforce quality — null foreign keys get quarantined to a rejects table, not dropped silently.
- Gold serves the question — Gold is shaped for consumers, not for purity: pre-aggregated by device, hour and region, as materialised tables the dashboard can hit without touching Silver.
🎯 Interview tip. Say why Bronze is immutable. Every team that transforms on the way in eventually needs to reprocess history and discovers the original is gone — that is the reason the layer exists.
22. Fragile Incremental Pipelines
Scenario. Your team writes custom Python code for incremental updates using watermarks and manual upserts. The pipeline frequently breaks when data arrives late (out of order).
How would Delta Live Tables (DLT) simplify this using its declarative syntax (APPLY CHANGES INTO for CDC) and built-in data quality expectations (EXPECT)?
- rename The current name: DLT is now Lakeflow Declarative Pipelines, and APPLY CHANGES INTO has been renamed AUTO CDC with the same syntax.
- AUTO CDC handles order — AUTO CDC handles out-of-order arrival for you — you declare the key and a SEQUENCE BY column, and it applies changes in sequence order regardless of arrival order.
- Expectations — Expectations move quality into the declaration: EXPECT with a constraint, and a choice to warn, drop the row, or fail the update, with the pass rate visible per run.
🎯 Interview tip. The real argument is that you stop writing orchestration. You declare the result, and the framework works out the dependency graph, the incremental logic and the retries — which is where the hand-rolled version keeps breaking.
23. Dashboard Performance SLA
Scenario. Your Tableau dashboard queries a 5 TB Delta table and takes 15 seconds to load. The business SLA requires sub-3-second response times.
How would you enable the Photon engine, and what cluster sizing/physical tuning (OPTIMIZE with Z-Order on filter columns) would you pair with it to meet the SLA?
- Serve it from SQL — Point Tableau at a serverless SQL warehouse rather than an all-purpose cluster: Photon is on by default there, it starts in seconds, and the result cache serves repeat queries instantly.
- Lay the data out — Then fix the layout so the query stops reading 5 TB — cluster on the columns the dashboard filters by, so file skipping does the work before Photon does.
- Pre-aggregate — If it still misses, the honest answer is that a dashboard should not scan a 5 TB fact table at all: build a Gold aggregate at the dashboard's grain.
🎯 Interview tip. Sequence matters and interviewers listen for it — layout first, engine second, hardware last. Scaling the warehouse to fix a full-table scan is the expensive way to not solve the problem.
24. High-Frequency File Ingestion
Scenario. Thousands of small JSON files land in S3 every hour. You need to process them continuously and incrementally.
Compare Auto Loader (file discovery) vs. Structured Streaming (processing). Can they be used together? Walk through a specific code/architecture approach for this incremental ingestion.
- Not alternatives — They are not competitors — Auto Loader is a Structured Streaming source. cloudFiles is the format you read, and the streaming engine is what processes it.
- How discovery works — Its value is discovery at scale: instead of listing a bucket with millions of objects, it uses file notifications and tracks what it has seen in RocksDB, so cost does not grow with directory size.
- The architecture — Architecture: readStream with cloudFiles and a schema location into Bronze, then a second stream from Bronze into Silver — with checkpoints on both so each restarts exactly where it stopped.
🎯 Interview tip. Mention Trigger.AvailableNow. For files arriving hourly you rarely want a cluster running all day; that trigger gives you streaming semantics and checkpointing on a scheduled batch.
25. Full Production Performance Triage
Scenario. Your production Spark job runs extremely slowly during peak business hours. You must fix it immediately.
Walk through your step-by-step triage process: (a) What do you check first in the Spark UI (Stages, Tasks, Shuffle Read/Write, Skew)? (b) How do you identify data skew in a specific key? (c) Which optimization would you apply first (salting, broadcast, repartition, or enabling AQE)—and justify your sequence of actions.
- Find the stage — Start at the Stages tab and sort by duration. In the slowest stage compare max task time against the median — if max is far above median, one task is doing everyone's work.
- Prove the skew — Prove it in the data rather than guessing: group by the join key, count, and order descending. A handful of keys holding most of the rows is your skew, and null is very often the worst offender.
- Fix in order — Then fix cheapest first — enable AQE, which splits skewed partitions and switches small joins to broadcast automatically. Broadcast the small side explicitly next. Salt only if skew survives both.
🎯 Interview tip. Order the actions by cost to reverse. AQE is a flag, broadcast is a hint, salting rewrites your logic and the person after you has to maintain it — so it is the last resort, not the clever first answer.
No comments:
Post a Comment