The Databricks Certified Data Engineer Professional Exam (Databricks-Certified-Professional-Data-Engineer)

In the Databricks Git folders integration, when merge conflicts arise in notebooks, the UI provides a visual diff editor that highlights conflicting code segments. Users can manually choose which changes to keep from each branch, edit directly in the notebook UI, and remove conflict markers.

After resolving, the engineer must mark the conflict as resolved, save, and commit the final version.

This process ensures that both contributors’ valid code segments are merged correctly and version history is maintained.

Forcing a push (C) or deleting notebooks (B) introduces data loss or versioning issues. Aborting without review (A) violates collaborative best practices. Therefore, D is the only correct and Databricks-approved way to resolve notebook merge conflicts.

In Delta Lake , the property delta.deletedFileRetentionDuration controls how long deleted data files are retained before being permanently removed during a VACUUM operation.

By default, this retention duration is set to 7 days .

To comply with stricter retention requirements, organizations can explicitly update the table property using an ALTER TABLE statement.

Option A uses the correct SQL command:

ALTER TABLE my_table SET TBLPROPERTIES ( ' delta.deletedFileRetentionDuration ' = ' interval 15 days ' )

This updates the Delta table metadata so that all future operations respect the 15-day retention policy for deleted files.

Why not the others?

B : This code incorrectly tries to set the property via the DeltaTable API. Delta’s Python API does not expose direct attributes like deletedFileRetentionDuration; instead, properties must be set through ALTER TABLE or DataFrameWriter options.

C : VACUUM ... RETAIN specifies a one-time file cleanup action (e.g., retaining 15 hours of history), not a persistent retention policy. It cannot be used to set a continuous retention duration .

D : Setting spark.conf applies a session-level configuration and does not permanently update the table’s retention metadata. Once the session ends, this configuration is lost.

Therefore, Option A is the correct and documented approach for persistently enforcing a 15-day deleted file retention period in Delta Lake.

This is the correct answer because the JSON posted to the Databricks REST API endpoint 2.0/jobs/create defines a new job with a name, an existing cluster id, and a notebook task. However, it does not specify any schedule or trigger for the job execution. Therefore, three new jobs with the same name and configuration will be created in the workspace, but none of them will be executed until they are manually triggered or scheduled. Verified References: [Databricks Certified Data Engineer Professional], under “Monitoring and Logging” section; [Databricks Documentation], under “Jobs API - Create” section.

Databricks Auto Loader simplifies and automates the process of loading data into Delta Lake. The default execution mode of the Auto Loader identifies new files by listing the input directory. It incrementally and idempotently loads these new files into the target Delta Lake table. This approach ensures that files are not missed and are processed exactly once, avoiding data duplication. The other options describe different mechanisms or integrations that are not part of the default behavior of the Auto Loader.

Deletion vectors (DVs) in Delta Lake optimize delete operations by marking deleted rows logically in metadata rather than rewriting Parquet files. When a DELETE statement is executed, affected rows are tracked by DVs in the transaction log. The data remains in the underlying files but is filtered out during query reads. This improves performance for frequent deletes and updates since file rewrites are deferred. Physical data removal only occurs when a VACUUM command is later executed. The Databricks documentation confirms: “With deletion vectors, deleted rows are marked in metadata and skipped at read time, avoiding file rewrites.” Thus, rows are marked as deleted in metadata—not in files.

Delta Lake ' s Liquid Clustering is an advanced feature that improves query performance by dynamically clustering data without requiring costly compaction steps like traditional Z-ordering.

When performing writes to a Liquid Clustered table, some write operations automatically maintain clustering, while others do not .

Explanation of Each Option:

(A) spark.writestream.format( ' delta ' ).mode( ' append ' ) (Correct Answer)

Reason: Streaming writes (writestream) do not support Liquid Clustering because streaming data arrives in micro-batches.

Since Liquid Clustering needs efficient global reorganization of files, streaming append operations don ' t provide sufficient data volume at a time to be effectively clustered.

Delta Lake documentation states that Liquid Clustering is only supported for batch writes.

(B) CTAS and RTAS statements

Reason: CREATE TABLE AS SELECT (CTAS) and REPLACE TABLE AS SELECT (RTAS) are batch operations and can enforce Liquid Clustering.

These operations create or replace a table based on a query result, and since they are batch-based, Liquid Clustering applies.

(C) INSERT INTO operations

Reason: INSERT INTO is supported for Liquid Clustering because it is a batch operation.

While it may not be as efficient as MERGE or COPY INTO, clustering is applied upon execution.

(D) spark.write.format( ' delta ' ).mode( ' append ' )

Reason: Batch append operations are supported for Liquid Clustering.

Unlike streaming append, batch writes allow the optimizer to re-cluster data efficiently.

Conclusion:

Since streaming append operations do not support Liquid Clustering , option (A) is the correct answer.

Lakeflow Declarative Pipelines (DLT) supports data quality enforcement using @dlt.expect, @dlt.expect_or_drop, and @dlt.expect_all.

@dlt.expect applies a rule and records whether rows pass or fail the condition but does not drop failing rows . Instead, failing rows can be written to a quarantine table.

@dlt.expect_or_drop enforces that only rows passing the condition flow downstream, dropping bad records automatically.

In this case, the requirement is:

Good rows (reading < 120) go to the silver table .

Bad rows (reading > = 120) go to a quarantine table .

Bad rows should not be included in silver.

The correct implementation is Option A , where:

The silver table uses @dlt.expect to validate reading < 120. These rows flow normally.

The quarantine table applies an expectation for reading > = 120, ensuring bad records are captured separately.

Other options are incorrect:

Option B/D: These either use expect_or_drop incorrectly or apply wrong conditions, leading to dropped rows without quarantining properly.

Option C: Uses expect_or_drop for both tables, which would discard bad rows instead of persisting them into a quarantine table.

Thus, Option A meets the business requirement to split good and bad data streams while ensuring both are captured for auditing and processing.

DBFS is a file system protocol that allows users to interact with files stored in object storage using syntax and guarantees similar to Unix file systems 1 .  DBFS is not a physical file system, but a layer over the object storage that provides a unified view of data across different data sources 1 .  By default, the DBFS root is accessible to all users in the workspace, and the access to mounted data sources depends on the permissions of the storage account or container 2 .  Mounted storage volumes do not need to have full public read and write permissions, but they do require a valid connection string or access key to be provided when mounting 3 .  Both the DBFS root and mounted storage can be accessed when using %sh in a Databricks notebook, as long as the cluster has FUSE enabled 4 .  The DBFS root does not store files in ephemeral block volumes attached to the driver, but in the object storage associated with the workspace 1 .  Mounted directories will persist saved data to external storage between sessions, unless they are unmounted or deleted 3 . References:  DBFS ,  Work with files on Azure Databricks ,  Mounting cloud object storage on Azure Databricks ,  Access DBFS with FUSE

Databricks Jobs supports defining multi-task workflows that include notebooks, SQL statements, and Python wheel tasks. These can be configured with retry policies, dependency chains, and failure notifications. The correct practice, as stated in the documentation, is to use the Jobs REST API (/jobs/create) or Databricks Asset Bundles to define multi-task jobs, and then trigger them programmatically using /jobs/run-now, CLI, or SDK. This allows the team to maintain full job history, handle retries automatically, and receive alerts via configured email notifications. Using /jobs/runs/submit creates one-off ad hoc runs without maintaining dependency visibility. Therefore, options B and C together satisfy the operational, automation, and governance requirements.