The Databricks Certified Data Engineer Associate Exam (Databricks-Certified-Data-Engineer-Associate)

 To return the number of null values in the member_id column, the best option is to use the count_if function, which counts the number of rows that satisfy a given condition. In this case, the condition is that the member_id column is null. The other options are either incorrect or not supported by Spark SQL. Option A will return the number of non-null values in the member_id column. Option B will not work because there is no count_null function in Spark SQL. Option D will not work because there is no null function in Spark SQL. Option E will not work because there is no count_null function in Spark SQL. References:

Built-in Functions - Spark SQL, Built-in Functions

count_if - Spark SQL, Built-in Functions

The query performing a streaming hop from raw data to a Bronze table is identified by using the Spark streaming read capability and then writing to a Bronze table. Let ' s analyze the options:

Option A: Utilizes .writeStream but performs a complete aggregation which is more characteristic of a roll-up into a summarized table rather than a hop into a Bronze table.

Option B: Also uses .writeStream but calculates an average, which again does not typically represent the raw to Bronze transformation, which usually involves minimal transformations.

Option C: This uses a basic .write with .mode( " append " ) which is not a streaming operation, and hence not suitable for real-time streaming data transformation to a Bronze table.

Option D: It employs spark.readStream.load() to ingest raw data as a stream and then writes it out with .writeStream, which is a typical pattern for streaming data into a Bronze table where raw data is captured in real-time and minimal transformation is applied. This approach aligns with the concept of a Bronze table in a modern data architecture, where raw data is ingested continuously and stored in a more accessible format.

The Continuous Pipeline Mode for Delta Live Tables allows the pipeline to run continuously and process data as it arrives. This mode is suitable for streaming ingest and CDC workloads that require low-latency updates. The Development mode for Delta Live Tables allows the pipeline to run on a dedicated cluster that is not shared with other pipelines. This mode is useful for testing and debugging the pipeline logic before deploying it to production. Therefore, the correct answer is B, because the pipeline will run continuously on a dedicated cluster until it is manually stopped, and the compute resources will be released only after the pipeline is shut down. References: Databricks Documentation - Configure pipeline settings for Delta Live Tables, Databricks Documentation - Continuous vs. triggered pipeline execution, Databricks Documentation - Development vs. production mode.

 Data Explorer is a graphical interface that allows you to browse, create, and manage data objects such as databases, tables, and views in your workspace. You can also review and modify the permissions on these data objects using Data Explorer. To access Data Explorer, you can click on the Data icon in the sidebar, or use the %sql magic command in a notebook. You can then select a database and a table, and click on the Permissions tab to view and edit the access control lists (ACLs) for the table. You can also use SQL commands such as SHOW GRANT and GRANT to query and modify the permissions on a Delta table. References:

Data Explorer

Access control for Delta tables

SHOW GRANT

[GRANT]

dbfs:/user/hive/warehouse Thereby showing " dbfs:/user/hive/warehouse/customer360.db

 The location of the customer360 database depends on the value of the spark.sql.warehouse.dir configuration property, which specifies the default location for managed databases and tables. If the property is not set, the default value is dbfs:/user/hive/warehouse. Therefore, the customer360 database will be located in dbfs:/user/hive/warehouse/customer360.db. However, if the property is set to a different value, such as dbfs:/user/hive/database, then the customer360 database will be located in dbfs:/user/hive/database/customer360.db. Thus, more information is needed to determine the correct response.

Option A is not correct, as dbfs:/user/hive/database/customer360 is not the default location for managed databases and tables, unless the spark.sql.warehouse.dir property is explicitly set to dbfs:/user/hive/database.

Option B is not correct, as dbfs:/user/hive/warehouse is the default location for the root directory of managed databases and tables, not for a specific database. The database name should be appended with .db to the directory path, such as dbfs:/user/hive/warehouse/customer360.db.

Option C is not correct, as dbfs:/user/hive/customer360 is not a valid location for a managed database, as it does not follow the directory structure specified by the spark.sql.warehouse.dir property.

Databases and Tables

[Databricks Data Engineer Professional Exam Guide]

Lakehouse Federation is the Databricks feature built for querying external systems without moving all data into Databricks . Databricks documentation describes it as the platform for query federation , enabling users to run queries against multiple external data sources while keeping governance centralized through Unity Catalog . Databricks also documents support for external systems such as Amazon Redshift and other databases through connections and foreign catalogs, allowing read-only access to external data while managing permissions in Unity Catalog. This aligns directly with the requirement to minimize duplication and still maintain centralized governance. Databricks Connect is for local development against Databricks compute, not federated querying. MLflow is for machine learning lifecycle management. Delta Lake is a storage format and table layer, not a federation framework. Therefore, when the goal is unified governance across Databricks and external systems like MySQL, Redshift, and BigQuery without first ingesting the data, Lakehouse Federation is the documented answer. Databricks does note that for high-volume production ingestion, managed connectors may sometimes be preferred, but for direct querying without data movement, Lakehouse Federation is the correct feature.

Bronze tables are the first layer of a medallion architecture, which is a data design pattern used to organize data in a lakehouse. Bronze tables contain raw data ingested from various sources, such as RDBMS data, JSON files, IoT data, etc. The table structures in this layer correspond to the source system table structures “as-is”, along with any additional metadata columns that capture the load date/time, process ID, etc. The only transformation applied to the raw data in this layer is to apply a schema, which defines the column names and data types of the table. The schema can be inferred from the data source or specified explicitly. Applying a schema to the raw data enables the use of SQL and other structured query languages to access and analyze the data. Therefore, option E is the correct answer. References: What is a Medallion Architecture?, Raw Data Ingestion into Delta Lake Bronze tables using Azure Synapse Mapping Data Flow, Apache Spark + Delta Lake concepts, Delta Lake Architecture & Azure Databricks Workspace.

In Databricks Asset Bundles , the databricks.yml file located at the repository root serves as the single authoritative configuration file for the entire project. This file defines key elements such as the bundle name, deployment targets (for example, dev, test, prod), workspace configurations, permissions, and resource definitions including jobs and pipelines. Databricks documentation specifies that there must be one primary databricks.yml file at the root , ensuring consistency and simplifying deployment processes. However, to support modularity and maintainability, this root configuration file can reference additional YAML files using the include mapping. This allows teams to organize configurations across directories while still maintaining a single source of truth. Option A is incorrect because multiple root-level configuration files are not supported as independent authorities. Option C is invalid since configuration files are not restricted to hidden directories. Option D introduces ambiguity and contradicts the requirement for a single authoritative configuration. Therefore, using one root-level configuration file with optional includes is the correct and recommended approach.

Option D is the correct answer because it enables the Serverless feature for the SQL endpoint, which allows the endpoint to automatically scale up and down based on the query load. This way, the endpoint can handle more concurrent queries and reduce the time it takes to return results. The Serverless feature also reduces the cold start time of the endpoint, which is the time it takes to start the cluster when a query is submitted to a non-running endpoint. The Serverless feature is available for both AWS and Azure Databricks platforms.

 Databricks SQL Serverless, Serverless SQL endpoints, New Performance Improvements in Databricks SQL