Databricks Certified-Data-Engineer-Professional - Databricks Certified Data Engineer Professional

The data engineering team is migrating an enterprise system with thousands of tables and views into the Lakehouse. They plan to implement the target architecture using a series of bronze, silver, and gold tables. Bronze tables will almost exclusively be used by production data engineering workloads, while silver tables will be used to support both data engineering and machine learning workloads. Gold tables will largely serve business intelligence and reporting purposes. While personal identifying information (PII) exists in all tiers of data, pseudonymization and anonymization rules are in place for all data at the silver and gold levels.
The organization is interested in reducing security concerns while maximizing the ability to collaborate across diverse teams.
Which statement exemplifies best practices for implementing this system?

• A. Because all tables must live in the same storage containers used for the database they're created in, organizations should be prepared to create between dozens and thousands of databases depending on their data isolation requirements.
• B. Storinq all production tables in a single database provides a unified view of all data assets available throughout the Lakehouse, simplifying discoverability by granting all users view privileges on this database.
• C. Isolating tables in separate databases based on data quality tiers allows for easy permissions management through database ACLs and allows physical separation of default storage locations for managed tables.
• D. Because databases on Databricks are merely a logical construct, choices around database organization do not impact security or discoverability in the Lakehouse.
• E. Working in the default Databricks database provides the greatest security when working with managed tables, as these will be created in the DBFS root.

Why are Pandas UDFs often preferred over traditional PySpark UDFs in performance-critical applications involving large datasets?

• A. They allow row-level execution of functions in Python with native Spark optimization, removing the need for columnar execution.
• B. They eliminate the JVM-Python boundary by bypassing serialization entirely, thereby avoiding data conversion overhead.
• C. They leverage Apache Arrow to enable vectorized operations between the JVM and Python runtimes, reducing serialization costs and improving computational efficiency.
• D. They minimize memory usage by streaming each row individually through a lightweight Python wrapper, avoiding batch processing overhead.

A data engineer is creating a data ingestion pipeline to understand where customers are taking their rented bicycles during use. The engineer noticed that, over time, data being transmitted from the bicycle sensors fail to include key details like latitude and longitude. Downstream analysts need both the clean records and the quarantined records available for separate processing.
The data engineer already has this code:
import dlt
from pyspark.sql.functions import expr
rules = {
"valid_lat": "(lat IS NOT NULL)",
"valid_long": "(long IS NOT NULL)"
}
quarantine_rules = "NOT({})".format(" AND ".join(rules.values()))
@dlt.view
def raw_trips_data():
return spark.readStream.table("ride_and_go.telemetry.trips")
How should the data engineer meet the requirements to capture good and bad data?

• A. @dlt.table(name="trips_data_quarantine")
  def trips_data_quarantine():
  return (
  spark.readStream.table("raw_trips_data")
  .filter(expr(quarantine_rules))
  )
• B. @dlt.table(partition_cols=["is_quarantined", ])
  @dlt.expect_all(rules)
  def trips_data_quarantine():
  return (
  spark.readStream.table("raw_trips_data")
  .withColumn("is_quarantined", expr(quarantine_rules))
  )
• C. @dlt.table
  @dlt.expect_all_or_drop(rules)
  def trips_data_quarantine():
  return spark.readStream.table("raw_trips_data")
• D. @dlt.view
  @dlt.expect_or_drop("lat_long_present", "(lat IS NOT NULL AND long IS NOT NULL)") def trips_data_quarantine():
  return spark.readStream.table("ride_and_go.telemetry.trips")

A workspace admin has created a new catalog called finance_data and wants to delegate permission management to a finance team lead without giving them full admin rights. Which privilege should be granted to the finance team lead?

• A. Make the finance team lead a metastore admin.
• B. MANAGE privilege on the finance_data catalog.
• C. GRANT OPTION privilege on the finance_data catalog.
• D. ALL PRIVILEGES on the finance_data catalog.

A data engineer, User A, has promoted a new pipeline to production by using the REST API to programmatically create several jobs. A DevOps engineer, User B, has configured an external orchestration tool to trigger job runs through the REST API. Both users authorized the REST API calls using their personal access tokens.
Which statement describes the contents of the workspace audit logs concerning these events?

• A. Because the REST API was used for job creation and triggering runs, a Service Principal will be automatically used to identity these events.
• B. Because these events are managed separately, User A will have their identity associated with the job creation events and User B will have their identity associated with the job run events.
• C. Because the REST API was used for job creation and triggering runs, user identity will not be captured in the audit logs.
• D. Because User B last configured the jobs, their identity will be associated with both the job creation events and the job run events.
• E. Because User A created the jobs, their identity will be associated with both the job creation events and the job run events.