最も有効な Certified-Data-Engineer-Professional 試験参考書

A data engineer has configured their Databricks Asset Bundle with multiple targets in databricks.yml and deployed it to the production workspace. Now, to validate the deployment, they need to invoke a job named my_project_job specifically within the prod target context.
Assuming the job is already deployed, they need to trigger its execution while ensuring the target- specific configuration is respected. Which command will trigger the job execution?

• A. databricks job run my_project_job --env prod
• B. databricks execute my_project_job -e prod
• C. databricks bundle run my_project_job -t prod
• D. databricks run my_project_job -t prod

A Structured Streaming job deployed to production has been experiencing delays during peak hours of the day. At present, during normal execution, each microbatch of data is processed in less than 3 seconds. During peak hours of the day, execution time for each microbatch becomes very inconsistent, sometimes exceeding 30 seconds. The streaming write is currently configured with a trigger interval of 10 seconds.
Holding all other variables constant and assuming records need to be processed in less than 10 seconds, which adjustment will meet the requirement?

• A. Use the trigger once option and configure a Databricks job to execute the query every 10 seconds; this ensures all backlogged records are processed with each batch.
• B. The trigger interval cannot be modified without modifying the checkpoint directory; to maintain the current stream state, increase the number of shuffle partitions to maximize parallelism.
• C. Increase the trigger interval to 30 seconds; setting the trigger interval near the maximum execution time observed for each batch is always best practice to ensure no records are dropped.
• D. Decrease the trigger interval to 5 seconds; triggering batches more frequently may prevent records from backing up and large batches from causing spill.
• E. Decrease the trigger interval to 5 seconds; triggering batches more frequently allows idle executors to begin processing the next batch while longer running tasks from previous batches finish.

The following code has been migrated to a Databricks notebook from a legacy workload:

The code executes successfully and provides the logically correct results, however, it takes over
20 minutes to extract and load around 1 GB of data.
Which statement is a possible explanation for this behavior?

• A. %sh triggers a cluster restart to collect and install Git. Most of the latency is related to cluster startup time.
• B. Instead of cloning, the code should use %sh pip install so that the Python code can get executed in parallel across all nodes in a cluster.
• C. %sh executes shell code on the driver node. The code does not take advantage of the worker nodes or Databricks optimized Spark.
• D. Python will always execute slower than Scala on Databricks. The run.py script should be refactored to Scala.
• E. %sh does not distribute file moving operations; the final line of code should be updated to use %fs instead.

To identify the top users consuming compute resources, a data engineering team needs to monitor usage within their Databricks workspace for better resource utilization and cost control.
The team decided to use Databricks system tables, available under the System catalog in Unity Catalog, to gain detailed visibility into workspace activity. Which SQL query should the team run from the System catalog to achieve this?

• A. SELECT sku_name,
  identity_metadata.created_by AS user_email,
  COUNT(usage_quantity) AS total_dbus
  FROM system.billing.usage
  GROUP BY user_email, sku_name
  ORDER BY total_dbus DESC
  LIMIT 10
• B. SELECT sku_name,
  identity_metadata.created_by AS user_email,
  SUM(usage_quantity * usage_unit) AS total_dbus
  FROM system.billing.usage
  GROUP BY user_email, sku_name
  ORDER BY total_dbus DESC
  LIMIT 10
• C. SELECT identity_metadata.run_as AS user_email,
  SUM(usage_quantity) AS total_dbus
  FROM system.billing.usage
  GROUP BY user_email
  ORDER BY total_dbus DESC
  LIMIT 10
• D. SELECT sku_name,
  usage_metadata.run_name AS user_email,
  SUM(usage_quantity) AS total_dbus
  FROM system.billing.usage
  GROUP BY user_email, sku_name
  ORDER BY total_dbus DESC
  LIMIT 10

A data engineer needs to productionize a new Spark application written by teammate. This application has numerous external dependencies, including libraries, and requires custom environment variables and Spark configuration parameters to be set. Which two methods will help the data engineer accomplish the task? (Choose two.)

• A. Install libraries on DBFS
• B. Add libraries to compute policies
• C. Create init scripts on DBFS.
• D. Use compute policies to set system properties, environment variables, and Spark configuration parameters.
• E. Use secrets in init scripts to store configuration data