Databricks Certified Associate Developer for Apache Spark 3.0 Questions

dumps2022-01-07T03:48:26+08:00

dumps Databricks Databricks Certified Associate Developer for Apache Spark 3.0 questions 0 Comments

Databricks Certified Associate Developer for Apache Spark 3.0 questions are the best material for you to pass the test. There are 60 multiple-choice questions in real Databricks Certified Associate Developer for Apache Spark 3.0 exam, and you have 120 minutes to take the test. The minimum passing score of the test is 70%, which means you need to answer 42 questions correctly. Free Databricks Certified Associate Developer for Apache Spark 3.0 questions are available below.

Page 1 of 5

1. +------+--------------------+--------------------+-------------------+

itemsDf.filter(col(supplier).not_contains('X')).select(supplier).distinct()

itemsDf.select(~col('supplier').contains('X')).distinct()

itemsDf.filter(not(col('supplier').contains('X'))).select('supplier').unique()

itemsDf.filter(~col('supplier').contains('X')).select('supplier').distinct()

itemsDf.filter(!col('supplier').contains('X')).select(col('supplier')).unique()

2. Which of the following describes the role of the cluster manager?

The cluster manager schedules tasks on the cluster in client mode.

The cluster manager schedules tasks on the cluster in local mode.

The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.

The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.

The cluster manager allocates resources to the DataFrame manager.

3. Which of the following code blocks adds a column predErrorSqrt to DataFrame transactionsDf that is the square root of column predError?

transactionsDf.withColumn("predErrorSqrt", sqrt(predError))

transactionsDf.select(sqrt(predError))

transactionsDf.withColumn("predErrorSqrt", col("predError").sqrt())

transactionsDf.withColumn("predErrorSqrt", sqrt(col("predError")))

transactionsDf.select(sqrt("predError"))

4. The code block displayed below contains an error. When the code block below has executed, it should have divided DataFrame transactionsDf into 14 parts, based on columns storeId and transactionDate (in this order). Find the error.

Code block:

transactionsDf.coalesce(14, ("storeId", "transactionDate"))

The parentheses around the column names need to be removed and .select() needs to be appended to the code block.

Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block. (Correct)

Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .select() needs to be appended to the code block.

Operator coalesce needs to be replaced by repartition and the parentheses around the column names need to be replaced by square brackets.

Operator coalesce needs to be replaced by repartition.

5. +------+-----------------------------+-------------------+

itemsDf.withColumn('attributes', sort_array(col('attributes').desc()))

itemsDf.withColumn('attributes', sort_array(desc('attributes')))

itemsDf.withColumn('attributes', sort(col('attributes'), asc=False))

itemsDf.withColumn("attributes", sort_array("attributes", asc=False))

itemsDf.select(sort_array("attributes"))

6. The code block displayed below contains an error. The code block should save DataFrame transactionsDf at path path as a parquet file, appending to any existing parquet file. Find the error.

Code block:

transactionsDf.format("parquet").option("mode", "append").save(path)

The code block is missing a reference to the DataFrameWriter.

save() is evaluated lazily and needs to be followed by an action.

The mode option should be omitted so that the command uses the default mode.

The code block is missing a bucketBy command that takes care of partitions.

Given that the DataFrame should be saved as parquet file, path is being passed to the wrong method.

7. Which of the following code blocks returns a DataFrame with an added column to DataFrame transactionsDf that shows the unix epoch timestamps in column transactionDate as strings in the format

month/day/year in column transactionDateFormatted?

Excerpt of DataFrame transactionsDf:

transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="dd/MM/yyyy"))

transactionsDf.withColumnRenamed("transactionDate", "transactionDateFormatted", from_unixtime("transactionDateFormatted", format="MM/dd/yyyy"))

transactionsDf.apply(from_unixtime(format="MM/dd/yyyy")).asColumn("transactionDateFor matted")

transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate", format="MM/dd/yyyy"))

transactionsDf.withColumn("transactionDateFormatted", from_unixtime("transactionDate"))

8. "MM d (EEE)"

9. Which of the following statements about stages is correct?

Different stages in a job may be executed in parallel.

Stages consist of one or more jobs.

Stages ephemerally store transactions, before they are committed through actions.

Tasks in a stage may be executed by multiple machines at the same time.

Stages may contain multiple actions, narrow, and wide transformations.

10. The code block displayed below contains an error. The code block should configure Spark so that DataFrames up to a size of 20 MB will be broadcast to all worker nodes when performing a join.

Find the error.

Code block:

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 20)

Spark will only broadcast DataFrames that are much smaller than the default value.

The correct option to write configurations is through spark.config and not spark.conf.

Spark will only apply the limit to threshold joins and not to other joins.

The passed limit has the wrong variable type.

The command is evaluated lazily and needs to be followed by an action.

Page 2 of 5

11. Which of the following code blocks returns a copy of DataFrame transactionsDf where the column storeId has been converted to string type?

transactionsDf.withColumn("storeId", convert("storeId", "string"))

transactionsDf.withColumn("storeId", col("storeId", "string"))

transactionsDf.withColumn("storeId", col("storeId").convert("string"))

transactionsDf.withColumn("storeId", col("storeId").cast("string"))

transactionsDf.withColumn("storeId", convert("storeId").as("string"))

12. Which of the elements that are labeled with a circle and a number contain an error or are misrepresented?

1, 10

1, 8

7, 9, 10

1, 4, 6, 9

13. load

14. Which of the following code blocks returns a new DataFrame with the same columns as DataFrame transactionsDf, except for columns predError and value which should be removed?

transactionsDf.drop(["predError", "value"])

transactionsDf.drop("predError", "value")

transactionsDf.drop(col("predError"), col("value"))

transactionsDf.drop(predError, value)

transactionsDf.drop("predError & value")

15. +------+----------------------------------+-------------------+

Code block:

itemsDf.withColumnRenamed("itemNameElements", split("itemName"))

itemsDf.withColumnRenamed("itemNameElements", split("itemName"))

All column names need to be wrapped in the col() operator.

Operator withColumnRenamed needs to be replaced with operator withColumn and a second argument "," needs to be passed to the split method.

Operator withColumnRenamed needs to be replaced with operator withColumn and the split method needs to be replaced by the splitString method.

Operator withColumnRenamed needs to be replaced with operator withColumn and a second argument " " needs to be passed to the split method.

The expressions "itemNameElements" and split("itemName") need to be swapped.

16. "transactionId", "supplier"

17. +-------+-------------+---------+-----+

Code block:

sc.union([transactionsDfMonday, transactionsDfTuesday])

The DataFrames' RDDs need to be passed into the sc.union method instead of the DataFrame variable names.

Instead of union, the concat method should be used, making sure to not use its default arguments.

Instead of the Spark context, transactionDfMonday should be called with the join method instead of the union method, making sure to use its default arguments.

Instead of the Spark context, transactionDfMonday should be called with the union method.

Instead of the Spark context, transactionDfMonday should be called with the unionByName method instead of the union method, making sure to not use its default arguments.

18. The code block displayed below contains an error. The code block should return DataFrame transactionsDf, but with the column storeId renamed to storeNumber. Find the error.

Code block:

transactionsDf.withColumn("storeNumber", "storeId")

Instead of withColumn, the withColumnRenamed method should be used.

Arguments "storeNumber" and "storeId" each need to be wrapped in a col() operator.

Argument "storeId" should be the first and argument "storeNumber" should be the second argument to the withColumn method.

The withColumn operator should be replaced with the copyDataFrame operator.

Instead of withColumn, the withColumnRenamed method should be used and argument "storeId" should be the first and argument "storeNumber" should be the second argument to that method.

19. Which of the following is the idea behind dynamic partition pruning in Spark?

Dynamic partition pruning is intended to skip over the data you do not need in the results of a query.

Dynamic partition pruning concatenates columns of similar data types to optimize join performance.

Dynamic partition pruning performs wide transformations on disk instead of in memory.

Dynamic partition pruning reoptimizes physical plans based on data types and broadcast variables.

Dynamic partition pruning reoptimizes query plans based on runtime statistics collected during query execution.

20. Which of the following code blocks reads JSON file imports.json into a DataFrame?

spark.read().mode("json").path("/FileStore/imports.json")

spark.read.format("json").path("/FileStore/imports.json")

spark.read("json", "/FileStore/imports.json")

spark.read.json("/FileStore/imports.json")

spark.read().json("/FileStore/imports.json")

Page 3 of 5

21. +------------- +--------- +----- +------- +--------- +---- +

transactionsDf.schema.print()

transactionsDf.rdd.printSchema()

transactionsDf.rdd.formatSchema()

transactionsDf.printSchema()

print(transactionsDf.schema)

22. Which of the following statements about Spark's configuration properties is incorrect?

The maximum number of tasks that an executor can process at the same time is controlled by the spark.task.cpus property.

The maximum number of tasks that an executor can process at the same time is controlled by the spark.executor.cores property.

The default value for spark.sql.autoBroadcastJoinThreshold is 10M

The default number of partitions to use when shuffling data for joins or aggregations is 300.

The default number of partitions returned from certain transformations can be controlled by the spark.default.parallelism property.

23. Which of the following is the deepest level in Spark's execution hierarchy?

Job

Task

Executor

Slot

Stage

24. spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf")

25. Which of the following code blocks displays the 10 rows with the smallest values of column value in DataFrame transactionsDf in a nicely formatted way?

transactionsDf.sort(asc(value)).show(10)

transactionsDf.sort(col("value")).show(10)

transactionsDf.sort(col("value").desc()).head()

transactionsDf.sort(col("value").asc()).print(10)

transactionsDf.orderBy("value").asc().show(10)

26. The code block displayed below contains an error. The code block is intended to join

DataFrame itemsDf with the larger DataFrame transactionsDf on column itemId. Find the error.

Code block:

transactionsDf.join(itemsDf, "itemId", how="broadcast")

The syntax is wrong, how= should be removed from the code block.

The join method should be replaced by the broadcast method.

Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.

The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.

broadcast is not a valid join type.

27. The code block displayed below contains an error. The code block is intended to write DataFrame transactionsDf to disk as a parquet file in location /FileStore/transactions_split, using column

storeId as key for partitioning. Find the error.

Code block:

transactionsDf.write.format("parquet").partitionOn("storeId").save("/FileStore/transactions_s plit")A.

The format("parquet") expression is inappropriate to use here, "parquet" should be passed as first argument to the save() operator and "/FileStore/transactions_split" as the second argument.

Partitioning data by storeId is possible with the partitionBy expression, so partitionOn should be replaced by partitionBy.

Partitioning data by storeId is possible with the bucketBy expression, so partitionOn should be replaced by bucketBy.

partitionOn("storeId") should be called before the write operation.

The format("parquet") expression should be removed and instead, the information should be added to the write expression like so: write("parquet").

28. Which of the following code blocks reads in the JSON file stored at filePath as a DataFrame?

spark.read.json(filePath)

spark.read.path(filePath, source="json")

spark.read().path(filePath)

spark.read().json(filePath)

spark.read.path(filePath)

29. Which of the following statements about garbage collection in Spark is incorrect?

Garbage collection information can be accessed in the Spark UI's stage detail view.

Optimizing garbage collection performance in Spark may limit caching ability.

Manually persisting RDDs in Spark prevents them from being garbage collected.

In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector.

Serialized caching is a strategy to increase the performance of garbage collection.

30. Which of the following describes Spark's way of managing memory?

Spark uses a subset of the reserved system memory.

Storage memory is used for caching partitions derived from DataFrames.

As a general rule for garbage collection, Spark performs better on many small objects than few big objects.

Disabling serialization potentially greatly reduces the memory footprint of a Spark application.

Spark's memory usage can be divided into three categories: Execution, transaction, and storage.

Page 4 of 5

31. The code block displayed below contains an error. The code block should display the schema of DataFrame transactionsDf. Find the error.

Code block:

transactionsDf.rdd.printSchema

There is no way to print a schema directly in Spark, since the schema can be printed easily through using print(transactionsDf.columns), so that should be used instead.

The code block should be wrapped into a print() operation.

printSchema is only accessible through the spark session, so the code block should be rewritten as spark.printSchema(transactionsDf).

printSchema is a method and should be written as printSchema(). It is also not callable through transactionsDf.rdd, but should be called directly from transactionsDf. (Correct)

printSchema is a not a method of transactionsDf.rdd. Instead, the schema should be printed via transactionsDf.print_schema().

32. Which of the following code blocks returns the number of unique values in column storeId of DataFrame transactionsDf?

transactionsDf.select("storeId").dropDuplicates().count()

transactionsDf.select(count("storeId")).dropDuplicates()

transactionsDf.select(distinct("storeId")).count()

transactionsDf.dropDuplicates().agg(count("storeId"))

transactionsDf.distinct().select("storeId").count()

33. print(itemsDf.types)

B. itemsDf.printSchema()

C. spark.schema(itemsDf)

D. itemsDf.rdd.printSchema()

E. itemsDf.print.schema()

34. Which of the following code blocks removes all rows in the 6-column DataFrame transactionsDf that have missing data in at least 3 columns?

transactionsDf.dropna("any")

transactionsDf.dropna(thresh=4)

transactionsDf.drop.na("",2)

transactionsDf.dropna(thresh=2)

transactionsDf.dropna("",4)

35. The code block displayed below contains an error. The code block should return the average of rows in column value grouped by unique storeId. Find the error.

Code block:

transactionsDf.agg("storeId").avg("value")

Instead of avg("value"), avg(col("value")) should be used.

The avg("value") should be specified as a second argument to agg() instead of being appended to it.

All column names should be wrapped in col() operators.

agg should be replaced by groupBy.

"storeId" and "value" should be swapped.

36. The code block displayed below contains an error. The code block should trigger Spark to cache DataFrame transactionsDf in executor memory where available, writing to disk where insufficient

executor memory is available, in a fault-tolerant way. Find the error.

Code block:

transactionsDf.persist(StorageLevel.MEMORY_AND_DISK)

Caching is not supported in Spark, data are always recomputed.

Data caching capabilities can be accessed through the spark object, but not through the DataFrame AP

The storage level is inappropriate for fault-tolerant storage.

The code block uses the wrong operator for caching.

The DataFrameWriter needs to be invoked.

37. select

38. spark.read.json(filePath, schema=schema)

C. spark.read.json(filePath, schema=schema_of_json(json_schema))

D. spark.read.json(filePath, schema=spark.read.json(json_schema))

39. Which of the following describes the role of tasks in the Spark execution hierarchy?

Tasks are the smallest element in the execution hierarchy.

Within one task, the slots are the unit of work done for each partition of the data.

Tasks are the second-smallest element in the execution hierarchy.

Stages with narrow dependencies can be grouped into one task.

Tasks with wide dependencies can be grouped into one stage.

40. The code block displayed below contains an error. The code block should write DataFrame

transactionsDf as a parquet file to location filePath after partitioning it on column storeId.

Find the error.

Code block:

transactionsDf.write.partitionOn("storeId").parquet(filePath)

The partitioning column as well as the file path should be passed to the write() method of DataFrame transactionsDf directly and not as appended commands as in the code block.

The partitionOn method should be called before the write method.

The operator should use the mode() option to configure the DataFrameWriter so that it replaces any existing files at location filePath.

Column storeId should be wrapped in a col() operator.

No method partitionOn() exists for the DataFrame class, partitionBy() should be used instead.

Page 5 of 5

41. Which of the following statements about executors is correct?

Executors are launched by the driver.

Executors stop upon application completion by default.

Each node hosts a single executor.

Executors store data in memory only.

An executor can serve multiple applications.

42. Which of the following code blocks returns about 150 randomly selected rows from the 1000-row DataFrame transactionsDf, assuming that any row can appear more than once in the returned DataFrame?

transactionsDf.resample(0.15, False, 3142)

transactionsDf.sample(0.15, False, 3142)

transactionsDf.sample(0.15)

transactionsDf.sample(0.85, 8429)

transactionsDf.sample(True, 0.15, 8261)

43. 1.Which of the following code blocks silently writes DataFrame itemsDf in avro format to location fileLocation if a file does not yet exist at that location?

itemsDf.write.avro(fileLocation)

itemsDf.write.format("avro").mode("ignore").save(fileLocation)

itemsDf.write.format("avro").mode("errorifexists").save(fileLocation)

itemsDf.save.format("avro").mode("ignore").write(fileLocation)

spark.DataFrameWriter(itemsDf).format("avro").write(fileLocation)

44. Which of the following code blocks returns all unique values across all values in columns value and productId in DataFrame transactionsDf in a one-column DataFrame?

tranactionsDf.select('value').join(transactionsDf.select('productId'), col('value')==col('productId'), 'outer')

transactionsDf.select(col('value'), col('productId')).agg({'*': 'count'})

transactionsDf.select('value', 'productId').distinct()

transactionsDf.select('value').union(transactionsDf.select('productId')).distinct()

transactionsDf.agg({'value': 'collect_set', 'productId': 'collect_set'})

45. Which of the following code blocks reduces a DataFrame from 12 to 6 partitions and performs a full shuffle?

DataFrame.repartition(12)

DataFrame.coalesce(6).shuffle()

DataFrame.coalesce(6)

DataFrame.coalesce(6, shuffle=True)

DataFrame.repartition(6)

46. Which of the following code blocks creates a new 6-column DataFrame by appending the rows of the 6-column DataFrame yesterdayTransactionsDf to the rows of the 6-column DataFrame todayTransactionsDf, ignoring that both DataFrames have different column names?

union(todayTransactionsDf, yesterdayTransactionsDf)

todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True)

todayTransactionsDf.unionByName(yesterdayTransactionsDf)

todayTransactionsDf.concat(yesterdayTransactionsDf)

todayTransactionsDf.union(yesterdayTransactionsDf)

47. Which of the following describes slots?

Slots are dynamically created and destroyed in accordance with an executor's workload.

To optimize I/O performance, Spark stores data on disk in multiple slots.

A Java Virtual Machine (JVM) working as an executor can be considered as a pool of slots for task execution.

A slot is always limited to a single core. Slots are the communication interface for executors and are used for receiving commands and sending results to the driver.

48. dfDates = dfDates.withColumnRenamed("date", to_datetime("date", "yyyy-MM-dd HH:mm:ss"))

E. 1.dfDates = spark.createDataFrame([("23/01/2022 11:28:12",),("24/01/2022 10:58:34",)], ["date"])

Databricks Certified Associate Developer for Apache Spark 3.0 Questions

Databricks Certified Associate Developer for Apache Spark 3.0 Questions

Share this post

Leave a Reply Cancel reply

Related Posts