Which of the following code blocks will most quickly return an approximation for the number of distinct values in column division in DataFrame storesDF?

The correct answer is C. storesDF.agg(approx_count_distinct(col("division"), 0.15).alias("divisionDistinct")). The approx_count_distinct function accepts an optional rsd (relative standard deviation) parameter that controls the trade-off between accuracy and computation speed. A higher rsd value means less accuracy but faster execution because the underlying HyperLogLog++ algorithm uses…

Efficiently use Spark SQL and DataFrame API for data transformations and analysis.

Question

Options

AstoresDF.agg(approx_count_distinct(col("division")).alias("divisionDistinct"))
BstoresDF.agg(approx_count_distinct(col("division"), 0.01).alias("divisionDistinct"))
CstoresDF.agg(approx_count_distinct(col("division"), 0.15).alias("divisionDistinct"))
DstoresDF.agg(approx_count_distinct(col("division"), 0.0).alias("divisionDistinct"))
EstoresDF.agg(approx_count_distinct(col("division"), 0.05).alias("divisionDistinct"))

How the community answered

(54 responses)

B
4% (2)
C
91% (49)
D
2% (1)
E
4% (2)

Explanation

The approx_count_distinct function accepts an optional rsd (relative standard deviation) parameter that controls the trade-off between accuracy and computation speed. A higher rsd value means less accuracy but faster execution because the underlying HyperLogLog++ algorithm uses fewer bits of precision. Option C uses rsd=0.15, which is the highest value among all the choices, making it the least accurate but the fastest. Option D (rsd=0.0) would attempt maximum precision, making it the slowest. Option A uses the default rsd=0.05. Therefore, C returns the approximation most quickly.

Topics

#Spark SQL Functions#DataFrame Aggregations#Performance Optimization#Distinct Count Approximation

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice