The code block shown below should cache DataFrame storesDF only in Spark's memory. Choose the response that correctly fil ls in the numbered blanks within the code block to complete this task. Code…

The correct answer is E. 1. storesDF. To cache a DataFrame exclusively in Spark's memory (without spilling to disk), you must use persist(StorageLevel.MEMORY_ONLY). The simple cache() method in many Spark versions defaults to MEMORY_AND_DISK, which allows spilling to disk when memory is insufficient. The pattern…

Performance Tuning and Optimization

Question

The code block shown below should cache DataFrame storesDF only in Spark's memory. Choose the response that correctly fil ls in the numbered blanks within the code block to complete this task. Code block:

1.2(3).count()

Options

A
1. storesDF
B
1. storesDF
C
1. storesDF
D
1. storesDF
E
1. storesDF

How the community answered

(55 responses)

A
2% (1)
B
2% (1)
D
5% (3)
E
91% (50)

Explanation

To cache a DataFrame exclusively in Spark's memory (without spilling to disk), you must use persist(StorageLevel.MEMORY_ONLY). The simple cache() method in many Spark versions defaults to MEMORY_AND_DISK, which allows spilling to disk when memory is insufficient. The pattern storesDF.persist(StorageLevel.MEMORY_ONLY).count() explicitly specifies memory-only storage and calls count() to trigger the actual caching action (since Spark is lazily evaluated). Option E correctly fills blank 2 with persist and blank 3 with StorageLevel.MEMORY_ONLY.

Topics

#Spark Caching#DataFrame Persistence#Storage Levels#Performance Tuning

Community Discussion

No community discussion yet for this question.

Full DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice