Of the following situations, in which will it be most advantageous to store DataFrame df at the MEMORY_AND_DISK storage level rather than the MEMORY_ONLY storage level?

Question

Options

• AWhen all of the computed data in DataFrame df can fit into memory.
• BWhen the memory is full and it's faster to recompute all the data in DataFrame df rather than read
• CWhen it's faster to recompute all the data in DataFrame df that cannot fit into memory based on
• DWhen it's faster to read all the computed data in DataFrame df that cannot fit into memory from
• EThe storage level MENORY_ONLY will always be more advantageous because it's faster to read

• A
  5% (2)
• B
  3% (1)
• C
  3% (1)
• D
  90% (35)

Explanation

MEMORY_ONLY caches data entirely in RAM. If the data doesn't fit, Spark simply does not cache the overflow partitions - they are recomputed from scratch each time they are needed. MEMORY_AND_DISK caches what fits in RAM and spills the rest to disk. The key tradeoff is: reading from disk is slower than reading from memory, but it may still be faster than recomputing partitions from scratch (especially if computation is complex or data must be re-read from a slow source). Therefore, MEMORY_AND_DISK is most advantageous when reading the spilled data from disk is faster than recomputing it. If recomputation is cheap and fast, MEMORY_ONLY may be preferable despite the spillover.

No community discussion yet for this question.