GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #5
A Generative AI Engineer has written scalable PySpark code to ingest unstructured PDF documents and chunk them in preparation for storing in a Databricks Vector Search index. Currently, the two…
The correct answer is D. Create a unique identifier for each document, flatten the dataframe to one chunk per row and. Databricks Vector Search requires a Delta table where each row represents a single chunk and every row has a unique identifier. The correct workflow is: (1) Create a document-level unique identifier first (e.g., a hash of the filename), (2) flatten (explode) the array of chunks…
Question
A Generative AI Engineer has written scalable PySpark code to ingest unstructured PDF documents and chunk them in preparation for storing in a Databricks Vector Search index. Currently, the two columns of their dataframe include the original filename as a string and an array of text chunks from that document. What set of steps should the Generative AI Engineer perform to store the chunks in a ready-to- ingest manner for Databricks Vector Search?
Options
- AUse PySpark's autoloader to apply a UDF across all chunks, formatting them in a JSON structure
- BFlatten the dataframe to one chunk per row, create a unique identifier for each row, and enable
- CUtilize the original filename as the unique identifier and save the dataframe as is.
- DCreate a unique identifier for each document, flatten the dataframe to one chunk per row and
How the community answered
(42 responses)- A12% (5)
- B7% (3)
- C2% (1)
- D79% (33)
Explanation
Databricks Vector Search requires a Delta table where each row represents a single chunk and every row has a unique identifier. The correct workflow is: (1) Create a document-level unique identifier first (e.g., a hash of the filename), (2) flatten (explode) the array of chunks so each chunk occupies its own row, allowing a composite chunk ID (e.g., doc_id + chunk_index) to be derived, and (3) enable Delta Sync so the index stays current. Option B is close but creates unique IDs only after flattening, missing the document-level ID that enables chunk-level ID generation. Option A misuses autoloader, which is a streaming ingestion tool, not a transformation step. Option C fails because the filename is not unique per chunk and the array is not flattened, which Vector Search cannot ingest.
Topics
Community Discussion
No community discussion yet for this question.