nerdexam
Databricks

GENERATIVE-AI-ENGINEER-ASSOCIATE · Question #91

A Generative AI Engineer has successfully ingested unstructured documents and chunked them by document sections. They would like to store the chunks in a Vector Search index. The current format of…

The correct answer is B. Flatten the dataframe to one chunk per row, create a unique identifier for each row, and save to a. Vector Search indexes require one embedding per row, so a chunked dataframe must be flattened to one chunk per row with a unique row-level identifier before being saved to a Delta table.

Data Ingestion and Preparation for Vector Search

Question

A Generative AI Engineer has successfully ingested unstructured documents and chunked them by document sections. They would like to store the chunks in a Vector Search index. The current format of the dataframe has two columns: (i) original document file name (ii) an array of text chunks for each document. What is the most performant way to store this dataframe?

Options

  • ASplit the data into train and test set, create a unique identifier for each document, then save to a
  • BFlatten the dataframe to one chunk per row, create a unique identifier for each row, and save to a
  • CFirst create a unique identifier for each document, then save to a Delta table
  • DStore each chunk as an independent JSON file in Unity Catalog Volume. For each JSON file, the

How the community answered

(50 responses)
  • A
    10% (5)
  • B
    82% (41)
  • C
    2% (1)
  • D
    6% (3)

Why each option

Vector Search indexes require one embedding per row, so a chunked dataframe must be flattened to one chunk per row with a unique row-level identifier before being saved to a Delta table.

ASplit the data into train and test set, create a unique identifier for each document, then save to a

Splitting into train and test sets is a machine learning data preparation step with no relevance to vector store ingestion, and a document-level identifier is too coarse for chunk-level retrieval.

BFlatten the dataframe to one chunk per row, create a unique identifier for each row, and save to aCorrect

Flattening the dataframe so each row contains exactly one text chunk, assigning a unique identifier per row (not per document), and saving to a Delta table matches the required schema for Databricks Vector Search sync indexes, enabling efficient per-chunk embedding and retrieval.

CFirst create a unique identifier for each document, then save to a Delta table

Creating a unique identifier per document rather than per chunk means multiple chunks share the same ID, which violates the uniqueness constraint required for Vector Search index rows.

DStore each chunk as an independent JSON file in Unity Catalog Volume. For each JSON file, the

Storing chunks as independent JSON files in a Unity Catalog Volume requires a custom ingestion pipeline and is far less performant than a Delta table sync index for large-scale vector search workloads.

Concept tested: Delta table schema preparation for Vector Search ingestion

Source: https://docs.databricks.com/en/generative-ai/vector-search.html

Topics

#Data Preprocessing#Vector Search Indexing#RAG Architecture#Databricks Data Engineering

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full GENERATIVE-AI-ENGINEER-ASSOCIATE Practice