PROFESSIONAL-DATA-ENGINEER · Question #194
You have historical data covering the last three years in BigQuery and a data pipeline that delivers new data to BigQuery daily. You have noticed that when the Data Science team runs a query…
The correct answer is A. Re-create the tables using DDL. Partition the tables by a column containing a TIMESTAMP or DATE Type. Partitioning a BigQuery table by a TIMESTAMP or DATE column (option A) solves the root cause: when a query filters on a date column, BigQuery's partition pruning eliminates irrelevant partitions from the scan, so only the relevant days are read instead of the entire table…
Question
Options
- ARe-create the tables using DDL. Partition the tables by a column containing a TIMESTAMP or DATE Type.
- BRecommend that the Data Science team export the table to a CSV file on Cloud Storage and use Cloud Datalab to explore the data by reading the files directly.
- CModify your pipeline to maintain the last 3090 days of data in one table and the longer history in a different table to minimize full table scans over the entire
- DWrite an Apache Beam pipeline that creates a BigQuery table per day. Recommend that the Data Science team use wildcards on the table name suffixes to
How the community answered
(38 responses)- A71% (27)
- B8% (3)
- C18% (7)
- D3% (1)
Explanation
Partitioning a BigQuery table by a TIMESTAMP or DATE column (option A) solves the root cause: when a query filters on a date column, BigQuery's partition pruning eliminates irrelevant partitions from the scan, so only the relevant days are read instead of the entire table - directly reducing both query time and cost.
Why the distractors fail:
- B breaks the SQL requirement - Cloud Datalab reading raw CSV files is not a SQL workflow and adds operational friction for the Data Science team.
- C is a manual workaround that adds pipeline complexity and still doesn't prevent full scans within each table; it also doesn't scale cleanly as data grows.
- D per-day tables with wildcard queries are a legacy pattern that predates native partitioning; wildcards still risk scanning all matching tables and add naming/management overhead without the cost controls partitioning provides.
Memory tip: Think of partitioning as giving BigQuery a filing cabinet with labeled drawers - a date filter lets it open only the drawer it needs instead of dumping the entire cabinet on the floor. Anytime a GCP exam mentions "full table scans" + "cost" + "date filtering," partitioning by DATE/TIMESTAMP is almost always the answer.
Topics
Community Discussion
No community discussion yet for this question.