You are testing a Dataflow pipeline to ingest and transform text files. The files are compressed gzip, errors are written to a dead-letter queue, and you are using SideInputs to join data. You…

Question

Options

• AHave a staging table that is an append-only model, and then update the production table every three hours with the changes written to staging
• BHave a staging table that is an append-only model, and then update the production table every ninety minutes with the changes written to staging
• CHave a staging table that moves the staged data over to the production table and deletes the contents of the staging table every three hours
• DHave a staging table that moves the staged data over to the production table and deletes the contents of the staging table every thirty minutes

• A
  23% (10)
• B
  14% (6)
• C
  56% (24)
• D
  7% (3)

Explanation

Explanation/Reference: There are a lot of reference doc to tell about comparison between them https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-developing-and- testing#choose_correctly_between_side_inputs_or_cogroupbykey_for_joins https://cloud.google.com/blog/products/data-analytics/guide-to-common-cloud-dataflow-use-case-patterns-part-2 on either the ingestion or reporting pieces? Explanation/Reference: Following common extract, transform, load (ETL) best practices, we used a staging table and a separate production table so that we could load data into the staging table without impacting users of the data. The design we created based on ETL best practices called for first deleting all the records from the staging table, loading the staging table, and then replacing the production table with the contents. When using the streaming API, the BigQuery streaming buffer remains active for about 30 to 60 minutes or more after use, which means that you can’t delete or change data during that time. Since we used the streaming API, we scheduled the load every three hours to balance getting data into BigQuery quickly and being able to subsequently delete the data from the staging table during the load process. Building a script with BigQuery on the back end. https://cloud.google.com/blog/products/data-analytics/moving-a-publishing-workflow-to-bigquery-for-new-data-insights

No community discussion yet for this question.