DAS-C01 · Question #52
A large company receives files from external parties in Amazon EC2 throughout the day. At the end of the day, the files are combined into a single file, compressed into a gzip file, and uploaded to…
The correct answer is B. Split the number of files so they are equal to a multiple of the number of slices in the Amazon. Redshift's COPY command achieves maximum parallelism when the number of input files equals a multiple of the total number of slices across the cluster (not just the number of nodes). Each slice processes one file at a time in parallel, so having files = N × slices ensures all…
Question
A large company receives files from external parties in Amazon EC2 throughout the day. At the end of the day, the files are combined into a single file, compressed into a gzip file, and uploaded to Amazon S3. The total size of all the files is close to 100 GB daily. Once the files are uploaded to Amazon S3, an AWS Batch program executes a COPY command to load the files into an Amazon Redshift cluster. Which program modification will accelerate the COPY process?
Options
- AUpload the individual files to Amazon S3 and run the COPY command as soon as the files
- BSplit the number of files so they are equal to a multiple of the number of slices in the Amazon
- CSplit the number of files so they are equal to a multiple of the number of compute nodes in the
- DApply sharding by breaking up the files so the distkey columns with the same values go to the
How the community answered
(25 responses)- A4% (1)
- B76% (19)
- C8% (2)
- D12% (3)
Explanation
Redshift's COPY command achieves maximum parallelism when the number of input files equals a multiple of the total number of slices across the cluster (not just the number of nodes). Each slice processes one file at a time in parallel, so having files = N × slices ensures all slices are fully utilized with no idle capacity. Matching the number of nodes (Option C) is incorrect because nodes contain multiple slices. Waiting to upload individual files (Option A) does not address parallelism, and distributing by distkey (Option D) is a table design concern unrelated to COPY throughput.
Topics
Community Discussion
No community discussion yet for this question.