DAS-C01 · Question #42
A company that produces network devices has millions of users. Data is collected from the devices on an hourly basis and stored in an Amazon S3 data lake. The company runs analyses on the last 24…
The correct answer is A. In Apache ORC partitioned by date and sorted by source IP. Apache ORC (Optimized Row Columnar) partitioned by date and sorted by source IP is optimal for this workload. ORC is a highly compressed columnar format that dramatically reduces I/O and speeds up analytical queries - ideal for both the 24-hour anomaly detection and the 2-year…
Question
A company that produces network devices has millions of users. Data is collected from the devices on an hourly basis and stored in an Amazon S3 data lake. The company runs analyses on the last 24 hours of data flow logs for abnormality detection and to troubleshoot and resolve user issues. The company also analyzes historical logs dating back 2 years to discover patterns and look for improvement opportunities. The data flow logs contain many metrics, such as date, timestamp, source IP, and target IP. There are about 10 billion events every day. How should this data be stored for optimal performance?
Options
- AIn Apache ORC partitioned by date and sorted by source IP
- BIn compressed .csv partitioned by date and sorted by source IP
- CIn Apache Parquet partitioned by source IP and sorted by date
- DIn compressed nested JSON partitioned by source IP and sorted by date
How the community answered
(41 responses)- A66% (27)
- B20% (8)
- C10% (4)
- D5% (2)
Explanation
Apache ORC (Optimized Row Columnar) partitioned by date and sorted by source IP is optimal for this workload. ORC is a highly compressed columnar format that dramatically reduces I/O and speeds up analytical queries - ideal for both the 24-hour anomaly detection and the 2-year historical pattern analysis. Partitioning by date aligns perfectly with both query patterns (last 24 hours and historical date ranges), enabling partition pruning to skip irrelevant data. Sorting by source IP within each date partition accelerates lookups by IP. Compressed CSV and nested JSON are row-based formats that are far less efficient for large-scale analytics. Parquet partitioned by source IP would not align well with the time-based query patterns.
Topics
Community Discussion
No community discussion yet for this question.