In many small file scenarios, Spark will start many tasks. When there is a Shuffle operation in the SQL logic, the number of hash buckets will be greatly increased, which will seriously affect the…

In many small file scenarios, Spark will start many tasks. When there is a Shuffle operation in the SQL logic, the number of hash buckets will be greatly increased, which will seriously affect the performance. In Fusioninsight, the () operator is usually used for small file scenarios to merge the partitions generated by the small files in the Table to reduce the number of partitions, so as to avoid generating too many hash buckets during shuffle and improve performance.