We are running an iceberg query which is similar to -
select * from catalog_name.table ORDER by core_modifiedon DESC limit 100
This contains around 100M rows. The iceberg data is in s3 and we are using shared cluster setup with 2 cns.
When this query is run independently, it takes around 20 secs.
When this query is run with 4 concurrency, it takes around 100 secs.
What I have noticed is the time taken for SCAN is increasing.
For first run
- IOTaskExecTime: 11s454ms
- __MAX_OF_IOTaskExecTime: 11s661ms
- __MIN_OF_IOTaskExecTime: 11s247ms
For five concurrency
- IOTaskExecTime: 46s728ms
- __MAX_OF_IOTaskExecTime: 49s407ms
- __MIN_OF_IOTaskExecTime: 44s50ms