We are running an iceberg query which is similar to -
select * from catalog_name.table ORDER by core_modifiedon DESC limit 100
This contains around 100M rows. The iceberg data is in s3 and we are using shared cluster setup with 2 cns.
When this query is run independently, it takes around 20 secs.
When this query is run with 4 concurrency, it takes around 100 secs.
What I have noticed is the time taken for SCAN is increasing.
For first run
- IOTaskExecTime: 11s454ms
- __MAX_OF_IOTaskExecTime: 11s661ms
- __MIN_OF_IOTaskExecTime: 11s247ms
For five concurrency
- IOTaskExecTime: 46s728ms
- __MAX_OF_IOTaskExecTime: 49s407ms
- __MIN_OF_IOTaskExecTime: 44s50ms
- NonDefaultSessionVariables: {“sql_mode_v2”:{“defaultValue”:32,“actualValue”:2097184},“enable_materialized_view_rewrite”:{“defaultValue”:true,“actualValue”:false},“character_set_results”:{“defaultValue”:“utf8”,“actualValue”:“NULL”},“enable_adaptive_sink_dop”:{“defaultValue”:false,“actualValue”:true},“enable_profile”:{“defaultValue”:false,“actualValue”:true}}