sounds like compaction is failing due to not enough memory. Also it keeps failing, making the whole cluster overloaded
mysql> show proc '/compactions';
+-------------------------------------------------------+-------+---------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| Partition | TxnID | StartTime | CommitTime | FinishTime | Error |
+-------------------------------------------------------+-------+---------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------+
| native_db.internal_ad_master_pk.internal_ad_master_pk | 7 | 2023-10-04 09:49:43 | 2023-10-04 10:14:19 | 2023-10-04 10:14:22 | NULL |
| native_db.internal_ad_master.internal_ad_master | 1328 | 2023-10-04 13:04:46 | NULL | 2023-10-04 13:04:46 | Memory of process exceed limit. Compaction Used: 30489509576, Limit: 26750531542. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1329 | 2023-10-04 13:04:48 | NULL | 2023-10-04 13:04:48 | Memory of process exceed limit. Compaction Used: 32183245088, Limit: 26750531542. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1330 | 2023-10-04 13:04:50 | NULL | 2023-10-04 13:04:51 | Memory of process exceed limit. Compaction Used: 30415119746, Limit: 26750531542. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1331 | 2023-10-04 13:04:53 | NULL | 2023-10-04 13:04:53 | Memory of process exceed limit. Compaction Used: 31972355594, Limit: 26750531542. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1332 | 2023-10-04 13:04:55 | NULL | 2023-10-04 13:04:55 | Memory of process exceed limit. Compaction Used: 33508518874, Limit: 26750531542. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1333 | 2023-10-04 13:04:57 | NULL | 2023-10-04 13:04:57 | Memory of process exceed limit. Compaction Used: 32411659986, Limit: 26750531542. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1334 | 2023-10-04 13:04:59 | NULL | 2023-10-04 13:05:00 | Memory of process exceed limit. Compaction Used: 35152971936, Limit: 26465151098. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1335 | 2023-10-04 13:05:02 | NULL | 2023-10-04 13:05:10 | Memory of process exceed limit. Compaction Used: 26935626442, Limit: 26750531542. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1336 | 2023-10-04 13:05:12 | NULL | 2023-10-04 13:05:13 | Memory of process exceed limit. read and decompress page Used: 26697365096, Limit: 26465151098. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1337 | 2023-10-04 13:05:15 | NULL | 2023-10-04 13:05:37 | Memory of process exceed limit. Compaction Used: 26853060312, Limit: 26750531542. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1338 | 2023-10-04 13:05:39 | NULL | 2023-10-04 13:05:40 | Memory of process exceed limit. Compaction Used: 27171917488, Limit: 26465151098. Mem usage has exceed the limit of BE |
| native_db.internal_ad_master.internal_ad_master | 1339 | 2023-10-04 13:05:42 | NULL | 2023-10-04 13:05:42 | Memory of process exceed limit. Compaction Used: 27465543128, Limit: 26750531542. Mem usage has exceed the limit of BE |
+-------------------------------------------------------+-------+---------------------+---------------------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------+
13 rows in set (0.03 sec)