Starrocks Version 3.2.6-2585333
I was doing my initial load of data from postgres to starrocks using Kafka When my BE died.
I tried to restart the BE but it starts for only a few seconds before dying again - I stopped all of the other kafka service so that I only have Starrocks running in the container and it still dies after a few seconds.
This seems to be the main issue
E0427 17:53:06.947628 833 tablet_updates.cpp:405] get extra file size in primary table fail, tablet_id: 11124 status: Internal error: Iterate dir /opt/starrocks/be/storage/data/51/11124/130830931 Filesystem error: filesystem error: cannot get file size: No such file or directory [/opt/starrocks/be/storage/data/51/11124/130830931/index.l0.29912.0]
W0427 17:53:06.956574 834 stack_util.cpp:350] 2024-04-27 17:53:06.940983, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::filesystem::__cxx11::filesystem_error, trace:
@ 0xa64ad82 _ZNSt10filesystem9file_sizeERKNS_7__cxx114pathE.cold
@ 0x44a372e starrocks::TabletUpdates::_get_extra_file_size()
@ 0x44a4398 starrocks::TabletUpdates::get_tablet_info_extra()
@ 0x447a497 starrocks::Tablet::build_tablet_report_info()
@ 0x408134e starrocks::TabletManager::report_all_tablets_info()
@ 0x3ee1f28 starrocks::ReportOlapTableTaskWorkerPool::_worker_thread_callback()
@ 0xa62ebb4 execute_native_thread_routine
@ 0x7ff9a246bac3 (unknown)
@ 0x7ff9a24fd850 (unknown)
@ (nil) (unknown)
Part of the be.warning logs
W0426 13:22:26.938978 1 cpu_info.cpp:190] /sys/devices/system/node is not present - no NUMA support
W0426 13:24:40.661931 1 cpu_info.cpp:190] /sys/devices/system/node is not present - no NUMA support
E0426 13:26:10.271930 1 daemon.cpp:241] got signal: Terminated from pid: 0, is going to exit
W0426 13:26:36.710038 1 cpu_info.cpp:190] /sys/devices/system/node is not present - no NUMA support
E0426 13:30:01.990836 1 daemon.cpp:241] got signal: Terminated from pid: 0, is going to exit
W0426 13:31:28.091912 1 cpu_info.cpp:190] /sys/devices/system/node is not present - no NUMA support
W0426 16:06:48.635632 836 tablet_updates.cpp:1591] wait_for_version slow(6052ms) version:755.1 tablet:11187 #version:769 [1 755.1@767 756] pending: rowsets:6[id/seg/row/del/byte/compaction]: [694/1/400584/0/2.77 MB/253.23 MB],[763/1/8550/0/59.49 KB/255.94 MB],[764/1/15940/0/123.98 KB/255.88 MB],[765/1/10954/0/72.14 KB/255.93 MB],[766/1/5523615/0/34.18 MB/221.82 MB],[767/1/15026/0/95.28 KB/255.91 MB]
W0426 16:06:59.956595 836 tablet_updates.cpp:1591] wait_for_version slow(5742ms) version:759.1 tablet:11189 #version:773 [1 759.1@771 760] pending: rowsets:6[id/seg/row/del/byte/compaction]: [694/1/394631/0/2.73 MB/253.27 MB],[767/1/3242/0/22.24 KB/255.98 MB],[768/1/8529/0/54.27 KB/255.95 MB],[769/1/14582/0/93.82 KB/255.91 MB],[770/1/5580231/0/34.54 MB/221.46 MB],[771/1/2468/0/18.30 KB/255.98 MB]
W0426 16:19:46.389668 836 tablet_updates.cpp:1591] wait_for_version slow(9237ms) version:1190.1 tablet:11187 #version:1169 [41 1190.1@1167 1191] pending: rowsets:6[id/seg/row/del/byte/compaction]: [1203/1/4006/0/29.73 KB/255.97 MB],[1204/1/4923/0/31.08 KB/255.97 MB],[1205/1/1218/0/13.71 KB/255.99 MB],[1206/1/3207/0/29.48 KB/255.97 MB],[1207/1/8180845/0/51.46 MB/204.54 MB],[1208/1/4148/0/32.06 KB/255.97 MB]
W0426 16:20:02.520300 836 tablet_updates.cpp:1591] wait_for_version slow(8789ms) version:1194.1 tablet:11189 #version:1173 [41 1194.1@1171 1195] pending: rowsets:5[id/seg/row/del/byte/compaction]: [1208/1/14700/0/110.04 KB/255.89 MB],[1209/1/2924/0/24.98 KB/255.98 MB],[1210/1/5649/0/55.96 KB/255.95 MB],[1211/1/8198482/0/51.58 MB/204.42 MB],[1212/1/4864/0/38.85 KB/255.96 MB]
W0426 16:35:13.346009 836 tablet_updates.cpp:1591] wait_for_version slow(12351ms) version:1713.1 tablet:11187 #version:1364 [370 1713.1@1362 1714] pending: rowsets:6[id/seg/row/del/byte/compaction]: [1733/1/3706/0/27.19 KB/255.97 MB],[1734/1/3221/0/23.86 KB/255.98 MB],[1735/1/4200/0/31.67 KB/255.97 MB],[1736/1/4776/0/35.58 KB/255.97 MB],[1737/1/10738158/0/67.41 MB/188.59 MB],[1738/1/3871/0/27.38 KB/255.97 MB]
W0426 16:35:36.546325 836 tablet_updates.cpp:1591] wait_for_version slow(12006ms) version:1719.1 tablet:11189 #version:1367 [373 1719.1@1365 1720] pending: rowsets:7[id/seg/row/del/byte/compaction]: [1738/1/6886/0/53.98 KB/255.95 MB],[1739/1/479/0/5.19 KB/255.99 MB],[1740/1/11473/0/72.20 KB/255.93 MB],[1741/1/25333/0/166.23 KB/255.84 MB],[1742/1/29194/0/191.68 KB/255.81 MB],[1743/1/10757903/0/67.52 MB/188.48 MB],[1744/1/22552/0/146.92 KB/255.86 MB]
W0426 16:48:17.532574 836 tablet_updates.cpp:1591] wait_for_version slow(15882ms) version:2110.1 tablet:11187 #version:1005 [1121 2110.1@1003 2111] pending: rowsets:7[id/seg/row/del/byte/compaction]: [2076/1/302222/0/1.91 MB/254.09 MB],[2136/1/4914/0/32.28 KB/255.97 MB],[2137/1/4916/0/34.68 KB/255.97 MB],[2138/1/4937/0/37.85 KB/255.96 MB],[2139/1/3419/0/27.57 KB/255.97 MB],[2140/1/13731377/0/85.57 MB/170.43 MB],[2141/1/4488/0/30.70 KB/255.97 MB]
W0426 16:48:45.111755 836 tablet_updates.cpp:1591] wait_for_version slow(15613ms) version:2117.1 tablet:11189 #version:1010 [1123 2117.1@1008 2118] pending: rowsets:9[id/seg/row/del/byte/compaction]: [2087/1/395496/0/2.48 MB/253.52 MB],[2141/1/14209/0/88.22 KB/255.91 MB],[2142/1/249/0/3.27 KB/256.00 MB],[2143/1/3413/0/25.60 KB/255.98 MB],[2144/1/4912/0/37.12 KB/255.96 MB],[2145/1/6125/0/47.16 KB/255.95 MB],[2146/1/4919/0/35.93 KB/255.96 MB],[2147/1/13660592/0/85.16 MB/170.84 MB],[2148/1/4922/0/30.26 KB/255.97 MB]
W0426 17:20:38.392560 836 tablet_updates.cpp:1591] wait_for_version slow(6024ms) version:3175.1 tablet:11187 #version:1320 [1876 3175.1@1318 3176] pending: rowsets:6[id/seg/row/del/byte/compaction]: [2140/1/13731377/0/85.57 MB/170.43 MB],[3217/1/3249/0/16.06 KB/255.98 MB],[3218/1/4159/0/16.25 KB/255.98 MB],[3219/1/4935/0/31.63 KB/255.97 MB],[3220/1/5979564/0/34.05 MB/221.95 MB],[3221/1/4625/0/21.32 KB/255.98 MB]
W0426 17:21:10.105254 836 tablet_updates.cpp:1591] wait_for_version slow(6454ms) version:3189.1 tablet:11189 #version:1332 [1878 3189.1@1330 3190] pending: rowsets:6[id/seg/row/del/byte/compaction]: [2147/1/13660592/0/85.16 MB/170.84 MB],[3231/1/3444/0/24.95 KB/255.98 MB],[3232/1/3921/0/28.11 KB/255.97 MB],[3233/1/4923/0/36.11 KB/255.96 MB],[3234/1/6111336/0/34.78 MB/221.22 MB],[3235/1/4956/0/23.24 KB/255.98 MB]
W0426 17:31:03.047030 836 tablet_updates.cpp:1591] wait_for_version slow(5333ms) version:3437.1 tablet:11126 #version:1222 [2235 3437.1@1220 3438] pending: rowsets:31[id/seg/row/del/byte/compaction]: [48/1/59894/0/2.16 MB/253.84 MB],[91/1/63357/0/2.51 MB/253.49 MB],[780/1/6571/0/649.71 KB/255.37 MB],[1502/1/8202/0/649.28 KB/255.37 MB],[1510/1/8178/0/792.57 KB/255.23 MB],[1748/1/6517/0/653.68 KB/255.36 MB],[1752/1/6724/0/688.45 KB/255.33 MB],[1813/1/8234/0/679.82 KB/255.34 MB],[1979/1/7886/0/741.96 KB/255.28 MB],[1980/1/7873/0/728.79 KB/255.29 MB],[1987/1/6908/0/655.80 KB/255.36 MB]…,[3480/1/1873/0/93.39 KB/255.91 MB],[3481/1/1284/0/88.65 KB/255.91 MB],[3482/1/3437/0/255.11 KB/255.75 MB],[3483/1/9466/0/631.85 KB/255.38 MB],[3484/1/9450/0/513.04 KB/255.50 MB],[3485/1/3465/0/313.15 KB/255.69 MB],[3486/1/8913/0/773.27 KB/255.24 MB],[3487/1/5225177/0/170.38 MB/85.62 MB],[3488/1/6024/0/408.16 KB/255.60 MB]
W0426 17:38:02.849318 836 tablet_updates.cpp:1591] wait_for_version slow(11050ms) version:3599.1 tablet:11187 #version:992 [2625 3599.1@990 3600] pending: rowsets:6[id/seg/row/del/byte/compaction]: [2140/1/13731377/0/85.57 MB/170.43 MB],[3649/1/6128/0/38.08 KB/255.96 MB],[3650/1/6076/0/39.37 KB/255.96 MB],[3651/1/2431/0/14.35 KB/255.99 MB],[3652/1/9332601/0/53.32 MB/202.68 MB],[3653/1/6109/0/37.15 KB/255.96 MB]
W0426 17:38:50.759980 836 tablet_updates.cpp:1591] wait_for_version slow(10858ms) version:3612.1 tablet:11189 #version:1001 [2629 3612.1@999 3613] pending: rowsets:6[id/seg/row/del/byte/compaction]: [2147/1/13660592/0/85.16 MB/170.84 MB],[3662/1/3028/0/20.07 KB/255.98 MB],[3663/1/5320/0/32.41 KB/255.97 MB],[3664/1/6111/0/43.57 KB/255.96 MB],[3665/1/9534412/0/54.44 MB/201.56 MB],[3666/1/3151/0/26.79 KB/255.97 MB]
W0426 17:41:37.972746 836 tablet_updates.cpp:1591] wait_for_version slow(5359ms) version:3676.1 tablet:11124 #version:1072 [2624 3676.1@1070 3677] pending: rowsets:43[id/seg/row/del/byte/compaction]: [46/1/56723/0/2.09 MB/253.91 MB],[88/1/61110/0/2.44 MB/253.56 MB],[780/1/6496/0/641.26 KB/255.37 MB],[1502/1/8200/0/648.12 KB/255.37 MB],[1510/1/8264/0/799.41 KB/255.22 MB],[1748/1/6560/0/657.88 KB/255.36 MB],[1752/1/6604/0/675.43 KB/255.34 MB],[1813/1/8189/0/673.88 KB/255.34 MB],[1979/1/7900/0/740.97 KB/255.28 MB],[1980/1/7960/0/739.34 KB/255.28 MB],[1988/1/6993/0/666.66 KB/255.35 MB]…,[3724/1/1573/0/143.42 KB/255.86 MB],[3725/1/617/0/86.07 KB/255.92 MB],[3726/1/624/0/60.68 KB/255.94 MB],[3727/1/1589/0/105.22 KB/255.90 MB],[3728/1/1528/0/106.95 KB/255.90 MB],[3729/1/1585/0/84.37 KB/255.92 MB],[3730/1/606/0/48.59 KB/255.95 MB],[3731/1/5411471/0/179.65 MB/76.35 MB],[3732/1/1595/0/94.98 KB/255.91 MB]
W0426 17:56:16.818560 836 tablet_updates.cpp:1591] wait_for_version slow(13791ms) version:4081.1 tablet:11189 #version:1089 [3013 4081.1@1087 4082] pending: rowsets:7[id/seg/row/del/byte/compaction]: [2147/1/13660592/0/85.16 MB/170.84 MB],[4138/1/4887/0/33.05 KB/255.97 MB],[4139/1/4899/0/32.26 KB/255.97 MB],[4140/1/3161/0/20.35 KB/255.98 MB],[4141/1/4157/0/25.42 KB/255.98 MB],[4142/1/12180699/0/69.95 MB/186.05 MB],[4143/1/6098/0/36.48 KB/255.96 MB]
W0426 17:57:45.505843 836 tablet_updates.cpp:1591] wait_for_version slow(15276ms) version:4119.1 tablet:11187 #version:806 [3330 4119.1@804 4120] pending: rowsets:8[id/seg/row/del/byte/compaction]: [2140/1/13731377/0/85.57 MB/170.43 MB],[4176/1/2955/0/16.68 KB/255.98 MB],[4177/1/4427/0/17.28 KB/255.98 MB],[4178/1/6128/0/31.28 KB/255.97 MB],[4179/1/6127/0/41.04 KB/255.96 MB],[4180/1/2457/0/16.32 KB/255.98 MB],[4181/1/12274477/0/70.49 MB/185.51 MB],[4182/1/2956/0/20.58 KB/255.98 MB]
W0427 08:18:07.778064 1 cpu_info.cpp:190] /sys/devices/system/node is not present - no NUMA support
E0427 08:18:23.071069 1 daemon.cpp:241] got signal: Terminated from pid: -728744528, is going to exit
W0427 08:18:31.740396 847 tablet_updates.cpp:1591] wait_for_version slow(15692ms) version:4835.1 tablet:11124 #version:1262 [3593 4835.1@1261 4835.1] pending: rowsets:41[id/seg/row/del/byte/compaction]: [46/1/56723/0/2.09 MB/253.91 MB],[88/1/61110/0/2.44 MB/253.56 MB],[780/1/6496/0/641.26 KB/255.37 MB],[1502/1/8200/0/648.12 KB/255.37 MB],[1510/1/8264/0/799.41 KB/255.22 MB],[1748/1/6560/0/657.88 KB/255.36 MB],[1752/1/6604/0/675.43 KB/255.34 MB],[1813/1/8189/0/673.88 KB/255.34 MB],[1979/1/7900/0/740.97 KB/255.28 MB],[1980/1/7960/0/739.34 KB/255.28 MB],[1988/1/6993/0/666.66 KB/255.35 MB]…,[3731/1/5411471/0/179.65 MB/76.35 MB],[3840/1/8427/0/646.47 KB/255.37 MB],[4141/1/68744/0/2.67 MB/253.33 MB],[4203/1/68459/0/2.93 MB/253.07 MB],[4280/1/74920/0/2.73 MB/253.27 MB],[4673/1/907195/0/30.63 MB/225.37 MB],[4752/1/79500/0/2.64 MB/253.36 MB],[4830/1/76820/0/2.03 MB/253.97 MB],[4907/1/81492/0/2.52 MB/253.48 MB]
W0427 08:21:01.270928 1 cpu_info.cpp:190] /sys/devices/system/node is not present - no NUMA support
W0427 08:21:17.852825 846 tablet_updates.cpp:1591] wait_for_version slow(11881ms) version:4835.1 tablet:11126 #version:1262 [3593 4835.1@1261 4835.1] pending: rowsets:35[id/seg/row/del/byte/compaction]: [48/1/59894/0/2.16 MB/253.84 MB],[91/1/63357/0/2.51 MB/253.49 MB],[780/1/6571/0/649.71 KB/255.37 MB],[1502/1/8202/0/649.28 KB/255.37 MB],[1510/1/8178/0/792.57 KB/255.23 MB],[1748/1/6517/0/653.68 KB/255.36 MB],[1752/1/6724/0/688.45 KB/255.33 MB],[1813/1/8234/0/679.82 KB/255.34 MB],[1979/1/7886/0/741.96 KB/255.28 MB],[1980/1/7873/0/728.79 KB/255.29 MB],[1987/1/6908/0/655.80 KB/255.36 MB]…,[3655/1/10487/0/767.89 KB/255.25 MB],[3684/1/10075/0/734.89 KB/255.28 MB],[4148/1/66316/0/2.75 MB/253.25 MB],[4211/1/69536/0/2.91 MB/253.09 MB],[4287/1/75109/0/2.77 MB/253.23 MB],[4683/1/1364291/0/44.78 MB/211.22 MB],[4759/1/78159/0/2.56 MB/253.44 MB],[4843/1/83199/0/2.19 MB/253.81 MB],[4907/1/68241/0/2.28 MB/253.72 MB]
E0427 08:22:54.379297 1 daemon.cpp:241] got signal: Terminated from pid: 2126583408, is going to exit
W0427 08:23:37.598146 1 cpu_info.cpp:190] /sys/devices/system/node is not present - no NUMA support
W0427 08:24:31.183040 861 tablet_updates.cpp:1591] wait_for_version slow(16817ms) version:4835.2 tablet:11124 #version:1263 [3593 4835.2@1262 4835.2] pending: rowsets:1[id/seg/row/del/byte/compaction]: [4908/1/7428598/0/254.63 MB/1.37 MB]
W0427 08:25:14.347487 861 tablet_updates.cpp:1591] wait_for_version slow(10463ms) version:4835.2 tablet:11126 #version:1264 [3593 4835.2@1262 4836] pending: rowsets:2[id/seg/row/del/byte/compaction]: [4908/1/7428706/0/254.44 MB/1.56 MB],[4909/1/4102/0/386.81 KB/255.62 MB]
W0427 08:26:36.548173 861 tablet_updates.cpp:1591] wait_for_version slow(45774ms) version:4852.1 tablet:11187 #version:1276 [3600 4852.1@1272 4855] pending: rowsets:11[id/seg/row/del/byte/compaction]: [4918/1/17163/0/109.34 KB/255.89 MB],[4919/1/8538/0/64.17 KB/255.94 MB],[4920/1/27935/0/182.77 KB/255.82 MB],[4921/1/43023/0/300.77 KB/255.71 MB],[4922/1/6581/0/48.73 KB/255.95 MB],[4923/1/3161/0/24.41 KB/255.98 MB],[4924/1/2453/0/18.39 KB/255.98 MB],[4925/1/28859903/0/173.40 MB/82.60 MB],[4926/1/2538/0/17.39 KB/255.98 MB],[4927/1/18239/0/117.10 KB/255.89 MB],[4928/1/250/0/3.93 KB/256.00 MB]
W0427 08:26:36.548981 861 storage_engine.cpp:943] Trace:
0427 08:25:20.726552 (+ 0us) storage_engine.cpp:947] start to perform update compaction
0427 08:25:20.726692 (+ 140us) storage_engine.cpp:955] found best tablet 11187
Metrics: {}
W0427 08:27:37.030292 861 tablet_updates.cpp:1591] wait_for_version slow(3166ms) version:4879.1 tablet:11187 #version:39 [4845 4879.1@37 4880] pending: rowsets:3[id/seg/row/del/byte/compaction]: [4925/1/28859903/0/173.40 MB/82.60 MB],[4953/1/239467/0/1.49 MB/254.51 MB],[4954/1/3651/0/29.92 KB/255.97 MB]
W0427 09:00:01.613147 861 tablet_updates.cpp:1591] wait_for_version slow(5164ms) version:5951.1 tablet:11189 #version:1119 [4850 5951.1@1117 5952] pending: rowsets:5[id/seg/row/del/byte/compaction]: [4915/1/28789914/0/172.88 MB/83.12 MB],[6037/1/4874/0/39.01 KB/255.96 MB],[6038/1/3682/0/28.16 KB/255.97 MB],[6039/1/6292875/0/39.83 MB/216.17 MB],[6040/1/3133/0/19.00 KB/255.98 MB]
W0427 09:00:43.709488 861 tablet_updates.cpp:1591] wait_for_version slow(6996ms) version:5972.1 tablet:11187 #version:1142 [4849 5972.1@1140 5973] pending: rowsets:6[id/seg/row/del/byte/compaction]: [4925/1/28859903/0/173.40 MB/82.60 MB],[6058/1/4759/0/40.19 KB/255.96 MB],[6059/1/4919/0/43.49 KB/255.96 MB],[6060/1/2452/0/21.14 KB/255.98 MB],[6061/1/6312427/0/40.00 MB/216.00 MB],[6062/1/2912/0/24.74 KB/255.98 MB]
W0427 09:15:28.950490 861 tablet_updates.cpp:1591] wait_for_version slow(8566ms) version:6457.1 tablet:11189 #version:1020 [5453 6457.1@1018 6458] pending: rowsets:5[id/seg/row/del/byte/compaction]: [4915/1/28789914/0/172.88 MB/83.12 MB],[6550/1/3626/0/20.99 KB/255.98 MB],[6551/1/3713/0/29.79 KB/255.97 MB],[6552/1/8840648/0/56.92 MB/199.08 MB],[6553/1/4881/0/35.66 KB/255.97 MB]
W0427 09:16:24.594455 861 tablet_updates.cpp:1591] wait_for_version slow(9684ms) version:6478.1 tablet:11187 #version:1042 [5453 6478.1@1040 6479] pending: rowsets:5[id/seg/row/del/byte/compaction]: [4925/1/28859903/0/173.40 MB/82.60 MB],[6572/1/18689/0/118.77 KB/255.88 MB],[6573/1/54915/0/361.24 KB/255.65 MB],[6574/1/8973068/0/57.71 MB/198.29 MB],[6575/1/23148/0/161.95 KB/255.84 MB]
W0427 09:18:14.617585 861 tablet_updates.cpp:1591] wait_for_version slow(4139ms) version:6523.1 tablet:11187 #version:1088 [5453 6523.1@1086 6524] pending: rowsets:5[id/seg/row/del/byte/compaction]: [4925/1/28859903/0/173.40 MB/82.60 MB],[6574/1/8973068/0/57.71 MB/198.29 MB],[6619/1/10047/0/61.68 KB/255.94 MB],[6620/1/471718/0/3.33 MB/252.67 MB],[6621/1/3185/0/25.97 KB/255.97 MB]
W0427 09:25:12.928895 861 tablet_updates.cpp:1591] wait_for_version slow(7945ms) version:6823.1 tablet:11126 #version:1035 [5804 6823.1@1033 6824] pending: rowsets:52[id/seg/row/del/byte/compaction]: [4919/1/33428/0/1.47 MB/254.53 MB],[4952/1/12992/0/706.01 KB/255.31 MB],[4958/1/12797/0/965.43 KB/255.06 MB],[4961/1/12792/0/1.16 MB/254.84 MB],[4964/1/12766/0/1.01 MB/254.99 MB],[5189/1/11146/0/800.98 KB/255.22 MB],[5190/1/9545/0/753.91 KB/255.26 MB],[5192/1/11114/0/925.45 KB/255.10 MB],[5193/1/9200/0/671.68 KB/255.34 MB],[5372/1/14186/0/767.11 KB/255.25 MB],[5378/1/12087/0/753.12 KB/255.26 MB]…,[6917/1/626/0/74.37 KB/255.93 MB],[6918/1/943/0/48.11 KB/255.95 MB],[6919/1/623/0/44.81 KB/255.96 MB],[6920/1/956/0/102.51 KB/255.90 MB],[6921/1/628/0/48.18 KB/255.95 MB],[6922/1/933/0/53.96 KB/255.95 MB],[6923/1/611/0/43.29 KB/255.96 MB],[6924/1/9458433/0/356.54 MB/-100.54 MB],[6925/1/946/0/79.64 KB/255.92 MB]
W0427 09:25:41.545120 861 tablet_updates.cpp:1591] wait_for_version slow(7572ms) version:6841.1 tablet:11124 #version:1054 [5803 6841.1@1052 6842] pending: rowsets:54[id/seg/row/del/byte/compaction]: [4919/1/33479/0/1.47 MB/254.53 MB],[4952/1/13023/0/709.49 KB/255.31 MB],[4958/1/12674/0/949.76 KB/255.07 MB],[4961/1/12776/0/1.15 MB/254.85 MB],[4964/1/12753/0/1.00 MB/255.00 MB],[5189/1/11113/0/803.54 KB/255.22 MB],[5190/1/9531/0/754.13 KB/255.26 MB],[5192/1/11107/0/924.46 KB/255.10 MB],[5193/1/9200/0/667.95 KB/255.35 MB],[5372/1/14193/0/772.89 KB/255.25 MB],[5378/1/12170/0/757.79 KB/255.26 MB]…,[6935/1/649/0/72.87 KB/255.93 MB],[6936/1/609/0/52.16 KB/255.95 MB],[6937/1/936/0/100.97 KB/255.90 MB],[6938/1/629/0/69.13 KB/255.93 MB],[6939/1/951/0/51.92 KB/255.95 MB],[6940/1/622/0/57.34 KB/255.94 MB],[6941/1/949/0/63.76 KB/255.94 MB],[6942/1/9394170/0/354.42 MB/-98.42 MB],[6943/1/625/0/65.94 KB/255.94 MB]
W0427 13:20:33.228695 861 tablet_updates.cpp:1591] wait_for_version slow(6310ms) version:17886.1 tablet:11124 #version:1228 [16672.1 17886.1@1226 17887] pending: rowsets:115[id/seg/row/del/byte/compaction]: [6942/1/9394170/0/354.42 MB/-98.42 MB],[13381/1/1286731/0/42.11 MB/213.89 MB],[17983/1/942/0/66.63 KB/255.93 MB],[17985/1/943/0/59.63 KB/255.94 MB],[17986/1/620/0/40.86 KB/255.96 MB],[17987/1/313/0/34.35 KB/255.97 MB],[17988/1/945/0/80.33 KB/255.92 MB],[17989/1/624/0/36.11 KB/255.96 MB],[17990/1/936/0/59.07 KB/255.94 MB],[17991/1/939/0/67.74 KB/255.93 MB],[17992/1/628/0/80.54 KB/255.92 MB]…,[18088/1/318/0/34.29 KB/255.97 MB],[18089/1/644/0/51.98 KB/255.95 MB],[18090/1/941/0/66.16 KB/255.94 MB],[18091/1/953/0/58.23 KB/255.94 MB],[18092/1/634/0/67.85 KB/255.93 MB],[18093/1/942/0/54.65 KB/255.95 MB],[18094/1/952/0/116.77 KB/255.89 MB],[18095/1/8056307/0/277.74 MB/-21.74 MB],[18096/1/629/0/67.12 KB/255.93 MB]
W0427 13:25:37.254652 861 tablet_updates.cpp:1591] wait_for_version slow(7107ms) version:18109.1 tablet:11126 #version:1011 [17110 18109.1@1009 18110] pending: rowsets:112[id/seg/row/del/byte/compaction]: [6924/1/9458433/0/356.54 MB/-100.54 MB],[13620/1/1859164/0/59.77 MB/196.23 MB],[18212/1/960/0/37.84 KB/255.96 MB],[18214/1/949/0/76.14 KB/255.93 MB],[18215/1/640/0/37.85 KB/255.96 MB],[18216/1/945/0/42.44 KB/255.96 MB],[18217/1/933/0/66.94 KB/255.93 MB],[18218/1/938/0/45.30 KB/255.96 MB],[18219/1/949/0/68.33 KB/255.93 MB],[18220/1/630/0/41.86 KB/255.96 MB],[18221/1/318/0/32.37 KB/255.97 MB]…,[18314/1/943/0/72.86 KB/255.93 MB],[18315/1/936/0/76.18 KB/255.93 MB],[18316/1/943/0/101.56 KB/255.90 MB],[18317/1/721/0/89.58 KB/255.91 MB],[18318/1/861/0/94.36 KB/255.91 MB],[18319/1/318/0/38.52 KB/255.96 MB],[18320/1/954/0/76.27 KB/255.93 MB],[18321/1/7597816/0/260.88 MB/-4.88 MB],[18322/1/639/0/52.41 KB/255.95 MB]
W0427 13:31:10.811489 861 tablet_updates.cpp:1591] wait_for_version slow(3298ms) version:18341.1 tablet:11124 #version:1247 [17109 18341.1@1245 18342] pending: rowsets:11[id/seg/row/del/byte/compaction]: [6942/1/9394170/0/354.42 MB/-98.42 MB],[18095/1/8056307/0/277.74 MB/-21.74 MB],[18548/1/948/0/97.70 KB/255.90 MB],[18549/1/317/0/34.33 KB/255.97 MB],[18550/1/666/0/57.95 KB/255.94 MB],[18551/1/1202/0/61.92 KB/255.94 MB],[18552/1/949/0/71.40 KB/255.93 MB],[18553/1/1092/0/72.74 KB/255.93 MB],[18554/1/1113/0/79.03 KB/255.92 MB],[18555/1/1755805/0/57.74 MB/198.26 MB],[18556/1/638/0/55.22 KB/255.95 MB]
W0427 17:13:49.682960 861 tablet_updates.cpp:1591] wait_for_version slow(7854ms) version:28094.1 tablet:11124 #version:1382 [26727 28094.1@1380 28095] pending: rowsets:126[id/seg/row/del/byte/compaction]: [6942/1/9394170/0/354.42 MB/-98.42 MB],[18095/1/8056307/0/277.74 MB/-21.74 MB],[18555/1/1755805/0/57.74 MB/198.26 MB],[28288/1/613/0/44.25 KB/255.96 MB],[28290/1/947/0/54.62 KB/255.95 MB],[28291/1/616/0/50.75 KB/255.95 MB],[28292/1/947/0/74.73 KB/255.93 MB],[28293/1/633/0/35.74 KB/255.97 MB],[28294/1/943/0/105.50 KB/255.90 MB],[28295/1/625/0/41.11 KB/255.96 MB],[28296/1/637/0/47.00 KB/255.95 MB]…,[28403/1/945/0/51.17 KB/255.95 MB],[28404/1/624/0/30.19 KB/255.97 MB],[28405/1/948/0/72.29 KB/255.93 MB],[28406/1/617/0/54.09 KB/255.95 MB],[28407/1/953/0/97.81 KB/255.90 MB],[28408/1/310/0/33.12 KB/255.97 MB],[28409/1/955/0/70.36 KB/255.93 MB],[28410/1/8104780/0/251.34 MB/4.66 MB],[28411/1/621/0/46.44 KB/255.95 MB]
W0427 17:16:37.370168 861 tablet_updates.cpp:1591] wait_for_version slow(8281ms) version:28216.1 tablet:11126 #version:1048 [27180 28216.1@1046 28217] pending: rowsets:125[id/seg/row/del/byte/compaction]: [6924/1/9458433/0/356.54 MB/-100.54 MB],[18321/1/7597816/0/260.88 MB/-4.88 MB],[28413/1/4711/0/275.95 KB/255.73 MB],[28414/1/893/0/69.04 KB/255.93 MB],[28416/1/673/0/51.68 KB/255.95 MB],[28417/1/948/0/87.74 KB/255.91 MB],[28418/1/637/0/69.45 KB/255.93 MB],[28419/1/945/0/59.61 KB/255.94 MB],[28420/1/739/0/67.04 KB/255.93 MB],[28421/1/830/0/58.60 KB/255.94 MB],[28422/1/316/0/30.98 KB/255.97 MB]…,[28528/1/943/0/71.43 KB/255.93 MB],[28529/1/934/0/75.47 KB/255.93 MB],[28530/1/640/0/52.69 KB/255.95 MB],[28531/1/951/0/81.76 KB/255.92 MB],[28532/1/945/0/100.86 KB/255.90 MB],[28533/1/947/0/85.14 KB/255.92 MB],[28534/1/942/0/48.91 KB/255.95 MB],[28535/1/10349323/0/333.44 MB/-77.44 MB],[28536/1/632/0/37.43 KB/255.96 MB]
W0427 17:48:02.307286 861 tablet_updates.cpp:1591] wait_for_version slow(9031ms) version:29671.1 tablet:11124 #version:1230 [28455 29671.1@1228 29672] pending: rowsets:27[id/seg/row/del/byte/compaction]: [6942/1/9394170/0/354.42 MB/-98.42 MB],[18095/1/8056307/0/277.74 MB/-21.74 MB],[28955/1/81424/0/2.83 MB/253.17 MB],[29063/1/83354/0/2.64 MB/253.36 MB],[29164/1/77215/0/2.49 MB/253.51 MB],[29261/1/75904/0/2.26 MB/253.74 MB],[29367/1/81415/0/2.77 MB/253.23 MB],[29466/1/75541/0/2.46 MB/253.54 MB],[29986/1/963/0/24.28 KB/255.98 MB],[29987/1/638/0/36.93 KB/255.96 MB],[29988/1/967/0/54.53 KB/255.95 MB]…,[29996/1/632/0/66.85 KB/255.93 MB],[29997/1/963/0/77.83 KB/255.92 MB],[29998/1/635/0/50.60 KB/255.95 MB],[29999/1/929/0/89.71 KB/255.91 MB],[30000/1/632/0/58.86 KB/255.94 MB],[30001/1/620/0/40.58 KB/255.96 MB],[30002/1/927/0/74.72 KB/255.93 MB],[30003/1/10722384/0/345.57 MB/-89.57 MB],[30004/1/616/0/70.40 KB/255.93 MB]
W0427 17:53:06.943413 833 stack_util.cpp:350] 2024-04-27 17:53:06.934845, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::filesystem::__cxx11::filesystem_error, trace:
@ 0xa64ad82 _ZNSt10filesystem9file_sizeERKNS_7__cxx114pathE.cold
@ 0x44a372e starrocks::TabletUpdates::_get_extra_file_size()
@ 0x44a3a06 starrocks::TabletUpdates::data_size()
@ 0x407b785 starrocks::TabletManager::update_root_path_info()
@ 0x3ef66a1 starrocks::StorageEngine::get_all_data_dir_info()
@ 0x3ee010f starrocks::ReportDiskStateTaskWorkerPool::_worker_thread_callback()
@ 0xa62ebb4 execute_native_thread_routine
@ 0x7ff9a246bac3 (unknown)
@ 0x7ff9a24fd850 (unknown)
@ (nil) (unknown)
E0427 17:53:06.947628 833 tablet_updates.cpp:405] get extra file size in primary table fail, tablet_id: 11124 status: Internal error: Iterate dir /opt/starrocks/be/storage/data/51/11124/130830931 Filesystem error: filesystem error: cannot get file size: No such file or directory [/opt/starrocks/be/storage/data/51/11124/130830931/index.l0.29912.0]
W0427 17:53:06.956574 834 stack_util.cpp:350] 2024-04-27 17:53:06.940983, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::filesystem::__cxx11::filesystem_error, trace:
@ 0xa64ad82 _ZNSt10filesystem9file_sizeERKNS_7__cxx114pathE.cold
@ 0x44a372e starrocks::TabletUpdates::_get_extra_file_size()
@ 0x44a4398 starrocks::TabletUpdates::get_tablet_info_extra()
@ 0x447a497 starrocks::Tablet::build_tablet_report_info()
@ 0x408134e starrocks::TabletManager::report_all_tablets_info()
@ 0x3ee1f28 starrocks::ReportOlapTableTaskWorkerPool::_worker_thread_callback()
@ 0xa62ebb4 execute_native_thread_routine
@ 0x7ff9a246bac3 (unknown)
@ 0x7ff9a24fd850 (unknown)
@ (nil) (unknown)
E0427 17:53:06.958189 834 tablet_updates.cpp:3174] get extra file size in primary table fail, tablet_id: 11124 status: Internal error: Iterate dir /opt/starrocks/be/storage/data/51/11124/130830931 Filesystem error: filesystem error: cannot get file size: No such file or directory [/opt/starrocks/be/storage/data/51/11124/130830931/index.l0.29912.0]
W0427 18:35:29.745079 861 tablet_updates.cpp:1591] wait_for_version slow(4427ms) version:31766.2 tablet:11124 #version:1151 [30629 31766.2@1150 31766.2] pending: rowsets:4[id/seg/row/del/byte/compaction]: [6942/1/9394170/0/354.42 MB/-98.42 MB],[18095/1/8056307/0/277.74 MB/-21.74 MB],[30003/1/10722384/0/345.57 MB/-89.57 MB],[32120/1/2127545/0/68.55 MB/187.45 MB]
W0427 20:11:11.105031 850 stack_util.cpp:350] 2024-04-27 20:11:11.103672, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::filesystem::__cxx11::filesystem_error, trace:
@ 0xa64ad82 _ZNSt10filesystem9file_sizeERKNS_7__cxx114pathE.cold
@ 0x44a372e starrocks::TabletUpdates::_get_extra_file_size()
@ 0x44a3a06 starrocks::TabletUpdates::data_size()
@ 0x407b785 starrocks::TabletManager::update_root_path_info()
@ 0x3ef66a1 starrocks::StorageEngine::get_all_data_dir_info()
@ 0x3efc6a8 starrocks::StorageEngine::_start_trash_sweep()
@ 0x407200b starrocks::StorageEngine::_garbage_sweeper_thread_callback()
@ 0xa62ebb4 execute_native_thread_routine
@ 0x7ff9a246bac3 (unknown)
@ 0x7ff9a24fd850 (unknown)
@ (nil) (unknown)
E0427 20:11:11.108469 850 tablet_updates.cpp:405] get extra file size in primary table fail, tablet_id: 12338 status: Internal error: Iterate dir /opt/starrocks/be/storage/data/21/12338/259545308 Filesystem error: filesystem error: cannot get file size: No such file or directory [/opt/starrocks/be/storage/data/21/12338/259545308/index.l0.81.0]
W0427 20:17:21.811383 861 tablet_updates.cpp:1591] wait_for_version slow(3244ms) version:425.1 tablet:12289 #version:434 [1 425.1@432 426] pending: rowsets:3[id/seg/row/del/byte/compaction]: [349/1/918181/401068/3.17 MB/259.75 MB],[431/1/401068/0/1.55 MB/254.45 MB],[432/1/5866/0/73.64 KB/255.93 MB]
W0427 20:21:35.795596 861 tablet_updates.cpp:1591] wait_for_version slow(5785ms) version:429.1 tablet:12338 #version:437 [1 429.1@435 430] pending: rowsets:4[id/seg/row/del/byte/compaction]: [432/1/1775/0/15.57 KB/255.98 MB],[433/1/8063/0/60.32 KB/255.94 MB],[434/1/2253517/0/12.87 MB/243.13 MB],[435/1/9937/0/76.59 KB/255.93 MB]
W0427 20:23:57.623831 861 tablet_updates.cpp:1591] wait_for_version slow(6373ms) version:506.1 tablet:12338 #version:515 [1 506.1@513 507] pending: rowsets:6[id/seg/row/del/byte/compaction]: [508/1/11278/0/74.77 KB/255.93 MB],[509/1/9786/0/69.12 KB/255.93 MB],[510/1/5219/0/40.99 KB/255.96 MB],[511/1/2838/0/26.25 KB/255.97 MB],[512/1/2868249/0/16.64 MB/239.36 MB],[513/1/2684/0/24.33 KB/255.98 MB]
W0427 20:25:09.167536 861 tablet_updates.cpp:1591] wait_for_version slow(8537ms) version:545.1 tablet:12336 #version:554 [1 545.1@553 545.1] pending: rowsets:3[id/seg/row/del/byte/compaction]: [550/1/3383/0/21.21 KB/255.98 MB],[551/1/2806/0/16.99 KB/255.98 MB],[552/1/3045188/0/17.79 MB/238.21 MB]
W0427 20:39:00.719089 861 tablet_updates.cpp:1591] wait_for_version slow(14518ms) version:1086.1 tablet:12338 #version:1052 [50 1086.1@1050 1087] pending: rowsets:4[id/seg/row/del/byte/compaction]: [512/1/2868249/0/16.64 MB/239.36 MB],[1098/1/3241/0/27.25 KB/255.97 MB],[1099/1/2158677/0/14.28 MB/241.72 MB],[1100/1/2469/0/21.71 KB/255.98 MB]
W0427 20:39:58.970141 861 tablet_updates.cpp:1591] wait_for_version slow(5927ms) version:1114.1 tablet:12336 #version:1080 [50 1114.1@1079 1114.1] pending: rowsets:4[id/seg/row/del/byte/compaction]: [552/1/3045188/0/17.79 MB/238.21 MB],[991/1/273924/0/1.86 MB/254.14 MB],[1127/1/6178/0/46.88 KB/255.95 MB],[1128/1/2000425/0/13.39 MB/242.61 MB]
W0429 14:34:40.247648 1 cpu_info.cpp:190] /sys/devices/system/node is not present - no NUMA support
W0429 14:34:54.605307 815 utils.cpp:95] Fail to get master client from cache. host=10.5.0.2 port=9020 code=THRIFT_RPC_ERROR
W0429 14:34:54.606249 818 utils.cpp:95] Fail to get master client from cache. host=10.5.0.2 port=9020 code=THRIFT_RPC_ERROR
W0429 14:34:54.632707 818 task_worker_pool.cpp:758] Fail to report workgroup to 10.5.0.2:9020, err=-1
W0429 14:34:54.632411 815 task_worker_pool.cpp:613] Fail to report task to 10.5.0.2:9020, err=-1
W0429 14:34:54.606441 819 utils.cpp:95] Fail to get master client from cache. host=10.5.0.2 port=9020 code=THRIFT_RPC_ERROR
W0429 14:34:54.634644 819 task_worker_pool.cpp:807] Fail to report resource_usage to 10.5.0.2:9020, err=-1
W0429 14:34:55.635615 819 utils.cpp:95] Fail to get master client from cache. host=10.5.0.2 port=9020 code=THRIFT_RPC_ERROR
W0429 14:34:55.635825 819 task_worker_pool.cpp:807] Fail to report resource_usage to 10.5.0.2:9020, err=-1
W0429 14:34:56.227546 816 utils.cpp:95] Fail to get master client from cache. host=10.5.0.2 port=9020 code=THRIFT_RPC_ERROR