Unable to create tables disk size

Infrastructure and Operations
1

I have currently deployed a starrocks cluster with 1 FE and 6 BE nodes in a share nothing topology in k8s.

I am using starrocks/fe-ubuntu, starrocks/be-ubuntu and tag 3.3-latest

Problem
When following the guide to load data via insert, I get an error

] failed to find any backend with qualified disk usage from 5 candidate backends, needAvailable=true,

Expected
I can create the table and insert data

Toubleshooting

  1. I notice the be nodes are healthy, but reporting NO STORAGE
mysql> show backends;
+-----------+---------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------+----------------------------------------+-------------------+-------------+----------+-------------------+------------+------------+------------------+----------+
| BackendId | IP                                                      | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg | Version       | Status                                 | DataTotalCapacity | DataUsedPct | CpuCores | NumRunningQueries | MemUsedPct | CpuUsedPct | DataCacheMetrics | Location |
+-----------+---------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------+----------------------------------------+-------------------+-------------+----------+-------------------+------------+------------+------------------+----------+
| 10001     | test-be-0.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:28:12 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
| 10002     | test-be-1.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:28:37 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
| 10003     | test-be-2.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:28:47 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
| 10042     | test-be-4.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:29:52 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
| 10047     | test-be-5.test-be.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2024-09-16 20:30:07 | 2024-09-16 20:32:47 | true  | false                | false                 | 0         | 0.000 B          | 1.000 B       | 0.000 B       | 0.00 %  | 0.00 %         |        | 3.3.3-312ed45 | {"lastSuccessReportTabletsTime":"N/A"} | 0.000 B           | 0.00 %      | 1        | 0                 | 0.00 %     | 0.0 %      | N/A              |          |
+-----------+---------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------+----------------------------------------+-------------------+-------------+----------+-------------------+------------+------------+------------------+----------+
  1. The storage_root_path is properly set in the /opt/starrocks/be/conf/be.conf location and being set properly when running a curl on the variables
$ curl http://test-be-0.test-be.svc.cluster.local:8040/varz | grep storage_root_path
storage_root_path=/opt/starrocks/be/storage
  1. The storage/disk is mounted correctly and there is a good amount of space
# df -h | grep storage; exit
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdd        9800G  6.3M  9800G   1% /opt/starrocks/be/storage

Any thoughts?

2

I also have that kind of issue but not exactly.

I tried writing /opt/starrocks/be/storage directory and double checked the configuration files (be.conf and fe.conf) to ensure there are no misconfigurations. But didn’t get success, you can try the same and let me know if working.

Thanks

3

yes this does not working upon doing the same. I can manually write using linux touch or mkdir, but starrocks thinks the nodes have no space

4

You need to add in the storage_root_path in the be.conf

5

that is done, see #2 and #3

6

fixed. hostname in the database was wrong for some reason,
solution,

ALTER SYSTEM MODIFY FRONTEND HOST ‘${BAD HOSTNAME}’ TO ‘${CORRECT HOSTNAME}’