During INSERT/UPDATE operations, I get the following errors frequently - too many tablet versions (>1000)

Working with Data
1

UPDATE was called on a table many times. This resulted in an error - too many tablet versions (>1000). And it was suggested to decrease the concurrent jobs or increase the batch size. What is a tablet version? When does it increase or decrease?

2

There is a formal definition of a tablet on the docs website. For your situation, think of it as another avro file. Compactation hasn’t happened yet to convert a group of them together into a parquet files.

Generally speaking, don’t treat your olap like a oltp.

3

Every ingestion indicates a version. StarRocks uses MVCC to manage the ingestion.

too many tablet versions

Mostly comes from the issue that the ingestion is too frequent, or the tablet number is large.
Suggestions:

  1. Check the tablet number. For partition, it’s better to less than 100GB. For tablet, 1GB per tablet is recommended.
  2. Check the ingestion frequency. 100 MB per stream load is recommended.
4

How many updates are you doing per second? Batch? Are they individual updates or bulk?

5

good afternoon, i have the same problem Too many versions and i didn’t know how to solve it

6

decrease the concurrent jobs or increase the batch size. You’ll also have to wait for the compaction job to run so it can consolidate all the versions / small files into larger file.

7

Thanks! decreasing the concurrent jobs helped. In my case, the batchUpdate was being used which triggered one SQL statement per INSERT instead I optimized the INSERT statement to have multiple records and trigger that.