Skip to content

feat/cir 20293 - #66611

Open
dzr171712 wants to merge 2 commits into
apache:masterfrom
dzr171712:fix/CIR-20293
Open

feat/cir 20293#66611
dzr171712 wants to merge 2 commits into
apache:masterfrom
dzr171712:fix/CIR-20293

Conversation

@dzr171712

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: None

Related PR: None

Internal tracking issue: CIR-20293

Problem Summary:

In production, operators often need to trigger BASE, CUMULATIVE, or FULL
compaction for a single abnormal tablet.

Before this change, there was no SQL-level and auditable way to trigger
compaction for a specific tablet.

The existing approaches have the following limitations:

  1. Calling the BE HTTP API requires users to know the BE address.
    It also bypasses the unified SQL audit path.
  2. The existing ADMIN COMPACT TABLE syntax works at partition granularity.
    It is too coarse for handling a single abnormal tablet.
  3. The existing partition-level command requires global ADMIN privilege and
    is mainly suitable for integrated storage and compute deployments.

How it was fixed:

This PR adds a new SQL syntax:

ADMIN COMPACT TABLET <tablet_id>
WHERE TYPE = 'BASE|CUMULATIVE|FULL';
The FE resolves the database, table, partition, index, tablet, and replicas
from the specified tablet ID. It then reuses the existing compaction task
request and submits the task to the corresponding BE nodes.
The implementation includes:
Added parser support for the tablet-level syntax.
Added FE command handling for tablet-level compaction.
Added tablet metadata resolution and privilege validation.
Added local deployment support in Env.
Added cloud deployment support in CloudEnv.
Reused the existing BE compaction task handling without changing BE code.
Classified the command as StmtType.ADMIN so that it is recorded in the
FE audit log.
Preserved the behavior of the existing partition-level compaction syntax.
Behavior changes
Before this PR:
Tablet compaction required direct access to the BE HTTP API.
Users had to know the BE address.
The operation was not triggered through the unified SQL audit path.
SQL compaction was only available at table partition granularity.
After this PR:
Users can trigger compaction for a single tablet through FE SQL:
ADMIN COMPACT TABLET 1786334146168
WHERE TYPE = 'CUMULATIVE';
The command supports:
ADMIN COMPACT TABLET <tablet_id> WHERE TYPE = 'BASE';

ADMIN COMPACT TABLET <tablet_id> WHERE TYPE = 'CUMULATIVE';

ADMIN COMPACT TABLET <tablet_id> WHERE TYPE = 'FULL';
The command can be executed by users with global ADMIN privilege or the
required table-level ALTER privilege.
The operation is recorded in the FE audit log with:
StmtType=ADMIN
This change only adds a more precise compaction trigger. It does not change
the data format, compaction algorithm, or existing BE compaction behavior.
The existing partition-level syntax remains unchanged.
The main possible impact is that manually triggering compaction can consume
BE CPU, memory, and I/O resources. The request is still subject to the
existing BE compaction task scheduling and concurrency controls.
Features added
This PR adds SQL-level, tablet-granularity compaction triggering.
The feature is needed because production troubleshooting often identifies
an abnormal tablet rather than an entire table partition. Operators should
be able to repair or compact only the affected tablet without directly
calling a BE HTTP endpoint or compacting unrelated tablets.
Refactoring
No unrelated refactoring was performed.
The implementation reuses the existing AdminCompactTableCommand,
CompactionTask, TCompactionReq, and BE task worker flow as much as
possible. The changes are limited to parser support, FE command dispatch,
tablet metadata lookup, privilege checking, and cloud/local backend routing.
Optimization
This PR improves the operational granularity of manual compaction.
Previously, operators had to compact a complete partition or call the BE
HTTP API directly. The new SQL command allows FE to route the request by
tablet ID, avoiding unnecessary compaction of unrelated tablets and
providing unified privilege control and audit logging.
Release note
Add SQL support for triggering BASE, CUMULATIVE, and FULL compaction for a
single tablet:
ADMIN COMPACT TABLET <tablet_id>
WHERE TYPE = 'BASE|CUMULATIVE|FULL';
Check List (For Author)
Test
Regression test

Unit Test

Manual test (add detailed scripts or steps below)

No need to test or manual test. Explain why:
This is a refactor/code format and no logic has been changed.

Previous test can cover this change.

No code files have been changed.

Other reason


Regression tests:
Added a local deployment regression test for tablet-level cumulative
compaction.
Added a cloud deployment regression test for tablet-level cumulative
compaction.
Verified that the compaction success time is updated.
Verified that the number of data rowsets decreases after compaction.
Verified that table data remains correct after compaction.
Unit tests:
Added parser coverage for the existing partition-level compaction syntax.
Added parser coverage for the new tablet-level compaction syntax.
Verified that both statements are parsed as
AdminCompactTableCommand.
Verified that both statements use StmtType.ADMIN.
Manual test:
Built the FE successfully:
NPM_CONFIG_CACHE=/tmp/doris-npm-cache ./build.sh --fe
Started the FE and verified that the FE and BE were alive:
SHOW FRONTENDS;
SHOW BACKENDS;
Triggered tablet-level cumulative compaction:
ADMIN COMPACT TABLET 1786334146168
WHERE TYPE = 'CUMULATIVE';
Verified that the command returned successfully through FE SQL.

Verified from the BE log that the compaction task was submitted:

successfully submit task|type=COMPACTION|signature=1786334146168
Verified that the FE audit log recorded the SQL statement with:
StmtType=ADMIN
Stmt=ADMIN COMPACT TABLET 1786334146168 WHERE TYPE = 'CUMULATIVE'
Verified that the existing partition-level compaction syntax remains
compatible.Behavior changed:

No.

Yes. Added a new tablet-level SQL compaction command while keeping
the existing partition-level command behavior unchanged.

Does this need documentation?

No.

Yes. This PR introduces a new user-facing SQL syntax. A
documentation PR will be submitted separately.


Check List (For Reviewer who merge this PR)

Confirm the release note

Confirm test cases

Confirm document

Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants