TiDB Monthly Update for May 2020

Releases

We have three releases this month:

• TiDB 4.0 RC.2 Release Notes
  • New features
    • Add support for the BACKUP and RESTORE commands to back up and restore data #16960
    • Support pre-checking the data volume in a single Region before committing and pre-splitting the Region when the data volume exceeds the threshold #16959
    • Support recording the Cop_time information in slow logs and the SLOW_LOG table #16904
  • Bug fixes
• TiDB 3.0.14 Release Notes
  • New features
    • Add the schema name column and the table name column to the query results of the ADMIN SHOW DDL JOBS statement 16428
    • Enhance the RECOVER TABLE syntax to support recovering truncated tables #15458
  • Bug fixes
• TiDB Operator 1.1.0 GA Release Notes

Blog posts

Nick Cameron has written Large Transactions in TiDB that describes how we implemented support for large transactions.

Fei Yang's article TiCDC: Replication Latency in Milliseconds for 100+ TB Clusters dives deep into why we built TiCDC and how we implement its key features.

In TiDB Dashboard: Easier Troubleshooting for Distributed Databases, Wenxuan Shi, Ming Zhang, and Shuang Chen walks you through the dashboards's main widgets: Key Visualizer, SQL statement analysis, slow query viewing, cluster diagnostics, log search, and instance profiling.

Notable pull requests

Planner

• Adopt another way to estimate the cost of join reorder
• Add a proposal document for the clustered index feature
• Support the IN expression for the Range partition

DDL

• Support exchanging partitions

Dashboard

• Support persisting the Key Visualizer data
• Support customizing the public path prefix behind a reverse proxy
• Add GZIP for the static server
• Support highlighting log search keywords

Execution

• Improve the performance of min/max aggregation functions by using the sliding window
• Add an apply cache to speed up NestedLoopJoin
• Support GROUP_CONCAT(ORDER BY)

Tools

• Improve the robustness of Backup & Restore (BR) #247 #298 #7917
• BR supports views and sequences
• TiCDC supports the cyclic replication
• TiDB Data Migration (DM) supports the leader eviction

Transaction

• Check the size of a lock column family when considering large CFs
• Pre-split Regions during two-phase commit execution to avoid hotspots

Engine

• Titan supports the compaction filter

Scheduling

• Add an engine filter to exclude special stores

Kubernetes

• Support noise reduction for auto-scaling
• Support arbitrary topology-based HA in TiDB Scheduler

Notable issues

Planner

• To improve the usability of SQL Plan Management
• To support the adaptive query execution

Tools

• To simplify DM configuration
• To fix the issue that Dumpling cannot dump heavily skewed data in parallel
• Sync-diff-inspector needs a better checksum algorithm
• To pre-split a Region before the Region becomes a write hotspot to avoid the server is busy error in TiKV
• To make the writes of a large transaction more graceful

Engine

• To support auto-compaction in the Raw KV mode

Scheduling

• To separate the effect of balance-region-limit on balance Region and merge Region

Kubernetes

• To support incremental backup and restore with BR

Call for participation

Wanna participate with us? Take a look at the following help-wanted issues, which are mentoring available:

• To provide a more convenient way to modify the TiKV data directory
• TiCDC should support more data formats in a message queue sink
• Support not merging some small Regions
• Call For Participation: SIG DDL 2020/Q1 Plan

New contributors

We’d like to welcome the following new contributors to TiDB and thank them for their work!

tidb:

• @crodjer

tikv:

• @longfangsong

pd:

• @ccw1996
• @TennyZhuang

docs:

• @Hangzhi

docs-cn:

• @glkappe
• @ica10888
• @AdamKorcz

tidb-operator:

• @PengJi
• @vincent178

parser:

• @PengJi

This Month in TiKV

For more detailed and comprehensive information about TiKV, we have TiKV Monthly Update. The following covers May.

This Month in TiKV - May 2020