ANALYTICAL DATABASE / SOURCE READING / 27 DEEP DIVES
Doris architecture
Follow Apache Doris from classic FE/BE architecture to cloud separation, through query, write, optimisation, execution, storage, metadata, high availability, modelling, tuning, lakehouse, multitenancy, observability best practices, production operations, and source contribution notes. The source guides are currently maintained in Chinese.
The source notes for this track are currently maintained in Chinese.Curriculum
27 deep dives
- 00Apache Doris architecture overviewA Chinese deep dive into Doris FE/BE topology, metadata, execution, storage, and external-system boundaries.
- 01SQL query lifecycleTrace a SQL query from the MySQL protocol through FE optimisation, scheduling, BE execution, and storage reads.
- 02Write lifecycleFollow Doris write paths through transactions, tablets, rowsets, and segment generation.
- 03Nereids optimizerStudy Doris logical planning, rules, costing, and physical plan generation through the Nereids Cascades framework.
- 04Pipeline execution engineBreak down fragments, pipelines, operators, and BE tasks in Doris' parallel execution engine.
- 05Vectorized executionExplain how blocks, columns, SIMD, and expression execution drive Doris vectorized query processing.
- 06Tablet, Rowset, and Segment storageUnderstand Doris columnar storage through tablets, rowsets, segments, pages, and scan boundaries.
- 07Compaction and version managementStudy version chains, publish flow, compaction, and visibility in Doris storage.
- 08Storage indexesMap how short-key indexes, ZoneMap, Bloom filters, Bitmap indexes, and other filters reduce scan cost.
- 09Catalog metadata and JournalUnderstand Doris metadata hierarchy, persistence, and recovery through Catalog and Journal internals.
- 10Data ingestion systemCompare Doris ingestion methods and their transaction lifecycles across Stream Load, Routine Load, Broker Load, and related paths.
- 11FE high availability and replicationExplain FE roles, BDBJE election, Journal replication, and high-availability behavior.
- 12BE replica consistency and repairStudy BE replica consistency, clone, repair, and tablet health management.
- 13RPC communication frameworkMap the responsibilities of Thrift, Brpc, Protobuf, and FE/BE communication paths.
- 14Resource management and Workload GroupUnderstand resource isolation, memory control, and query governance through Workload Group, MemTracker, and RuntimeFilter.
- 15Doris Cloud storage-compute separationCompare Doris Classic and Cloud architecture, including object storage, local cache, and elastic compute trade-offs.
- 16Monitoring, performance, and troubleshootingTurn metrics, query slowness, ingestion issues, replica problems, and resource symptoms into a Doris troubleshooting playbook.
- 17Table design and data modellingTranslate business access patterns into Doris data models, sort keys, partitioning, bucketing, indexes, and colocate choices.
- 18Slow SQL and Query Profile tuningUse slow SQL evidence, EXPLAIN, and Query Profile to move from symptoms to operator-level bottlenecks.
- 19Materialized views and query accelerationDesign query acceleration with synchronous rollups, async materialized views, transparent rewrite, refresh strategy, and resource isolation.
- 20Real-time ingestion, updates, and deletesBuild a production method for CDC, idempotent labels, version buildup, Unique Key updates, delete semantics, and Routine Load.
- 21Lakehouse and federated queryUnderstand External Catalogs, partition pruning, Data Cache, small-file governance, and when to accelerate lake data inside Doris.
- 22Observability and semi-structured analyticsDesign observability analytics for logs, traces, metrics, VARIANT fields, inverted indexes, and hot/cold retention.
- 23Production operations and disaster recoveryOrganise capacity, tablet health, scaling, FE/BE availability, backup, recovery, upgrades, and incident response for Doris production.
- 24Security, governance, and multitenancyBuild a Doris governance model across accounts, roles, LDAP/Ranger, Workload Groups, SQL blocking, audit, and masking.
- 25Source contribution and debuggingUse Query IDs, EXPLAIN, Profile, and source maps to reproduce, debug, test, and contribute to Apache Doris.
- 26Observability data best practicesA production playbook for modelling and operating logs, traces, metrics, retention, indexes, ingestion, rollups, and troubleshooting in Doris.