STREAM PROCESSING / SOURCE READING / 15 DEEP DIVES
Flink internals
Follow the runtime call path from APIs and scheduling through networking, state, and observability. The source guides are currently maintained in Chinese.
The source notes for this track are currently maintained in Chinese.Curriculum
15 deep dives
- 00Apache Flink architecture overviewA Chinese deep dive into Flink's runtime topology, component responsibilities, and execution path.
- 01Job submission and schedulingTrace how a JAR or SQL job becomes distributed tasks through Flink's submission and scheduling path.
- 02End-to-end stream dataflowFollow records from source to transform and sink through operators, task chains, and the network layer.
- 03Checkpoint and SavepointUnderstand the boundaries and collaboration between checkpoints, savepoints, barriers, and recovery.
- 04State and StateBackendStudy how Flink state is organised, stored, and rebuilt through the State API and state backends.
- 05Windows and timersSee how windows, triggers, timer services, and time semantics decide when stream computation runs.
- 06Watermarks and event timeLearn how watermarks define out-of-order boundaries and safely advance event-time computation.
- 07Table/SQL planner and CalciteConnect Table/SQL, Calcite planning, optimisation, and Flink physical execution plans.
- 08BackpressureUnderstand backpressure through symptoms, propagation paths, and network flow control.
- 09Network stack and Netty ShuffleExplore Netty Shuffle, result partitions, and credit-based flow control for cross-task transport.
- 10TaskManager memory modelBreak down heap, direct, managed, and network memory in Flink's TaskManager model.
- 11Resource schedulingLearn how ResourceManager, slots, resource declarations, and scheduling map plans to cluster capacity.
- 12Connector framework and Source/Sink APIsStudy connector lifecycles, concurrency, and consistency through FLIP-27 Source and Sink APIs.
- 13State recovery and rescalingExplain recovery, key groups, repartitioning, and rescaling trade-offs for evolvable jobs.
- 14Monitoring, performance, and troubleshootingTurn metrics, flame graphs, backpressure, checkpoints, and symptoms into an executable troubleshooting sequence.