GE Compiler Core Flow — From AscendIR to Executable Model【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/geIntroduces transformation chain AscendIR experiences after entering compiler, and final product form. Compiler is multi-stage flow, each step has clear responsibility and design constraints.1. Compilation Flow OverviewGE compiler compiles AscendIR into OM model throughCompilerStages, divided into four stages: preprocessing, graph optimization, engine partition and build:1.1 CompilerStages: Four Major Compilation StagesGE compiler entry isGraphManager, it manages graph lifecycle: AddGraph → Build → Run. Core compilation flow organized throughCompilerStagesstructure into four stages:struct CompilerStages { GraphPrepare preparer; // Preprocessing: normalization, Shape inference GraphOptimize optimizer; // Optimization: graph-level optimization, engine optimization EnginePartitioner partitioner; // Partition: divide subgraphs by engine GraphBuilder builder; // Build: memory planning, stream allocation, task generation };This design follows each stage one responsibility principle, adopts modular flow similar to traditional compilers (such as LLVM Pass Manager).1.2 Complete Compilation Flow1.3 Three-Stage Optimization DesignGE divides graph optimization into three stages (PreRunOptimizeOriginalGraph→PreRunOptimizeSubGraph→PreRunAfterOptimizeSubGraph):Stage One (OriginalGraph Optimization): Before engine partitioning, perform engine-independent general optimization on the complete graph. At this time, all operators have not been assigned to specific engines, and the optimizer can freely perform cross-engine operator fusion and elimination.Stage Two (SubGraph Optimization): After engine partitioning, each subgraph is assigned to a specific engine (such as FE fusion engine), and each engine performs engine-specific optimization on the subgraph assigned to itself. This step is multi-threaded parallel — subgraphs of different engines do not interfere with each other.Stage Three (AfterOptimizeSubGraph Optimization): After subgraph optimization is merged back into the whole graph, perform post-optimization from a whole graph perspective. Subgraph boundaries may have prevented some optimizations that need to cross subgraphs, and can be reviewed again after merging.Three-stage optimization achieves a balance between generality and performance: engine-specific optimization (such as FE fusion) needs to be executed after partitioning, while the overhead of partition-merge also needs to be controlled.1.4 Tuning Mode: Compiler Breakpoint MechanismGE supports a special Build Mode (BUILD_MODE_TUNING), allowing pausing at different compilation stages:BUILD_STEP_BEFORE_UB_MATCH: Pause before UB matchingBUILD_STEP_AFTER_UB_MATCH: Pause after UB matchingBUILD_STEP_AFTER_BUILD: Pause after buildThis enables AOE (Ascend Optimization Engine) to inject its own tuning logic at intermediate stages, and then resume compilation. This is a compiler plugin mechanism, similar to GCCs plugin interface or LLVMs pass insertion points.2. Graph-Level Optimization: Pass System2.1 Pass InfrastructureGEs optimization Passes fall into two categories:GraphPass: Runs on the whole graph unit, managed byPassManagerfor sequential execution (passes/pass_manager.h). Caller registers throughAddPass(name, pass), thenRun(graph)executes in order.NodePass (BaseNodePass): Runs on node unit, traverses each node of the graph throughGEPassframework (passes/base_pass.h).GEPassprovides complex re-traversal mechanism:GEPass::Run(names_to_passes) { For each node in graph: For each NodePass: pass.Run(node) If pass modified graph structure: Collect nodes needing re-traversal (nodes_need_re_pass_) Collect nodes needing immediate re-traversal (nodes_need_re_pass_immediately_) If there are nodes needing re-traversal: Re-traverse these nodes }Optimization Pass may modify graph structure (add/delete nodes), causing subsequent nodes to see a graph different from before. GEPasssAddRePassNodeandAddImmediateRePassNodemechanism allows Pass to declare this new node needs to be processed again by other Passes. The immediate re-traversal (ImmediateRePass) capability enables certain modifications to be immediately seen by subsequent Passes in the current round, avoiding performance overhead of multiple rounds of iteration.2.2 Organization of Optimization PassesGEs optimization Passes execute in multiple batches, distributed at different stages. Core Passes include:OptimizeStage1(Pre-partition optimization):Pass Sub-stageKey PassPurpose1. Graph Structure OrganizationMergeInputMemcpyPass, SwitchDataEdgesBypassNormalize control flow1. Constant OptimizationConstantFuseSamePass, CommonSubexpressionEliminationPassEliminate redundant constants1. Data OptimizationFuseDataNodesWithCommonInputPassMerge Data nodes with same input1. Transform OptimizationPermutePass, SameTransdataBreadthFusionPass, TransOpBreadthFusionPassFormat transform optimization1. Variable OptimizationVariableOpPassVariable acceleration2. Node-level OptimizationConstantFoldingPass, CastRemovePass, ReshapeRemovePass etc.Node elimination and simplification3. Control Flow TransformSwitchToStreamSwitchPass, MergeToStreamMergePass, AttachStreamLabelPassControl flow → Stream control3. Dynamic BatchingMultiBatchPass, SubgraphMultiDimsPassMulti-dimension dynamic inferenceOptimizeStage2(Post-merge optimization):Stage2 Passes process the merged whole graph, at this time subgraph boundaries have been eliminated:InnerIdentityDeletePass: Delete intermediate Identity nodesHcclContinuousMemcpyPass: Communication operator continuous memory copy optimizationConstantFoldingPass(Round 2): After merging may have new constant folding opportunitiesCondRemovePass / AssignRemovePass: Condition/Assignment node eliminationAtomicAddrCleanPass: Atomic clear address managementSubgraphPass: Handle memory conflicts between subgraphsAttachStreamLabelPass: Stream label allocationLabelAllocator: Functional operator label allocationBufferPoolMemoryPass: Buffer pool memory optimizationParallelGroupPass: Parallel group processingConcatNotaskPass: Concat no-task optimization2.3 Necessity of Two-Stage OptimizationEngine partitioning changes the graphs topology structure, this is the core reason requiring two-stage optimization.Stage1 runs before partitioning, can safely perform:Constant folding (does not depend on engine information)Common subexpression elimination (does not depend on engine information)Control flow transformation (needs to know all control flow nodes)Stage2 runs after partitioning and merging, at this time needs to handle:Memcpy nodes introduced by subgraph boundariesNew constant folding opportunities after engine-specific optimizationMemory read-write conflicts between subgraphsGEs three-stage optimization flow guarantees predictability of optimization order — this is crucial for a compiler that needs to support multiple AI framework backends. GE provides custom Pass extension point throughFusionPassExecutor(fusion/pass/fusion_pass_executor.h), allowing users to register custom fusion Passes.2.4 ATC Compilation Option EntryATC offline compilation entry inapi/atc/main_impl.ccmerges command line arguments and optional raw JSON configuration into one flat options map, then passes to GE Compiler. Raw JSON only parsescompile options; ATC will first inject raw value to correspondingFLAGS_*by CLI priority, reuse original CLI validation, parsing and side effects, then construct final options. After entering compiler, no longer distinguish option source.Related design see ATC Raw GE Options.3. Fusion Optimization3.1 Two Fusion RoutesGEs fusion optimization follows two routes:Route One: Hand-written Pattern Fusion(compiler/graph/fusion/)Implemented through Pattern Matcher framework for declarative fusion rules. Developers describe what kind of subgraph pattern should be fused, and framework is responsible for matching and replacing in target graph.Route Two: Auto Fusion(compiler/graph/optimize/autofuse/)Based on operator classification and dependency analysis, automatically identifies fusionable operator combinations. This subsystem (AutofuseOptimize) is called atAfterPrecisionRefinestage.3.2 Pattern Matcher Fusion FrameworkCore components of fusion framework (compiler/graph/fusion/):Matching algorithm (pattern_matcher.cc) adoptsbacktracking search:Start from Pattern graph output node, find type-matching node in target graphTraverse Pattern graph and target graph backward along data edges, match node by nodeIf some branch does not match, backtrack to last branch point and try next candidateAfter all branches match successfully, validate subgraph boundary validity (InnerSubgraphBoundary)Starting matching from output node is because output nodes are usually much fewer than intermediate nodes — output node type and quantity are Patterns most distinctive part. Starting from output can quickly prune, avoiding large amount of invalid intermediate node matching.3.3 FusionPassExecutor: Fusion Pass ExecutorFusionPassExecutor(fusion/pass/fusion_pass_executor.h) is responsible for executing fusion Passes registered throughREG_FUSION_PASSmacro. It is called at two positions in compilation flow:OptimizeOriginalGraph: Execute engine-level built-in fusion Pass custom PassRunCustomPassAfterOriginGraphOptimize: Execute user-registered custom PassCurrently custom Pass is no longer limited to C static registration. Python pass will also register toPassRegistrythrough bridge, then at execution phase be connected to existing main flow by three types of adapters:PythonFusionBasePassAdapterdirectly calls Pythonrun(graph, context)PythonPatternFusionPassAdapterreuses CPatternFusionPass::Run(), only callbacks Python onPatterns / MeetRequirements / Replacementthree hooksPythonDecomposePassAdapterreuses CDecomposePass::Run(), only callbacks Python onMeetRequirements / Replacementtwo hooksThis design guarantees existing execution semantics ofFusionPassExecutor,PassRegistry,PatternFusionPassandDecomposePassdo not need to set up a parallel scheduling framework for Python.PythonPatternFusionPassbesides adapter protocol also provides a layer of expression-style syntactic sugar: users can declare pattern expression throughpatternmethod, and can definereplacement(self, inputs)returning replacement expression. Python layer will automatically create ESGraphBuilder, graph input, graph output and pattern capture, finally still return originalPattern/Graphobjects to C bridge. Multiplepatternmethods will be synthesized into multiple patterns returned by legacypatterns(self); old explicitpatterns(self)is still compatible, but cannot be mixed withpatternmethods. This layer of encapsulation only changes Python-side usability, does not change C pass execution flow and matching semantics.For mechanism explanation and development steps面向开发者, see Fusion Pattern Pass Mechanism.To lower customFusionBasePassintegration cost,ge/fusion/graph_fuse_inspector_utils.haddsGraphFuseInspectorUtilspublic utility class. It converges key steps originally scattered inComputeGraph::IsSupportFuse,FusionUtils::WillCauseCycleIfFuse,FusionUtils::UpdateToCycleDetectorand fusion statistics logic into two open capabilities:CanFuse(nodes_before_fuse, failed_reason): Execute fusionability validation (attribute consistency cycle detection), failure reason returned throughfailed_reason.ReportFuse(nodes_before_fuse, nodes_after_fuse, ctx): Called after graph modification and before releasing old nodes, usepass_nameinctxto mark new node fusion source, update cycle detector and record fusion debugging; whennodes_after_fuseis empty indicates only deleting nodes.InSubgraphRewriteraddedReplace(subgraph, replacement, ctx)overload, chainingCanFuseandReportFuseinto unified graph modification flow: check fusionability before modification, report fusion result after modification, then delete old nodes.3.4 Auto Fusion (AutofuseOptimize)Auto fusion executes after precision adjustment and before format adjustment, timing choice is critical: precision is already determined (no more Cast insertion), but format is not yet fixed (still has transformation space).Auto fusion subsystem (compiler/graph/optimize/autofuse/) contains complete subdirectory structure:ascendc/(AscendC operator fusion),ascir/,att/,codegen/,compiler/,optimize/etc, indicating it not only makes fusion decisions, but also involves code generation of fused operators — this is a complete path from operator classification to code generation.4. Engine Partitioning4.1 Necessity of Engine PartitioningAscend devices have multiple execution engines, each engine responsible for different types of operators:EngineResponsibilityTypical Operatorsnn_engine (AIcoreEngine)AI Core matrix computationMatMul, Conv, SoftmaxVectorEngineVector computationElementWise operationscpu_engine (HostCpu)Host CPU executionOperators not supporting device executionhccl_engineCollective communicationAllReduce, Broadcastdvpp_engineDigital visual preprocessingImage/video processingffts_engineFFT operationsFrequency domain transformrts_engineRuntime servicesStreamSwitch, StreamActiveOperators of different engines cannot be placed in same execution sequence, therefore need to assign operators to correct execution engine through engine partitioning.4.2 Partitioning FlowEngine partitioning is completed byEnginePartitioner(partition/engine_partitioner.h), flow as follows:Key step analysis:StagePartition: For training graphs, partition graph by training stage (forward/backward/update). This is training-scenario specific requirement — different stages may use different optimization strategies.EnginePlacer: Determine execution engine for each operator. This step determines by querying operator registration information (OpDescsGetOpKernelLibName()). Assignment strategy implemented inengine_place.cc.DynamicShapePartition: Dynamic Shape partitioning (partition/dynamic_shape_partition.cc) divides graph into known Shape and unknown Shape subgraphs. Unknown Shape subgraphs need special runtime scheduling (device-side Shape computation).Two-level Partitioning (Composite Atomic):kCompositeEnginePartitioning: First partition by composite engine (such as FE fusion engine)kAtomicEnginePartitioning: Then partition by atomic engineReason for two-level partitioning is: fusion engine needs to first see complete fusionable region, atomic engine partitioning is after fusion optimization.4.3 Cluster-Based Partitioning AlgorithmEnginePartitioneruses Cluster-based partitioning algorithm:Initialization: Create a Cluster for each nodeMarking: Mark each Clusters engine according to engine assignment resultMerging: If adjacent Clusters belong to same engine, and no second path (HasSecondPath), then mergeSplitting: Insert Placeholder/End node pairs according to merged Cluster boundariesHasSecondPath checkis algorithms key: if multiple data paths exist between two Clusters, cannot simply merge — merging will change other paths data flow.4.4 Multi-threaded Parallelism of Subgraph OptimizationAfter partitioning, each subgraph is assigned to different engine, through thread pool parallel optimization:OptimizeSubGraphWithMultiThreads: ThreadPool executor(16 threads) for each subgraph: executor.commit(ProcessSubGraphWithMultiThreads)Each thread independently calls enginesOptimizeFusedGraphmethod for one subgraph. Optimization of different subgraphs does not depend on each other — this is guaranteed by partitioning algorithm (subgraphs connect through Placeholder/End, structure completely independent).5. Build Stage5.1 GraphBuilder: Build EntryGraphBuilder(build/graph_builder.h) is build stage entry, core method isBuild():5.2 ModelBuilder: Model BuildingModelBuilder(build/model_builder.h) is responsible for:Stream Allocation(StreamAllocator): Allocate execution streams for operators in graphMemory Planning: Determine memory offset for each tensorWeight Merging(MergeWeights): Merge all weights into one continuous memory regionBuild Model Definition: Serialize graph structure into Model Protocol Buffer5.3 Stream Allocation (StreamAllocator)StreamAllocator(build/stream/stream_allocator.h) is responsible for:Stream Allocation Design Philosophy:Ascend devices Stream is an ordered queue of device-side operations — operations in one stream execute sequentially, different streams can parallelize. Stream allocation core contradiction is:Parallelism vs Sync Overhead.More streams → More parallel opportunities → But need more sync EventsFewer streams → Less sync overhead → But lower parallelismGEs strategy is:First allocate logical streams by engine and stream label (AssignLogicalStreams)Insert sync nodes between logical streams (Event/Notify)Split overly long streams based on task count (SplitStreams)Optimize sync Event reuse (ReuseEvent)StreamSplitHelperstructure tracks task count and split state on each stream. When tasks on some stream exceed hardware limit, automatically split into multiple physical streams.5.4 Memory Planning (GraphMemoryAssigner)GraphMemoryAssigner(build/memory/graph_mem_assigner.h) implements memory reuse planning:Memory Allocator Hierarchy:MemAssigner (Interface) ├── HybridMemAssigner (Hybrid Allocator) │ ├── MaxBlockMemAssigner (Max Block Allocator - Priority) │ └── BinaryBlockMemAssigner (Binary Block Allocator) ├── DynamicBatchMemAssigner (Dynamic Batch Memory) └── VariableMemoryAssigner (Variable Memory)Memory Reuse Strategy:GE uses block-based memory reuse (BlockMemAssigner), core idea is:Organize tensors by lifecycleIf two tensors lifecycles do not overlap, they can share same memory blockManage continuous memory regions throughMemoryBlockabstractionGEs memory planning is a static analysis, needs to handle multiple memory types (HBM, P2P, Host), and supports zero copy (ZeroCopy) optimization — input tensors can directly use user-provided memory, without additional copy.GraphMemSplitter(memory/graph_mem_splitter.h) is responsible for finer-grained memory splitting at graph level, handling memory sharing and isolation between subgraphs.5.5 TaskGenerator: Task GenerationTaskGenerator(build/task_generator.h) converts optimized graph into executable task sequence:GenerateTask: Generate corresponding hardware task for each nodeSupport Fusion Nodes: Fusion operators (such as TBE fusion operators) generate single taskSupport FFTS Nodes: FFTS operators have special task generation pathMulti-threaded Generation: Use thread pool to parallel generate tasksDuring task generation, each operatorsOpKernelLibNamedetermines which execution engine to use to generate task. GeneratedTaskDef(Protocol Buffer format) contains:Operator binary (TBE Kernel / AscendC Kernel)Input output offsetsStream IDWorkspace size and offset5.6 Compilation Product: GeRootModelCompilation final product isGeRootModel, it contains:Root Graph: Original ComputeGraph (contains compiled metadata)Subgraph Model Mapping(SubgraphInstanceNameToModel): Each subgraphs correspondingGeModelEachGeModelcontains:Task sequence (ModelTaskDef)Weight data (weight_buffer)TBE Kernel storage (tbe_kernel_store)Memory layout information (stream count, event count, memory size)Stream allocation resultThis product after serialization to OM (Offline Model) format, can be directly loaded and executed on Ascend device.6. Operator Compilation6.1 Online Compilation MechanismOperator compilation occurs at two timings:During Engine Subgraph Optimization: Fusion engine (FE) calls operator compiler atOptimizeFusedGraphstageModelBuilder Stage:CompileSingleOpcalls TBE/AscendC compiler for each operator needing compilationOpCompileAdapterunderopcompiler/directory provides operator compilation adapter interface. Operator compilation detailed process is not inside GE — GE calls external compiler (such as TBEsop_tilingop_build) to generate operator binary.6.2 TBE Kernel StoreCompiled operator binary is stored inTBEKernelStore(model_builder.h), finally serialized to OM file. EachTBEKernelcontains:Operator nameCompiled binaryInput output description7. Model CacheGE supports model cache (build/model_cache.h), avoids repeated compilation:BuildModel: ModelCache.Init(root_graph) if ModelCache.TryLoadModelFromCache(): return cached_model // Cache hit else: DoBuildModel() // Normal compilation ModelCache.TryCacheModel() // Cache resultCache key is graphs hash — throughComputeHashForConstNodescalculate SHA256 hash for constant nodes, as part of cache key.8. Shape OptimizationGEs Shape optimization converts dynamic shape to static shape as much as possible:Constant Folding and InferShape Collaboration: Shape inference and constant folding alternate until convergence. For example, Shape → Gather(indices[1,0,2,3]) → Reshape chain, if Datas shape is [3,4,5,6], first do Shape inference to derive Reshape output as dynamic, then do constant folding to eliminate Shape and Gather, finally do inference again for Reshapes output — at this time Reshapes shape input is already const [4,3,5,6], output becomes static.While Loop Shape Inference: For While operators body subgraph multiple inference iterations, use previous inferences output Shape as next input, until two results are consistent. This fixed-point iteration strategy infers as much static information as possible.Dynamic Gear: For scenarios where shape has regular variations (such as different batch size), throughMapIndex Caseoperators split one dynamic graph into N static subgraphs. Each inference selects corresponding subgraph based on input shape execution, obtaining dynamic shape flexibility through static subgraph sinking.9. Weight OptimizationWeight Merging: Merge scattered weight data into continuous memory region, making load phase more efficientConst Deduplication: Through binary comparison discover Const nodes with same weight, let them share memory10. Compiler Design CharacteristicsGE compilers most significant characteristic isexplicit engine partitioning and stream allocation— this is because Ascend hardware has clear heterogeneous engines (AI Core, Vector Core, Host CPU), unlike other hardware platforms that mainly have one unified execution unit.GE adopts single-layer IR, compilation is fast. Scheduling information passes through operator Tiling parameters, rather than decided in compiler. This makes GEs compiler more concise, transferring scheduling complexity to operator implementation.11. Key Design Decision SummaryThree-stage Optimization: Before partitioning, after partitioning (subgraph level), after merging. Partitioning changes graph structure, must do different optimizations at different timings.Cluster-Based Partitioning: Greedy algorithm based on adjacent Cluster merging, in practice can effectively complete engine partitioning.Multi-threaded Subgraph Optimization: After partitioning subgraphs are mutually independent, naturally supports parallelism. 16 thread pool (default) can significantly accelerate compilation of large-scale graphs.Two-stage Engine Partitioning (Composite Atomic): First let fusion engine see complete fusionable region, then do fine atomic engine partitioning.Stream Allocation Event Reuse: Event is hardware resource (limited quantity), throughReuseEventmechanism reuse Event at different time segments, reduce hardware resource consumption.Memory Allocation Block Abstraction: ThroughMemoryBlockmanage continuous memory regions, support tensors with non-overlapping lifecycles sharing memory blocks. Compared to simple each tensor independent allocation, this can significantly reduce total memory footprint.Compiler-produced GeRootModel contains complete execution plan — task sequence, memory layout, stream allocation result — this is exactly the blueprint next chapter runtime module needs to load and execute: How does runtime understand these compilation products, and drive entire computation flow on Ascend device?【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考