[Compiler Refactor 8] Migrate all operators to use the new OpExecConfig class#1817
Merged
Conversation
zuozhiw
added a commit
that referenced
this pull request
Feb 2, 2023
…Plan implementation (#1807) This PR does the following changes: 1. Refactors the `Workflow` class to use the new `PhysicalPlan` class. After this change, all helper functions related to traversing the physical plan DAG is inside `PhysicalPlan`. 2. In all classes dealing with the actual physical plan, use `LayerID` (physical operator ID) instead of `OperatorID`(logical operator ID) to identify an operator. Specifically, `OperatorID` consists of `(workflowID, operatorID)` (logical operator). `LayerID` consists of `(workflowID, operatorID, layerID)`, one logical operator can corresponds to multiple physical operators (such as aggregate and visualization). 3. Changes from #1794 and #1817 are also merged into master in this branch. Previously, these two PRs are merged into this base branch. In the previous two PRs, all operators are updated to adopt the new `OpExecConfig` API. 4. After all operators adopt the new `OpExecConfig` API, the old code are all cleaned up. Specifically, many old `XxxOpExecConfig` classes are no longer needed and they are all unified into the new OpExecConfig` API. 5. Refactors the `WorkflowPipelinedBuilder` class to use the new `PhysicalPlan` API. Separate the logic of deiciding a region and adding a materialization operator with a new `MaterializationRewriter` class. 6. Refactors the compilation phase into a logical plan building and a physical plan building phase. In the physical plan building phase, adds a new `PartitionEnforcer` to decide the shuffle policies of each link. The old ad-hoc way to decide shuffle policies (`DeploymentFilter`) is removed.
yangzhang75
pushed a commit
to yangzhang75/texera
that referenced
this pull request
Jun 22, 2026
…Plan implementation (apache#1807) This PR does the following changes: 1. Refactors the `Workflow` class to use the new `PhysicalPlan` class. After this change, all helper functions related to traversing the physical plan DAG is inside `PhysicalPlan`. 2. In all classes dealing with the actual physical plan, use `LayerID` (physical operator ID) instead of `OperatorID`(logical operator ID) to identify an operator. Specifically, `OperatorID` consists of `(workflowID, operatorID)` (logical operator). `LayerID` consists of `(workflowID, operatorID, layerID)`, one logical operator can corresponds to multiple physical operators (such as aggregate and visualization). 3. Changes from apache#1794 and apache#1817 are also merged into master in this branch. Previously, these two PRs are merged into this base branch. In the previous two PRs, all operators are updated to adopt the new `OpExecConfig` API. 4. After all operators adopt the new `OpExecConfig` API, the old code are all cleaned up. Specifically, many old `XxxOpExecConfig` classes are no longer needed and they are all unified into the new OpExecConfig` API. 5. Refactors the `WorkflowPipelinedBuilder` class to use the new `PhysicalPlan` API. Separate the logic of deiciding a region and adding a materialization operator with a new `MaterializationRewriter` class. 6. Refactors the compilation phase into a logical plan building and a physical plan building phase. In the physical plan building phase, adds a new `PartitionEnforcer` to decide the shuffle policies of each link. The old ad-hoc way to decide shuffle policies (`DeploymentFilter`) is removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is a continuation of #1794 . In this PR, all the operators are migrated to use the new
OpExecConfigclass. This PR involves changes to more complicated operators such as sources, aggregations, and joins.