Tuesday, July 16, 2024

Spark execution plans

 Spark execution plan: Spark's Catalyst optimizer creates plans

  • Logical Plan: abstract syntax tree (AST) and this doesn’t have how data divided into partitions, what algorithm would use
  • Physical Plan: how the data will be partitioned, which specific algorithms will be used, and how the results will be returned
  • Execution Plan: physical operation that involves shuffling data, reading data, filtering data, or performing computations


No comments: