Spark submit on YARN - Execution flow

When a Spark job is submitted to YARN, the following steps occur:

1. The client submits the job to the YARN ResourceManager (RM) via the YARN client.

2. The RM allocates a container for the ApplicationMaster (AM) on one of the cluster nodes. The AM is responsible for managing the lifecycle of the Spark application on YARN.

3. The AM starts on the allocated container and registers itself with the RM.

4. The AM negotiates resources with the RM to allocate containers for Spark executors.

5. The RM assigns containers to the AM, which in turn launches the Spark executors on these containers.

6. The executors register themselves with the AM.

7. The AM submits tasks to the executors and monitors their progress.

8. The executors execute the tasks and report the results back to the AM.

9. The AM aggregates the results and updates the job status.

10. Once the job is completed or terminated, the AM releases the resources and notifies the RM.

The ApplicationMaster is responsible for coordinating the execution of the Spark application on YARN. It negotiates with the RM to allocate resources (containers) for the Spark executors and manages their lifecycle. It also monitors the progress of the executors and handles failures by requesting the RM to re-allocate containers if needed.

The ApplicationManager is a component within the RM that receives requests from clients to submit applications. It is responsible for accepting application submissions, allocating resources to the applications, and tracking their status. The ApplicationManager works closely with the ApplicationMaster to manage the execution of Spark applications on YARN.

The relationship between the ApplicationMaster and the executors is that the AM is responsible for launching and managing the lifecycle of the executors. It negotiates with the RM to allocate resources (containers) for the executors and monitors their progress. The executors register themselves with the AM and execute the tasks assigned by the AM. The AM communicates with the executors to collect the results and update the job status.