Tuesday, July 16, 2024

Spring annotations and Basic in 5 mins - Spring interview refresh

Spring annotations  and Basics

  • @component - is the most generic one and  @repository, @ service, @controller are supposed to be used in their specific layers
  • @repository - So when you tag any component with at repository, spring would automatically add the exception translation for Jdbc exceptions.So whenever a Jdbc exception happens, then it needs to be translated into the specific spring exception.
  • spring beans are not thread safe
  • scope: singleton(default scope),  prototype, request and session.
  • request and session are both applicable only in web application context.
  • gang of four singleton or basic singleton:  it means one instance per class loader.
  • The spring singleton is one instance per container, one instance per application context. if you multiple application context than you will have multiple objects
  • Constructor injection: to create  immutable beans and for mandatory dependencies
  • Setter injection: Ex: internationalize something, which are not required while creating and setter injection for optional dependencies.
  • @Autowire: if only one implementation than by type. otherwise by name of the variable
  • NosuchBeanException: bean not scaned means 
  • NouniqueBeanDefination: if multiple implementation per interface and variable name is interface name
  • @Autowire
  • ClassA classAImpl;
    • Search by type  ClassA if multiple bean on same type , match by variable name "classAImpl" , if no match found then by name used @primary or match by qulifier
  • @primary : on class to tell use this implementation class for injection to solve NOsuchBeanException
  • @qulifier("") : used on class ann injection to identify  bean
  • BOM - bill of materials dependency  to have same spring versions.  all versions will take from dependency Management

 

Spark submit on YARN - Execution flow


When a Spark job is submitted to YARN, the following steps occur:

1. The client submits the job to the YARN ResourceManager (RM) via the YARN client.

2. The RM allocates a container for the ApplicationMaster (AM) on one of the cluster nodes. The AM is responsible for managing the lifecycle of the Spark application on YARN.

3. The AM starts on the allocated container and registers itself with the RM.

4. The AM negotiates resources with the RM to allocate containers for Spark executors.

5. The RM assigns containers to the AM, which in turn launches the Spark executors on these containers.

6. The executors register themselves with the AM.

7. The AM submits tasks to the executors and monitors their progress.

8. The executors execute the tasks and report the results back to the AM.

9. The AM aggregates the results and updates the job status.

10. Once the job is completed or terminated, the AM releases the resources and notifies the RM.


The ApplicationMaster is responsible for coordinating the execution of the Spark application on YARN. It negotiates with the RM to allocate resources (containers) for the Spark executors and manages their lifecycle. It also monitors the progress of the executors and handles failures by requesting the RM to re-allocate containers if needed.


The ApplicationManager is a component within the RM that receives requests from clients to submit applications. It is responsible for accepting application submissions, allocating resources to the applications, and tracking their status. The ApplicationManager works closely with the ApplicationMaster to manage the execution of Spark applications on YARN.


The relationship between the ApplicationMaster and the executors is that the AM is responsible for launching and managing the lifecycle of the executors. It negotiates with the RM to allocate resources (containers) for the executors and monitors their progress. The executors register themselves with the AM and execute the tasks assigned by the AM. The AM communicates with the executors to collect the results and update the job status.

 

HDFS - Edit logs and FsImage

 HDFS - Edit logs and FsImage

What is DistCp? - It is a tool that is used for copying a very large amount of data to and from Hadoop file systems in parallel

Default replication factor? By default, the replication factor is 3. There are no two copies that will be on the same data node. Usually, the first two copies will be on the same rack, and the third copy will be off the shelf. It is advised to set the replication factor to at least three so that one copy is always safe, even if something happens to the rack.

The two types of metadata that NameNode server stores are in Disk and RAM.

Edit logs: all transaction logs like added block, deleted blocks, replication details after last fsimage file created. The updates are periodically merged to FSImage.

FsImage: file stored in OS system and contains the complete directory stucture of hdfs. Name node reads when starting to memory


Spark execution optimisers - Catalyst and Tungsten


Spark uses two engines to optimize and run the queries - Catalyst and Tungsten, in that order. Catalyst basically generates an optimized physical query plan from the logical query plan by applying a series of transformations like predicate pushdown, column pruning, and constant folding on the logical plan. 

This optimized query plan is then used by Tungsten to generate optimized code, that resembles hand written code, by making use of Whole-stage Codegen functionality introduced in Spark 2.0. This functionality has improved Spark's efficiency by a huge margin from Spark 1.6, which used the traditional Volcano Iterator Model 

Spark execution plans

 Spark execution plan: Spark's Catalyst optimizer creates plans

  • Logical Plan: abstract syntax tree (AST) and this doesn’t have how data divided into partitions, what algorithm would use
  • Physical Plan: how the data will be partitioned, which specific algorithms will be used, and how the results will be returned
  • Execution Plan: physical operation that involves shuffling data, reading data, filtering data, or performing computations


How Spark executer handles threads

 How spark executer handles threads:

Spark executor handles threads by determining the number of tasks that can be run in parallel based on the number of cores, scheduling tasks to available threads, ensuring thread safety, handling task failures, and managing resources

1. Number of cores

2. Scheduling

3. Thread Safety

4. Fault Tolerance

5. Resource Management

How to fix Lazy evaluation overhead

Technic to handle Lazy evaluation overhead 

1. Caching and Persisting RDDs

2. Using Broadcast Variables 

3. Avoiding Operations that Cause Shuffling : Operations like groupByKey and reduceByKey 

4. Using the right data structures**: DataFrames and Datasets 

5. Tuning Spark configurations: auto scale confgs

6. Checkpointing


Spark - Shuffle Optimisation:

Spark -  Shuffle Optimisation:

  • spark.sql.shuffle.partitions parameter  - 200mb
  • spark.serializer to org.apache.spark.serializer.KryoSerializer faster than java serial and deserial
  • Reduce Disk I/O  - spark.shuffle.compress — whether the engine would compress shuffle outputs or not. Default value is “true”.
  • spark.shuffle.spill.compress — whether to compress intermediate shuffle spill files or not. Default value is “true”.
  • spark.io.compression.codec codec for compressing the data, which is snappy by default.

For smaller datasets , generally snappy works well with most of the datasets but there is emerging compression codec “zstld” which surpasses snappy performance. Details are here.This has been introduced by Facebook and attached is the link for details.

Optimize Spark’s In-memory computation: Spark uses memory to store intermediate results during shuffling. Adjust the memory usage for shuffling by tuning the spark.memory.fraction parameter. By default, this parameter is set to 0.6, which means that 60% of the executor memory is used for storage/caching and 40% is used for execution


BigData - Distributed System Design Patterns

Below are thlist of interesting topic/technics which are used/considered while building Hadoop or any Distributed systems.

Bloom Filters  - used to check key available in big/huge dataset

High-water mark index -  index refers, index all followers are written

Lease - lock release on resource

Heartbeat - Worker periodically send signal to master to indicate there availability. 

Fence -  Put a 'Fence' around the previous leader to prevent it from doing any damage or causing corruption. Fencing is the idea of putting a fence around a previously active leader so that it cannot access cluster resources and hence stop serving any read/write request. The following two techniques are used:

  • Resource fencing
  • Node fencing

Examples: HDFS uses fencing to stop the previously active NameNode from accessing cluster resources, thereby stopping it from servicing requests.

High-water mark index

Distributed systems keep multiple copies of data for fault tolerance and higher availability. To achieve strong consistency, one of the options is to use a leader-follower setup, where the leader is responsible for entertaining all the writes, and the followers replicate data from the leader.

Quorum - used for HA. 

Write-ahead log (WAL) -  log file where mater node write/append data 

Circuit Breaker Micro Service design Pattern

 

Circuit Breaker Pattern - it acts as a safeguard against service failures by monitoring x, setting thresholds, and temporarily halting/Stopping traffic to failing services. It helps prevent cascading failures and maintains system stability, ensuring reliable performance in distributed architectures.

Closed State: Initially, the circuit breaker is in a Closed state, allowing requests through.

Open State: If a certain number of requests fail (like timeouts or errors), the breaker "trips" to an Open state. This stops calls to the failing service, giving it time to recover.

Half-Open State: After a cooldown period, the breaker enters a Half-Open state, allowing a limited number of test requests through. If these succeed, it goes back to Closed; if not, it returns to Open.



Top Micro services Design Patterns in 5 mins

 




Monday, December 13, 2021

Critical RCE 0day (ZERO day) in Apache Log4j library was reported (CVE-2021-44228) - Fix

Critical RCE 0day in Apache Log4j library was reported (CVE-2021-44228) and Apache suggested to upgrade your log4j to version 2.15.x:  https://threatpost.com/zero-day-in-ubiquitous-apache-log4j-tool-under-active-attack/176937/

https://logging.apache.org/log4j/2.x/

The Log4j team has been made aware of a security vulnerability, CVE-2021-44228, that has been addressed in Log4j 2.15.0.

For those who cannot upgrade to 2.15.0, in releases >=2.10, this vulnerability can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.


Wednesday, July 21, 2021

Google Cloud Dataflow Pipeline using JdbctoBigQuery template "timezone region not found" issue - Fix

Sometimes Dataflow pipeline with JdbctoBigQuery template may not work due to : "error occurred at recursive SQL level 1 ORA-01882: timezone region not found"


Error details:

Error
2021-07-21T06:22:13.292218504ZError message from worker: java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.sql.SQLException: Cannot create PoolableConnectionFactory (ORA-00604: error occurred at recursive SQL level 1 ORA-01882: timezone region not found ) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:197) org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:168)

Steps to Fix the issue:
1. extract oracle/jdbc/defaultConnectionProperties.properties file from ojdbcXXXXX.jar file.
    jar xf o ojdbcXXXXX.jar oracle/jdbc/defaultConnectionProperties.properties
2. add oracle.jdbc.timezoneAsRegion=false propery
3. then update jdbc jar file with updated defaultConnectionProperties.properties
    jar uf ojdbcXXXXX.jar oracle/jdbc/defaultConnectionProperties.properties