Static flume interceptor

 tier1.sources  = source1
 tier1.channels = channel1
 tier1.sinks = sink1

 tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
 tier1.sources.source1.zookeeperConnect = 135.250.193.206:2181
 tier1.sources.source1.topic = flum_test
 tier1.sources.source1.groupId = flume
 tier1.sources.source1.channels = channel1

 tier1.sources.source1.interceptors = i1
 tier1.sources.source1.interceptors.i1.type = static
 tier1.sources.source1.interceptors.i1.key = one
 tier1.sources.source1.interceptors.i1.value = 1

 tier1.channels.channel1.type = memory
 tier1.channels.channel1.capacity = 10000
 tier1.channels.channel1.transactionCapacity = 1000

 tier1.sinks.sink1.type = hdfs
 #tier1.sinks.sink1.hdfs.path = /tmp/kafka/%{topic}/%y-%m-%d
 tier1.sinks.sink1.hdfs.path =hdfs://135.250.193.206:8020/user/hive/warehouse/spm_database.db/%{topic}/%{one}
 tier1.sinks.sink1.hdfs.rollInterval = 5
 tier1.sinks.sink1.hdfs.rollSize = 0
 tier1.sinks.sink1.hdfs.rollCount = 0
 tier1.sinks.sink1.hdfs.fileType = DataStream
 tier1.sinks.sink1.channel = channel1

Flume config - without source

agent1.sinks = sink1
agent1.channels = channel1

# Describe channel here
agent1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
agent1.channels.channel1.brokerList = 135.250.193.206:9092
agent1.channels.channel1.topic = flum_test
agent1.channels.channel1.zookeeperConnect = 135.250.193.206:2181
agent1.channels.channel1.parseAsFlumeEvent=false
                                                                                                                                                                             
agent1.channels.channel1.transactionCapacity = 1000000
agent1.channels.channel1.checkpointInterval = 30000
agent1.channels.channel1.maxFileSize = 2146435071
agent1.channels.channel1.capacity= 10000000

agent1.sinks.sink1.channel = channel1

# Describe sink1                                                                                                                                                                                                                          
agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.hdfs.path =  hdfs://135.250.193.206:8020/user/hive/warehouse/spm_database.db/flum_test/year=%Y/month=%m/day=%d
agent1.sinks.sink1.hdfs.useLocalTimeStamp = true
agent1.sinks.sink1.hdfs.filePrefix = LogCreateTest

agent1.sinks.sink1.hdfs.rollInterval = 600
agent1.sinks.sink1.hdfs.rollSize = 0
agent1.sinks.sink1.hdfs.rollCount = 10000
agent1.sinks.sink1.hdfs.batchSize = 10000
agent1.sinks.sink1.hdfs.txnEventMax = 40000
agent1.sinks.sink1.hdfs.fileType = DataStream
agent1.sinks.sink1.hdfs.maxOpenFiles=50
agent1.sinks.sink1.hdfs.appendTimeout = 10000
agent1.sinks.sink1.hdfs.callTimeout = 10000
agent1.sinks.sink1.hdfs.threadsPoolSize=100
agent1.sinks.sink1.hdfs.rollTimerPoolSize = 1

Steps to install flume

service iptables stop
service ip6tables stop
sudo yum install flume-ng flume-ng-agent flume-ng-doc
sudo yum install flume-ng-agent
sudo yum install flume-ng-doc


If flume not comming up then run below commands:
sudo service cloudera-scm-agent hard_stop
service cloudera-scm-agent start

scheduling workflows using oozie coordinator

1. Create folder and create lib under new folder in hdfs
hadoop fs -mkdir /user/oozie/shekhar
hadoop fs -mkdir /user/oozie/shekhar/lib
2. Copy workflow.xml and coordinator.xml file to hdfs location /user/oozie/shekhar
hadoop fs -put workflow.xml /user/oozie/shekhar/
3. Copy your jars to hdfs location /user/oozie/shekhar/lib
hadoop fs -put sparkanalitics-1.jar /user/oozie/shekhar/lib/
4. Create coordinatorjob.properties and simpleSparkTest.sh file in current dir (not in HDFS)
5. Run using oozie command in current dir
$ oozie job -oozie http://localhost:11000/oozie -config coordinatorjob.properties -submit
job: 0000673-120823182447665-oozie-hado-C

logs will generates in /root/oozie-oozi/0000673-120823182447665-oozie-hado-C folder

Suspending the coordinator job
$ oozie job -oozie http://localhost:11000/oozie -suspend 0000673-120823182447665-oozie-hado-C
Resuming a Coordinator Job
$ oozie job -oozie http://localhost:11000/oozie -resume 0000673-120823182447665-oozie-hado-C
Killing a Coordinator Job
$ oozie job -oozie http://localhost:11000/oozie -kill 0000673-120823182447665-oozie-hado-C
Rerunning a Coordinator Action or Multiple Actions
$ oozie job -rerun 0000673-120823182447665-oozie-hado-C [-nocleanup]
[-refresh][-action 1,3-5] [-date 2012-01-01T01:00Z::2012-05-31T23:59Z, 2012-11-10T01:00Z, 2012-12-31T22:00Z]
-action or -date is required to rerun. If neither -action nor -date is given, the exception will be thrown.

Checking the Status of a Coordinator/Workflow job or a Coordinator Action
$ oozie job -oozie http://localhost:11000/oozie -info 0000673-20823182447665-oozie-hado-C

The info option can display information about a workflow job or coordinator job or coordinator action.
============================
workflow.xml


   
   
       
            root@cooper.alcatel.com
            /var/lib/hadoop-hdfs/simpleSparkTest.sh
       
       
       
   
   
        Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
   
   

=============================
coordinator.xml




${workflowPath}


       
============================
simpleSparkTest.sh

cd /var/lib/hadoop-hdfs
spark-submit --jars ./utils-common-1.0.0.jar --master yarn --class alu.ausdc.analitics.sparkEx.log4j.Main sparkanalitics-1.jar 20-01-2015
===============================

coordinatorjob.properties

#run for every 5 sec
frequency=5
startTime=2012-08-31T20\:20Z
endTime=2013-08-31T20\:20Z
timezone=GMT+0530

nameNode=hdfs://cooper.alcatel.com:8020
jobTracker=cooper.alcatel.com:8021
queueName=default
inputDir=${nameNode}/data.in
outputDir=${nameNode}/out
user.name=training
# give workflow xml path here
workflowPath=${nameNode}/user/oozie/shekhar
# give coordinator xml path here
oozie.coord.application.path=${nameNode}/user/oozie/shekhar

Hodoop installation videos

Hodoop installation videos URL: http://www.harjeet-kumar.co.nr/

UML and Design pattern

Use case diagrams: Depicts the typical interaction between external users (actors) and the system. The
emphasis is on what a system does rather than how it does it. A use case is a summary of scenarios for a
single task or goal. An actor is responsible for initiating a task. The connection between actor and use case is a
communication association.
Class diagrams: Class diagram technique is vital within Object Oriented methods. Class diagrams describe the types of objects in the system and the various static relationships among them. Class diagrams also show the
attributes and the methods. Class diagrams have the following possible relationships:
􀂃 Association: A relationship between instances of 2 classes.
􀂃 Aggregation: An association in which one class belongs to a collection (does not always have to be a
collection. You can also have cardinality of “1”). This is a part of a whole relationship where the part can
exist without the whole. For example: A line item is whole and the products are the parts. If a line item is
deleted then the products need not be deleted.

Many-to-Many Ex: Category, Experiment

􀂃 Composition: An association in which one class belongs to a collection (does not always have to be a
collection. You can also have cardinality of “1”). This is a part of a whole relationship where the part cannot
exist without the whole. If the whole is deleted then the parts are deleted. For example: An Order is a whole
and the line items are the parts. If an order is deleted then all the line items should be deleted as well (ie
cascade deletes).
One to many or One-to-One: Ex: Experiment, Wavelength

􀂃 Generalization: An inheritance link indicating that one class is a super class of the other. The Generalization
expresses the “is a” relationship whereas the association, aggregation and composition express the “has a”
relationship.
Ex: Promotions; coupons, one stop

􀂃 Dependency: A dependency is a weak relationship where one class requires another class. The dependency
expresses the “uses” relationship. For example: A domain model class uses a utility class like Formatter etc.
Sequence diagrams: Sequence diagrams are interaction diagrams which detail what messages are sent and
when. The sequence diagrams are organized according to time. The time progresses as you move from top to
bottom of the diagram. The objects involved in the diagram are shown from left to right according to when they
take part.
Package diagrams: To simplify complex class diagrams you can group classes into packages.
Façade pattern: The façade pattern provides an interface to large subsystems of classes. A common design goal is to minimize the communication and dependencies between subsystems. One way to achieve this goal is to introduce a façade object that provides a single, simplified interface.

The Front Controller suggests that we only have one Servlet (instead of having specific Servlet for each specific
request) centralising the handling of all the requests and delegating the functions like validation, invoking business
services etc to a command or a helper component. For example Struts framework uses the command design
pattern to delegate the business services to an action class.
Benefits
􀂃 Avoid duplicating the control logic like security check, flow control etc.
􀂃 Apply the common logic, which is shared by multiple requests in the Front controller.
􀂃 Separate the system processing logic from the view processing logic.
􀂃 Provides a controlled and centralized access point for your system.

Command pattern: The Command pattern is an object behavioral pattern that allows you to achieve complete
decoupling between the sender and the receiver. (A sender is an object that invokes an operation, and a receiver is an object that receives the request to execute a certain operation. With decoupling, the sender has no knowledge of the Receiver's interface.) The term request here refers to the command that is to be executed. The Command pattern also allows you to vary when and how a request is fulfilled. At times it is necessary to issue requests to objects without knowing anything about the operation being requested or the receiver of the request. In procedural languages, this type of communication is accomplished via a call-back: a function that is registered somewhere to be called at a later point.Commands are the object-oriented equivalent of call-backs and encapsulate the call-back function.

83: What is a business delegate? Why should you use a business delegate?
Problem: When presentation tier components interact directly with the business services components like EJB,
the presentation components are vulnerable to changes in the implementation of business services components.
Solution: Use a Business Delegate to reduce the coupling between the presentation tier components and the
business services tier components. Business Delegate hides the underlying implementation details of the business
service, such as look-up and access details of the EJB architecture.

What is a session façade?
A 84: Problem: Too many method invocations between the client and the server will lead to network overhead, tight
coupling due to dependencies between the client and the server, misuse of server business methods due to fine
grained access etc.
Solution: Use a session bean as a façade to encapsulate the complexities between the client and the server
interactions. The Session Facade manages the business objects, and provides a uniform coarse-grained service
access layer to clients.

Session façade is responsible for
􀂃 Improving performance by minimizing fine-grained method calls over the network.
􀂃 Improving manageability by reducing coupling, exposing uniform interface and exposing fewer methods to
clients.
􀂃 Managing transaction and security in a centralised manner.

how to use Jquery datataable

1. add jquery.dataTables.js and jquery.dataTables.css in your html. keep these files in proper folder.




2. keep below code in your html code.

AJax call with json response

Set Json response in Controller:

PrintWriter writer = response.getWriter();
JSONArray storeList = new JSONArray();
JSONObject store = null;
List list = new ArrayList();
list.add("RS");
list.add("EUR");
list.add("DOLLER");
for (String string : list) {
    store = new JSONObject();
    store.put("storeName", string);
    storeList.put(store);
}
JSONObject json = new JSONObject();
json.put("storeList", storeList);
sb = json.toString();
writer.write(sb);











ajax call:

Way 1:
$.ajax({
        url : currentPath+'/appName/partnerStoreAddress',
        data : 'store='+selectedValue+'&cbn='+cbnValue,
        dataType : 'json',
        success : function(json) {
            createAdressTable(json);
            $.unblockUI();
        },
        error:function(xhr,errorThrown){
              var r = jQuery.parseJSON(xhr.responseText);
              alert("Message: " + r.Message);
              alert("StackTrace: " + r.StackTrace);
              alert("ExceptionType: " + r.ExceptionType);
              alert(xhr.status);
              alert(errorThrown);
        }
    });

Way 2:

function setupAjaxForm(form_id, form_validations){
    var form = '#' + form_id;
   
    // setup jQuery Plugin 'ajaxForm'    
    var options = {
        target : '#refresh',  //if you want to refresh any div only
        dataType:  'json',
        beforeSubmit: function(){
        $.unblockUI();
         $.blockUI({
                message : ''
            })
            //pageRefresh = 'Y';
        },
        timeout:8000,
        success: function(json){
            $.unblockUI();
            $("#refresh").html($(html).find("#refresh"));
            $('#HPContactEmailList_filter').appendTo($('.themeheader'));
                       
            new setupAjaxForm('dummyForm'); 
           
        },
        error:function(xhr,errorThrown){
            $.unblockUI();
            alert('Selected item was not saved');
           
        }
    };
    $(form).ajaxForm(options);
}


$(document).ready( function () {
   
    $('#HPContactEmailList_filter').appendTo($('.themeheader'));
    new setupAjaxForm('dummyForm'); 
    }
);

Common mistake with Date

Use ‘dd/MM/yyyy’ instead of ‘dd/mm/yyyy’ when you are working with SimpleDateFormat to format. otherwise 'mm' will be considered as mints.

Can we access private methods in other class?..yes we can using reflection api

Yes...We can access private methods in other class using reflection API

Example code:

MyClass.java with private method
public class MyClass {

private String myPrivateMethod(String name) {
//Do something private
return "Hello "+;
}
}

Test class:  MainTest.java public class MainTest class{

private MyClass underTest;


public static void main(String []args)throws Exception {

MyClass underTest= new MyClass();

Class[] parameterTypes = new Class[1];
parameterTypes[0] = java.lang.String.class;

Method m = underTest.getClass().getDeclaredMethod("myPrivateMethod", parameterTypes);
m.setAccessible(true);

Object[] parameters = new Object[1];
parameters[0] = "Shekhar Reddy!";

String result = (String) m.invoke(underTest, parameters);
System.out.println(result )//Shekhar Reddy!

}


}
outPut: Hello Shekhar Reddy!

JUnit test case using Mock API

Mock API helps to create dummy object and returns back something to program when we are calling any method on that dummy object instead calling actual object.
Ex:

context.checking(new Expectations() {{
      oneOf (accountDAO).selectAccount(with("1234")); 
      will(returnValue(null));
  }});
 

Here we are asking Mock api return "null" value when I Call  
selectAccount method with 1234 value on AccountDAO object.

You can find very good example with maven integration in the below link.

http://onjavahell.blogspot.in/2009/05/good-unit-testing-with-jmock.html

Singleton object using Enum type

Since Java 1.5, there is a new approach to implement Singletons. Simply make it an enum type :

public enum MySingleton {

  INSTANCE;

  //Singleton method
  public void someMethod( ) {...}
}

Accessing the enum singleton :

MySingleton.INSTANCE.someMethod( );

New in spring 2.5 and Dependencies injection with Spring annotations (@Repository, @Service, @Autowired)

Click on below link to know what is new in spring 2.5

http://www.infoq.com/articles/spring-2.5-part-1
 Click below link to find simple example code with old and new bean configuration Dependencies injection with Spring annotations (@Repository, @Service, @Autowired). 
http://onjavahell.blogspot.in/2009/04/dependencies-injection-with-spring.html