In the recent year gesture recognition has become the most intuitive and effective communication technique for human interaction with machines. In this paper we are going to work on hand gesture recognition and interpret the meaning of it from video sequences. Our work takes place in following three phases: 1. Hand Detection & Tracking 2. Feature extraction 3. Gesture recognition. We have started proposed work with first step as applying hand tracking and hand detection algorithm to track hand motion and to extract position of the hand. Trajectory based features are being drawn out from hand and used for recognition process and hidden markov model is being design for each gesture for gesture recognition. Hidden Markov Model is basically a powerful statistical tool to model generative sequences. Our method is being tested on our own data set of 16 gestures and the average recognition rate we have got is 91%. With proposed methodology gives the better recognition results compare with the traditional approaches such as PCA, ANN, SVM, DTW and many more.