The ability to detect and track object of interest from sequence of frames is a critical and vital problem of many vision systems developed as yet. This paper presents a smart surveillance system that tracks objects of interest in a sequence of frames in their own defined respective boundaries. The objects of interest are registered or saved within the system. We have proposed a unique tracking algorithm using combination of SURF feature matching, Kalman filtering and template matching approach. Moreover, an efficient technique is proposed that is used to refine registered object image, extract object of interest and remove extraneous image area from it. The system will track registered objects in their respective boundaries using real time video generated through two IP cameras positioned in front of each other.