Vehicle surveillance system provides a large range of informational services for the driver and administrator such as multiview road and driver surveillance videos from multiple cameras mounted on the vehicle, video shots monitoring driving behavior and highlighting the traffic conditions on the roads. How to retrieval driver’s specific behavior, such as ignoring pedestrian, operating infotainment, near collision or running the red light, is difficult in large scale driving data. Annotation and retrieving of these video streams has an important role on visual aids for safety and driving behavior assessment. In a vehicle surveillance system, video as a primary data source requires effective ways of retrieving the desired clip data from a database. And data from naturalistic studies allow for an unparalleled breadth and depth of driver behavior analysis that goes beyond the quantification and description of driver distraction into a deeper understanding of how drivers interact with their vehicles. To do so, a model that classifies vehicle video data on the basis of traffic information and its semantic properties which were described by driver’s eye gaze orientation was developed in this paper. The vehicle data from OBD and sensors is also used to annotate the video. Then the annotated video data based on the model is organized and streamed by retrieval platform and adaptive streaming method. The experimental results show that this model is a good example for evidence-based traffic instruction programs and driving behavior assessment.