In this paper, we propose a novel algorithm based on directional local difference binary patterns useful for content based image indexing and retrieval. The popular and successful method local binary patterns (LBP) codify a pixel, based on the neighborhood gray values around the pixel. Another flavor of LBP is, center symmetric local binary patterns (CS-LBP), which is the base method for our proposed novel algorithm. The proposed method is based on the directional difference between neighboring pixels. The four directional local difference binary patterns (DLDBP) in 0o, 45o, 90o, and 135o directions are proposed. Then, we apply our method on benchmark image database Corel-1k. The proposed DLDBP (Directional Local Difference Binary Patterns) can also be used to represent a video, using a key frame in the video. We apply the proposed directional local difference binary patterns (DLDBP) key frame based algorithm, on a video database, which consists of ten videos of airplane, ten videos of sailing boat , ten videos of car, and ten videos are of war tank. The performance of proposed DLDBP (Directional Local Difference Binary Patterns) is compared with CS-LBP (Central Symmetric Local Binary Patterns) method. The performance of DLDBP key frame based method is compared with volume local binary patterns (VLBP) method.