Classifying an object captured in an image is useful for understanding the contents of the image and annotating it exactly with corresponding tags automatically is the problem faced recently. As the real world data set is highly imbalanced it degrades the performance of automatic image annotation and object detection. To prevail over this drawback we have proposed a new system for pattern matching and annotation which is based on the fusion of principles obtained from Fractal Transform and gentle AdaBoost algorithm. This paper, also tries to overcome deterioration in the performance occurring through imbalance dataset, different orientation, scaling in image annotation by choosing an over sampling method for learning the classifier. The proposed IFSMOTE classifier is initially trained up by setting a threshold value which helps to identify the objects correctly and an over-sampling technique based on fractal is used to classify the imbalanced dataset. Experimental results on the Flicker image dataset have shown superior performance results in terms of precision, recall and F-measure. This paper also presents the comparative results of our proposed system with other traditional image annotation algorithm like SVM, SMOTE and FSMOTE.