Nowadays the rise of the artificial intelligence is with high speed. Even we are far away from the moment when machines are going to make decisions instead of human beings, the development in some fields of artificial intelligence is astonishing. Deep neural networks are such a filed. They are in a big expansion in a new millennium. Their application is wide: they are used in processing images, video, speech, audio, and text. In the last decade, researches put special attention and resources in the development of special kind of neural networks, convolutional neural networks. These networks have been widely applied to a variety of pattern recognition problems. Convolutional neural networks were trained on millions of images and it is difficult to outperform the accuracies that have been achieved. On the other hand, when we have a small dataset to train the network, there is no success to do it from a scratch. This article exploits the technique of transfer learning for classifying the images of small datasets. It consists fine-tuning of the pre-trained neural network. Here in details is presented the selection of hyper parameters in such networks, in order to maximize the classification accuracy. In the end, the directions have been proposed for the selection of the hyper parameters and of the pre-trained network which can be suitable for transfer learning.