Only by means of XOR operation, a novel image encryption scheme is proposed based on a nonlinear chaotic map (NCM). There are two rounds in the proposed image encryption scheme. In each round of the scheme, the pixel gray values are modified from the first pixel to the last pixel firstly, and then the modified image is encrypted from the last pixel to the first pixel in the inverse order. In order to accelerate the encryption speed, every time NCM is iterated, n (n>3) bytes random numbers which are used to mask the plain-image can be gained. And to enhance the security, a small perturbation will be given to the parameters of the NCM based on the last obtained n bytes modified elements before next iteration. Experimental results and theory analysis show that the algorithm has a high security performance and a good efficiency.