Linear congruential generator has been widely applied to generate pseudo-random numbers successfully. This paper proposes a novel chaos-based image encryption scheme using affine modular maps, which are extensions of linear congruential generators, acting on the unit interval. A permutation process utilizes two affine modular maps to get two index order sequences for the shuffling of image pixel positions, while a diffusion process employs another two affine modular maps to yield two pseudo-random gray value sequences for a two-way diffusion of gray values. Experimental results are carried out with detailed analysis to demonstrate that the proposed image encryption scheme possesses large key space to frustrate brute-force attack efficiently and can resist statistical attack, differential attack, known-plaintext attack as well as chosen-plaintext attack thanks to the yielded gray value sequences in the diffusion process not only being sensitive to the control parameters and initial conditions of the considered chaotic maps, but also strongly depending on the plain-image processed.