Remember that a rainstorm is a three-dimensional object. Rain is falling from one or more large clouds onto an area on the ground. The rainbow is a small area on the sky, and the sun shines on many other raindrops than those that make the rainbow.

Every raindrop reflects some of the sunlight without significantly refracting it: some of the light is reflected at the front surface of the raindrop, some is reflected from the back if the angle of incidence is too small (smaller than the 42° associated with the rainbow).

Actually, all the raindrops are reflecting some light at small angles, because light is incident on the entire front surface of the raindrop. However only those raindrops within 42° of our antisolar line reflect light back to our eyes. We can think of the 42° of the rainbow as the upper limit at which the reflected light can hit our eyes.

Any raindrops outside the cone will necessarily reflect light back, but the light will either go over our heads or past us on the right or left.

And some of the raindrops inside our cone are reflecting colors to our eyes, but most are simply reflecting white light straight back to us or contributing to someone else's rainbow.