How can a guy tell me we are taking it to fast if he is the only one taking it fast?

I haven't touched him and he started feeling an touching on me the last few times and today was the 1st time I barely touched Dwn there and I moved my hand right away and he out my hand bck Dwn there!!!! Today I asked if he thinks we are going at an ok pace and he tells me we maybe going to fast? Hes the only one moving to fast I just go with it!!! All the advise I can get please!!