in the opening line who is "you"

It is just an invitation to the reader. The sense to me is kind of like he is pulling back a curtain to reveal to us a truth hidden behind the banality of everyday life. "Let us go then, you and I" ... in other words, I will bring you on a "visit" to the repetitive emptiness of the work of Prufrock and the women quoting Michaelangelo and others. There is no specific audience that the narrator is addressing.