How We Know Anything
We don't infer reality from our consciousness of sense data. We infer sense data from our consciousness of reality. Our senses are not barriers to direct contact with the world. They are our means of direct contact with the world. The sequence is reality -> consciousness -> sense data. (Ayn Rand got this right.)