The problem of whether things are perceived to be the same by different observers is a different question than whether what we perceive has a physical existence in and of itself. It is certainly the case that what we perceive is only a limited part of what exists, science revels that there are even more ways to perceive (e.g. IR radiation) as well as more things to perceive at the smaller scales (e.g. what would be perceived using a microscope). However, the fact that we perceive at all implies the existence of something irrespective of whether we perceive the whole thing or even that thing in the same way as others.

Also, the fact that we can create mathematical models of the physical world, and predict the behaviour of it again implies that there is something there to predict about. Especially as mathematical models can predict behaviour or effects never before observed. Importantly, this fact that there are things to be discovered about the external world unknown to any human implies again that there is something there for us to discover things about.

Consider this near trivial definitional proof: our eyes are physical and so is the light. Were the tables non-physical there would be no way for them to physically (that is by reflecting and absorbing bits of the light) interact by definition of what it is to be physical. The knowledge of the physicality of our eyes is enough to prove the physicality of the light and therefore the table.

Finally, and more importantly, even accepting that everything COULD be an illusion of some form, we have no evidence for this. Therefore, it is illogical to believe that is the case in the face of the fact that we have evidence for the physicality of the objects (such as the simple proof outlined above). It would be equally as rash as believing in Russell’s celestial teapot, or the theory that we are under constant observation by invisible elephants.