No, it is misunderstanding of hindus because they do not know why muslims worship towards qiblah. Kabaah is qiblah of muslims i.e. a direction of worship of muslims. It is not the direction of faith but only direction of worship. Our direction of faith is ALLAH swt and no one else.
The place where kabaah is built is the place where before the land was foam on water surface. It is believed to be that first place from where earth emerged. (tafsir ibn-e kathir).
Before kabaah, the qiblah of muslims was baitul muqadas i.e. masjid-e aqsa which Is not occupied by jews. When Muhammad s.a.w.w was given prophet hood, He used to offer namaz towards baitul muqadas but one day during namaz He s.a.w.w was ordered to worship towards kabbah, since that day we offer namaz towards kabbaah. So, if hindus think that kabaah is what we worship, then they should also find a counter argument for importance of bait-ul-muqadas for muslims which was first qiblah of muslims.
Second point is, if we were worshiping a building, bait-ul-muqadas was a huge and most loved building since our ancestors, why did we chose to worship a smaller building?
Third point to be taken into account is, if muslims were to worship only a building, the idols would have remained inside that building because worshiping a room does not mean anything.
Mostly hindus think that inside kabbah lives ALLAH, this is the most lousy argument because it does nto even stands with their own religion which they try to associate with islam. According to Hinduism, god is everywhere, how can they enclose their brahma into a room!!
Kaba is the house of ALLAH which means it is a pure place, pure of all evils. It does not mean ALLAH lives here, this arguments is against the fundamentals of islam itself.
Regarding hajre aswad, it is stone which was used by hazrat Ibrahim a.s. and his son hazrat ismael a.s. to build kabah, it was a stone descended from jannat. It is not a meteorite or any ordinary stone even then it is a stone which is also a stone on earth. It did not change. To worship any other being is shirk and such a person is considered excluded from islam once he does the shirk. So, these hindu allegations and day dreams are nothing but nonsense and they hold no ground because whatever is presented to defend the hindu point of view is countered by the basic fundamentals of islam.