I have no doubt that part of what brings visitors to Bilboa's museum is the building itself. It is also hardly a surprise that reasonable people will doubt that the building is full of great art if the building itself isn't great art.
Looking at classic museums, such as the MFA, the Louvre, the Met, we see ornate, classically balanced architecture; inside, there are paintings and murals, carvings and statues, quite independent of the actual displays. They celebrate 'beauty' or 'the muses' and speak directly to what the visitor is to expect of the art itself. Similar architecture speaks to 'knowledge' at the Boston public library, the NY public library main branch, the Widener library, etc. In times past, the balance of the architecture itself was supposed to be a tribute to the good and sound judgment of a bank or a courthouse; both of which would be adorned with decor speaking to wealth or justice, perhaps both at each location.
In short, there is nothing new in the building for such an institution being considered a piece of art itself; and so, modern art gets 'modern' buildings. It is no surprise that post-modernity demands a similar venue; and that people are supposed to base their expectations of the contents on the container.
The problem, to my mind, is that there is a great confusion about how to do all this on a budget... what to sacrifice, what to keep, and so on. When a budget is made, maintenance seems to be left out - and things that contribute to reducing maintenance costs are also left out. Further, the virtues of postmodernity are most often seen in the disposable, the malleable, while those of modernity are usually seen in the industrially bleak and unadorned - but traditional virtues require high labor costs and good materials and are thus more difficult to construct.
This leads to them being difficult to get past the board with the budget and the bean counters. Nobody thinks that they can put a wing on the MFA that actually uses the original marble construction of the MFA, that matches it in quality and substance - forget carvings and paintings. So naturally, the new sections are built in different style. It gets worse when that style itself proves expensive so that the quality of the construction is cut at key points. This is where you get products like the Stata center, which is much more cheaply constructed than the Great Dome, but even more cheaply constructed than would have been originally intended.