Swarovski, Urban Outfitters and Pandora are among the retailers currently occupying the building, but the latter said no decision has been made from its side