Prince Haakon and Princess Mette-Marit want to demolish nine of the 41 buildings on their Skaugum estate, and build a new modern house on the site.
The buildings to be razed include Rødlåven, a 199 square meter traditional red barn, that is cherished by locals who take walks in the surrounding hills.
"It's very disappointing," said Fredrikke Hegnar von Ubisch, the manager of the local Asker museum."These buildings represent another period, and one of them is very central to where we are doing our hiking, it's very visible. The reaction will be massive if it is destroyed."
The new controversy comes shortly after the Royal couple angered many Norwegians by announcing plans to take their two children out of the state school system and instead send them to expensive private schools, a move that some have seen as distancing them from their people.
The Royal couple also plan to knock down Bråtan, a traditional farmhouse built in the late 1800s.
"The buildings in question are in such poor condition that it is not in any way appropriate to rebuild them," Palace spokesman Marianne Hagen told Budstikka newspaper. "The foundations and the quality of the construction are not up to today's standards."
But von Ubisch, an expert in conservation, argued that the buildings could easily be restored and that the Royal Couple had a duty to set a good example.
"Because they are Royal and have the position they have, it's really important that they do the right thing," she said. "Many farmers in Norway have the same problems with these old buildings. There would be no problem using them today if they really wanted to."
She said that the view of Rødlåven from the surrounding hills was much loved in Asker.
"It's the most popular area for people to go hiking on Saturdays and Sunday, and everybody passes this building all the time," she said. "We all enjoy seeing this nice landscape with this picturesque red barn."