"It is important for me to hear what people really think," he told Norwegian media. "And if there is one place people really say what they think about most things, it's in the taxi. Right from the gut."
The stunt, which was captured on a hidden camera for the Labour leader's election campaign video, was the brain-child of Try, one of Norway's leading advertising agencies.
The Prime Minister wore dark glasses and an ordinary taxi driver's uniform for the shift, which took place in June. The passengers, some of whom had been instructed to wait for the taxi, took little time to rumble their driver's disguise.
"From this angle you really look like Stoltenberg," said one. "So, you have begun driving a taxi," says another. "Have you quit as prime minister?" asks a third.
The conversation in the filmed exchanges quickly moves onto politics.
The video has gone viral across the world, featuring in The Times, The Guardian, the Telegraph and the Daily Mail in the UK, as well as on the BBC. It featured in The Sydney Morning Herald in Australia, in the Huffington Post in the US, and on international television channels such as Al-Jazeera.
Mr Stoltenberg remains popular in Norway but his government is seen as tired by voters and is lagging behind the opposition centre-right in the run-up to next month's election.
Recent polls show that Labour is supported by 27.5 percent of the electorate, while the Conservative party is in the lead with 31.9 percent of the voters.
The stunt is reminiscent of the wooden soapbox UK Prime Minister John Major pulled out in 1992, when victory seemed similarly distant for the British Conservative party.
It is also a favourite ruse pulled by the benevolent potentates of fairy-tale and legend, such as Caliph Harun al-Rashid in the 1001 Nights or King Richard the Lion Heart in the Robin Hood stories.
Stoltenberg admits that he was sceptical when first presented with the idea by the agency's founder Kjetil Try, who is a personal friend.
"When the idea came up, I was not sure whether it could be done. I spent of lot of time contemplating it," he told VG.
He also appears to have had some problems getting used to his temporary new role.
"I know this city pretty well, but I'm used to being in the back seat, so I got a bit confused about which lane to be in. Some of my passengers were criticising me," he told VG. "I think I stepped on the clutch, but it turned out to be the break."
Asked by VG if he might become a taxi driver if he loses the election, Stoltenberg threw in his election pitch.
"I think the nation and Norwegian taxi passengers are best served if I am the prime minister and not a taxi driver."