Van Gorp slammed in a rebound in the 59th minute past Norway keeper Ingrid Hjelmseth who had cleared Tessa Wullaert's header following a cross from Belgium's Davina Philtjens.
The Norwegian players may understandably have felt aggrieved at the goal, with Van Gorp appearing to be in an offside position before arriving to tuck the rebounding ball into the net.
Eight minutes later, Cayman extended the lead to 2-0, heading in a long throw-in that had found its way through the Norwegian defence.
Ahead of the tournament in the Netherlands, Norway won nine of ten competitive meetings with Belgium.
But the Belgians, propelled by a noisy crowd, had the first chance when Van Gorp saw her left-footed effort stopped by Hjelmseth just three minutes into the game.
Norway then took the reins, with Ada Hegerberg heading over the crossbar and Caroline Graham Hansen slamming her volley over with just Belgian keeper Justien Odeurs to beat.
Belgium's Tine De Caigny then missed a pass from Wullaert right in front of the goal and Cayman sent her effort wide, while Andrine Hegerberg's shot was tipped away by Odeurs.
Norway tried to push after the two goals, but the Belgian defence resisted as Norwegian captain Maren Mjelde sent a free kick over five minutes from the end and Ada Hegerberg squandered a chance.
“I felt we were in the match and had chances. The goals came from nothing. It is very frustrating,” Norway winger Caroline Graham Hansen told broadcaster NRK.
Norway's Caroline Graham Hansen takes on Belgium midfielder Davina Philtjens. Photo: Berit Roald / NTB scanpix
Graham Hansen also felt she should have been awarded a penalty after rounding Odeurs and then going to ground with the score at 0-0 on 50 minutes.
“The ‘keeper took my legs away. The referees were unfortunately not at the level they should have been. It worked both ways. It’s a shame it has to be like this,” the attacking midfielder said.
Norway coach Martin Sjögren made no excuses for the result.
“We didn’t perform well enough. If we had, we’d have won. So we can talk about offsides, penalties and chances, but the result is 2-0 to Belgium, and we can’t get away from that,” he told NRK.
In the other Group A game on Thursday, the Netherlands beat Denmark 1-0. That results leaves Norway with a slim chance of qualification for the knockout stages, should they defeat their Scandinavian neighbours by a large enough margin and Belgium lose their own final group match-up against the Dutch.