The plane, a Norwegian flight from Stockholm's Arlanda Airport to Heraklion in Greece, had been due to take off at 9pm, when attention was drawn to the tattoo, which was thought by crew to be the flag of the Islamic State or a verse from the Quran.
According to Swedish tabloid Expressen, citing an airport employee, the cabin crew refused to fly if the man was onboard. The captain made the decision to remove the man and his female companion from the flight.
Darko Ali, commander at Arlanda's border police station, confirmed that cabin crew and a passenger “reacted to another passenger for some reason, and this caused concern. The captain decided that the person would not be allowed to remain on board and wanted that person to leave.”
The man was not arrested and is not suspected of any crime.
Public concern about the terror group has previously led to misunderstandings.
A 21st birthday party received police attention in Karlskrona, southern Sweden, last year, after a member of the public thought a pair of balloons spelling out the birthday girl's age were in fact a tribute to the group that has ravaged much of Syria and Iraq.
Also last year, Swedish police were called to confront an innocent volunteering group called Bearded Villains, after they were seen posing with their own black flag, which a passer-by wrongly identified as being an Isis banner.