Nash, 86, and Louis Nirenberg, 90, were awarded for "striking and seminal contributions to the theory of nonlinear partial differential equations (PDEs) and its applications to geometric analysis," the Norwegian Academy of Sciences and Letters said in a statement.
PDEs are equations first developed to describe physical phenomena which also help to analyse abstract geometrical objects.
"Far from being confined to the solutions of the problems for which they were devised, the results proven by Nash and Nirenberg have become very useful tools and have found tremendous applications in further contexts," the academy said.
Nash, from Princeton University and the Massachusetts Institute of Technology (MIT), is best known for his contribution to game theory — the mathematics of decision-making — which won him the Nobel economics prize in 1994.
His life story formed the basis of the 2001 film "A Beautiful Mind" in which his genius and struggle with mental illness was portrayed by Russell Crowe.
Canadian-born Nirenberg from the Courant Institute of Mathematical Science at New York University "has had one of the longest and most feted careers in mathematics," the prize committee said.
"Even though they did not formally collaborate on any papers, they influenced each other greatly during the 1950s," the committee added. "The results of their work are felt more strongly today than ever before."
The prize money of 6.0 million kroner ($767,000, €700,000) will be handed over by King Harald of Norway in a ceremony in Oslo on .
Named after the 19th century Norwegian mathematician Niels Henrik Abel, the prize was established by the Norwegian government in 2002 to award outstanding scientific work in the field of mathematics, a discipline not included among the Nobel prizes.