The Calgary-based company said it intends to start a claim for costs and damages under Nafta against the US. The dispute would be decided by a three-member arbitration panel to be established under the terms of the trade agreement.
A panel couldn't force approval of the pipeline, proposed to carry Canadian crude into the US, but it could award damages for lost profits and costs incurred by the company.
US environmental and labour groups say that companies could seek compensation for government decisions under similar provisions found in the Trans-Pacific Partnership - a 12-nation trade pact seven years in the making that would clear barriers to commerce among nations that produce 40 per cent of global economic output.
New Zealand, Australia, Brunei, Canada, Chile, Japan, Malaysia, Mexico, Peru, Singapore, the United States and Vietnam agreed to the Trans-Pacific Partnership (TPP) deal in October, though each government's approval could take months or even years.
The director of Chile's economic relations bureau said New Zealand will host the signing ceremony for the controversial deal next month.
A second country, Peru, has told Bloomberg its Deputy Trade Minister and TPP negotiator, Edgar Vasquez, will attend the ceremony here on February 4.
The date and location was revealed in a statement issued by Andres Rebolledo, the director-general of Chile's General International Economic Relations Bureau, following a meeting with the country's National Human Rights Institute to discuss the impact of the TPP on human rights in Chile.
The New Zealand Government would not confirm the statement, saying "a number of countries are still working out their domestic approval processes required before signature".
"Further details will be announced when and if they are confirmed," said Duty Minister Simon Bridges.
Prominent critic of the TPP, University of Auckland professor Jane Kelsey, said the way details had emerged reflected badly on the Government.
"Consistent with the Government's obsessive secrecy throughout the TPP process, we have to get confirmation of what is happening in our own country from offshore," Kelsey said.
The TPP deal is a top priority for President Barack Obama that must be ratified by a skeptical Congress before it can take effect.
Keystone supporters said conservationists were confusing the issue.
"President Obama's environmental extremist friends will look for any excuse to deny Americans natural resources, driving up the cost of energy and costing jobs," said AshLee Strong, a spokeswoman for House Speaker Paul Ryan, a Wisconsin Republican who backs passage of the trade agreement.
Communications Workers of America said TransCanada's NAFTA claim provides a case study in how the TPP arbitration process would give thousands more companies a new mechanism to challenge US laws and regulations.
Abuses of the system could be avoided if the TPP language were revised to address concerns between companies and nations, said Representative Sander Levin, a Michigan Democrat who had unsuccessfully proposed such changes to the Obama administration while the pact was being negotiated.
"TransCanada's challenge to the President's decision on the Keystone XL Pipeline through a NAFTA investment claim further highlights why we must be certain that the Trans-Pacific Partnership trade agreement addresses serious concerns," Levin said.
"A full and vigorous public debate is needed to identify problems like this one before the TPP agreement is signed."
Frank Benenati, a White House spokesman, didn't respond to a request for comment. Obama administration officials have said there are adequate protections in the TPP to prevent the arbitration process from being exploited, and the president has described the agreement as "the most pro-labour, pro-environment, progressive deal in history".
Martin O'Malley, seeking the Democratic nomination for president, tweeted that the Keystone challenge through NAFTA was "outrageous" and "an example of why I oppose #TPP".
"Trade deals shouldn't value corporate profits over national interests," he wrote.
If completed, the 12-nation Trans-Pacific Partnership would be the largest US free-trade agreement, covering an area with a combined annual economic output of more than US$28 trillion.
Under current treaties and agreements with more than 50 countries, about 9,500 US subsidiaries of foreign firms can pursue such suits against the US, according to data from Public Citizen, a Washington-based non-profit group.
Under the TPP, that would roughly double, with about 9,200 more subsidiaries getting rights to file trade claims.
Lori Wallach, head of Public Citizen's Global Trade Watch, said TransCanada's claim "is exactly the attack on US environmental policy that the president insisted could never happen" under the TPP.
"This particular Nafta investor-state attack basically destroys all of the administration's arguments in defense of the TPP expanding the regime to thousands more corporations who could attack the US," Wallach said.
TransCanada said it was seeking compensation "as a result of the US administration's breach" of its obligations under NAFTA, a deal signed by the US Canada and Mexico that took effect in 1994.
"TransCanada had every reason to expect its application would be granted," the company said.
"TransCanada invested billions of dollars in the Keystone XL project and the denial of permit deprived TransCanada of the value of that investment."
- Additional reporting: BusinessDesk