The constitutional amendment pushed by Yingluck's government had sought to change the Senate into a fully elected body. Currently about half its members are selected by a panel of judges and heads of independent state agencies.
Yingluck swept to power with an absolute majority in the lower house in 2011 elections after a military crackdown on Thaksin's supporters a year earlier left more than 90 people dead.
In broad terms, the ruling party followed procedures spelled out in the 2007 charter for changing the constitution. The court, however, criticized it for various violations that occurred during the amendment process.
The ruling effectively gives the court the power to block any significant political reform by any government, even if it holds an absolute legislative majority, as does Yingluck's.
The court's action is the latest chapter in the long-running political war between backers of Thaksin and his opponents in Thailand's old conservative political establishment, led by the military and supporters of the monarchy.
Thaksin was ousted after being accused of corruption and disrespect for Thailand's constitutional monarch, King Bhumibol Adulyadej.
The courts, which are bastions of royalism, have consistently ruled against Thaksin's interests. Appointed senators also have been among the leaders of the crusade against Thaksin.
"The political situation in Thailand right now can be summed up in these words: we are now living under a 'Constitutional Court' monarchy," said Assistant Professor Thamrongsak Lertpetchanan, a political scientist at Rangsit University.
With Thaksin's side holding an overwhelming electoral mandate, the court is "the last fortress of the establishment and the authoritarians," he said ahead of Wednesday's ruling. "This is a war between the legislative branch and the judicial branch."
Thaksin's supporters know this all too well. Court rulings in 2006 encouraged his opponents before the coup, and in 2008 forced out two pro-Thaksin prime ministers who were already under pressure from street demonstrations. Thaksin's opponents usually face lenient action.
Wednesday's ruling is likely to bolster activism by Thaksin's foes, who have been demonstrating in Bangkok's streets for several weeks. They have been reacting to an effort by Yingluck's government to include Thaksin in a proposed amnesty for political offenders. The government attempted to ram the legislation through Parliament but backed down in response to public outrage.
Thaksin's opponents, especially in the opposition Democrat Party, have attempted to build on that outrage to bring down the government. They are trying to have lawmakers who voted for the amnesty impeached.
"I'd like to thank the nine judges for delivering justice," said Somchai Sawaengkarn, a senator who helped bring the case before the court. "The verdict sets a precedent that shows the balance in Thai democracy that the legislative, judicial and executive branches must work to balance one another."
The court, while specific in its ruling on procedural errors made by the ruling party, was less clear in rejecting the contents of the amendment, although it said a provision allowing close relatives of lower house lawmakers to run for the Senate could lead to a lack of checks and balances and turn the legislature into "a husband and wife Parliament."