The material included tens of thousands of emails, the organization alleged.
Mueller's investigation is looking at whether any crimes were committed as part of what U.S. intelligence agencies say was systematic Russian meddling in the election Trump won.
Trump has consistently said there was "no collusion" with Russia, an assertion he repeated Friday.
Trump for America alleges that Mueller acquired private records without a warrant or subpoena.
There was no immediate reply Saturday to a request for comment from Mueller's office.
Transition documents are private property, not government records, the organization contends. The letter invokes federal law and decades of precedent to argue that Mueller overstepped.
The transfer of transition documents is "unlawful conduct that undermines the Presidential Transition Act of 1963," the letter said, "and will impair the ability of future presidential transition teams to candidly discuss policy and internal matters that benefit the country as a whole."
The Trump transition alleges that the handover was done by "career staff at the General Services Administration" and suggested that those employees may have had political motives.
Mueller's team, the letter said, "has extensively used the materials in question, including portions that are susceptible to claims of privilege, and without notifying TFA or taking customary precautions to protect TFA's rights and privileges."
The complaint on behalf of the Trump transition team suggests that the organization is preparing to challenge the origin of material that could shed light on contacts and discussions involving former White House national security adviser Michael Flynn and others.
Mueller's investigation has led to charges against two people and guilty pleas from two others, including Flynn.
The Trump organization claims that GSA had assured Trump for America that while it retained copies of transition records, it would not release them without consulting the organization.
The GSA provided facilities to the Trump transition team in the weeks before Trump's Jan. 20 inauguration.