The male same-sex partners began legal proceedings and last year a High Court judge ruled that the child, now 18 months old, should live with them.
The decision came to light after the surrogate mother and father appealed and a new judgment was published on Friday. None of the people involved can be named.
Court of Appeal judges ruled that the original decision to give custody to the gay couple with limited contact six times a year with the surrogate mother and father was correct.
Lord Justice McFarlane said that while surrogacy arrangements had no legal standing, the child's genetic relationships and welfare were the most important factors for deciding where he or she should live.
While the legal mother and father had the right "to change their minds" this did not necessarily mean that they had the right to keep the child, he said.
The child was conceived using sperm from one of the men and an egg from a Spanish donor, meaning one of the gay couple is a genetic relative of the child, but the birth mother and father are not.
Despite this they remain the child's legal mother and father, because no adoption or parental order has been made.
In the original ruling High Court judge Mrs Justice Theis had decided that "H's identity needs as a child of gay intended parents would be best met by living with a genetic parent".
Lord Justice McFarlane said the judge had been "critical of [the birth parents] for the way in which they had behaved in the later stages of the pregnancy and immediately after H's birth.
"She described them as having embarked on a deliberate and calculated course of conduct and as having continued to put obstacles in the way of [the same-sex couple] in seeking to establish a relationship with [the child].
"She referred to them as being rigid, taking a position and sticking to it, and as having little or no capacity to resolve disputes or negotiate their way through difficulties."
She also found that the birth parents were "less able to look at matters from the child's point of view".
However, the Court of Appeal also criticised the gay couple for "unwisely and unaccountably" generating publicity about the case, and made an order preventing them from speaking to the press.
The judge added that the case "demonstrates the risks involved when parties reach agreement to conceive a child which, if it goes wrong, can cause huge distress to all concerned".
Lord Justice McFarlane said the subject is being examined by the Law Commission which could reform the law to create a proper legal basis for surrogacy agreements.
The case follows another ruling made in 2015 where Justice Alison Russell said that a mother who had a baby for a gay couple and then refused them access had to hand the child over as she had entered into an informal surrogacy arrangement with them.
That ruling led to calls for the law to be reformed.
