Milne said that her dad saw the photo and told the store manager that "it's clearly not James" and the manager replied "well they look the same".
She said that despite the evidence her brother was still refused entry into the store.
"I have never been so disgusted and mad in my life," Milne wrote in a Facebook post.
Milne's post has been shared more than 92,000 times and had hundreds pledging to boycott the retailer.
One person on Facebook, Beverley Wilson, wrote, "Absolutely disgusting!!!! Everyone in the area should boycott that shop. It is discrimination."
Another wrote, "This makes me so sick!! I work with children who have Down syndrome they are the most kindest people I have ever met they don't judge nor put other people down so for JB to do this is disgusting this manger should be fired on the spot!!."
James has received a lot of support on social media.
"Disgusted. Stand strong, not everyone is sooo ignorant. I too have a son with a brain injury, and I would cheerfully knock someone out who would dare be negative!!!! Rock on James xx," said Terry Bowles.
Milne said that her Mum called up the manager of the store, demanding that he apologize to her son, to which he replied that "he would never, ever, ever get an apology" from him and that he had "the right to stop anyone he pleased from entering the store".
Since then the chief executive of JB Hi-Fi has sent an apology letter to James.
