He told the court he usually worked 11 hours a day, six days a week, trying to earn extra money to support his family and his retired parents.
He said that shortly after his son's birth, he became "stressed" and began suffering from insomnia.
"Queenstown is a really hard place to live, especially with two kids. It's hard to find accommodation ... the rents are really high.
"A lot of things were on my mind."
He did not share his concerns because it was "too hard" for him to talk to anyone.
That afternoon, he and his infant son were in bed and the man had only been asleep for about half an hour when the baby began crying.
He described feeling "angry and frustrated" and said he was "half asleep" when he reached his right arm out and picked the infant up by his left elbow.
He then moved the child in front of him and "squeezed" him around his mid-section for 10 to 15 seconds.
When the baby started crying loudly the man noticed his son's left arm "was looking dislocated".
His wife entered the room, noticed the baby's arm injury and questioned the defendant, who denied doing anything to cause it.
They attended a medical centre before going to Lakes District Hospital, after which the child was transferred to Southland Hospital, where a "skeletal survey" identified the rib injuries, on either side of his spine.
On March 14, the baby was transferred to Christchurch with his mother and underwent CT and MRI scans before having surgery on his arm during which two pins were inserted.
He was released from hospital on March 23.
The defendant was first spoken to by police at Southland Hospital, and alleged the injuries had occurred after his son slipped while the pair were showering together two days earlier.
In a later DVD interview with police, played in court, the defendant admitted his actions on the Sunday caused the injuries.
He told the court yesterday he was "in a state of shock" at the time and still found it hard to believe.
Since the incident he had sought treatment, which was ongoing.
Under cross-examination, the man accepted submissions from Crown prosecutor Riki Donnelly that he was experienced with handling "fragile" babies and knew how to do so properly.
The trial continues.