They had "nice chats about the past, about his family. He wanted to pick up my grandchild, to hold him, which he was a little bit shy to go to him. You could see he really reached out for touching a child at that moment,'' Brand recalled.
"When I got the message when he passed away, it was very sad for me,'' Brand told The Associated Press. "But I think he was successful and he did what he wanted to do. I wanted him to go in peace and I am thinking of the family today, what they go through.''
Brand started to work on Robben Island in 1978 when he was 18 and Mandela was 60. Mandela spent 18 of his 27 years in prison on Robben Island.
Eventually Brand grew to like Mandela and smuggled in special treats like bread and Mandela's favorite hair pomade. Brand even sneaked in Mandela's infant granddaughter so that the prisoner could hold her.
For his part, Mandela encouraged Brand to continue his education and maintained an interest in him and his family.
Years later, when Mandela was president, he took special care to single out Brand for recognition while Brand was a lowly civil servant while South Africa's new constitution was being drafted. Mandela flew in by helicopter and entered the room where members of parliament were debating the new constitution, said Brand.
Mandela went around the room shaking hands with parliamentarians but when he saw Brand, who was distributing documents, Mandela lifted his arms and warmly greeted him.
"He immediately made a big announcement to everyone: 'You know who is this person? This person was my warden, this person was my friend.''' Brand said he felt very humble and proud at that moment. After that when the parliamentarians went out for a group photo, Mandela insisted that Brand be in the photo. "He said 'No, no. You must stand next to me, we belong together.'''
Brand once again works on Robben Island which is no longer a prison but a bustling tourist attraction. He and other former guards and prisoners tell visitors about the new South Africa's racial reconciliation.
Brand said he shared a joke with Mandela about his final resting place.
"I say to Mandela that we must bury you on Robben Island. Then he just laughed. He said 'Why? For tourist attraction?' He said `You should have made money. But I think I must go to Qunu.' That was him making a joke.''
• Read our full coverage on Nelson Mandela.