"I've been nominated twice for Academy awards and I've never lost to a white person. The first time I lost to Denzel [Washington] and the second time I lost to Forest Whitaker," Smith said.
"When I see this list and series of nominations that come out - and everybody is fantastic and that's the complexity of this issue, everyone is beautiful and deserving and it's fantastic - but it feels like it's going the wrong direction."
Smith said he was "proud" of his wife's stance but had no idea she was planning to address the issue until after she spoke out online.
He laughed: "I was out of the country at the time. And I came home, 'What happened?'
"She's deeply passionate, and when she is moved, she has to go. I heard her words and I was knocked over. I was happy to be married to that woman."
Lee, who was awarded an honorary Oscar for lifetime achievement this season, and Pinkett-Smith have both said they will sit out the February 28 ceremony after no black actors were nominated in any of the 20 acting categories for a second straight year.
Lee said he had never used the word "boycott" to characterise his protest.
Lee, Smith, Pinkett-Smith and documentary director Michael Moore are the only major stars so far to have announced they will not attend the biggest night in the movie industry.
Their protest has sparked a major debate over the lack of diversity both at the Academy of Motion Picture Arts and Sciences, whose majority white male members vote on the Oscars, and in Hollywood at large.
Oscars host Chris Rock has also come under pressure from rapper 50 Cent and Fast & Furious actor Tyrese Gibson to quit, although the black comedian has made no public statements on the issue.
"There is no joke that he can crack. There is no way for him to seize the moment and come into this thing and say, 'I'm going to say this and say that I'm going to address the issue but then I'm still going to keep my gig as the host,' " Gibson told People magazine.
"The statement that you make is that you step down."
Mark Ruffalo, nominated for his performance in the newspaper drama Spotlight, told BBC News he was "weighing" whether to skip the ceremony.
But later in the day, he clarified that he will attend the Oscars "in support of the victims of clergy sexual abuse and good journalism" - the subjects of Spotlight.