One defector, who gives lectures on North Korean society in the South, was given an instruction from superiors to avoid calling the reclusive state a "nuclear regime".
They also said that they had been "encouraged" to treat issues such as North Korean political prison camps and human rights issues as taboo, according to the person.
Defectors also support the long-term vision of reunification, but their views are very different from the generally positive sentiments of the South Korean public, South China Morning Post reports.
Many defectors believe unification will only be possible with the toppling of the Kim regime.
Choi Seong-guk, an animator who fled to South Korea eight years ago, also said he felt his freedom of speech was under pressure when he spoke out against the inter-Korean reconciliation.
Choi, who draws satirical cartoons about the North's human rights violations, said he was "kicked out" of a South Korean television show once for making negative comments about Kim.
"They asked my opinion about Kim Jong-un's trip to South Korea. I told them we cannot believe him as people in North Korea always say 'peace will only come after all the capitalists in the South disappear'. We shouldn't be tricked by Kim Jong-un," Choi said.
"Then, the host said: 'Thank you very much' and kicked me out of the show shortly after. It took less than five minutes."
Choi said sponsorship by businesspeople for his artwork had also been cut, but he was not sure if it had to do with his political stance.
Eric Foley, a founder of aid group Voice of the Martyrs – Korea said the government told him to stop his missionary work of sending Bibles to the North. Instead, the governments in Pyongyang and Seoul said they were planning to hold a Christian event in the North to showcase "religious freedom" and would consider inviting them.
"They foresee there would be a time that they invite us to go to North Korea to do Christian work in a setting mutually agreed by the North and South Korean governments," Foley said.
"If the government wants to invite us to any of those activities, we will consider it. But in general our Christian activities should be conducted without mediation from the governments."
He added that from what he had heard from defectors, the cosying up between Kim and foreign governments would have little impact on the North Korean people.
Ji Hyeon-a, a high-profile defector who testified about her sufferings to the United Nations last year and attended a meeting with US Vice-President Mike Pence in February this year, also feels a similar pressure.
She said she was disappointed that the human rights issues were not mentioned in the inter-Korean summit, nor at the Trump-Kim summit.
"[None of the leaders] mentioned anything about [North Korean] human rights … Trump even described Kim as a citizen-loving leader, which felt quite unpleasant," Ji said.
"It made all 32,000 defectors in South Korea liars."
But Kim In-yong, another 33-year-old defector who is a research student in the South, supports the peace talks and said it was unrealistic to hope the Kim regime would collapse soon.
"The more North Korea is exposed [to the international community], the more they will be concerned about human rights issues and will try to take measures to meet international standards. It's going to be slow but we need to keep trying to change North Korean society."