"These findings suggest that for women early sexual experiences may shape future sexual encounters/relationships to a greater extent than for men," the report, an analysis of the third National Survey of Sexual Attitudes and Lifestyles, said.
Researchers carrying out the survey interviewed 6,669 women and 4,839 men aged between 16 and 74 who reported having at least one sexual partner in the past year.
The findings, published in the BMJ Open, also suggested that men and women who believed stereotypes such as the idea that men had a higher sex drive were more likely to conform to them - suggesting that they were using them to explain away their own experiences.
The report said: "Endorsing the assumption that 'people want less sex as they age' was associated with lack of interest in both genders.
"It might be that this belief contributes to a decline in interest, or - equally plausible - that those who lack interest adopt this attitude to avoid viewing their experience as problematic.
"Interestingly, men who endorsed the view that 'men have a higher sex drive than women' were significantly less likely to report lacking interest in sex, whereas women who agreed with this statement were more likely to do so."
In women only, low interest in sex was linked to having three or more partners in the past year, having children under five in the household, and not sharing a partner's sexual preferences.
Women whose general interests differed from those of their partners also experienced a lack of sex drive.
The paper's authors said the findings showed that examining the non-sexual aspects of relationships could add to understanding of people's level of interest in sex.
Co-author Dr Kirstin Mitchell, of the University of Glasgow, commented: "The findings on the strong association between open sexual communication and a reduced likelihood of sexual interest problems emphasise the importance of providing a broad sexual and relationships education rather than limiting attention only to adverse consequences of sex and how to prevent them."
Professor Cynthia Graham, of the Centre for Sexual Health Research at the University of Southampton and lead author on the paper, said: "Our findings show us the importance of the relational context in understanding low sexual interest in both men and women.
For women in particular, the quality and length of relationship and communication with their partners are important in their experience of sexual interest.
"It highlights the need to assess and - if necessary - treat sexual interest problems in a holistic and relationship - as well as gender-specific way."