Professor Karl Michaelsson, of Uppsala University, Sweden, led a team to investigate whether the mechanism for potential harm was consumption of the sugars, lactose and galactose found in high levels in milk, compared with much lower levels in other dairy products.
In animal studies, these elements have been linked to cell damage, impaired immunity and premature ageing.
Two large groups of 61,433 women and 45,339 men in Sweden completed food frequency questionnaires for 96 common foods including milk, yoghurt and cheese.
Women were tracked for an average of 20 years, during which time 15,541 died and 17,252 had a fracture, of whom 4259 had a hip fracture.
In women, drinking more milk did not cut the risk of broken bones, says the study published in The BMJ.
Women who drank more than three glasses of milk a day - on average 680ml - had almost double the risk of death than women who drank less than one glass of milk a day, around 60ml.
Men were tracked for an average of 11 years, during which time 10,112 died and 5066 had a fracture, with 1166 hip fracture cases.
Men who drank more milk also had a higher risk of death, although much lower than in women.
Tests on blood and urine found higher milk intake was linked to biomarkers of oxidative stress and inflammation.
In contrast, eating more dairy products such as yoghurt and cheese - with a low lactose content - led to lower death rates and fewer fractures, particularly in women.
Professor Michaelsson said the study design meant lactose and galactose could not be definitely implicated and the results "should be interpreted cautiously".
He said independent replication was needed before the findings "can be used for dietary recommendations".
Professor Sue Lanham-New, Head of the Department of Nutritional Sciences, University of Surrey, said the study had limitations, such as weak methods to assess lifestyle habits including bodyweight.
She said: "Also key is that all milk in Sweden is fortified with vitamin A. This may well be an important confounding factor.
"Milk and dairy products in the UK provide 50-60 per cent of the calcium in our diet. We know that low calcium intakes is a risk factor for osteoporosis.
"Individuals should still be encouraged to consume a balanced diet from the five key food groups of which milk and dairy are key."
Professor Brian Ratcliffe, professor of nutrition at Robert Gordon University in Aberdeen, said the possible role of alcohol intake was understated given that most of the extra deaths in milk drinkers were from heart disease.
Lowest milk consumption was associated with the highest consumption of alcohol, and "there is considerable evidence for a cardio-protective effect of alcohol at modest intakes", he said.
- Daily Mail