The lesions were characteristic of melanomas created in laboratories on the fish species Xiphophorus, the research found.
The study, published in the journal PLoS One, said the sampled fish were caught in a protected marine park area with no evidence of pollution, and therefore the likelihood the cancers were caused by carcinogenic pollutants was low.
"Further work needs to be carried out to establish the exact cause of the cancer but having eliminated other likely factors such as microbial pathogens and marine pollution, UV radiation appears to be the most likely cause," Dr Sweet said.
Previous studies have shown UV radiation can have detrimental effects on marine and freshwater organisms and can penetrate at deep as 60 metres, the researchers said.
The coral trout in this study were all captured in less than 20 metres of water.
However, the high levels of melanoma in the population could also be linked to a genetic defect, said Australian Institute of Marine Science research fellow, Michelle Huepel.
She said it could be a combination of UV radiation, genetics, and the southern Great Barrier Reef location where coral trout live at their temperature limit, which could compromise their immune systems.
Dr Huepel said commercial fishermen had observed the dark lesions on coral trout for many years, so the phenomenon was not a new one.
However, it was Dr Huepel who raised the alarm and sent samples to Dr Sweet for analysis.
She said coral trout can vary in colour and therefore the brown and black spots may not have sparked curiosity before.
The study did not find any fish with advanced cancer. The authors suggesting those fish would be less active and feeding less, and therefore would not have been caught.
More work would need to be done to examine how the disease spreads and whether it is benign or malignant.
Coral trout are an iconic and commercially important species, supporting a high-value fishing industry on the reef.
Dr Huepel said understanding the causes of the disease was important to the conservation and management of reefs.
-AAP