RNA indicates which part of the body cells came from -- different to DNA which would indicate who the cells belonged to.
Prof Buston told the jury Dr Sijen did not follow written instructions when conducting tests on the sample.
"When you bake a cake you should follow instructions. But if you don't follow instructions, you should explain why you are not following the instructions."
There were problems with timings and the test, he said. The instructions said the testing time should be for 90 seconds, rather than 30 seconds used in Dr Sijen's test.
She also used less "primer" to detect RNA, Prof Buston said.
"The method they are using for the detection of RNA is not really fit for purpose."
The test used by the institute was a "source of variability" and the results were not able to be reproduced, he said.
"The inconsistency of the results also concerns me.
"I would be highly reluctant to accept the results of the [tests] because of the technique that has been applied."
The trial in front of Justice Simon France continues.