McIlroy finished at 18-under-par, one ahead of Scott, who closed with a one-under-71.
Former champion John Senden closed with a Sunday-best 66 to claim outright third - and automatic qualification for next year's British Open - at 11-under.
Rhein Gibson and Brydan Macpherson also clinched Open berths by carding 69s on Sunday to share fourth spot.
Scott had been hoping to join Robert Allenby as the only the second player to complete the Triple Crown after also winning the Australian Masters and Australia PGA titles last month.
But Australia's first US Masters champion couldn't buy a putt on the back nine when he had countless opportunities to slam the door shut on McIlroy.
All up, Scott lipped out on five occasions during the final round.
McIlroy, a two-time major champion, wiped away Scott's four-stroke lead by the eighth hole with a birdie, eagle, birdie run to set the stage for a thrilling climax to the tournament.
Scott held his nerve to drain an eight-footer on nine to regain his outright lead as the back nine developed into a gripping matchplay struggle between Australia's world No.2 and the world No.6 from Northern Ireland.
Scott wasted golden opportunities to double or even triple his advantage when he missed six and four-foot birdie attempts on the 10th and 11th holes as the pressure intensified.
It became a putting duel as both rifled long irons to within close range to set up eagle chances on the par-5 13th.
McIlroy missed his putt and Scott, with the chance to go two up, left his on the lip as the pair settled for birdies.
Clinging to a one-shot lead, Scott had another big chance to close the door on McIlroy on the par-5 16th only to three-putt for par when a birdie or eagle would have left his challenger two or three behind with two holes remaining.
McIlroy dodged another bullet when he got up and down for a par three on the difficult 17th hole and Scott missed yet another eight-footer for birdie.
The succession of missed opportunities came back to haunt Scott on the final hole of the tournament.
New Zealand's Mark Brown finished in a tie for sixth place at eight-under after a three-under 69 in the final round. Fellow kiwi Mahal Pearce was four further shots back in a tie for 20th while Michael Long and Tim Wilkinson both finished in a share of 27th at two-under.
- AAP
