The Rovers scored first from captain Fergus Neil's free-kick in just the fifth minute from the left flank.
Right-midfielder Ross Willox rose above the defensive wall to nod the ball past Olympic goalkeeper Scott Basalaj.
Seconds after play resumed, Willox was back in the visitors' 18m box, curling a cross to Angus Kilkolly but this time Basalaj was equal to the occasion for the header although the ball deflected to forager Jean-Michel Paulin who drilled it back to the keeper's feet.
Veteran Olympic striker Luis Corrales howled like someone had extracted his toenail in the 17th minute when Blues veteran midfielder Chris McIvor tackled him from behind but play carried on among a few polite exchanges when he clutched his heavily strapped knee and was replaced in the second half.
The reality was defender Matt Bruin had Corrales' number in the first half at least.
Neil prevented an equaliser in the 24th minute when he headed a ball away from the right upright after a goalmouth scramble but the Greeks didn't have much impetus or shots at goal.
Willox was having a blinder of a game and was instrumental in feeding Kilkolly for the 2-0 lead in the 32nd minute.
Four minutes later, Kilkolly surged into the box to find himself in a one-on-one situation with Basalaj but the keeper won that battle as the ball chipped him on the shoulder to zip past the left upright amid howls of protests from the Bluewater Stadium faithful when referee Martin Roil ruled a goal-kick.
Just 30 seconds before halftime, Basalaj denied Kilkolly after he received a ball, snaked around a defender before drilling it to him. The striker had another attempt from the deflection but defender Jacob Masseurs stuck his foot out to repel him again.
As good as Kilkolly is, he cost his team a goal in the 48th minute when he selfishly took a shot from inside the top of the box as vice-captain Joshua Stevenson, unmarked to his left, was left remonstrating.
But the poor decision had bigger ramifications. From the ensuing clearance kick from Basalaj, Olympic striker Sam Blackburn became the beneficiary on the left flank. He took a shot from an oblique angle after a surge into the box to find a kind deflection off a defender's foot to clip the far post and roll into the net to narrow the margin, 2-1.
The momentum shift was painfully obvious when in the 54th minute Blake Weston came close to levelling terms as his header rolled just outside the far post.
Five minutes later, Olympic striker Daniel Mulholand had another shot from just inside the box from an obtuse angle but Blues keeper Ruben Parker Hanks collected it low at the near post.
From there on the game became a little testy, with players from both sides chucking handbags at each other although referee Roil could have prevented it by pinging players from grabbing shirts and shorts.
The Wellingtonians threatened to draw and push the game out on a pitch that was beginning to show wet scars but, to the relief of parochial cheering and clapping home fans, the referee blew up to ensure another round awaits in the cup.
