That led to a remarkable 30 more penalties in the shoot-out. Earlier, Southampton - second in the Premier League - came from behind to win 2-1 away to top-flight rivals Arsenal at the Emirates Stadium.
Arsenal manager Arsene Wenger made 10 changes to the side that beat Aston Villa 3-0 in the Premier League last weekend but when Alexis Sanchez put the Gunners 1-0 up with a superb 14th-minute free-kick, it seemed the home side were on their way to victory.
But the Gunners, with Saints old boy Calum Chambers the sole survivor from the Villa victory, were pegged back by Dusan Tadic's penalty six minutes later.
Southampton's Nathaniel Clyne then scored what turned out to be the winning goal with a blockbuster 35-yard strike five minutes before half-time.
Saints squandered several chances but did enough to knock Arsenal out of the tournament. Another all-Premier League tie saw Swansea overwhelm Everton 3-0 at the Liberty Stadium.
Sunderland, last season's beaten finalists, bowed out following a 2-1 defeat by fellow Premier League side Stoke City.
Third division Milton Keynes Dons, who knocked English giants Manchester United out in the last round, continued their League Cup progress as Benik Afobe scored twice in a 2-0 win over Bradford.
Meanwhile fourth-tier Shrewsbury followed up their second round victory over Leicester by knocking out Championship side Norwich, chasing promotion to the Premier League, with a 1-0 win over Norwich. Managerless Cardiff lost 3-0 at home Bournemouth, making while Derby saw off Championship rivals Reading 2-0.
Fulham, who recently sacked manager Felix Magath, won for the first time this season as the bottom of the Championship side beat third-tier Doncaster 2-1. Sheffield United saw off fellow third tier side Leyton Orient 1-0.