Venus and Harrison continued on from their stunning performance against defending Henri Kontinen and John Peers in their opening match on Monday morning, but had to recover from a slow start.
Venus lost his opening service game with the French pair returning serve well and creating opportunities with Mahut particularly lethal at the net.
However the Kiwi/American pair fought back to break Herbert's serve in the 6th game and were two points from the first set at 5-4. But the set went to a tie breaker and after going 3-0 up early; Mahut and Hugues Herbert closed it out 7-4 with the set taking 53 minutes.
The French pair had won a tight semi final against Venus and Harrison at the Cincinnati Masters in August and this was again a very close match.
Venus and Harrison returned well throughout and created a break point opportunity on Herbert's first service game in the second set and Venus put away a volley to take the early ascendency.
At 3-2 Harrison fought back from 0-40 to hold, saving four break points. They went on to close out the second set in 39 minutes.
The match went to a deciding Super tie break with Venus and Harrison converting on their third match point for a memorable victory in one hour and 40 minutes.
"We took care of our serves really well and also returned well and got into a lot of service games. They both have great serves and normally roll through their games quickly. So we were able to put pressure on them which helped us roll through our games a bit easier." Venus said.
Venus and Harrison received US$94,000 for participating in the tournament plus $36,000 for each round robin match win. If they go through the tournament unbeaten they will share US$486,000.
"Every match is a great opportunity for us and the money is a bit of a bonus and the points also. So we just try and focus on the things we are doing and hopefully we can keep getting better." Venus said.
Earlier third seeds Rojer and Tecau lost their second straight match, 7-6, 7-6 to 2nd seeds Henri Kontinen and John Peers who face a shootout on Friday morning against Hugues-Herbert and Mahut for the other semi final position in the group.