The new measure is expected to be in force this autumn.
Breaking rules while using one's mobile phones or earphones at the wheel is currently punishable by a €135 ($228) fine and the loss of three licence points. But Mr Borne said that current punishments had failed to change the worst offenders' habits.
"We don't wish to deprive anyone of their licence for months on end," however, he said, suggesting the confiscation would be temporary in all but the most serious cases.
One motorist told BFM TV: "In that case, I'm likely to lose my licence soon."
Recent polls suggest that 70 per cent of French motorists admit to having used their mobiles while driving and another report found that a total of six per cent use their phones while driving on the motorway, compared to four per cent in 2015.
At 15 per cent, the figure was much higher for truck drivers.
"Motorists have not yet fully realised how dangerous it is to use their phones when behind the wheel," Pascal Contremoulins, head of road safety at Sanef, a company responsible for operating French motorways, told French media last week.
"Fifteen per cent of fatal accidents on the motorway are due to inattentiveness."
However, Jean-Baptiste Iosca, a lawyer specialising on road safety, said that such "repressive" measures were not the way forward given that there were already "two million people in France on the roads without driving licences".
The only solution was to raise awareness, he said.
In the UK, motorists caught using a handheld phone have faced receiving six points on their licence and a £200 ($377) fine - up from the previous penalty of three points and £100 ($188).
Under a law passed in March, 2017, drivers can lose their licence if they receive 12 points within three years, or six points in the first two years after passing their test.
According to police figures obtained by road repair company AA last year, the stricter rules on mobile phone use at the wheel has cut the number of offences by half.
Around 39,000 fixed penalty notices (FPNs) were issued to drivers between March and December 2017 compared with 74,000 during the same period in 2016.
The 47 per cent decline is due to a combination of harsher punishments, road safety campaigns and a lack of enforcement due to reductions in traffic officer numbers, according to the AA.