The council-owned spot is close to a family picnic area but is swamped daily with people looking for sex.
It is also popular with dog walkers and cyclists thanks in part to its stunning views of London.
One person who partakes in public sex told The Sun: "It's a popular place for 'dogging'. It's like a sex playground and absolutely anything goes on in those woods. They are used for casual sex. There are condoms everywhere. You don't make dates, you just turn up and have sex with whoever turns up.
"Locals walk dogs, but everyone knows it's also used by people who don't have time for dates and who go there, get off and go. I don't know why Foster would come here."
Neighbours beside the site have long complained about the lewd acts taking place there.
Foster - who won 47 medals in a glittering swimming career that spanned five Olympics - first visited at 10.43am on November 21.
During a torrential downpour he was spotted briefly walking towards the woods.
Two days later he was again seen in his Range Rover as it parked up in the same car park at 11.26am.
But he did not get out and drove off in less than a minute. He was also seen in the same vehicle at 12.27pm on November 24. Half a dozen cars were already parked up with drivers inside.
Foster stepped out of his car and disappeared into the bushes. He could briefly be seen alone in the undergrowth and emerged less than four minutes later before driving away again.
Finally on November 25 he pulled into the almost empty car park at 11.10am, staying less than a minute. Several hours later a police car patrolled the car park, which has seen vigilante attacks in the past.
On Thursday night Foster said any allegations of improper conduct by him would be entirely false.
The Scratchwood site is rated highly on websites for people seeking no-strings sex with strangers. One reviewer said: "Dogging location, open space. Lots going on."
Throughout the week, The Sun watched as dozens of lone middle-aged people pulled up in their cars.
Many sat alone for several minutes before following others out of the car park and into the bushes.
Others appeared to be carrying out solo sex acts in their vehicles. At least one visitor was a man wearing a dress and woman's wig.