In the video, the two men jump over the security barriers, set off the alarm, get the lift to the top and then climb out of the building at a terrifying height.
They even film themselves touching the pyramid at the top of the skyscraper.
The pranksters were shocked to find the doors to the roof on the 50th floor were unlocked.
Looking at London below, the camera man said: "Guys I can't believe we are on here, it is crazy.
"Should we touch the top? I need to touch the top of the building. I have done it. I have touched the top of Canary Wharf.
"My heart is pounding. We can literally see the whole of London, just everything."
The men have urged others not to try and copy them, saying the building is now locked.
The video was uploaded by a YouTube user called Night Scape on Monday night.
A Canary Wharf Group spokesperson said: "We are actively investigating unauthorised entry into One Canada Square, on the Canary Wharf Estate.
"Security is taken extremely seriously at Canary Wharf. We invest heavily in our security and review the processes on a regular basis.
"Current arrangements have been tightened; our security procedures are being reviewed and an internal investigation conducted.
"We are working closely with the Metropolitan Police on this matter. Our priority is to ensure the safety of our tenants, employees and those visiting the Estate."