What totally surprised me, and tripped up countless people during our trip, was the meticulous way this must be done, and the additional shoe-based rules that go along.
When do I need to take my shoes off?
There is no hard and fast rule on where you must be shoeless, so your best guide is your eyes. Homes and hotels will typically always be shoeless as well as anywhere with a tatami mat floor.
As for shops and restaurants, your eyes are your best guide, so take a look around when entering a restaurant, shop or attraction. If there's a line of shoes or cubby outside or people are shoeless, that's your sign to follow suit.
How do I take off shoes?
Now, this is the moment that catches almost every tourist out.
Shoes should never touch an area dedicated for bare feet. However, bare feet shouldn't touch the ground dedicated for shoes. So, once you remove a shoe, you must step that foot onto the barefoot area, not back onto the ground. Then, repeat for the second foot.
Typically, places have a wooden pallet you step onto once you've removed a shoe. What many tourists do is step their shoes on here (a big no no), or step their bare feet onto the ground (again, not polite).
For this reason, the Japanese often choose shoes they can slip in and out of without untying laces.
Where do I put my shoes?
Again, if in a popular place, simply follow suit to what others have done. Use cubbys or lockers if they are provided and if lining them up on the ground, make sure they point towards the front door (this also makes it easier to put them back on when leaving).
A note on slippers
If the building requires you to remove shoes and has bathrooms (like a home or hotel), a pair of slippers are often provided in the toilet. As I learned the hard way that morning in the hotel, wearing "bathroom slippers" outside the bathroom will earn you an embarrassing tap on the wrist.
What happens if I'm unsure?
If in doubt, ask! The Japanese may have many rules, but they are often more than happy to give advice when it comes to customs.