Remember when you were 8 and you were riding your bicycle and you forgot to stop at the intersection? That’s why I’m failing you.
So let’s say you move to Japan and you need to be able to drive a car for some reason (work, you don’t live in a big city, etc). Hopefully you have a license in another country as this will make the process much easier; if you don’t happen to be in the possession of a license then you are going to have to do the same thing as every other Japanese person. So after you get all of your paperwork ready (translation of your license, ID card, photos) you go to the drivers license center for your written test. The test is usually easier than the Japanese version since they know that you already have some driving experience; the test is to make sure you know some of the different things you’ll experience on the road in Japan. After you go through this part you will schedule the driving portion of the test. This is where things get hairy.

I can see your failure from here
A typical course looks like the one above, four rectangles with winding roads traversing the interior of each one. When you go to take the test you will make a couple of turns and weave through one of the rectangles. The roads in the rectangles are much narrower than the main road since they are meant to mimic some of the crazy strips of concrete that are somehow still considered roads.
When it’s your turn to drive, you have to make sure that your first move is not to the driver door (on the right), but to the front of the car. You have to check under the front of the car, not because it’s something you’ll ever do again until you run over an old woman, to make sure that there aren’t any obstructions. Remember all the times you’ve done this? Yep. After this you can get in the car. From here you adjust everything (seat, mirrors) so that you can use them all correctly before you start on your 5 minutes of fun.
So what are they looking for? A small part of this is checking whether or not you can keep a vehicle and the road, but they’re not really looking for this; what they really want you do to is follow a rigid set of rules that, for the most part, you’re never going to follow again. When you are driving you, obviously, have to check mirrors. They don’t want you to just check the mirrors, they want you to show they that you are doing it. This means that you need to do more than just move your eyes, you must move your head enough (more than you might expect) so that it’s easily noticeable to the examiner. Don’t do this (even if you’re checking your mirrors) and you’ll likely fail, even if you do everything else very well. Stop at a light or a stop sign and the car is even slightly in the plane of the big white stop strip and you’re losing points. Touch, not cross, any other line and you’re losing points (e.g. switch to a turning lane and graze the line in between the two lanes). They think you didn’t have your turn signal on long enough (3 seconds before turning) you’re losing points.
This is quite well known for anyone living in Japan and pretty much expected. From drivinglikeass.com:
But the skills test is the killer. Taking the skills test at the official center means being prepared to fail at least three or four times: almost no one passes the first time.
Besides the usual “major” violations, which result in immediate cessation of the skills test (hitting a pole while parking, not stopping for at least three seconds at stop signs), there are all manner of “minor” violations that, in the aggregate, will result in failure, including:
Not checking under and behind your car for animals or small children prior to getting in
Not adjusting your seat, seatbelt, and mirrors before starting the car
Not putting the emergency break on when you finish the course
Not leaving enough room on the side of the road for bikes to pass you
Touching the curb when parking
…not to mention all manner of less common sense things such as:Failing to check mirrors in the “correct” order
Not responding with a polite “hai” (“yes” or “OK”) after the officer gives you instructions
If you don’t know the system then you’re going to run into a lot of issues; you could be an excellent driver, but if you’re not performing for them then you’ll probably end up with a big fail and the examiner will be quickly asking you if you can come the same time tomorrow, like you have nothing better to do than endlessly take a driving test.

You have a license? All that means is that you can drive in a closed course free of octopi.
This is another point that needs to be considered; they can (and do) fail you over small slights, which means that you need to take the test again and pay the center, again. Do they purposely fail people so that they can help the center receive more money? I can’t, and won’t, say since there isn’t really any way to prove that. With that said, however, they are a little quick to ask you when you can come back. This also goes to show that they’re not really worried about public safety nor your inability to control a large pile of metal.
“Sorry, but you failed. You did this and this wrong. So are you free tomorrow?”
There are many other reasons why they might fail you (mad, feel like it, don’t like you, feeling power-hungry), but these come with any job that gives any amount of power over another person; these reasons are applicable to a wide variety of jobs and locations and they’re probably a non-zero percentage of reasons for failing a test taker.
It can be a pain in the ass to take a driving test in Japan, but once you realize that what you’re doing is not testing potential, but your ability to perform and follow formalities. Once you understand this you’re already most of the way finished with the test.

I failed my driver’s test the first time because the woman said my tires didn’t have enough tread and refused to get in such an unsafe vehicle. I came back the next day with the same tires and passed.