The image I had of Japan was one of technological excellence, a pioneer in all things gadgets. How hard can getting a phone be? They must be bleeding smartphones and sim cards on almost every street corner and it must be harder to not be within range of free wifi than to be. Well, it turns out that getting a reasonably priced mobile phone is in fact quite difficult if not impossible and free wifi only exists in starbucks (and there you have to buy a drink, which one could argue is a positive thing). Any foreigner who comes to Japan basically has four options:
1) Don't get a phone. Depending on how long you spend here you might survive with your own phone, text messages cost about 30 cents for me on my Finnish phone so it's not the end of the world if you send a few every now and then.
2) Get a data sim. One cannot get only a sim card in Japan, they are always sold with a phone. However, one can buy a data only sim online. It costs 15,000 yen for 6 months, so about 20 eur per month. This actually seems like a decent option, but of course it means that all the people you want to communicate with have to have something like skype or whatsapp.
3) Get prepaid. According to everyone I've asked the cheapest carrier in Japan is Softbank, and as it happens Softbank also has the most comprehensive English website. On this website, it says that something called a prepaid phone exists (note that this is a prepaid phone, so you cannot buy a prepaid sim card). The way prepaid works is that you charge either 3000 or 5000 yen on to the sim, which validates the sim card for 2 months. After the 2 months, you have to charge 3000 or 5000 yen regardless of whether you need it or not. The beauty of this deal is that by paying 300 yen a month you get free messages, so assuming you don't make any calls this can be quite cheap (around 12 eur per month not including the phone and charger you have to buy).
But there is a twist, there are no prepaid phones anywhere to be found. The big Softbank stores with English speaking staff, Shibuya, Omotesando and Shinjuku, get three (yes, three) prepaid phones a day. These get sold out in the morning, so if you work during the day like any normal person, you will never be able to buy one. Furthermore, the smaller stores, which are absolutely everywhere but only have staff who speak Japanese, don't have any prepaid phones. As it turns out I got my phone at one of these Japanese only stores, and the whole process went pretty painlessly with my not so fluent Japanese skills coupled with creative use of a dictionary.
4) Join a two year we're-going-to-rip-you-off contract. These are the contracts that all Japanese people join, they can be cheaper or more expensive depending on the type of phone you choose to have and whether you want internet or not. The cheapest contract is something called the White plan, which costs 980 yen per month + 300 yen per month for free messages. So far it doesn't seem to cost more than the prepaid, but what increases the cost is the 10 000 yen you have to pay to be able to cancel it after one year. If you want unlimited internet the monthly few is somewhere above 4000 yen a month.
A useful thing to know is that you do not need a Japanese bank account to get one of these contracts, all you need is a credit card, you National Health Insurance card (someone want to explain this to me?) and your foreigner card.
There's no correct answer to which option one should pick, since it depends on what you are going to be using the phone for. Personally I picked the White plan, because I got the mobile phone for free and calling to other Softbank phones is free from 1 am to 9 pm (messages are free for the whole day). Besides, who wouldn't want to own a traditional water proof Japanese keitai which has free tv! For most people who already have a smartphone, I think the data sim is the way to go.
EDIT: Having talked with other people and researched this a bit further, it turns out it is possible to get data+calls sim card from a company called b-mobile. However, their website is all in Japanese which makes finding the correct plan a bit more difficult.