Ran into a bunch of frustrating problems with Tokyo Cabinet recently.

When my hash database was reaching 2GB in size, the datafile would become corrupt. What's scary is that at that stage, writes to the database seemingly disappeared into the void.

When reopening this corrupt DB, I'd get an "invalid meta data" exception. I googled for "tokyo cabinet invalid meta data" and didn't turn up anything useful. There are, of course, no forums to ask for help.

After re-reading javadocs, I noticed the TLARGE large database flag. Good.. enabled it and reran the data import. Only to have it crash again.

Turns out you ALSO need to _compile_ TC (NOT TC-java bindings) with the --enable-off64 flag enabled during the configure stage. Like so

./configure --enable-off64
make && sudo make install

If you had already compiled it previously, you also need to make clean first.

2 Comments »