I think paper size has a huge effect on the game contained on it. When reading a lot of text, I want a narrow-ish two-column so that my eye doesn't have to hunt for the start of the next line.
I like the idea of a square book but I think it would need to represent the game - for example an RPG based on Monopoly - which has a square board.