I don't do all the background and the worldbuilding before I start the story. What I do is I work out the bare minimum I need to start the story, and often that really is a bare minimum - it's a character in a situation, and I know nothing about the character, I know nothing about the situation, and then I think about it for a long time, and make notes about where I think the story is going to go and so on, but I don't really make notes to do with the background or the magic system or the world.