Another issue here is what is meant by 'package' in this context.
I have packages from elpa, melpa, etc in ~/.emacs.d/elpa. That directory also includes development versions of packages I'm working on, installed via package-install-file. Then there's all the bits and pieces of elisp that I've either written myself or found online. These are stored either in ~/.emacs.d/ or a subdirectory.
I also keep the emacs sources in .emacs.d/trunk, as I'm using the latest development version.
Everything in that directory is backed up in a private bitbucket repository, set to ignore .emacs.d/trunk and .emacs.d/elpa.
From Stefan's comment it sounds as if I should pull everything out of .emacs.d except init and elpa. But it's convenient for me to have emacs in its own contained world on my computer. Is there a risk of breaking something with this approach?