I tracked down some relevant logic in emacs-23.3 keyboard.c source.
Essentially, I comment out a bunch of really aggressive rules that cause the mouse_click modifier to be recorded in an event modifier ( "<" is current emacs behavior, and ">" is my modification):