Even if it is possible, I would strongly discourage doing it. What possible reason could you have for doing it?
I could be wrong (I spent about 3 min total looking into this), but this line in NSMenu, saying "// Create a NSMenuView to draw our menu items.", makes me think this might not be possible.
Considering NSMenuView contains code handling -mouseDown: et al, I'd suspect that to be the reason your approach doesn't work.
I strongly suggest you to implement this in a different way, e.g. by subclassing NSButton.