I see that some pop-ups (for example when you hover over some buttons in a
customize window) look different. I'd like to decrease their font, which is too
big, but I'm not managing to find the appropriate customization item or
variable. Does anyone know its name, or if decreasing this font is at all
possible?
It is possible. You can either use the customize interface:
M-x customize-face RET tooltip RET
and change the height attribute there, or add a line to your init file:
(set-face-attribute 'tooltip nil :height 100)