I'm working on some experimental patches to implement caching of color-name lookups and color-cell allocations.
Along the way I noticed that for visual class TrueColor, we never free color cells on the server, and we can (and do, but only in image.c) synthesize pixel values from RGB values instead of calling XAllocColor and making the round trip to the server. TrueColor is all my laptop display supports, and I gather it's supported on most hardware these days, so it may be worth taking advantage of these to cut out the extra traffic, and I can probably drop the color-cell allocation cache work...
Ken