You are changing the colors when you convert from CMYK to RGB and back. I think it is failing because you are combining the channels of a 3 channel image with a 4 channel image. I still don't understand why I cannot reproduce it though.
What happens when you use
What happens when you use
TransformColorSpace(ColorProfile.SRGB, ColorProfile.USWebCoatedSWOP) on the textImage and make that CMYK instead of RGB?