You are right the problem is the expression you are using. Using rgb(...) in fx is only for specifying a unique color and not for doing computations. You would have to rewrite your code to the following:
using (MagickImage image = new MagickImage("PageHeatmap.png")) { // Evaluate is faster if you just want to set a pixel to a value image.Evaluate(Channels.Green | Channels.Blue, EvaluateOperator.Set, 128); image.Fx("(1-u.r)*255", Channels.Red); image.Write("PageHeatmap.new.png"); }