2nd year student, need help with my model

I am working on a print-scan image steganography. Basically, imagine a watermark on images that is invisible. You could use it like a QR code, on printed material, but the cover image is completely upto you. This is a brief breakdown of the training process. We have an Encoder and Decoder. The Encoder takes an input image I and secret tensor S, generates a Residual image R. R = Encoder( I, S ) Encoded image E is defined as E = I + R

Where function Encoder is just a forward pass of the concatenation of I and S

Then image loss IL is defined as IL = I - transform(E) Where transform introduces noise and image defects you get while scanning

The Decoder takes in E and outputs S' S' = Decoder(E)

Secret loss SL is defined as SL = S - S'

Total Loss TL = IL + SL (oversimplified)

Now, what I realised is that, if I tried to use HSI color space, the hiding capabilities can improve.

Without modifying anything above, I simply converted I to I_hsi before forward pass.

My logic was that, given that my model architecture is sufficiently complex, my model can learn conversion functions implicitly. The Encoder generates a Residual such that it can hide better in the HSI color space.

I implemented this simple fix. My total Loss before this fix was 0.05 My total Loss after the fix was 1.73 (plateaud, same hyperparameters)

Is my logic of the fix wrong? What changes can I make to improve the loss value?