training controlnet to recover image,but i failed,can anyone help me?

[balala8]

I had the whim that I wanted to feed an rgb image into controlnet and output the image using sd.
It looks like the following image

Sounds interesting. It's like I tell controlnet what I want and sd generates the exact same image for me.

i use text2image pipeline with controlnet, using diffusers. The code was changed from here,https://huggingface.co/blog/train-your-controlnet. batch size is 32,gradient_accumulation_steps=2,num_train_epochs=50,learning_rate=1e-5.
and dataset images about 2000image and prompts.

Example input and prompt:

a pearl bracelet with a gold coin,on pink and blue fine silk, the silk is very smooth

Result after 2000 epochs:

I am reaching out to this knowledgeable community for insights or advice on why the resulting images lean towards a white palette and how I might refine the process to achieve more vivid and accurate color representation. Any guidance, technical suggestions, or shared experiences would be immensely appreciated as I navigate this innovative yet complex endeavor.