Run your program in your debugger and determine where your program crashes. Then work the conditions that cause it to crash there.

Having an instance of the MainWindow in your dialog as a member variable is unusual.
What happens when the existing red, green, or blue value is 255?