According to ASE:
"Brake fluid is a simple maintenance item, but one not to be overlooked. Contaminants such as water, rust, dirt, mineral oil or compounds from overheated glycol get into brake fluid and can cause damage throughout the braking system. Moisture can create bubbles within the brake lines that diminish stopping power, or it can cause corrosion within the caliper, which can seize pistons and eventually lead to a pull. Mineral oils can attack rubber seals and cause leaks; dirt, rust and debris can clog valves.
Remember that most brake fluid is a polyglycol formulation, which will absorb moisture. So, an open can of polyglycol brake fluid should be used immediately to avoid any inadvertent moisture absorption.
Check the condition of the fluid itself when checking fluid level. You want to see relatively clear fluid. Cloudy fluid is an indication of moisture, and dark fluid indicates debris. If the fluid looks layered, that indicates a silicone brake fluid was mistakenly mixed in with the polyglycol fluid. In any of these cases, the entire system should be flushed and refilled with the recommended fluid" ASE A5
Also copper contamination is one of the most common contaminants to the brake system due to the additives and base stock breaking down or accumulating moisture. Gylcol based fluid attracts moisture like a magnet does steel. Silicone based brake fluids do not have the moisture attraction problem. However, YOU SHOULD NOT MIX GYLCOL AND SILICONE FLUIDS
You don't just change your engine oil due to contamination. You also have to change your engine oil due to the base oil stock and the additives breaking down over time and due to heat.