Challenging yourself in a class that doesn't directly apply to your long term career is wasted effort.
Taking the toughest American Heritage and Religion classes was not going to help me in my IT career. If you want to challenge yourself, do it in the classes that matter most.