If you are saying there should be further restrictions you are vastly underestimating the resourcefulness of people who need/want money and have a means to convert something they have to cash. You're simply making it less efficient.
If you're saying they should have the penalty waived, then what was the point of putting it in an HSA?
Milton Friedman covers this all here starting at nine minutes:
https://www.youtube.com/watch?v=xtpgkX588nM