It depends what you mean by "grounded". In the case of suspending or revoking a license, I think it requires a civil case in front of an Administrative Law Judge--unless the pilot voluntarily surrendered it, of course. This would most likely happen after there was an accident or incident investigated by the NTSB, or possibly a serious deviation reported by ATC (and not self-reported to ASRS).
Less formally, if an employer, fellow crew member, instructor, etc. has serious doubts about whether a pilot is fit to fly, then given the risk-averse nature of the field, it's expected that the pilot would voluntarily ground himself as a precaution until he could resolve those concerns.
FWIW, I ran across this recording where ATC declared a pilot a "risk to aviation safety" and ordered her to land immediately--literally grounding her. I suspect FSDO took a serious interest in that afterward, but I don't know what the final result was.