Goku is literally brain-damaged (when he bumped his head when he was a child, erasing his Sayajin programming, that damaged his memory, therefore, his brain), that is a fact. However, that doesn't necessarily make him retarded. There is a difference between those terms.
EDIT: To clarify...