“打” is an action of hand, which is the basic semantic feature of “打”.
The verb phrase 打毛衣 is lexicalized to denote the action of "knitting." The lexicalization of this phrase is centered on the noun 毛衣. On the other hand, in the verb phrase 打衣服, the noun 衣服 doesn't serve the same function. Therefore, 打 only conveys the core meaning of hit in this phrase.
