This topic has been dealt in one of my last technical articles. The document shows as it’s possible to approach an incident investigation on the basis of common instructions used by shellcodes to achieve their goals. The analysis is conducted within a memory dump. In this regard have been developed some yara custom rules that can be used as an aid for experienced analyst to quickly locate malicious code. Note, however, that this is a very wide topic (have you ever heard of padding, permutations, encoding etc. etc.) and rules that can be good for certain situations may not be effective in others.
Read full article on InfoSec Institute website:
You can also download the Yara rules created for this research: