Answer
$$r'=2\theta\cos(\theta^2)\cos(2\theta)-2\sin(\theta^2)\sin(2\theta)$$
Work Step by Step
$$r=\sin(\theta^2)\cos(2\theta)$$
The derivative of $r$ is $$r'=\Big(\sin(\theta^2)\Big)'\cos(2\theta)+\sin(\theta^2)\Big(\cos(2\theta)\Big)'$$
According to the Chain Rule, we have
$\Big(\sin(\theta^2)\Big)'=\cos(\theta^2)(\theta^2)'=2\theta\cos(\theta^2)$
$\Big(\cos(2\theta)\Big)'=-\sin(2\theta)(2\theta)'=-2\sin(2\theta)$
Therefore, $$r'=2\theta\cos(\theta^2)\cos(2\theta)+\sin(\theta^2)\Big(-2\sin(2\theta)\Big)$$
$$r'=2\theta\cos(\theta^2)\cos(2\theta)-2\sin(\theta^2)\sin(2\theta)$$