Answer
$$\int_{0}^{\pi / 4} \int_{0}^{\cos \theta} 3 r^{2} \sin \theta d r d \theta=\frac{3}{16}$$
Work Step by Step
Given $$\int_{0}^{\pi / 4} \int_{0}^{\cos \theta} 3 r^{2} \sin \theta \ \ d r \ \ d \theta$$
so we have
\begin{aligned}
I&=\int_{0}^{\pi / 4} \int_{0}^{\cos \theta} 3 r^{2} \sin \theta \ d r \ d \theta\\
&=\int_{0}^{\pi / 4}\left[r^{3} \right]_{0}^{\cos \theta} \sin \theta \ d \theta\\
&=\int_{0}^{\pi / 4} \cos^{3}\theta \ \ \sin \theta \ \ d \theta\\
&=\left[-\frac{\cos ^{4} \theta}{4}\right]_{0}^{\pi / 4}\\
&=-\frac{1}{4}\left[\left(\frac{1}{\sqrt{2}}\right)^{4}-1\right]\\
&=\frac{3}{16}
\end{aligned}
