Answer
$$ \dfrac{2 \sqrt 2-1}{3}$$
Work Step by Step
Consider $I=\int_{0}^{\pi/2} \int_{0}^{\sin x} \cos x \sqrt {1+\cos^2 x} dy \ dx$
or, $=\int_{0}^{\pi/2} y \cos x [ \sqrt {1+\cos^2 x} ]_{0}^{\sin x} \cos x \ dx$
or, $=\int_{0}^{\pi/2} \sin x \cos x \sqrt {1+\cos^2 x} \ dx$
Let us consider $1+\cos^2 x=a \implies da=-2 \cos x \sin x \ dx$
Now, $I=\int_{0}^{\pi/2} \sin x \cos x \sqrt {1+\cos^2 x} \ dx=-\dfrac{1}{2} \int_{2}^{1} \sqrt a da \\=\dfrac{1}{2} \int_{1}^{2} \sqrt a da \\=[\dfrac{a^{3/2}}{3}]_1^2 \\= \dfrac{2 \sqrt 2-1}{3}$$
