Answer
$a)=\int\frac{x^2dx}{(x^2+a^2)^\frac{3}{2}} = \ln(\sqrt{x^2+a^2}+x)-\frac{x}{\sqrt{x^2+a^2}} + C$
$b)=\int\frac{x^2dx}{(x^2+a^2)^\frac{3}{2}}= arcsinh \left ( \frac{x}{a} \right )-\frac{x}{\sqrt{x^2+a^2}}+C$
Work Step by Step
$a)$ Integration by substitution
$I=\int\frac{x^2dx}{(x^2+a^2)^\frac{3}{2}}$
$\begin{Bmatrix}
x=a \tan\theta & x^2=a^2 \tan^2\theta \\
\frac{dx}{d\theta}=a\sec^2\theta & dx=a\sec^2\theta {d\theta}
\end{Bmatrix}$
$=\int\frac{a^2 \tan^2\theta a\sec^2\theta }{(a^2 \tan^2\theta+a^2)^\frac{3}{2}}{d\theta}$
$=\int\frac{a^3 \tan^2\theta \sec^2\theta }{(a^2 (\tan^2\theta+1))^\frac{3}{2}}{d\theta}$
$=\int\frac{a^3 \tan^2\theta \sec^2\theta }{(a^2 \sec^2\theta)^\frac{3}{2}}{d\theta}$
$=\int\frac{a^3 \tan^2\theta \sec^2\theta }{a^3 \sec^3\theta}{d\theta}$
$=\int\frac{ \tan^2\theta }{\sec\theta}{d\theta}$
$=\int\frac{ (\sec^2\theta-1) }{\sec\theta}{d\theta}$
$=\int\frac{ \sec^2\theta }{\sec\theta}{d\theta}-\int\frac{ 1}{\sec\theta}{d\theta}$
$=\int{\sec\theta}{d\theta}-\int{\cos\theta}{d\theta}$
$=\ln(\sec\theta+\tan\theta)-\sin\theta +C_1$
$\sec\theta=\frac{{\sqrt{x^2+a^2}}}{a}$
$\tan\theta=\frac{x}{a}$
$\sin\theta =\frac{x}{\sqrt{x^2+a^2}}$
$=\ln(\frac{{\sqrt{x^2+a^2}}}{a}+\frac{x}{a})-\frac{x}{\sqrt{x^2+a^2}}-\ln(a) + C_1 $
$=\ln({\sqrt{x^2+a^2}}+{x})-\ln(a)-\frac{x}{\sqrt{x^2+a^2}}+ C_1$
$C=C_1-\ln(a)$
$I=\ln(\sqrt{x^2+a^2}+x)-\frac{x}{\sqrt{x^2+a^2}} + C $
$b)$ By the hyperbolic substitution x=a sinh t
$I=\int\frac{x^2dx}{(x^2+a^2)^\frac{3}{2}}$
$\begin{Bmatrix}
x=a \sinh t & x^2=a^2 \sinh^2t \\
\frac{dx}{dt}=a\cosh t & dx=a\cosh t {dt}
\end{Bmatrix}$
$=\int\frac{a^2 \sinh^2t {a}\cosh t }{(a^2 \sinh^2t+a^2)^\frac{3}{2}}{dt}$
$=\int\frac{a^3 \sinh^2t \cosh t }{(a^2 ( \sinh^2t+1))^\frac{3}{2}}{dt}$
$=\int\frac{a^3 \sinh^2t \cosh t }{(a^2 \cosh^2t)^\frac{3}{2}}{dt}$
$=\int\frac{a^3 \sinh^2t \cosh t}{a^3 \cosh^3t}{dt}$
$=\int\frac{ \sinh^2t }{\cosh^2t}{dt}$
$=\int\tanh^2t{dt}$
$=\int1-\sec h^2t{dt}$
$=\int1{dt}-\int\sec h^2t{dt}$
$=t-\tanh t+C$
$\sin\theta=\frac{x}{a}$
$\tan\theta =\frac{x}{\sqrt{x^2+a^2}}$
$t=arcsinh \left ( \frac{x}{a} \right )$
$I=arcsinh \left ( \frac{x}{a} \right )-\frac{x}{\sqrt{x^2+a^2}}+C$
