Answer
$$\frac{-x}{\sqrt{x^{2}-1}}+\ln |x+\sqrt{x^{2}-1}|+C$$
Work Step by Step
Given $$\int \frac{x^{2}}{\left(x^{2}-1\right)^{3 / 2}} d x $$
Let
$$ x=\sec u\ \ \ \ \ \to\ \ \ dx=\sec u \tan udu$$
Then
\begin{align*}
\int \frac{x^{2}}{\left(x^{2}-1\right)^{3 / 2}} d x&=\int \frac{\sec ^{2} u(\sec u \tan u) d u}{\left(\sec ^{2} u-1\right)^{3 / 2}}\\
&= \int \frac{\sec ^{2} u(\sec u \tan u) d u}{\left(\tan ^{2} u\right)^{3 / 2}}\\
&=\int \frac{\sec ^{3} u d u}{ \tan ^{2} u }\\
&=\int \sec u\csc^2 udu
\end{align*}
Use integration by parts with $$U= \sec udu,\ \ \ dv=\csc^2 udu $$
Then
\begin{align*}
\int \frac{x^{2}}{\left(x^{2}-1\right)^{3 / 2}} d x&=\int \frac{\sec ^{2} u(\sec u \tan u) d u}{\left(\sec ^{2} u-1\right)^{3 / 2}}\\
&= \int \frac{\sec ^{2} u(\sec u \tan u) d u}{\left(\tan ^{2} u\right)^{3 / 2}}\\
&=\int \frac{\sec ^{3} u d u}{ \tan ^{2} u }\\
&=\int \sec u\csc^2 udu\\
&=-\sec u \cot u+\int \sec u \tan u \cot u \, d u\\
&=-\sec u \cot u+\int \sec u \, d u\\
&=-\sec u \cot u+\ln |\sec u+\tan u|+C\\
&= \frac{-x}{\sqrt{x^{2}-1}}+\ln |x+\sqrt{x^{2}-1}|+C
\end{align*}
