Answer
$(-1,1)$
Work Step by Step
The function $f$ is increasing on an interval $\text{I}$ if:
$$f'(x) \gt 0~~\text{on I}$$
Using the fundamental theorem of calculus it follows:
$$f(x)=\int_{0}^{x}(1-t^{2})\cos^{2}(t)dt \to f'(x)=(1-x^{2})\cos^{2}(x)$$
$$f'(x) \gt 0 \to (1-x^{2})\cos^{2}(x) \gt 0$$
Notice that $$\cos^{2}(x) \geq 0$$ for all $x \in \mathbb R$ so the sign of $f'$ is the sign of $(1-x^{2})$.
$$(1-x^{2})\cos^{2}(x) \gt 0 \to (1-x^{2}) \gt 0 \to (1-x)(1+x) \gt 0 \to (1-x) \gt 0 ~~ \text{and}~~(1+x) \gt 0 $$
$$\to x \lt 1~~ \text{and}~~ x \gt -1 $$
The intersection gives:
$$-1 \lt x \lt 1$$
On this interval $\cos^2 x>0$.
The solution in interval notation is $(-1,1)$.
