Answer
$$ \frac{3MR^2}{10} $$
Work Step by Step
Have to start by dividing the cone into very thin, approximately cylindrical slices of height $dx$. Using the assumed-to-be-known formula for the moment of inertia of a solid, uniform cylinder (available in table 9.2), one can find the moment of inertia $dI$ of the slice (rotating around the $x$-axis, as seen in the picture):
$$ dI = \Big( \frac{r^2}{2} \Big) dm $$
In order to express $dI$ in terms of the known values $R$, $h$ and $M$ and the integration variable $x$, one has to relate the slice's radius $r$ and mass $dm$ to these terms. The mass $dm$ is the slice's volume times its density, $(\rho)dV$ and the formula for the volume of a cylinder is also assumed to be known:
$$ dV = (\pi r^2)dx $$
The density of the slice $\rho$ is the same as the density of the entire cone, which can be written as the ratio between the cone's mass $M$ and volume $V$ (again, formula assumed to be known):
$$ \rho = \frac{M}{V} = \frac{M}{\frac{\pi R^2 h}{3}} = \frac{3M}{\pi R^2 h} $$
Thus, $dm$ can now be written as follows:
$$ dm = (\rho)dV = \Big( \frac{3M}{\pi R^2 h} \Big) (\pi r^2)dx = \Big( \frac{3Mr^2}{R^2h} \Big) dx $$
Time to insert this into the previous expression for $dI$:
$$ dI = \Big( \frac{r^2}{2} \Big) dm = \Big( \frac{r^2}{2} \Big) \Big( \frac{3Mr^2}{R^2h} \Big) dx = \Big( \frac{3Mr^4}{2R^2h} \Big) dx $$
The radius $r$ increases linearly with $x$, like so:
$$ r = \frac{R}{h} x $$
Inserting this into the most recent expression for $dI$ yields:
$$ dI = \Big( \frac{3MR^2}{2h^5} x^4 \Big) dx $$
Integrating both sides provides the following integral:
$$ I(x) = \frac{3MR^2}{2h^5} \int x^4 \, \mathrm{d}x $$
Finally, integrating from $x = 0$ to $x = h$ to get the formula:
$$ I = \frac{3MR^2}{2h^5} \int_0^h x^4 \, \mathrm{d}x = \frac{3MR^2}{10} $$
