Answer
$V\left( r \right) = - \dfrac{{mG}}{{2Rr}}\left( {\sqrt {{R^2} + {r^2}} - \left| {R - r} \right|} \right)$
Work Step by Step
Using the integral for the sphere in Exercise 48 (b), we have
$V\left( {0,0,r} \right) = - \dfrac{{mG}}{{4\pi }}\mathop \smallint \limits_{\phi = 0}^\pi \mathop \smallint \limits_{\theta = 0}^{2\pi } \dfrac{{\sin \phi {\rm{d}}\theta {\rm{d}}\phi }}{{\sqrt {{R^2} + {r^2} - 2Rr\cos \phi } }}$
However, in our present case, the surface is a hemisphere of radius $R$. So, we need to modify this integral for $0 \le \phi \le \dfrac{\pi }{2}$. Thus,
$V\left( r \right) = - \dfrac{{mG}}{{4\pi }}\mathop \smallint \limits_{\phi = 0}^{\pi /2} \mathop \smallint \limits_{\theta = 0}^{2\pi } \dfrac{{\sin \phi {\rm{d}}\theta {\rm{d}}\phi }}{{\sqrt {{R^2} + {r^2} - 2Rr\cos \phi } }}$
$V\left( r \right) = - \dfrac{{mG}}{{4\pi }}\mathop \smallint \limits_{\theta = 0}^{2\pi } {\rm{d}}\theta \mathop \smallint \limits_{\phi = 0}^{\pi /2} \dfrac{{\sin \phi {\rm{d}}\phi }}{{\sqrt {{R^2} + {r^2} - 2Rr\cos \phi } }}$
$V\left( r \right) = - \dfrac{{mG}}{{4\pi }}\left( {2\pi } \right)\mathop \smallint \limits_{\phi = 0}^{\pi /2} \dfrac{{\sin \phi {\rm{d}}\phi }}{{\sqrt {{R^2} + {r^2} - 2Rr\cos \phi } }}$
$V\left( r \right) = - \dfrac{{mG}}{2}\mathop \smallint \limits_{\phi = 0}^{\pi /2} \dfrac{{\sin \phi {\rm{d}}\phi }}{{\sqrt {{R^2} + {r^2} - 2Rr\cos \phi } }}$
Write $u = {R^2} + {r^2} - 2Rr\cos \phi $. So, $du = 2Rr\sin \phi d\phi $. The integral becomes
$V\left( r \right) = - \dfrac{{mG}}{{4Rr}}\mathop \smallint \limits_{u = {R^2} + {r^2} - 2Rr}^{{R^2} + {r^2}} \dfrac{1}{{\sqrt u }}{\rm{d}}u$
$V\left( r \right) = - \dfrac{{mG}}{{4Rr}}\mathop \smallint \limits_{u = {{\left( {R - r} \right)}^2}}^{{R^2} + {r^2}} \dfrac{1}{{\sqrt u }}{\rm{d}}u$
$V\left( r \right) = - \dfrac{{mG}}{{4Rr}}\mathop \smallint \limits_{u = {{\left( {R - r} \right)}^2}}^{{R^2} + {r^2}} {u^{ - 1/2}}{\rm{d}}u$
$V\left( r \right) = - \dfrac{{mG}}{{2Rr}}\left( {{u^{1/2}}|_{{{\left( {R - r} \right)}^2}}^{{R^2} + {r^2}}} \right)$
$V\left( r \right) = - \dfrac{{mG}}{{2Rr}}\left[ {{{\left( {{R^2} + {r^2}} \right)}^{1/2}} - {{\left( {{{\left( {R - r} \right)}^2}} \right)}^{1/2}}} \right]$
Therefore,
$V\left( r \right) = - \dfrac{{mG}}{{2Rr}}\left( {\sqrt {{R^2} + {r^2}} - \left| {R - r} \right|} \right)$
