Depending upon your version of MacOS you might have to install additional tools in order to compile R packages from source code:
https://cran.r-project.org/bin/macosx/tools/
The error message seems to suggest it is trying to install version 1.7-3. You might have to download the newest source ( Rserve_1.8-6.tar.gz) manually from here: