You can try if it works when you use this line
install.packages('Rserve',,"http://rforge.net/",type="source")
If not you have to compile a new version of Rserve (1.8.6 as of now) before you can use it. To do that you will have to install Rtools on you machine. Please refer two my entry about KNIME and R how to do that: