A Virtual Private Server (VPS) is a virtual machine that is sold as a service. It runs its own OS. The functionality can be compared to the one of a dedicated server. On the other hand, Remote Desktop Protocol (RDP) is software that allows remotely accessing computers over special Windows-based apps.
The VPS is much like a real physical server in terms of functionality. It has its unique IP address and it can be managed independently. This is one of the most common web hosting methods. You can pay for a VPS account and host your website in a virtual environment. In principle, you can even run different operating systems concurrently in your virtual server. Of course, these operating systems are virtualized. Hence, you can have a virtual server that runs Windows and/or Linux.
To interact with different users, the RDP comes in handy. It enables data transfer between users and the server. The communications take place through encrypted channels. It runs Windows only.
Hence, deciding which type of server is better for you is not straightforward. It all depends on what you exactly need. If your aim is to provide hosting for websites, you need a VPS. If all you need is a server with top-level configuration, RDP might be your options.