I have two systems, and I want to transfer data from one system to another while doing my project. I do not have an internet connection, and I do not have a modem. Is there any possibility of connecting the two systems without a modem and by using single RJ45 cable?
If the two devices are in the same place (i.e. you don't need to connect them over the internet) you wouldn't use a modem, you'd use a hub or switch - a home modem router would work, but that's just because they are normally also a switch, and I guess this is where the confusion lies.
You can simply connect two devices together and they will create something of an ad hoc network.
Normally to connect devices directly you need a RJ45 crossover cable (which isn't the same as a normal RJ45 cable), but if both devices have Gigabit Ethernet ports then they should work using a standard cable (thanks to afrazier for the reminder).
You have not mentioned what you wish to connect, so I'm going to assume it's two Windows PCs. In which case Microsoft has a handy article for you.
The basics are: