Yes, there is a software/hardware that can achieve this, and relatively cheaply. Asterisk is an open-source (free) PBX software, that can do what you need, and more. Asterisk is usually used on linux, although there are pre-packaged virtual machine appliances available. There are also lots of Asterisk-derivatives, like PbxInAFlash, TrixBox, and others, that are packaged with various add-ons.
In addition, you need a hardware to connect your phones to a computer. Like LinkSys PAP2 (really old, but can be bought second-hand cheaply), which can connect two phones. There are bigger devices that can connect more phones, or you can buy a dozen PAP2s (they are network-connected, and scale ok).
However, judging your expertise from your question, putting everything together does require a certain level of technical expertise that you probably don't have. So getting outside help is probably a good idea. But you certainly don't need to pay your telco's prices.
What you will pay for
Cost of the hardware: Computer + VoIP adapters to connect phones to the PBX. Depends on the number of phones, but generally $20 per phone is a fair price if you need 2-10. Any generic computer made in the last 5 years will handle the 5-10 simultaneous calls ok.
Cost of network traffic. Depends of number of simultaneous calls and call quality. Your basic home-level broadband internet connection can probably handle 5 simultaneous calls ok.
Cost of DID (incoming phone number). Generally very cheap ($5/mo), but if you need vanity number (1-800-CALL-ME), then the sky is the limit.
Cost of incoming calls (per minute, depends on country, generally way cheaper than what your telco quotes).
Cost of the technician setting things up.