Just as the title suggests.
Is a public IP exposed if a BitTorrent client is open but not downloading?
I'm not completely sure what you mean by trackable, but a lot of bittorrent clients will open a port on your router firewall using UPnP making you detectable and potentially exposing your computer to anyone who has found an exploitable bug in the networking code of the bittorrent client.
It's not that likely there will be such a bug, but it is not outside the realms of possibility that a carefully malformed request to your client could trigger some kind of bug.
As to "trackable", it is possible that due to the above hole in your router firewall the presence of a bittorrent client could be inferred and detected. But "trackable" infers detection of your location and activities; the first is possible, the second is very unlikely.
Yes, a malicious client can probe you, though your torrent software should return an error to them if you are not participating in the swarm that they are targeting.
The common case here, is that you were participating in a swarm, and other clients have learned your IP. A malicious party enters the swarm, and gets your IP from another client. They then attempt to connect to you. Because these kinds of malicious clients are not interested in actually obtaining chunks of a file by the most expedient means possible, they can be modified to just keep trying to connect, in case the target has that particular torrent paused and will become active again later.
There are a couple ways to deal with this, depending on whether you use UPNP. if you do, set your software to use a randomly selected port, and restart it. if not, you will have to periodically change the port in your client and router port forward rules. Your best bet, if you can afford it, is a privacy proxy for torrents.
The benefit of a proxy over a vpn is that your torrent software directly interacts with the proxy, rather than leaving it up to system routing. if the proxy is not accessible, then the software just doesn't work, but if your VPN is down, your routing will just pass the traffic out of your home connection instead of using the tunnel. The downside to a proxy (and an upside for VPN) is that your ISP can see your traffic if you don't use bittorrent encryption, and can tell that its Bittorrent traffic (but not what the content is) more clearly than they can if you are passing your traffic through a tunnel (which they can still identify, but less explicitly).