As far as I can find out, show translucent selection rectangle off used to work in Vista for both the desktop and Windows Explorer.
However, since Windows 7 the Windows Explorer ignores this setting and always uses the filled selection rectangle, no matter its setting. This is no doubt a new bug introduced in Windows 7.
Sorry to be negative, but in my opinion the most one can do is file a bug report with Microsoft as regarding this issue.
Unfortunately, there's no submit a bug report option directly. The way one does it is to call Microsoft Support and open a support request (can also be done online). If they can reproduce your issue and verify that it's actually a bug they will file a bug report.
[EDIT]
I found a way of forcing Explorer to honor the Translucent Selection Rectangle setting that is found in the registry in the key :
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced
You should set the value of ListviewAlphaSelect
to zero (0) to disable the Translucent Selection Rectangle.
Unfortunately this change is ignored by the list-view control of Windows 7 (DirectUIHWND).
One must therefore force Windows 7 to use the old list-view control from Windows Vista.
The following article contains a registry hack which (incidentally) will accomplish this:
How to Disable Full Row Select in Explorer in Windows 7.
It links to a zip file that contains two .reg files : One effects the change and the other undoes it. I tried it inside a vanilla virtual machine and it worked for me.
I would still suggest to create a system restore point before using this hack.
show translucent selection rectangle
off, and I just noticed this! Confirmed.