From the perspective here, it appears to be mostly down to ignorance and prejudice (as others have stated). There are those who are ignorant of what it is and does, believing that they already know and others have prejudiced views of Microsoft and everything they do. For the more general public, I imagine it's more likely because there is just no compelling reason.
As it stands right now, Silverlight is a god-send for those who want to develop Rich Internet Applications (RIA) without buying expensive tools or learning a new language. Silverlight uses known .NET concepts to the extent that source code can quite often be shared between desktop applications and Silverlight applications. It is this aspect of Silverlight that is compelling and it will only be through developers taking advantage of this by creating must-have applications that consumers will be convinced to download and use it.
Of course, developers must have an audience and that's where the cross-platform shortcomings have a big impact. However, I believe those shortcomings will be more effectively addressed soon, making Silverlight more viable, and as Microsoft partners with sites like Netflix, Silverlight should become more widely used. I believe it is currently installed on around 30% of systems, which is not close to Flash at 95% but it's certainly nothing to be sniffed at.
With all that said, as some have pointed out, the days of browser plug-ins like Flash and Silverlight may well be numbered except in some domain-specific areas as we move to a more open set of standards for defining rich content. Hopefully, the likes of Flash and Silverlight will eventually become programming models for such open standards, allowing us to use the tools that are familiar to produce the content we want without worrying about which user has which system, browser, and plug-in.
For consumers, the reason to download Silverlight is to access the Silverlight application that you otherwise couldn't. If you don't need that application, you don't need Silverlight. Simple as that.