Selecting (and sizing) the correct file system is more important than anything else, not only for security but for tons of other reasons people do not usually recognize. Without a file system all processing would go to null.
Very well put response by Olli, and the OP is much dated, but file systems are my pet peeve I could not stay away. superuser.com is not something I visited before, I am not an admin, but I signed up and I am going to visit more.
Things changed a lot since 2011, but even back then I formatted USB cards FAT, and used USB drives to carry 4Gb+ files around. The reason of course was compatibility not security (so much for S in SD, but I use passwords on my 7z's), and I never really carried anything bigger than a CD ISO, they were mostly for SQL scripts and daily-hourly diffs of already encrypted database snapshots squeezed to near death by 7-Zip.
These days I wear any SD out faster than anyone I know. I have a USB stick in some production machines at my employer for hourly automated backup, formatted FAT. I keep an eye on them every day though and - you guessed - back them up religiously by hand (they are off-line secured building ITAR stuff). SSD leveled some of the playing field but I still do not trust them as much as regular HD, and SD is worse than optical. They go bad in an instant and the loss is total.
Any file system which invites the host OS to write randomly to it (NTFS, Recycle Bin) is bad news for an SD. Also, unmounting it helps a lot, no OS is going try accessing unmounted storage, so any file system will do as long as the SD includes a script to unmout itself (one of the standard files on every SD on mine).
Reading an SD is still slow today, so I would recommend something like disk dump (dd) to grab the entire image when mirroring instead of file-by file. dd also let you know when there is something wrong, so your file manager won't go kaboom.
Of course, if your primary purpose is to extend the life of some penny-stock you are going about your business the wrong way. I do what I do not for extending the life of an SD but to keep it from going bad when I am not watching, and there is the difference.
I avoid ext4 or any journaling FS on SD because I do not care when they go bad writing to them, but it sure hurts when a day or so later I cannot read them!