I often pull out the USB stick without having to first eject USB, but so far I have not had any problems.
Would this lead to problems in the future?
Yes, it can cause problems.
If your USB drive is NTFS-formatted, Linux's ntfs-3g is usually unable to read any files you added just before unplugging it, I presume because the journal is dirty.
Further, Safely Remove provides a useful check: if any files on the drive are open (which may have your unsaved data in applications), it will fail, reminding you to save.
If write caching is enabled (which is not the default), it's worse: any data you thought you just wrote to the drive may not be actually written yet. If you pull out the drive before that happens, your data will be lost.
If you're just reading files from a USB drive, it probably won't hurt. The risk is of data loss rather than hardware damage (you're cutting power when you pull out the drive, regardless of whether you used Safely Remove). Still, it's best practice to use Safely Remove.
As long as a reading/writing operation isn't performing, pulling out the usb without ejecting it shouldn't damage anything.
But be aware that it can cause problems between operating systems. If you pull out the usb in Windows you won't be able to use it with Linux.