I think you'll like rsyncrypto.
Use rsyncrypto to encrypt files from your plaintext directory to your encrypted directory, and decrypt files from your encrypted directory and your plaintext directory, using keys that you keep locally.
Use rsync to synchronize between your encrypted directory and the remote host.
The rsyncrypto implementation you can download now from Sourceforge not only handles changes in bytes, but also insertions and deletions.
With rsyncrypto, all encryption keys never leave the local computer.
"The remote server should preferably not even know the directory structure"
In that case, you'll want to use the
That makes each encrypted file name is a random string of characters,
and by default all mangled file names are stored in a single directory.
The true file names and folder names are stored in the (encrypted) file named "filemap".
Related: "Is there an encrypted version control system?"