0

How copy messages between two IMAP servers and skip messages that already exist in the target mailbox?

I am trying to implement copying using imapfilter but the issue is that each run I get a number of duplicates - this happens when a filter is based on all messages but when I use unseen filter.

What I am trying to achieve:

  1. The source server(s) is the production server I use for daily emails etc.
  2. The target server is a backup server that should store all emails
  3. The assumption is that if a message was deleted on the source server it shall still be kept on the target.
  4. I want the imapfilter job to be run every day, maybe often (schewduled as crone job)
  5. Incremental message selection based on un_seen filter is an option but when if an email is marked as read before the backup job reaches it then such message would not be copied - I guess all messages should be analyzed

I have the following config.lua script.

---------------
--  Options  --
---------------

options.timeout = 120
options.subscribe = true
options.create = true
options.charset = 'UTF-8'

target_folder = 'MailArchive/'

source1 = IMAP {
    server = 'imap.mail.org',
    username = '[email protected]',
    password = password1,
    ssl = 'tls1.3',
}

source2 = IMAP {
    server = 'imap.mail.com',
    username = '[email protected]',
    password = password2,
    ssl = 'tls1.3',
}

target = IMAP {
    server = 'localhost',
    username = 'user',
    password = password3,
    port = 143,
}

----------------------
-- Backup procedure --
----------------------

sources = { source1, source2 }

local function copy_imap_folder(_src_acc, _src_box, _trg_acc, _trg_box)
    if string.upper(_src_box) ~= 'TRASH' 
    and string.upper(_src_box) ~= 'SPAM' 
    and string.upper(_src_box) ~= 'JUNK' 
    and string.upper(_src_box) ~= 'INFECTED ITEMS' then
    print('Processing mailbox: ' ..  _src_box)
    print('Copying to folder: ' .. _trg_box)
--  local newemails = _src_acc[_src_box]:is_unseen() -- not used
    local newemails = _src_acc[_src_box]:select_all()
    newemails:copy_messages(_trg_acc[_trg_box])
    newemails = _trg_acc[_trg_box]:is_unseen()
    newemails:mark_seen()
    end
end

for _, src in ipairs(sources) do
    print('Processing account: ' .. src._account.username .. '@' .. src._account.server)
    mailboxes, folders = src:list_all()
    for _, mbox in ipairs(mailboxes) do
    targetmailbox = target_folder .. mbox
    copy_imap_folder(src, mbox, target, targetmailbox)
    end        
end

Getting to the point, my questions are as follows

  1. How to copy without duplicates - should first read message_id from source, then from target, do a set difference, and copy only the messagfes not existing mails? Is there some out of the box function for this or I should implement all this by myself in lua?

  2. Why do I get duplicates when using all_message filter but not when un_seen filter? Does it has enything to do with the number of messages? Or unread emails are handled defferently?

Best regards, Sebastian

1 Answer 1

0

What I understood from your requirement is: you want to copy only UNSEEN messages to target folder avoiding duplicates.

For this you need to work on UID of messages in source server/folder. Because UID of message is unique in particular folder. Also, you need to store the UID "say, N" of last message from source folder copied to target folder. In next copy cycle you need to copy all the source folder messages with UID greater than "N".

In this copy process, UIDVALIDITY associated with folder should also be used. More details can be given if required.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .