4

I wanted to convert my D:\ partition with all my data from NTFS to ext4 and the internet told me that I should do the following:

  1. Shrink all partitions to a minimum size
  2. Back up my data (done by cloning the partition in GParted)
  3. Format the partition to ext4 (Not done yet, but not that hard)
  4. Copy the backup to the drive
  5. Delete backup and grow all partitions

And I don't know how to do part 4. Can I really copy a whole drive like I would do for folders? Are there tools specialized in this? How can I make sure I don't miss any folders hidden by the system or symbolic links etc.? I have an Ubuntu live USB-Stick to work from if that is important.

4
  • 2
    You don’t know how to copy files?
    – Ramhound
    Nov 26, 2017 at 17:29
  • I do know how to copy files. The question is about if I can copy whole drives like I do for files or if there are special things to take care of to not break the system.
    – piegames
    Nov 26, 2017 at 20:39
  • There are tools that read each sector of a drive and outputs an image of said drive.
    – Ramhound
    Nov 26, 2017 at 20:40
  • So you're all going to do this using one single drive? Hope you have a backup. May 17, 2023 at 22:20

1 Answer 1

9

1. FreeFileSync GUI tool (Windows, Mac, Linux)

To ensure you don't miss any data, I highly highly recommend using a good file-copy tool like FreeFileSync. It is graphical, no cost, free and open source, and cross-platform (Windows, Mac, Linux). Just be sure not to screw it up and copy backwards, deleting everything! That's not hard to get right, however, since it's graphical and very user-friendly.

Just set up your folders with source on the left and destination on the right, then click the gear at the top right and choose the "Mirror" option to mirror from left to right, then press the "Compare" button at the top left. Once it determines what it needs to copy, it will pop up an info screen showing what it's going to do (how many files it will create, overwrite, or delete, etc.). Ensure you see nothing next to the trash can (indicating what is to be deleted), or that you see expected files to be deleted, as marked with the trash can and a minus sign, and then press the "Synchronize" button to begin. Voila! Easy to use. Here's a screenshot: enter image description here

2. rsync Command-line tool (Linux, Windows with Cygwin) (AKA: "How to use rsync"):

If you are doing the copy from Linux, you may also use FreeFileSync of course, but if your'e a command-line guru you might like rsync better. It's a command-line-only tool, and harder to use, but still very powerful, albeit not quite as powerful as FreeFileSync since it doesn't handle two-way syncing very well. Here's a few of my favorite rsync commands:

  1. COPY from source to destination, leaving anything in the destination that isn't conflicting still intact:

     sudo rsync -rah --stats --info=progress2 '/path/to/source' '/path/to/destination'
    

    Optionally, add --verbose, or just -v (or -rahv in my case) to also get a full list of exactly what will be copied.

    Optionally, add --dry-run to show what would happen but NOT ACTUALLY DO IT YET.

    Optionally, you may also log the entire list of files to be copied to a stdout.log file like this, and any errors to a stderr.log file like this. Here I remove the progress indicator by removing --info=progress2, just so the log files don't get polluted with progress info everywhere:

     (sudo rsync -rah --verbose --dry-run --stats '/path/to/source' '/path/to/destination' | tee -a '/path/to/stdout.log') 3>&1 1>&2 2>&3 | tee -a '/path/to/stderr.log'
    

    Remove --dry-run from the above when ready to actually make the copy. Be sure to review the stats paragraph printout (which printed because you had --stats) at the end of the stdout.log file before running the script for real, to make sure it all looks good.

    Here's an example of a --stats output obtained from doing the MIRROR option below. Notice it is deleting some files. Pay careful attention to that, as deleting files can't be undone (at least not without special software recovery tools). Just keep in mind it is permanently deleting them, NOT sending them to some sort of recycle bin or something, so watch out!

    Number of files: 102,033 (reg: 89,638, dir: 9,867, link: 2,502, special: 26)
    Number of created files: 3 (reg: 3)
    Number of deleted files: 9,861 (reg: 9,360, dir: 474, link: 27)
    Number of regular files transferred: 25
    Total file size: 8.12G bytes
    Total transferred file size: 35.30M bytes
    Literal data: 35.30M bytes
    Matched data: 0 bytes
    File list size: 917.43K
    File list generation time: 0.007 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 38.33M
    Total bytes received: 1.02M

    sent 38.33M bytes received 1.02M bytes 7.15M bytes/sec
    total size is 8.12G speedup is 206.37

  2. To MIRROR from source to destination, which means also deleting anything in destination that is not in source (notice the "Number of deleted files" line in the stats output shown just above), add --delete --delete-excluded (even though we aren't excluding anything yet, once you learn about the --exclude=PATTERN syntax you'll also realize you want the --delete-excluded flag).

    First, the safe dry-run version:

     (sudo rsync -rah --verbose --dry-run --stats --delete --delete-excluded '/path/to/source' '/path/to/destination' | tee -a '/path/to/stdout.log') 3>&1 1>&2 2>&3 | tee -a '/path/to/stderr.log'
    

    And now the danger danger danger--could delete everything if you do source and destination backwards version (so check the stats output at the end of stdout.log before continuing):

     (sudo rsync -rah --verbose --stats --delete --delete-excluded '/path/to/source' '/path/to/destination' | tee -a '/path/to/stdout.log') 3>&1 1>&2 2>&3 | tee -a '/path/to/stderr.log'
    

    And now the version that doesn't save to any log files in case you want to just see the live progress and the stats paragraph at the end, but NOT have a log of individual files copied or deleted in stdout.log, or errors printed in stderr.log:

     sudo rsync -rah --info=progress2 --stats --delete --delete-excluded '/path/to/source' '/path/to/destination'
    
  3. And this is just the beginning of rsync. Read the manual: man rsync, or online: https://linux.die.net/man/1/rsync, and keep in mind you can mix and match many of the options I've shown above. Other really useful things to look into include setting up a source.txt file, and an exclude.txt file, to include and exclude, respectively, lists of files and paths. Do this with the option --files-from='/path/to/source.txt' and --exclude-from='/path/to/exclude.txt', respectively. This is extremely useful. Lastly, this goes really well with the --relative option. In the end, a full Bash script might look something like this:

      (sudo rsync $OPTIONS $SRC $DEST | tee -a $LOG_STDOUT) 3>&1 1>&2 2>&3 | tee -a $LOG_STDERR
    

    ...where $SRC is your source path (in this case, a relative path for all files in the source.txt file, such as simply / to make everything relative to your root directory), $DEST is your destination path, $LOG_STDOUT is your path to where you'd like to log your stdout info, $LOG_STDERR is your path to where you'd like to log your stderr info, and $OPTIONS might look something like this:

     -rah --verbose --stats --relative --info=progress2 --delete --delete-excluded --dry-run --files-from='/path/to/source.txt' --exclude-from='/path/to/exclude.txt'
    

    And, of course, as stated above, you might want to remove --verbose, --relative, --info=progress2, --delete and --delete-excluded, and --dry-run, as desired, depending on what you're doing.

Important: for exFAT and FAT filesystems, where you want to use rsync...

See also my answer here: Best settings for using rsync with FAT and exFAT filesystems

See also:

  1. [my Q&A] How to install FreeFileSync on an Ubuntu Live USB installation disk
  2. [my answer] Ask Ubuntu: How to show the transfer progress and speed when copying files with cp?
  3. [my answer] Unix & Linux: Is it possible to see cp speed and percent copied?
  4. [my answer] Ask Ubuntu: How to change permission of a drive in an external hard disk?
12
  • 1
    I just tried FreeFileSync based on this answer and it works really well for my use-cases (mirroring and synching directories with over 1 million files each.
    – Reb.Cabin
    Dec 11, 2019 at 20:48
  • 1
    -r is redundant as -a already means -rlptgoD.
    – Dagelf
    Jun 10, 2020 at 10:46
  • 1
    @Dagelf, no, that's not always true. From man rsync, under the --files-from=FILE section: "The --archive (-a) option’s behavior does not imply --recursive (-r), so specify it explicitly, if you want it." So, since my last example uses the --files-from option, and it's better to be safe than sorry, just get in the habit of explicitly declaring -r even when you use -a. Aug 19, 2020 at 9:17
  • 1
    Under normal circumstances, though, yes, you are right. man rsync says: -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X). Aug 19, 2020 at 9:20
  • 2
    rsync is such a tricky beast I have learned to just be as explicit as possible all the time. Aug 19, 2020 at 9:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .