34

I’m a front-end developer, and I keep my front-end projects in OneDrive. Due to how Node.js organizes dependencies, each project has a node_modules directory with tens of megabytes of tiny files which take forever to sync.

What can I do to avoid synchronizing node_modules directories while keeping the projects in OneDrive?

3

10 Answers 10

14

I know this is an older question, but I ran into the same issue. The solution I came across was to first create an empty node_modules folder and then sync that with the cloud.

Once the empty folder was synced, I then went into

OneDrive settings → Choose folders

and deselected the node_modules folder.

Then when you run npm install in the project root folder, OneDrive will detect a conflict with the contents of the folder in the cloud. It will not sync.

Image of Sync Error

The downside of this approach is that OneDrive produces sync errors, but it seemed to work well for my needs.

3
  • 3
    As of March 2019 this is still the best answer that I've found. It isn't ideal but it is functional.
    – Taul
    Mar 12, 2019 at 22:04
  • 1
    As of July 2019: If you are getting sync errors about 'The file or folder already exists', go to OneDrive web UI, and remove the node_modules folder there. If it's there, OneDrive complains that they are different. If it's not in cloud, there is no difference and OD doesn't complain. Jul 24, 2019 at 12:47
  • 1
    @TomášKafka Then the folder will start syncing again after a restart of OneDrive.
    – renklus
    Apr 10, 2020 at 0:32
12

Background

OneDrive doesn’t allow to exclude directories by name, and that’s something that won’t change soon:

[OneDrive’s representative response on a UserVoice request:] Not right now

Some people recommend to uncheck directories you don’t want to sync in OneDrive settings, but that became impossible with OneDrive Files on-demand.

Solution

However, you can solve this by tweaking things on the Node.js’ side. Make the node_modules directory a file symlink to a different place:

# Open any place outside OneDrive
cd D:\node_dependencies
# Make a symlink target. After linking, node_modules for your project 
# will be installed here
mkdir node_modules_for_my_project

# Open the project directory
cd <the project directory>
# Make a *file* link from node_modules to the newly created directory.
# You will need to delete existing node_modules directory if it exists
cmd /C "mklink node_modules D:\node_dependencies\node_modules_for_my_project"

The important piece here is that you create a file symlink, not directory one. OneDrive won’t recognize and sync this symlink, whereas Node.js will work with it as intended:

A screenshot showing that npm install and Node.js’s require work as intended

Tested with OneDrive v17.3.7101.1018 and OneDrive Files on-demand enabled.

Drawbacks

This is not a universal solution. A drawback of this is that Explorer, Powershell and other tools won’t recognize node_modules as a directory:

A screenshot showing that Explorer shows node_modules as an unrecognized file with a shortcut icon

However, Node.js-based code editors will read it just fine:

Screenshot of Visual Studio code with node_modules directory expanded

4
  • 1
    This does not seem to work. I'm getting Cannot find module errors when trying to launch the app (using Typescript). Dec 5, 2017 at 13:45
  • @PierrickBignet A guess: you indirectly run some native code which isn't designed to process symlinks, that's why creating symlink requires privileges.
    – InQβ
    Sep 14, 2019 at 1:41
  • Sadly, this didn't work for me. npm install regularly failed with maximum call stack size exceeded, and attempting to install to the default node_modules folder, copy all files to the symlink location, then create the symlink afterwards caused failures when attempting things like npm run start etc. Oct 14, 2020 at 20:31
  • OneDrive doesn't follow junctions so if you do mklink /J ... it will work in explorer, powershell, etc.
    – Sam Hasler
    May 30, 2023 at 16:32
5

I've found that the best solution is creating a 'local' folder outside the OneDrive folder where the project will be saved and any modules will be installed. This is the actual project.

To sync it to OneDrive I manually create a 'parallel' folder and copy only the content I want using 'RoboCopy'. You can then schedule this task.

The files & folders names/wildcards to ignore were copied from the .gitignore file

The solution is based on a few articles:

Schedule RoboCopy task and simple examples

RoboCopy documentation by Microsoft

you can also use GUI to create the .bat task file to be scheduled. GUI by microsoft & ChoEazyCopy GUI which is much better

2
  • Because of this very issue (being unable to exclude bin, obj, node_modules, and other developer-type files and folders), I don't like working within my OneDrive (or Dropbox, etc.) directories. I do a similar thing, only I use Beyond Compare scripts and the Task Scheduler.
    – kodybrown
    Mar 29, 2022 at 23:20
  • In general, this is the best strategy because as a developer you really don't your compiler and other tools getting slowed down or files/folders getting locked by background syncing. You also usually want to choose when to make a backup of your current project. I use TreeComp instead that performs very fast file incremental folder syncing and comparison (handy for cherry picking and finding unexpected file changes) and retains file attributes. Jun 12, 2022 at 14:11
1

For Typescript users out there, add node_modules to your path to avoid module resolution issues.

In tsconfig.json add :

"paths": {
  [... some paths ...],
  "*": [
    "node_modules/*"
  ]
}

Edit: Still have some issues. I can build the project but ts-node and nodemon fire Cannot find module errors. Well, this is still an issue then :(

0

After reading these proposed options, I ended up deciding to put everything outside of OneDrive and create a script with Robocopy to backup to OneDrive whenever there was a change in the projects folder.

Something like:

robocopy SOURCE DEST /mir /xd node_modules
0

Just delete the files or folders in OneDrive that you don't want to sync , they're still there in your local disk.

0
0

I use a development folder that OneDrive doesn't back up and a backup folder OneDrive does back up and copy the development folder to the backup folder with a batch file. The following is essentially an example / step-by-step of @RanST's answer.

Create a batch file.

@ECHO ON
ECHO Copy started.
robocopy "{{SOURCE}}\PROJECTS" "{{DESTINATION}}\PROJECTS" /xd node_modules .serverless /s /z /r:3 /w:10 /tbd /np /ts 
ECHO Copy completed!
PAUSE

/xd = exclude. Simply add the folder names to exclude.

/s = include subfolders exclude empty directories.

/z = retryable. /r:3 = 3 retries. /w:10 = wait 10 secs between retries.

/tbd = wait for a share names if they caused a retry error

/np = don't display progress

/ts = add a timestamp to the log

The Microsoft Documentation explains all the options.

Sync the parent.

I wanted a backup that didn't need to be updated every new project, so I chose to back up a parent folder and maintain the file structure when I created the development specific folders.

Main synced folder

/MY_WHOLE_BACKUP/.../PROJECTS/clients/client1/parentProject/subProject/root/myApp/app.js

DON'T simplify the path

/MY_LOCAL_FILES/.../PROJECTS/client1/myApp/app.js

DO keep it all

/MY_LOCAL_FILES/.../PROJECTS/clients/client1/parentProject/subProject/root/myApp/app.js

Schedule the task.

Search for and open the Task Scheduler in Windows.

Click Create Task....

Name the task.

Click the Triggers tab.

Click the New... button.

Adjust the schedule to your preference.

Click OK.

Click the Actions tab.

Click the New... button.

Browse for your batch file.

Click OK.

BOOM!

Test and configure as needed. I hope this helps.

0

In 2024, if you're still encountering issues and previous solutions haven't worked, here's an alternative method:

  1. Move your source files to a local folder that syncs with the cloud, such as C:\onedrive\mynodeproject.

  2. Select a different folder for storing your node_modules. For example, you could use C:\temp\myproject.

  3. Open a command prompt as an administrator. (Refer to additional notes if you lack admin privileges.)

  4. Create the new storage folder by executing:

    mkdir C:\temp\myproject
    
  5. Establish a junction point named "src" at this location, pointing to your source files:

    mklink /J C:\temp\myproject\src C:\onedrive\mynodeproject
    
  6. Create a symbolic link for package.json to ensure it points to your original package.json in the source directory:

    mklink C:\temp\myproject\package.json C:\onedrive\mynodeproject\package.json
    
  7. If a node_modules directory already exists, relocate its contents to C:\temp\myproject\node_modules.

  8. Attempt to install your modules using npm. For instance:

    npm install SomeLib
    
  9. To start your application, instead of using the current directory (.), specify the path to your source files. For example:

    node src/index.js
    

    or simply:

    node src
    

Please ensure to replace placeholder values like SomeLib with the actual names of the libraries you intend to install.

Note on Administrative Privileges:

Important Consideration Regarding package.json: If you choose not to sync your package.json from the source by omitting the symbolic link creation, be aware that npm will generate a new package.json file in your temporary local folder (e.g., C:\temp\myproject). This may lead to discrepancies between your project's dependencies and scripts in your source directory and those in the temporary directory. It's crucial to ensure consistency in your project setup to avoid potential issues during development and deployment.

Note on Syncing package.json: If you prefer not to have your package.json file automatically sync with every change, you can opt to only create a junction point for your project directory. This approach bypasses the need for a symbolic link to package.json, thus avoiding the necessity for administrative privileges if Developer Mode is enabled and configured correctly.

Adapting to Path Changes and Invocation Methods: Utilizing this method may necessitate adjustments to your project's paths and the way certain calls are invoked. For instance, if you are using Electron or similar frameworks, you might need to modify the package.json to correct paths or establish new commands to accommodate the altered directory structure. This approach is not a one-size-fits-all solution but rather an alternative strategy to manage your project files in the cloud. It serves as a workaround until cloud service providers offer more flexible options for excluding specific files and folders from synchronization according to user preferences.

-1

I have found that using dev containers can fix this issue if you can put the time into creating your environment.

Specifically this part.

1
  • Welcome to Super User! Please quote the essential parts of the answer from the reference link(s), as the answer can become invalid if the linked page(s) change.
    – DavidPostill
    Sep 25, 2020 at 13:12
-1

An alternative might be to use your OneDrive storage but employ a smarter cloud backup tool like rclone to perform your file synchronization.

The free MIT licensed rclone supports over 70 different storage providers including OneDrive with rich file and folder filtering rules so setting up a filter to exclude node_module folders (or any other file/folder pattern) is fairly straight forward.

Since you're in control of when your files get synced, you can safely avoid file and folder locking issues that otherwise force you to store duplicate copies of project folders. You manage your own encryption keys and can enable many other features such as file compression. You can even union together multiple local and/or cloud file systems to present as one.

rclone is supported on Windows, macOS, linux and FreeBSD although be aware of the file name limitations for OneDrive.

The rclone community has a support forum if you have any related questions.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .