speed up file transfers on NTFS

 Private URL for my reference - https://chatgpt.com/c/6a814e36-5198-83ee-8072-7555b22d45d7

Related posts - set up file sharing on Mac ,  ntfs write on MacOS

When copying files from a USB SSD on a Mac to a USB HDD on Linux over the network, the files initially start at 9.8 MBps since it is a 100 Mbps network, but later the transfer speed drops to 4.5 MBps and sometimes to 2.5MBps.

ChatGPT asked me the model of the hard disk, and said, "That particular 4 TB Toshiba Canvio Advance is a 5400 RPM 2.5-inch portable HDD powered entirely from USB. A healthy 5400 RPM drive should still sustain roughly 80–120 MB/s sequential writes, which is far above your 100 Mbps network limit (~10 MB/s). So under normal circumstances the HDD alone should not reduce a transfer from 9.8 MB/s to 2.5 MB/s." And suggested

iostat -xm 1

"If you see the HDD at 100% utilization (%util) while the network transfer has slowed, you've found the bottleneck."

But that was not the case. In the next iteration, the NTFS filesystem was narrowed down as the culprit - mount showed 

/dev/sdc2 on /media/sssvv/TOSHIBA EXT type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096,uhelper=udisks2)

and ChatGPT said, "type fuseblk - This means the drive is not using the newer kernel NTFS3 driver. It is mounted through FUSE, which on Mint 22.1 typically means ntfs-3g (or a compatible FUSE-based NTFS implementation)."

To use the kernel ntfs3 driver instead,

uname -r
cat /proc/filesystems | grep ntfs
modinfo ntfs3

which showed the driver was present, but not loaded. So,

sudo modprobe ntfs3
# and to verify,
cat /proc/filesystems | grep ntfs

sudo mkdir -p /mnt/ntfs-test
sudo mount -t ntfs3 /dev/sdc2 /mnt/ntfs-test

That worked (read-only).

To find why ntfs3 was not loaded,

apt policy udisks2
udisks2:
  Installed: 2.10.1-6ubuntu1.3

"With udisks2 2.10.1-6ubuntu1.3, the built-in defaults already specify: ntfs3, ntfs - meaning udisks2 should try ntfs3 first and only fall back to ntfs (which typically ends up as ntfs-3g/FUSE)."

Verified with

grep -R ntfs_drivers /etc/udisks2 /usr/share/udisks2 2>/dev/null
/etc/udisks2/mount_options.conf:ntfs_drivers=ntfs3,ntfs
/etc/udisks2/mount_options.conf.example:# ntfs_drivers=ntfs3,ntfs

Then I found I had these lines in /etc/fstab - 

# External NTFS disk

UUID=fc6965f0-d088-48df-a082-b413ceb02a1c  /mnt/toshiba  ntfs-3g  rw,uid=33,gid=33,umask=022,allow_other,nofail,x-systemd.automount  0  0

So maybe that was the culprit. ChatGPT asked me to change it to

UUID=fc6965f0-d088-48df-a082-b413ceb02a1c  /mnt/toshiba  ntfs3  rw,uid=33,gid=33,umask=022,nofail,x-systemd.automount  0  0

and noted that

• allow_other is primarily a FUSE option. It's not needed with ntfs3 and may generate warnings if left in place.
  
  
• uid=33,gid=33 means all files will appear owned by user/group 33 (www-data on Ubuntu/Mint). Keep it if that's intentional.
  
  
• If you want your desktop user to own the files, use your UID/GID instead (typically 1000:1000).

So I did that, and found that mount now shows

/dev/sdc2 on /media/sssvv/TOSHIBA EXT type ntfs3 (rw,nosuid,nodev,relatime,uid=1000,gid=1000,iocharset=utf8,uhelper=udisks2)

So the problem was probably due to the fstab entry, which I edited.

Now I get 9.8 MB/s for large files over the network - limited by the 100 Mbps network. Tested with a 2.2 GB file.

Even faster than this would be: copy from the Mac's internal drive to a Mac-formatted external drive, then eject the external drive, mount it on Linux (mounted automatically) and copy from that Mac-formatted external drive to the NTFS formatted external drive mounted on Linux - 70+ MB/s for 2.5 GB.