top | item 33637192

Migrating 1 terabyte of files from OneDrive to Nextcloud

122 points| herrherrmann | 3 years ago |herrherrmann.net | reply

141 comments

order
[+] prmph|3 years ago|reply
I once lost some important files stored on OneDrive. These were pretty marge media files. There were downloaded as a zip file, apparently successfully. Then, I deleted those files from OneDrive.

Imagine my surprise some time later when I went to access those files. Upon expanding the zip file, I saw that the media files in question had been replaced by some text files, with their content indicating that the download of those media files failed, and instructing me to try to download them again.

What the heck, so I could download a thousand files, which realistically I can't go through one-by-one to check if any have been replaced by a text file, and OndeDrive does not alert me to the fact that some files could not be downloaded.

I already had issues with the slowness and clunkiness of OneDrive, but this was the last straw. I'm migrating all my data from there.

[+] nh2|3 years ago|reply
I can confirm this 100%.

We have multiple customers that send us data via OneDrive, and for all of them multi-file downloads do not work.

Microsoft's ZIP-Streamer to download multiple files simply does not work.

The text files contain exceptions with contents such as:

    PK^C^D^T^@^H^H^@^@<80>L^PS^@^@^@^@^@^@^@^@^@^@^@^@9^@^@^@__myfile.png_Error.txtThis file cannot be downloaded. 
    ExceptionType: ZipMeTAException. 
    CorrelationId: 08132d8f-77b7-4c75-a66b-345e8a15c340, 
    UTC DateTime: 8/16/2021 9:36:53 AMPK^G^H<FB>%Mw<A4>^@^@^@<A4>^@^@^@PK^C^D^T^@^H^H^@^@<80>L^PS^@^@^@^@^@^@^@^@^@^@^@^@-^@^@^@myfile.png<89>PNG
This is nondeterministic per file, so you have to either manually click every single file in a folder to download everything to avoid ZIP streaming (you apparently cannot use any API to download files that were just shared via link and no other auth), or repeatedly "download the ZIP to convergence" -- which is exactly as silly as it sounds.
[+] sebazzz|3 years ago|reply
Since Microsoft has been outsourcing development and QA for some of their consumer products quality has gone downhill.

The picture browser is barely usable. Keeps scrolling back to the top. The iOS file provider sometimes doesn't respond. Downloading 500 photos locally on an iPad results in a unresponding app.

[+] chii|3 years ago|reply
> the media files in question had been replaced by some text files

this is why a download program should verify with a hash (which should be embedded into the name of the file, or some kind of tracking metadata).

[+] neamar|3 years ago|reply
My experience with Nextcloud was underwhelming. Regular crashes, very slow sync if you have thousands of files, full reupload on any change and ignored files that do not get synced with no warnings. I had made five accounts for my family and each of them got a separate blocking bug. The worst part was the syncing of shared folders.

I ended up using Seafile, which is mostly open source and has beek rock solid for the past two years. I'm not looking back!

[+] tomaskafka|3 years ago|reply
+1 here. Nextcloud is a pile of PHP scripts, while Seafile splits and diffs individual file blocks, and absolutely flies at 'whatever the lower of of your network and drive speed is', handling any file sizes you can throw at it.

On the other hand, Seafile is also a binary distributed by Chinese team, and if I was a Chinese secret service, I'd build in all the backdoors I can, which I guess already happened.

[+] preya2k|3 years ago|reply
This! He says OneDrive is buggy, but that’s nothing compared to Nextcloud. He’s in for a big disappointment.

Seafile is a much better choice if you’re only looking for FileSync (without the whole App Store thing).

Another new and already better and faster contender: ownCloud OCIS, which is a complete Go rewrite of ownCloud and is already very fast and efficient and works like a charm for file sharing.

[+] bayindirh|3 years ago|reply
As a sysadmin managing Nextcloud on an on premise VPS, we have none of the problems you mentioned. We have 20+ accounts, tons of shared files, and whatnot.

Just upgraded to Nextcloud 25 (literally 20 minutes ago), and no problems whatsoever.

I deployed it on a virtualized server which is not extremely powerful, and everybody seems to enjoy the productivity boost it brought into the team.

[+] kelnos|3 years ago|reply
Surprising that your and the OP's experiences are so different. Based on the description, I think it's safe to assume the OP also has thousands of files, some of them very large.

I wonder if the issue is with Nextcloud itself, or with the particular hosting provider.

[+] Gigachad|3 years ago|reply
As polished as the UI looks. You can’t cover up the fact it’s a PHP turd under the hood.
[+] Octabrain|3 years ago|reply
Not relate with the post but a rant against OneDrive a couple of weeks ago came to my mind:

My partner uses Windows 11. She works and also studies in college. In both places they use Office360 so, she has her laptop set up with her Microsoft accounts and switches between college one and work when needs it. The other day, she came for help asking why the college OneDrive was getting full when she only has documents. Turns out that the fucking OneDrive decided to synchronize the whole user personal folder (where she has very personal stuff) into the college's OneDrive. Surprisingly, disabling such a wrong behavior was not straight forward as I expected. Microsoft has managed to make it ambiguous enough for having to spend a fair amount of time on such a simple thing.

I don't know it's just another example of them trying to shovel down your throat their cloud services or just some obscure plan to simply have access to all your very personal data (or both).

[+] pixel16|3 years ago|reply
This sounds like College IT is using Known folder redirection to have files backed up seamlessly to Onedrive.

Source: Msft Employee

[+] ubermonkey|3 years ago|reply
I remain baffled at how presumptuous Microsoft is about OneDrive and its behavior.
[+] vondur|3 years ago|reply
One of the big issues we see with Office365 in the browser are people working with two different Office365 accounts in the same browser, which causes a lot of issues. Known folder redirection, which is the ability for us to turn on automatic syncing of the Desktop, Documents and Pictures folders on Windows machines has been pretty helpful for us. So many people dump everything on their desktop and if the computer dies or files have been accidentally deleted it saves them. Known folder redirect is now rolling out to Mac clients, which is more of a pain, as some of our users sign into iCloud and sync those same folders.
[+] trinix912|3 years ago|reply
I know way too many people whom this has happened to. High school accounts, college accounts, even work accounts.

Even when it's a personal account, that feature messes up the C:\Users\username folder layout so then some apps save things into the non-OneDrive desktop folder and there's no way for an ordinary user to find them. Not to mention syncing broken program shortcuts (.lnk files) across devices and so on. This whole synchronization feature is a sysadmin nightmare slapped on top of a filesystem that wasn't designed to support it.

[+] rbut|3 years ago|reply
Our business migrated from OneDrive for business to a self-hosted Seafile [1] setup (we run it in a docker).

Seafile has a syncing client, and a drive client (Seadrive) for viewing files as if it was a shared drive.

Since moving to Seadrive I have not had a single support issue, nor heard of a single issue syncing, whereas I'd have them weekly with OneDrive to the point we'd have to delete their local files and re-setup OneDrive from scratch.

OneDrive truly is an abomination built on top of archaic Sharepoint.

[1] https://www.seafile.com/en/home/

[+] shellac|3 years ago|reply
> OneDrive truly is an abomination built on top of archaic Sharepoint.

OneDrive _for business_ is built on Sharepoint. Onedrive ... erm ... not for business (?) is an entirely separate product, and they don't interoperate.

Which is another baffling decision by Microsoft branding, since for regular users they make no distinction. It only becomes apparent when you discover you can't work with people outside the organisation.

[+] tacker2000|3 years ago|reply
Ive been using seafile for over 5 yrs in my business and it has so far worked flawlessly!

There are only very rare hiccups with binary files such as excel sheets or PSD files, where seafile will create a “SFConflict” file. Havent figured out how to fix that het.

[+] clnq|3 years ago|reply
I think the author misrepresented the NAS option a bit. I have the same Sinology DS220j and from my experience, you definitely do not need to constantly install/manage software, configure network access and renew certificates. Yes, these tasks pop up every now and then, but I don't think you can say you're "very busy" with something that takes 1 hour/year.

There are also significant benefits of running your own NAS - no big tech company has your documents, and they can't lock you out on a whim (such as if your Google or Microsoft account gets suspended for sometimes unfair reasons). The access speed is much higher with the data on LAN, and therefore there's little need to keep the data duplicated on your machine. And you have offline access. Security options are generally better than with most cloud providers - you can configure the firewall to deny access to anyone but whitelisted IPs, or do the many other things you can do if you own your firewall.

Finally, it's not true to say that all of your files would be gone on a NAS if something happened to the server. This goes back to the saying about data copies "two is one and one is none". You can set up backups to another NAS overnight, or use one of the cheap cloud storage services not meant for random access like Backblaze B2. Of course, the NAS itself is vulnerable physically - if it gets caught in a fire, stolen, or flooded for example.

[+] herrherrmann|3 years ago|reply
Hey, author here!

I agree that objectively, it’s not much work to maintain the Synology NAS once everything is correctly set up. But I personally just got tired after attempting to configure the network access alone for the nth time. My goal was to make the NAS accessible remotely (i.e. from outside my home network) and I felt like that was already a big task (comparing different DynDNS providers, port forwarding, asking my internet provider for a static IP address). And the recurring tasks (mostly renewing certificates for the , as far as I remember) would annoy me every year or so.

And I agree with the advantages you listed about a NAS setup! That’s why I tried one out, after all. I did run into some scary security situations (once the DynDNS was set up, someone was immediately trying to log in via brute force), but nothing too major, fortunately. And I never got the network setup right (e.g. to use the local network speed when I’m home and still access my files the same way when I’m away via the internet). So, these were only theoretical advantages that I could never really get to myself (I’m sure other people figured it out, but I didn’t exactly find good documentation on these things online either).

And regarding your last point, I also generally agree! Although it feels silly to me to have a NAS at home and then still do regular backups via another storage/backup service elsewhere. I wouldn’t want to back up 1+ TB of files every day or even week, although there might be diffing solutions for that.

Anyway, I hope this answer makes it clearer why I decided against a (Synology) NAS in the end. I’m sure other peeps figured it out and are happy with their Synology setup, but for my purposes I’ve never really gotten there.

[+] iansinnott|3 years ago|reply
Nextcloud on Hetzner has been great in my experience. Running for a few months now as a Dropbox replacement and very happy. Mobile app, mac app, it all seems to work exactly as expected.

My only gripe is that they do not allow Nextcloud external storage[0] but that's understandable given what they're selling.

[0]: https://docs.nextcloud.com/server/latest/admin_manual/config...

[+] herrherrmann|3 years ago|reply
Good to hear that you haven’t run into any issues for a few months, at least! Some peeps warned me that Nextcloud is also quite buggy, but I didn’t run into any issues yet myself.
[+] ilyt|3 years ago|reply
I used syncthing[1] for that althought use case is a bit different - syncthing is mostly to sync the data between many devices (p2p) without option to have partial sync on a directory; you can have more than one synced directories but "one big directory" will be "one big directory" on every machine.

So I have my phone pics synced to my NAS and PC, my blog's markdown files between server and my computers, another dir for some music etc. There is also option for simple versioning in case you need to dig out the old files but it's not integrated with OS as it is "just a directory".

Disadvantages is that you need at least one other device running to sync as it is P2P (with some community-ran server to forward if you're NATed out of sight) but other than that, literally zero problems.

[1] https://syncthing.net/

[+] solstice|3 years ago|reply
> no partial sync

That's not really true. You can specify which files you don't want synced. So, on my phone i can say "don't send me files that match the following pattern" https://docs.syncthing.net/users/ignoring.html

So in your example you could make a subfolder "big files" and exclude that subfolder from being synchronized to your mobile device.

I also love syncthing. I've got it set up to transfer my phone's pictures to my PC where I regularly sort through them and weed them out. And I've got a shared folder that's replicated across all my devices that contains my Obsidian vault, my KeePass file and other useful things

[+] fevangelou|3 years ago|reply
As a current user of Dropbox, Google Drive & OneDrive I can definitely say that Dropbox beats the other two (and probably any other similar service) in both network transfers (up/down) and indexing time (especially if you consider LAN sync too for when mirroring data between devices). The level of file/folder movement you can do in Dropbox while it's sync'ing data is unparalleled. If you try doing the same e.g. with Google Drive, you will either end up with duplicate files or even lose data overall.

But we all know that this probably comes at a potential cost, e.g. privacy related concerns (anyone remember the controversial Dropbox board appointment of C. Rice back in the day?), services getting hacked etc.

So knowing the above and knowing that with Nextcloud, Owncloud, Syncthing etc. you fully control your cloud backup, any (perhaps) lower performance takes a step back for sure. But then again, if you're hosting on a server that's geographically close to you & on a solid datacenter with great peering (e.g. Hetzner, Scaleway/Online.net or OVH for Europeans), performance can be on par or perhaps even better compared to "established" backup cloud services.

[+] psadauskas|3 years ago|reply
Several years ago, I had a personal Dropbox account with lots of stuff in it. I joined a company that also had a corporate shared Dropbox. Since I couldn't sync multiple Dropbox accounts with the same client on a single computer, I joined my personal Dropbox account to the company one.

Later, when I left that job, the company removed me from the corporate Dropbox account. Dropbox removed all the corporate files from my local machine, and also all my personal ones. And deleted my personal files from my personal dropbox account, too. Luckily, I had another backup I was able to restore.

Now, I never link personal and corporate accounts of any kind, and stopped using Dropbox once they stopped supporting Linux.

[+] leipert|3 years ago|reply
Personally, I really love Syncthing https://syncthing.net/, if you do not need an iOS client.

I have it set up on all my families windows machines, my personal machines and between two NAS. One of the NAS does nightly backups to backblaze.

[+] bayindirh|3 years ago|reply
For using Syncthing on iOS, there's Moebius sync, which encapsulates the syncthing client as a one-time paid application.

Works pretty flawless.

[+] pizza234|3 years ago|reply
Owncloud/Nextcloud... beware. It's significantly worse than the other services.

I've tried really hard to make it work (including self-hosting), but it's just poor software, for a few reasons.

Worst problem, which I haven't experienced with the other services, is that if there is a connection problem (this may happen routinely, if servers have scheduled downtime), files may go out of sync. Result is that, ridiculously, one may get conflicts even if they're using a single client in total. There was an issue opened, which has been ignored and autoclosed for a while.

Second, lack of centralization causes a bit of a mess. One never knows which version the servers ship (although one may not care); hosts may be less realiable, and finally, hosts may change frequently (I've see a few changes in the list of free servers) so one is never sure how long the service will be offered.

Also, the configuration shipped with the Nginx version was not great. Overall, I think that the software "isn't great", and I don't trust it for personal data.

[+] kornhole|3 years ago|reply
If comparing on costs, hosting it on home HW is by far the best way to go. I run Rsync and keep the backup in my shed so it is off-site.

Nextcloud provides way more than data backups. It is best in class for personal internet services. My family and friends use my server for syncing their contacts, calendars, tasks, kanban, music, RSS, bookmarks, passwords, notes, and finances to all their devices. Ours is also setup with Talk for voice and video conferencing, polls, forms, phone tracking, CMS, mail, maps, activity pub social media, and there are a ton of other possible apps. It is perfect solution to avoid Goopple.

[+] alvarezbjm-hn|3 years ago|reply
Why didn't you stay with pcloud, again? You mentioned encryption was uncomfortable, but in the end you selected nextcloud which doesn't have reliable encryption.

*I am a pcloud user since 5 years ago

[+] tmikaeld|3 years ago|reply
I’ve used pcloud for 7 years, it’s been very stable and i even use it for backups (700GB daily).

It’s the only solution I’ve been able to send at consistent 1Gbps speeds to (except when it’s many tiny files).

About encryption, you unlock it once per restart and it stays open. It’s not like you have to enter password every time you move a file..

(For the backups i use rClone with encryption)

[+] herrherrmann|3 years ago|reply
You’re right, pCloud is not a bad alternative! I remember being annoyed by the password prompt for the encryption feature, but another user here wrote that they only need it once after every restart, which seems fine to me now. Back when I tried it for the first time (2 years ago) I decided against it because the OneDrive migration was not fully working (many files were skipped and it wasn’t transparent which ones or what was the reason), and the support couldn’t help me figure out any viable solution to migrate all my files. Nowadays with the NAS the migration would have been easier, I suppose.
[+] vxNsr|3 years ago|reply
Aside from Dropbox do any of the mentioned providers offer auto save functionality for docs and spreadsheets? For me that’s really the killer feature (that and the fact that I paid $100 for 7 years of service using MS’s employee discount offering for everyone during the pandemic).

Every time people talk about self hosting or using a different cloud provider my biggest concern is auto save. And I haven’t heard anyone address it.

[+] gurchik|3 years ago|reply
I was looking into how I could set up Hetzner Storage Box as a storage device for Nextcloud. Somehow I didn’t even realize that Hetzner offers their own hosted Nextcloud instances (Storage Share).
[+] fundatus|3 years ago|reply
In case you still want to build it yourself: you can use sshfs to mount a Storage box on your Hetzner server instance.
[+] bfung|3 years ago|reply
iCloud+ 2TB: 9.99 €

does the macOS and iOS backup. Click buy and pretty much done. Also have my parents data backed up with family sharing, I don’t have to be family IT support!

If you have other devices… Tim Cook recommends replacing those with Apple devices /sarcasm

Only requirement not met is not having a plan that supports more than 2T.

Yes, Apple owns my life. But happy about it for now, good value.

[+] qiller|3 years ago|reply
I’m happy with it in general, but it feels like iCloud has a higher sync latency than some other services like OneDrive. And because it is a black box, there is no way to force sync it. I would drop a file for example to sync up to my old MacBook that acts as a CNC controller, walk over and OneDrive would reliably be there. iCloud however…
[+] piskerpan|3 years ago|reply
iCloud is good stuff, at least for ease of use. I use it to store my photos and passwords. Photos are then automatically synced to my Mac, which is then backed up to Time Machine. Same goes for the iCloud Keychain.

You could say I’m 100% on Apple working decently, but with 3 copies I feel relatively safe.

I wish there was a third party tool to occasionally verify the whole library to ensure photos don’t rot or just disappear.

[+] herrherrmann|3 years ago|reply
Ah, you reminded me that I did actually try out iCloud as well! I was unhappy with how the sync worked on macOS – basically lots of magic in the background without any idea what is happening and how long it might take. And you couldn’t even turn off the laptop in between sync attempts, as far as I remember? And their online services are quite far behind the competition as well (no proper folder sharing without requiring the receiver to have an Apple account, for example).

But yeah, I’m still keeping the lowest-tier iCloud subscription to sync my photos via Apple Photos, which is quite nice to have.

[+] SXX|3 years ago|reply
If only iCloud had E2E encryption ...
[+] throwaway0x7E6|3 years ago|reply
>good value

one terabyte of HDD space ≈ $12

[+] JZL003|3 years ago|reply
What about backblaze b2 with rclone, I'm sure there's a front end somewhere and with the clousflare cdn it's no egress cost. Or clousflares storage amounts
[+] teo_zero|3 years ago|reply
So this article is about looking for a perfect solution, not finding it, and finally compromising on some of them, like end-to-end encryption, ease of migration...?
[+] dspillett|3 years ago|reply
> if anything happened to my trusty server, all my files would potentially be gone

This is what a good backup solution is for.

While Hetzner's product and other hosted NextCloud instances come with some protection baked in I would still strongly suggest an off-service backup rather than keeping all your eggs in one basket.

(I'm assuming here that NextCloud is being used as the primary storage, rather than a backup location for other storage, if everything else is elsewhere already then the need for an extra full backup solution is reduced)

[+] herrherrmann|3 years ago|reply
Interesting! So, you’d still back up your full cloud storage elsewhere (Backblaze has been mentioned by other users), even though your cloud storage provider does backups themselves (e.g. Hetzner claims to do several backups every day)?
[+] thejosh|3 years ago|reply
I backup (encrypted) to multiple places, including to Google Drive using a Turkish billing account on another google account. 2tb works out to about $30USD per year.

In terms of actual computer backups, thanks to btrfs+btrbk I can incremental backup every hour over LAN to my server and also to my attached USB drive. With compression/snapshots this is pretty small.

[+] gzer0|3 years ago|reply
Thanks for the tip about the Turkish 2tb for $30/yr. Very good deal. Just did this. :)
[+] kcartlidge|3 years ago|reply
Re the article and PCloud, the way the crypto folder works is that you get a mapped PCloud drive similar to OneDrive, DropBox, etc., and within there you get a "Crypto Folder". When you try and access that, it's empty. You then unlock it. However the article is wrong in that you are not prompted every time. You are prompted until it locks again, and you are in control of whether it locks automatically or not.

I've used PCloud for years now. I only unlock my Crypto when I need files from inside it, keep it unlocked and accessible for as long as I need, then lock it again. It's actually a really good way of working. Less valuable files are always available in the main mapped drive so working with them is easy, but the more secure option is available too.

It also supports Linux properly, and whilst not as fast as DropBox is certainly faster than Google Drive and OneDrive. And it allows multiple synced folders in addition to the mapped drive.

I moved on from Google Drive when I found that some of my documents, which I'd specifically requested be mirrored locally, were found to contain http links to the online version rather than an actual local copy. Meaning some of my local backups contained useless stub files. And the total inability of the web UI to give me simple folder size details was annoying - almost like they were encouraging me to keep filling the space I had.

I moved on from OneDrive when it wouldn't let me store files it thought might be dangerous (including `.js`, though I suspect they fixed that years ago). I also stored some older PHP code files in there (old enough to predate my git usage). When I looked at them they had an extra line of text added to the top of each file. I can't remember if it was a hashbang, comment, or text, as it was too long ago. I just remember my shock that they would change my uploaded files automatically.

Conclusion: after trying all those the article did (except, oddly enough given the article, NextCloud) I found PCloud to be ideal. For me.