talking from the exist-db side of things, I m curious to hear about your problems with large .xar packages and the package-manager. I have >5gb .xars and do not experience any of the problems that you describe. Copying and indexing 20gb of unzipped xml is not instantaneous, but one wouldn’t expect it to be so.
I have some ideas on where the problems with your xars might be though. If anybody would be willing to share one of the problematic .xars i would be happy to take a look.
On a general note, if you encounter issues, please report them on our issue trackers on GitHub. As an open-source project, we can’t promise when a fix will make it into the next release, but we surely can’t fix what we don’t know about. We are however currently working on the package-manager code so now is a good time to open those tickets, or better yet open a PR with your code fixes.
P.S.: I have noticed that error logging of the package-manager isn’t working 100% as it is supposed to, but so far I m lacking a way to consistently reproduce the problem.