Sync fails: "too many open files" (original matter addressed but bulk upload introduced another w/ similar symptoms)

[gcala]

Hi, sync seems broken on my laptop. It fails with a series of "Too many open files" messages in the client. It stops for a moment and then restarts the whole process from beginning.

I don't know if this error is related to desktop client or machine where it runs.

Any idea? Thanks

Client configuration

Client version: 2.5.1 git (rev. b37cbe)

Operating system: Arch Linux

OS language: Italian

Qt version used by client package (Linux only, see also Settings dialog): 5.12.0

Client package (From Nextcloud or distro) (Linux only): community/nextcloud-client 2.5.1-2

Server configuration

Operating system: Arch Linux

Web server: apache 2.4.37

Database: mariadb 10.1.37

PHP version: 7.3.0

Nextcloud version: 15.0.2

Logs

1. Client logfile: https://cuteworks.it/nextcloud/s/KzWbigWWwNjRk2n

2. Web server error log: no errors

3. Server logfile: no errors

[benjamin-leiding]

I have the same issue on my Linux system and it seems to be only triggered when I process larger numbers of very small files, e.g., small pictures or git-repositories. Client 2.3.3 works, so it has to be a problem that was introduced with version 2.5.x. I also tried the dev-branch for the nextcloud-client -> same issue.

Client configuration

Client version: 2.5.1
Operating system: Xubuntu 18.04 LTS
OS language: English

Server configuration
Managed Nextcloud (Owncube.com) - Version: 15.0.0

Logs
Web server error log: to my knowledge -> no errors
Server logfile: to my knowledge -> no errors
Client logfile:
20190109_1148_owncloud.log.0.gz
20190109_1200_owncloud.log.0.gz
20190109_1200_owncloud.log.1.gz
20190109_1212_owncloud.log.0.gz
20190109_1213_owncloud.log.0.gz

Let me know whether you need more info for debugging purposes!

[GieltjE]

Could it be this crashes on Windows machines? When we try to sync large numbers of new files (2k+) it just crashes after a while.

[benjamin-leiding]

Yes, I was also able to reproduce the same results on a virtual machine with Win 7 and the 2.5.1 client for Windows. Unfortunately, I cannot provide logs at the moment. But the Linux logs should already help the dev team once they take care of this issue.

@GieltjE Could you maybe also upload logfiles to make it easier for the dev team to fix our issue?

[GieltjE]

The client log just logs sync complete, a friend of mine will hopefully post the windows logs soon.

We also noticed that it happens on lots of small files, when the files get bigger it doesn't crash every time.

@bleidingGOE, the windows logbook should provide the most information

[dennmuel]

Confirming the "too many files" error on Linux Mint. However the client does not restart but instead either dies without notification or causes the operating system to freeze.
As a workaround one can disable sync for the causing folders, so that at least all others get synced.

[benjamin-leiding]

It has been 2 month since @gcala opened the issue - the dev team did not even respond despite us providing several logs and precise descriptions. Is nextcloud still actively developed?

[GieltjE]

According to github there are enough commits in the server code, but the desktop client just seems to be getting updated translations....
We can't even force the clients to update to new versions resulting in major parts of the userbase running outdated crap....

[camilasan]

Hi All,

Note that Github isn't a support portal. We provide support to customers only. Github is our developer platform. If you're not a customer you can ask the community for help on help.nextcloud.com.

We of course try to deal with bugs as quick as we can, but it can take time as we have lots of other things on our plate. You are welcome to contribute to this free and open source product, to submit patches that fix bugs for example. We try to respond quickly to pull request from contributors.

Have a great day!
Camila

[lbdroid]

I ran into this issue today after a distro upgrade. The version I was running prior to the upgrade was 57bc791 . I am running that version again, and it is not affected by this REGRESSION.

If anybody cares enough about this problem to work on it, I'd suggest bisecting with that commit as a starting point.

@camilasan : your response comes off as hostile and rude. Not a particularly good way to encourage people to help in this project. I mean after all, how dare people want to ASSIST the project by reporting and discussing bugs in lieu of supporting it financially.

[jmjos]

Hello!

I also encountered the problem. It's not a bug within nextcloud, rather a system feature limiting the number of open files. E.g. see https://easyengine.io/tutorials/linux/increase-open-files-limit/ for a fix.

The issue can be closed.

[GieltjE]

It's not a Linux only problem, we only use Windows clients.

[benjamin-leiding]

I already tried this on my ubuntu 18.04 LTS - did not work. Moreover, the problem also occurs on Windows.

…

On 27 June 2019 10:00:14 CEST, JM Joseph ***@***.***> wrote: Hello! I also encountered the problem. It's not a bug within nextcloud, rather a system feature limiting the number of open files. E.g. see [https://easyengine.io/tutorials/linux/increase-open-files-limit/](https://easyengine.io/tutorials/linux/increase-open-files-limit/) for a fix. The issue can be closed. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #1035 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[wioxjk]

Experiencing the same problem with 2.6.1 on the following distros:

Ubuntu 18.04, 19.10
Ubuntu Budgie 18.04, 19.10
Trisquel 8, 9
Debian 9, 10
Parabola
Windows 8, 8.1, 10
Linux Mint 18, 19
OpenSuSE Tumbleweed

sadly, changing ulimit does not work.

The problem does not occur on 2.3.1

Photo and videofolder is 900 GB

[todoneunl]

+1 on Ubuntu 19.10.

I'm trying to sync a 25GB photo folder. 'Too many open files', 100% CPU and application freeze. I tried it with or without hidden files.

Seems like a huge issue and I'm perplexed that this is not taken seriously and still not fixed.

[wioxjk]

Seems like a huge issue and I'm perplexed that this is not taken seriously and still not fixed.

I agree, I've asked in the official Nextcloud channels but I have not even gotten a reply there.