I was dumping ~gigs of files of approximately 3-6megs a pop (my music collection) so I could track the files that I want to listen to when I'm on the go. I had the git watch command running from the assistant branch.

I was getting something along the lines of...

/Users/jtang/annex/.git/annex/tmp/: openTempFile: resource exhausted (Too many open files)

and

git-annex: createPipe: resource exhausted (Too many open files)

I also noticed that I somehow ended up with 256 ssh-agent's running on one of my machines, I'm not sure if the two issues are related or not, I had not noticed this type of behaviour up until recently.

Also this was appearing in the logs

x00:annex jtang$ tail -f .git/annex/daemon.log
(scanning...) Already up-to-date.
kqueue: Too many open files

To be precise, I suspect that the kqueue limit is 256, I had 325 files in the 'queue', I ended up doing a git annex add manually and all was fine.

Yes, this is a known problem with kqueue, it has to keep every directory in the tree open. On inotify I have a note that it may need to fork off extra watcher processes to deal with this. Of course that adds significant complication.

In the meantime, you may be able to increase your system's maximum allowed number of open files per process somehow.

(I doubt that the ssh-agent is related; git-annex does not use ssh-agent directly anyway..)

Comment by http://joeyh.name/ Mon Jun 25 15:42:48 2012
On the system limits side, I think if you want to make it more approachable by more users then adjusting system limits might scare users away. On the note of the ssh-agents spawning like no tomorrow on my machine, it turned out that i had a symlink from my .bashrc to .bash_profile, I guess I should not be too lazy and have two seperate files.

Doing,

sudo sysctl -w kern.maxfilesperproc=400000

Somewhat works for me, git-annex watch at least starts up and takes a while to scan the directory, but it's not ideal. Also, creating files seems to work okay, when I remove a file the changes don't seem to get pushed across my other repos, running a sync on the remote repo fixes things.

Jimmy, sounds like I could use something like this to get the current limit:

sysctl kern.maxfilesperproc

Probably prints "sysctl kern.maxfilesperproc = 256" or such.. can you verify? Once I have the limit, I can make the kqueue code use subset of it, and print out a message when it needs to be increased, like the inotify code does.

(Also, the kqueue code only opens directories, not files, so unless you have 400000 directories, that's a little high.)


On file removal not propigating, does this still happen? When you remove a file does a git commit automatically happen, or is that broken with kqueue?

Comment by http://joeyh.name/ Wed Jul 4 12:32:44 2012

In relation to the system limits,

laplace:~ jtang$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 10240

Also, the maxfiles for the whole system is

laplace:~ jtang$ sysctl kern.maxfiles
kern.maxfiles: 12288

the above was the defaults as far as I recall. What you probably would be interested is the ulimits that the user see

laplace:~ jtang$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

I would imagine the limit that you are looking for is 256. Hope this helps.


On the point about deletions not being propagated, it does do a commit. I suspect that the kqueue code is just not picking up the changes and pushing the changes out. The watch command on a single annex with no remotes functions as expected.

On kFreeBSD, I get this:

$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 11095

But ulimit still has 1024 limit, so you'd need to adjust both, as root. Messy..

Comment by http://joeyh.name/ Sun Jul 8 18:13:58 2012
Comments on this page are closed.