manifests not added to gc if two instances of package manger running at same time

[adamsiembida]

I'm having an issue where projects are not showing up as "active manifests" when I run gc. It is my understanding that any projects created should show up under gc as active manifests until they are deleted from the filesystem.

I'm using Windows 10.

The problem seems to occur when two instances of the juila console are used at the same time:

1. Start a julia console using the desktop shortcut that was created during installation.
2. Start the package manager ]and create a new shared environment activate --shared test1.
3. Add a package to the test1 project add DSP.
4. Open up a second terminal in the same manner, without closing the first one.
5. Inside that terminal create another shared environment called test2. ]then activate --shared test2.
6. [edit] Add a package to the test2 project add DSP.
7. Close both terminals.
8. Open a new terminal in the same manner as before, and enter the package manger ].
9. Type gc.

At this point I would expect both test1 and test2 to show up under the gc's "active manifests" heading, but only test1 is showing up.

I did some more testing and tried deleting test1 from my filesystem. I ran the gc command, and the package manager deleted the DSP and dependent packages from my computer, even though test2 is still using it.

[StefanKarpinski]

Yeah, we need some file locking. @vtjnash: what's the best portable way to do discretionary file locking?

[vtjnash]

Imho, Pidfile.jl :)

[StefanKarpinski]

Ok, so if we were to use that, we'd have to make it a stdlib and then make Pkg depend on it. Sigh, seems like a lot of work.

[StefanKarpinski]

Actually, I don't entirely understand why this is happening. The usage file is opened in append mode, a couple of lines are written to it and then it's immediately closed. This script doesn't seem to be sufficient to create a race condition between these two appends. Perhaps it's because you don't actually ever add anything to test2?

[fredrikekre]

Perhaps it's because you don't actually ever add anything to test2?

Yea, this was my first thought too. The Project.toml/Manifest.toml files are not created by activate, only when you add something.

[adamsiembida]

Perhaps it's because you don't actually ever add anything to test2?

Sorry, I must have missed adding that in the description of the issue. I actually did add the DSP package to test2 as well, specifically for that reason.

I just repeated the process again to make sure and ran into the same issue (I used different environment names just to make sure). I will edit my post.

[StefanKarpinski]

After you've done that can you send the contents of ~/.julia/logs/manifest_usage.toml before and after the gc operation?

[adamsiembida]

Before

[["C:\\Users\\Adam\\.julia\\environments\\v1.0\\Manifest.toml"]]
time = 2018-10-18T18:30:05.318Z
[["C:\\Users\\Adam\\.julia\\environments\\test1\\Manifest.toml"]]
time = 2018-10-18T18:30:55.416Z
[["C:\\Users\\Adam\\.julia\\environments\\test1\\Manifest.toml"]]
time = 2018-10-18T18:30:59.307Z

[edit] Now delete the test1 folder through the file explorer.

Open julia terminal and run gc:

[["C:\\Users\\Adam\\.julia\\environments\\v1.0\\Manifest.toml"]]
time = 2018-10-18T18:36:10.419Z

[vtjnash]

Append mode is not posix append mode (issue JuliaLang/julia#3374)

[StefanKarpinski]

Append mode is not posix append mode (issue JuliaLang/julia#3374)

Nevertheless, that does not appear to be the issue here. What seems to be happening is that the gc call is somehow not retaining the record of the test1 environment. Weirder still is that the test1 environment seems to be deleted by the gc call? Is that really what's happening here, @w1res? Or did you delete test1 yourself at some point in this process?

[adamsiembida]

Weirder still is that the test1 environment seems to be deleted by the gc call? Is that really what's happening here, @w1res? Or did you delete test1 yourself at some point in this process?

I deleted test1 before running gc. I edited that post to be clearer.

1. I set up the two environments using the steps described in the original post.
2. Then posted the output of the ~/.julia/logs/manifest_usage.toml file. (It seems to have some error at this point).
3. Then I deleted test1 using the Windows file explorer.
4. I restarted julia just to be safe.
5. Then I ran gc.
6. Then posted the output of the ~/.julia/logs/manifest_usage.toml file again. (I'm guessing there should be some mention of test2 at this point, since that project is still on the disk, but there isn't. Not surprising since there wasn't any mention of test2 immediately after creating the two environments either).

The things that seem to be weird is that after creating both environments is that there seem to be two entries for test1 instead of one entry for test1, and one for test2.

The reason I would delete test1 and run gc was to show that after deleting the only project that seemed to be tracked properly, and running gc it would go ahead and remove the packages even though they were still used in test2. I thought this might differentiate whether the project is still being tracked properly but just not displaying properly in the gc command, or whether it is just not being tracked at all (this seems to be the case).

[StefanKarpinski]

I deleted test1 before running gc.

That would seem to explain the whole thing, wouldn't it? How can an environment that you've deleted preserve any packages?

[KristofferC]

The problem here just seems to be that "test2" didn't get tracked due to a race condition with appending the environment usage to the same file from multiple processes.

[adamsiembida]

I deleted test1 before running gc.

That would seem to explain the whole thing, wouldn't it? How can an environment that you've deleted preserve any packages?

Because the test2 environment which still exists is using the package--the bug is that test2 never got tracked like it should because I had two julia windows open at the same time.

[StefanKarpinski]

Ok, so the issue here is that the usage of test2 is not being correctly recorded after all. @vtjnash does that seem like it could be the deviation in behavior of our append mode from POSIX?