Verify repository exists before attempting to clone it

[flip111]

This works ...

» git push
Username for 'https://github.com': flip111
Password for 'https://flip111@github.com':
Everything up-to-date

Doesn't work

» spago publish
Reading Spago workspace configuration...

✅ Selecting package to build: foldable-traversable-extra

Downloading dependencies...
Building...
           Src   Lib   All
Warnings     0     0     0
Errors       0     0     0

✅ Build succeeded.

Passed preliminary checks.
Pushing tag 'v0.0.4' to the remote
Username for 'https://github.com': flip111
Password for 'https://flip111@github.com':
Building again with the build plan from the solver...
Building...
           Src   Lib   All
Warnings     0     0     0
Errors       0     0     0

✅ Build succeeded.


✅ Ready for publishing. Calling the registry..


✅ Registry accepted the Publish request and is processing...

Logs from the Registry pipeline:
  2024-01-22T13:44:36.796Z Received Publish request, job id: c15beb1c-580f-4f2e-a269-c7d77a4f43bf
  2024-01-22T13:44:36.806Z Fetching {
  "githubOwner": "flip111",
  "githubRepo": "foldable-traversable-extra"
}
  2024-01-22T13:44:36.973Z Failed to clone git tag: Cloning into '/tmp/tmp-2210958-ehZufsMfSvEF/foldable-traversable-extra'...
fatal: could not read Username for 'https://github.com': No such device or address
  2024-01-22T13:44:36.975Z Failed to clone repository flip111/foldable-traversable-extra at ref v0.0.4

❌ Registry finished processing the package, but it failed. Please fix it and try again.

I don't have 2fa authenticated on github yet, not sure if that's related but i just mentioned it in case.

[f-f]

This is a Registry error, see the logs when we query the API using the job id from your logs:

$ curl https://registry.purescript.org/api/v1/jobs/c15beb1c-580f-4f2e-a269-c7d77a4f43bf | jq .

{
  "createdAt": "2024-01-22T13:44:36.794Z",
  "finishedAt": "2024-01-22T13:44:36.974Z",
  "jobId": "c15beb1c-580f-4f2e-a269-c7d77a4f43bf",
  "jobType": "publish",
  "logs": [
    {
      "jobId": "c15beb1c-580f-4f2e-a269-c7d77a4f43bf",
      "level": "ERROR",
      "message": "Failed to clone git tag: Cloning into '/tmp/tmp-2210958-ehZufsMfSvEF/foldable-traversable-extra'...\nfatal: could not read Username for 'https://github.com': No such device or address",
      "timestamp": "2024-01-22T13:44:36.973Z"
    },
    {
      "jobId": "c15beb1c-580f-4f2e-a269-c7d77a4f43bf",
      "level": "ERROR",
      "message": "Failed to clone repository flip111/foldable-traversable-extra at ref v0.0.4",
      "timestamp": "2024-01-22T13:44:36.975Z"
    }
  ],
  "packageName": "foldable-traversable-extra",
  "ref": "v0.0.4",
  "success": false
}

That tag was created at 13:43, and the logs happen at 13:44, so I think this is a bug?
I'll move the issue to the registry repo.

[f-f]

@thomashoneyman any clue what might be happening here? The spago config seems alright

[thomashoneyman]

The first error is strange:

Failed to clone git tag: Cloning into '/tmp/tmp-2210958-ehZufsMfSvEF/foldable-traversable-extra'...\nfatal: could not read Username for 'https://github.com': No such device or address",

We clone the repository early in publishing:

registry-dev/app/src/App/API.purs

Line 370 in ef989a1

{ path: packageDirectory, published: publishedTime } <- Source.fetch source tmp existingMetadata.location payload.ref

Here's the implementation of that fetch call:

registry-dev/app/src/App/Effect/Source.purs

Lines 78 to 81 in ef989a1

	clonePackageAtTag = do
	let url = Array.fold [ "https://github.com/", owner, "/", repo ]
	let args = [ "clone", url, "--branch", ref, "--single-branch", "-c", "advice.detachedHead=false", repoDir ]
	withRetryOnTimeout (Git.gitCLI args Nothing) >>= case _ of

Under the hood it's just calling git on the command line:

registry-dev/app/src/App/CLI/Git.purs

Lines 74 to 80 in ef989a1

	-- | Run the `git` tool via the command line.
	gitCLI :: Array String -> Maybe FilePath -> Aff (Either String String)
	gitCLI args cwd = do
	result <- liftAff $ _.getResult =<< Execa.execa "git" args (_ { cwd = cwd })
	pure case result.exit of
	Normally 0 -> Right (String.trim result.stdout)
	_ -> Left (result.stdout <> result.stderr)

[thomashoneyman]

@f-f @flip111 The error is not good, but I would guess this is happening because the repository field in the Spago config is incorrect:

https://github.com/flip111/purescript-foldable-traversable-extra/blob/d7e3c78766831c1d8e52091e34226bcdbb591209/spago.yaml#L18

It lists the repository as foldable-traversable-extra but the actual repository has a purescript prefix:
https://github.com/flip111/purescript-foldable-traversable-extra

[flip111]

Thank you for looking into this. I changed the repo name and it worked. Let's keep this issue open to make a better error message.

[f-f]

Ah indeed! Thanks Thomas. I guess both the registry and spago should check for the existence of the repo before doing their thing.

[thomashoneyman]

What was actually output to the console from the Spago publish? The registry generally comments this line:

"Failed to clone repository flip111/foldable-traversable-extra at ref v0.0.4"

What error would y'all want instead? Some kind of preflight check that the repository exists, and an error "Repository flip111/foldable-traversable-extra does not exist."?

[f-f]

Some kind of preflight check that the repository exists, and an error "Repository flip111/foldable-traversable-extra does not exist."?

Yeah, errors from git clone are pretty generic, so we can offer a much better error just running this simple check

[f-f]

The errors returned to the user are the ones appearing in the API call, and the first one is pretty confusing:
"Failed to clone git tag: Cloning into '/tmp/tmp-2210958-ehZufsMfSvEF/foldable-traversable-extra'...\nfatal: could not read Username for 'https://github.com': No such device or address"

[thomashoneyman]

I think a simple git ls-remote would be a good preflight check, with interaction disabled via GIT_TERMINAL_PROMPT=0 since if the repo doesn't exist GitHub will prompt for login in case it's private.