Input file handling

[Pijukatel]

The point of this issue is to define CLI behavior with respect to the input files so that it is compatible with the rest of the Apify tooling.

1. CLI respects the env var to get input key

Ordered from highest priority to lowest:
ACTOR_INPUT_KEY
APIFY_INPUT_KEY
CRAWLEE_INPUT_KEY
default="INPUT"

The CLI decides what is the desired input key based on what is saved n the env vars and then it starts the Actor with all of those three env vars set to this value.

Example 1:

If

ACTOR_INPUT_KEY is missing
APIFY_INPUT_KEY= "CUSTOM_INPUT"
CRAWLEE_INPUT_KEY is missing

then CLI will start the actor in with following env vars set:

ACTOR_INPUT_KEY="CUSTOM_INPUT"
APIFY_INPUT_KEY="CUSTOM_INPUT"
CRAWLEE_INPUT_KEY="CUSTOM_INPUT"

Example 2:
If

ACTOR_INPUT_KEY is missing
APIFY_INPUT_KEY is missing
CRAWLEE_INPUT_KEY is missing

then CLI will start the actor in with following env vars set:

ACTOR_INPUT_KEY="INPUT"
APIFY_INPUT_KEY="INPUT"
CRAWLEE_INPUT_KEY="INPUT"

Example 3:
If

ACTOR_INPUT_KEY="A"
APIFY_INPUT_KEY="B"
CRAWLEE_INPUT_KEY="C"

then CLI will start the actor in with following env vars set:

ACTOR_INPUT_KEY="A"
APIFY_INPUT_KEY="A"
CRAWLEE_INPUT_KEY="A"

(This is to avoid edge cases where Crawlee-only-based code might not be even aware of the SDK-based env vars that have higher priority)

2. When called with -i argument and creating temp input file, it should use the name set in the env variable mentioned above:

Example when the env vars are not defined or they have value = "INPUT"

• -i {"a":"b"} should create temp file in the kvs called INPUT.json
• -i some_path/custom_input.custom_suffix should create temp copy of the file in the kvs called INPUT.custom_suffix

These temp files are deleted after cli is finished.
If there was a preexisting file of the same name, the cli will temporarily replace it and restore it afterwards.
Restoration does not happen if the file was modified by the Actor.

3. CLI does not care about duplicate input files, SDK should handle it

For example, there is already INPUT file in the default kvs and cli creates INPUT.json there. It is not CLI issue, but SDK should detect it and raise an error.

Some context:
https://apify.slack.com/archives/C02JQSN79V4/p1761920132606859
https://apify.slack.com/archives/C010Q0FBYG3/p1762945429512269

[Pijukatel]

Well, based on this answer
https://apify.slack.com/archives/C010Q0FBYG3/p1762969791479429?thread_ts=1762945429.512269&cid=C010Q0FBYG3

The cli is not supposed to just read the ENV var, but actually set it so that the SDK knows what to use as the input file.

Which would mean that the SDK is not supposed to guess the input file, but take the one that is defined in the ENV var. In nothing is defined, it should take only INPUT (not INPUT.json, or INPUT.bin to INPUT.TXT ...) If you want to define a different input file, change the env var

[B4nan]

Which would mean that the SDK is not supposed to guess the input file

Where exactly is the guessing? It just uses the env var, its default if not available.

The cli is not supposed to just read the ENV var, but actually set it so that the SDK knows what to use as the input file.

SDK should not depend on CLI, CLI can override the default behavior via env vars, but it's not required. I dont see how "just read" and "read and write" is different - if it reads the value, than the value is already correct, why should it write again? It should set it only if the value differs from the default and is provided via some CLI arguments

It all feels a bit too artificial to consider anything other than INPUT.json locally. I think the only case we are not so sure about is the extensions, it's not about the env var really, right? You get INPUT in the env var, and it should mean INPUT.json locally, I think that's how it always worked. And we should use extensions locally, so its easy to read those files. Maybe that changed in python tooling recently, if so, IMO it was wrong. Having extensions improves the DX of local development, that's the sole reason to have them.

If your question was "should the env var hold the extension", the answer is a nope afaik. And while the KVS supports anything, not just JSON, I dont think this applies to the inputs really, e.g. all the CLI related docs mention the inputs being JSON. If you'd like to have some other format, you wouldn't use the input, but a generic KVS item.

[Pijukatel]

Where exactly is the guessing? It just uses the env var, its default if not available.

It is guessing in situations where multiple files with the same basename and different suffixes exist in the kvs.
For example: INPUT, INPUT.json, INPUT.txt -> SDK will pick one of those

What should be expected behavior in such a case? How will SDK know which one of those was temporarily created by CLI? That is why the CLI should set full name in the env var and the SDK should respect it

[Pijukatel]

If your question was "should the env var hold the extension", the answer is a nope afaik. And while the KVS supports anything, not just JSON, I dont think this applies to the inputs really, e.g. all the CLI related docs mention the inputs being JSON. If you'd like to have some other format, you wouldn't use the input, but a generic KVS item.

It would be easier, but is that really the case? Are there no Actors that use non-JSON inputs?

[Pijukatel]

It all feels a bit too artificial to consider anything other than INPUT.json locally. I think the only case we are not so sure about is the extensions, it's not about the env var really, right? You get INPUT in the env var, and it should mean INPUT.json

So when we have just INPUT file (without suffix) in the storage, the SDK should ignore it?

[B4nan]

It would be easier, but is that really the case? Are there no Actors that use non-JSON inputs?

I'd say it's not a use case the getInput method is for. AFAIK the platform will also understand only a JSON input, we also have an input schema that cannot work with anything else than a JSON. The fact that the KVS backend can handle any file doesn't mean input can be any file.

So when we have just INPUT file (without suffix) in the storage, the SDK should ignore it?

IMO the INPUT key means INPUT.json locally. cc @vladfrangu since you were implementing the memory storage in JS, is that how it works?

[Pijukatel]

And we should use extensions locally, so its easy to read those files. Maybe that changed in python tooling recently, if so, IMO it was wrong. Having extensions improves the DX of local development, that's the sole reason to have them.

The problem with the suffix-based FilesystemStorageClient is that for a single KEY, there are competing file options (INPUT, INPUT.json, INPUT.txt, INPUT.bin...). In the Python tooling, this was changed in a way that there is no room for interpretation; one key can map only to one file - there is no guessing. This is more aligned with how the storage client works on the platform, and it is better except for manual edits on the file system, where the suffix helps a little, but is not really needed.

[B4nan]

there are competing file options

But how exactly would those files be created? I can only understand a direct usage of KVS API, which again is not something to consider here, the normal workflow is a user writes the INPUT.json by hand (or the SDK/CLI infers it based on the input schema defaults). If they want any other format than JSON, they shouldn't use the input key, it's not designed for that.

this was changed in a way that there is no room for interpretation

Which fixed one confusing part but broke a lot of others, no? 🙃

[vladfrangu]

Btw looking at the memory storage in JS, I'm semi certain that having INPUT, INPUT.txt and INPUT.json will actually yield UB 🙃 (as in, who knows which file will be loaded, probably depends on the order the fs returns them)

EDIT: confirmed, what is exactly what happens 🙃

[vladfrangu]

We don't really care for the extension in the JS Memory Storage impl, its there only for storing it on the FS for users to inspect later. We outright ignore the extension for the internal key of the record

[B4nan]

yield UB

What is UB? 👀 unknown or unhandled something? :]

[vladfrangu]

Undefined behavior