[Store] Integrate Symfony's finder for more flexible loaders

[aszenz]

I have this current code to index all non-empty markdown files, I feel this use case is common enough to warrant some better DX by integrating the finder directly but curious to hear alternative solutions on making the loader more flexible.

        $documentsLoader = new TextFileLoader();
        $vectorizer = new Vectorizer(
            $this->aiPlatform->getPlatform(),
            AIPlatform::EMBEDDING_MODEL
        );
        $docFiles = Finder::create()->files()->in($folderPath)->name('*.md')->size('> 0')->getIterator();
        $docsArray = \array_map(
            fn(\SplFileInfo $file) => $file->getRealPath() ?: throw new \RuntimeException('File path not found'),
            iterator_to_array($docFiles)
        );
        $output->writeln(\sprintf('Indexing %d documents...', \count($docsArray)));
        $indexer = new Indexer(
            loader: $documentsLoader,
            vectorizer: $vectorizer,
            store: $this->vektor,
            source: $docsArray,
            transformers: [
                new RecursiveCharacterTextTransformer(
                    separators: ['#', '##', '\n', ' '],
                    chunkSize: 1000,
                )
            ]
        );
        $indexer->index();

[chr-hertel]

What was your direction of travel here?
I thought about having a DirectoryLoader before - that could be some kind of chained loader with different inner loaders per file type or so. and also with #1149 i tried some kind of typed source approach.