robots.txt is requested for every (55000 times) added url to sitemap

[ametad]

Dear people of Spatie,

For our site we use this package to crawl a little, and for a large part we manually add the urls to the sitemap like so:

        $generator = SitemapGenerator::create(config('app.url'))
            ->maxTagsPerSitemap(10000)
            ->configureCrawler(function (Crawler $crawler) {
                $crawler
                    ->setUserAgent(config('app.name').'/'.app()->version())
                    ->setDelayBetweenRequests(150)
                    ->setParseableMimeTypes(['text/html']);
            })
            ->shouldCrawl(function (UriInterface $url) {

                // Only internal urls
                if (parse_url(config('app.url'), PHP_URL_HOST) !== $url->getHost()) {
                    return false;
                }

                // Don't crawl the models directly, we will add them later on.
                if (Str::startsWith($url->getPath(), '/specific/path/to/')) {
                    return false;
                }

                // No pagination
                if (Str::contains($url->getQuery(), 'page=')) {
                    return false;
                }

                return true;
            })
            ->hasCrawled(function (Url $url) {

                // Ignore urls with trailing slash
                if (Str::endsWith($url->url, '/')) {
                    return;
                }

                return $url;
            });

        // 55.000 models from database
        ModelName::all()->each(function ($model) use ($generator) {
            $generator->getSitemap()
                ->add(
                    Url::create($model->url->show)
                        ->setLastModificationDate($model->updated_at)
                );
        });

        $generator->writeToFile(/*...*/);

My problem is that for every url added to the sitemap ($generator->getSitemap()->add($url)) there is a request to robots.txt on my server as well. Look at my logs, this goes on for about 55.000 times:

127.0.0.1 - - [19/Oct/2021:07:43:21 +0000] "GET /robots.txt HTTP/1.0" 200 379 "-" "-"
127.0.0.1 - - [19/Oct/2021:07:43:21 +0000] "GET /robots.txt HTTP/1.0" 200 379 "-" "-"
127.0.0.1 - - [19/Oct/2021:07:43:21 +0000] "GET /robots.txt HTTP/1.0" 200 379 "-" "-"
127.0.0.1 - - [19/Oct/2021:07:43:21 +0000] "GET /robots.txt HTTP/1.0" 200 379 "-" "-"
127.0.0.1 - - [19/Oct/2021:07:43:21 +0000] "GET /robots.txt HTTP/1.0" 200 379 "-" "-"
127.0.0.1 - - [19/Oct/2021:07:43:21 +0000] "GET /robots.txt HTTP/1.0" 200 379 "-" "-"
127.0.0.1 - - [19/Oct/2021:07:43:21 +0000] "GET /robots.txt HTTP/1.0" 200 379 "-" "-"

This is a bit of a drain on the server I rather not have. Is there a way that this can be reduced to just one time a request for the robots.txt and use this in memory over and over again?

[ametad]

Found a solution, it does work like this:

        // 55.000 models from database
        $sitemap = $generator->getSitemap();
        ModelName::all()->each(function ($model) use ($sitemap) {
            $sitemap->add(
                Url::create($model->url->show)
                    ->setLastModificationDate($model->updated_at)
            );
        });