Crawler doesn't work unless I use a production URL

[benjamincrozat]

Hello!

In my code, I have a scheduled command to generate a new sitemap daily. It looks extremely simple:

SitemapGenerator::create('https://example.com')
    ->writeToFile(public_path('sitemap.xml'));

You'll notice one thing though: I hard coded the production environment's URL. When I do SitemapGenerator::create('/') or even SitemapGenerator::create(config('app.url')), the sitemap isn't generated at all.
It bothers me because I need to run tests to make sure the sitemap is correctly generated. My test would be something like "create 10 fake posts and make sure the sitemap contains their URLs".
I always make sure the local URL is accessible. Despite that, even if I hard code https://example.com.test, the crawler won't work.

I just can't figure out why this is occurring. It worked perfectly before Laravel 7 (I have no idea if it's really related though).

Thanks in advance for your help.

[kimobot]

Also experiencing this issue, using Laravel 5.8.

[illmatic]

I'm experiencing the same issue. Though regardless of what URL i use, my sitemap file is blank.
Tried

1. SitemapGenerator::create('https://URL')->writeToFile(public_path('sitemap.xml'));
2. SitemapGenerator::create('http://URL')->writeToFile(public_path('sitemap.xml'));
3. SitemapGenerator::create(env('APP_URL'))->writeToFile(public_path('sitemap.xml'));

Here is a snapshot of my composer file

        "laravel/framework": ">6.14",
        "spatie/laravel-sitemap": "^5.7",

Resulting file contents

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"> </urlset>

[jbajou]

same on one of my production website while other website works perfectly and I cannot understand the difference between the two sites...

[manshu]

I've the same issue. Did anyone find any alternative?

[jbajou]

anyone made progress on this already ? I'm still investigating from time to time, haven't find anything yet.

[ThomasVeld]

I'm having the same problem

[manshu]

I added links manually. There was no other way to do it.

[eNzyOfficial]

Same problem here

[rmsite]

Same problem here

Laravel 7

[rmsite]

Work for me if I use '/' in the path,

SitemapGenerator::create(URL::to('/').'/')->writeToFile(public_path('sitemap.xml'));

OR

SitemapGenerator::create('http://domain.test/')->writeToFile(public_path('sitemap.xml'));

[TranceDevCode]

Thank you so much, this work for me bro!

SitemapGenerator::create('http://domain.test/')->writeToFile(public_path('sitemap.xml'));

[qdanik]

I have the same issue

"laravel/framework": "^6.2",
"spatie/laravel-sitemap": "^5.7"

[ghost]

Work for me if I use '/' in the path,

SitemapGenerator::create(URL::to('/').'/')->writeToFile(public_path('sitemap.xml'));

OR

SitemapGenerator::create('http://domain.test/')->writeToFile(public_path('sitemap.xml'));

This worked for me too.

So, what you could try is using something like ngrok (which I use myself) to expose your local or testing environment to the web, and point the crawler to that URL. This is how I got it to work.

[ardavan-ansari]

I was using valet and I had this issue, I switched to php artisan serve and used the resulting url like so :
SitemapGenerator::create( 'http://127.0.0.1:8000/' )
Finally got it working.

If you aren't using valet, you can use php -S localhost:8000 to create a PHP built in server and use that instead.

[bassigbc]

Same issue, doesn't work if get URL string from a function:

$URL = \URL::to('/'); SitemapGenerator::create( $URL )->writeToFile(public_path('sitemap.xml'));
generate a file with no urls.

$URL = "https://mysite.com"; SitemapGenerator::create( $URL )->writeToFile(public_path('sitemap.xml'));
works fine.

$URL content is the same in both cases

[padders]

I have the same issue with Laravel Valet. It doesn't seem to pick up the .test sites at all. Using php artisan serve instead as mentioned by ardavan-ansari worked well though.

Given that it's generating on a different url to the live site, a handy feature would be to be able to automate the find replace as part of the process. This would allow us to schedule the creation and update a static site.

[hsdmr]

Hello everyone. I had the same problem. So I decided to add the links manually. There are thousands of links in my project and I keep the links in a separate database to be SEO compatible.

My database table is as follows.

Schema::create('slugs', function (Blueprint $table) {
            $table->id();
            $table->string('owner');
            $table->string('slug')->unique();
            $table->string('seo_title')->nullable();
            $table->longText('seo_description')->nullable();
            $table->tinyInteger('index')->default(1);
            $table->tinyInteger('follow')->default(1);
            $table->string('language')->default('tr_TR');
            $table->softDeletes();
            $table->timestamps();
        });

I created the artisan command as explained in the read me section. Below is my code in the command.

public function handle()
    {
        $sitemapIndex = SitemapIndex::create(); 
        Slug::select('owner')->groupBy('owner')->get()->each(function(Slug $slug) use ($sitemapIndex){
            $sitemap =  Sitemap::create();
            Slug::where('owner','=',$slug->owner)->get()->each(function(Slug $slug) use ($sitemap){
                if($slug->owner=='page') $sitemap->add(Url::create('/'));    //write homepage in pages sitemap
                if($slug->index==1) $sitemap->add($slug->slug);               //print only indexed pages
            });
            $sitemap->writeToFile(public_path($slug->owner.'-sitemap.xml'));
            $sitemapIndex->add('/'.$slug->owner.'-sitemap.xml');
        });
        $sitemapIndex->writeToFile(public_path('sitemap_index.xml'));
    }

I hope it helps you too.

[ehsantalkhouncheh]

I have same issue none of your solutions work for me :(

Laravel 8

[cihankusmez]

same issue

Laravel 9

[abbasmashaddy72]

Same Issue no solutions working

[umairparacha00]

@freekmurze Facing the same issue.

[nabihsiblini]

Facing the same issue here with Laravel 10.33, I solved it by adding the following in the 'guzzle_options' into 'config/sitemap.php':
RequestOptions::VERIFY => false,

SitemapGenerator::create(config("app.url"))
->configureCrawler(function ($crawler) {
$crawler->ignoreRobots();
})
->hasCrawled(function (Url $url) {
$url->url = urldecode($url->url); // for arabic links
return $url;
})
->writeToFile(public_path("sitemap.xml"));