Help generating wikidumps

[deedeenelson994]

Hi all! I'm new to the community and fairly new to Linux and python altogether, so I could use some help getting started. Everything seemed to work on my old Ubuntu VM but I can't access it anymore and I'm struggling to get either SaveWeb or MediaWiki Dump Generator to work on my new VM (Ubuntu 23.04). If someone could provide me with step-by-step instructions that a beginner would be able to understand, that would be greatly appreciated. Here's the error message I've been getting regardless of which project I'm trying to archive:

wikiteam3dumpgenerator https://groundedforlife.fandom.com --xml --images --curonly
Checking API... https://groundedforlife.fandom.com/api.php
API is OK: https://groundedforlife.fandom.com/api.php
Checking index.php... https://groundedforlife.fandom.com/index.php
check_index(): Trying Special:Version...
POST https://groundedforlife.fandom.com/index.php {'title': 'Special:Version'} 301
index.php available probability: 20% (0.2)
Checking index.php... https://groundedforlife.fandom.com/index.php
check_index(): Trying Special:Version...
POST https://groundedforlife.fandom.com/index.php {'title': 'Special:Version'} 301
index.php available probability: 20% (0.2)
Checking index.php... https://groundedforlife.fandom.com/
check_index(): Trying Special:Version...
POST https://groundedforlife.fandom.com/ {'title': 'Special:Version'} 301
index.php available probability: 20% (0.2)
Error in index.php.
Please, provide a correct path to index.php or use --xmlrevisions or --xmlapiexport. Terminating.

[robkam]

The step-by-step instructions, as simple as we can make them, are at README.md

You need to explain to us how to reproduce the problem you're having.

I don't get the same problem when I enter, in a terminal window:
dumpgenerator https://groundedforlife.fandom.com/ --xml --images --curonly

Checking API... https://groundedforlife.fandom.com/api.php
API is OK:  https://groundedforlife.fandom.com/api.php
Checking index.php... https://groundedforlife.fandom.com/index.php
index.php is OK
No --path argument provided. Defaulting to:
  [working_directory]/[domain_prefix]-[date]-wikidump
Which expands to:
  ./groundedforlifefandomcom-20230924-wikidump
--delay is the default value of 0.5
There will be a 0.5 second delay between HTTP calls in order to keep the server from timing you out.
If you know that this is unnecessary, you can manually specify '--delay 0.0'.
#########################################################################
# Welcome to DumpGenerator 0.4.0-alpha by WikiTeam (GPL v3)             #
# More info at: https://github.com/elsiehupp/wikiteam3                  #
#########################################################################

#########################################################################
# Copyright (C) 2011-2023 WikiTeam developers                           #
#                                                                       #
# This program is free software: you can redistribute it and/or modify  #
# it under the terms of the GNU General Public License as published by  #
# the Free Software Foundation, either version 3 of the License, or     #
# (at your option) any later version.                                   #
#                                                                       #
# This program is distributed in the hope that it will be useful,       #
# but WITHOUT ANY WARRANTY; without even the implied warranty of        #
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the         #
# GNU General Public License for more details.                          #
#                                                                       #
# You should have received a copy of the GNU General Public License     #
# along with this program.  If not, see <http://www.gnu.org/licenses/>. #
#########################################################################

Analysing https://groundedforlife.fandom.com/api.php
Trying generating a new dump into a new directory...
https://groundedforlife.fandom.com/api.php

Retrieving the XML for every page from the beginning


Retrieving the XML for every page

Loading page titles from namespaces = all
Excluding titles from namespaces = None
31 namespaces found
    Retrieving titles in the namespace 0
    Retrieving titles in the namespace 1
    Retrieving titles in the namespace 2
    Retrieving titles in the namespace 3
    Retrieving titles in the namespace 4
    Retrieving titles in the namespace 5
    Retrieving titles in the namespace 6
    Retrieving titles in the namespace 7
    Retrieving titles in the namespace 8
    Retrieving titles in the namespace 9
    Retrieving titles in the namespace 10
    Retrieving titles in the namespace 11
    Retrieving titles in the namespace 12
    Retrieving titles in the namespace 13
    Retrieving titles in the namespace 14
    Retrieving titles in the namespace 15
    Retrieving titles in the namespace 420
    Retrieving titles in the namespace 421
    Retrieving titles in the namespace 1200
    Retrieving titles in the namespace 1201
    Retrieving titles in the namespace 1202
    Retrieving titles in the namespace 828
    Retrieving titles in the namespace 829
    Retrieving titles in the namespace 2900
    Retrieving titles in the namespace 2901
    Retrieving titles in the namespace 110
    Retrieving titles in the namespace 111
    Retrieving titles in the namespace 500
    Retrieving titles in the namespace 501
    Retrieving titles in the namespace 502
    Retrieving titles in the namespace 503
Titles saved at... groundedforlifefandomcom-20230924-titles.txt
403 page titles loaded
    (She's Got) Kegs, 1 edit
    Action Mountain High, 1 edit
    All Apologies, 1 edit
    All the Young Nudes, 1 edit
    Amy, 1 edit
    Baby, You Can't Drive My Car, 1 edit
    Baby Come Back, 1 edit
    Bang on a Drum, 1 edit
    Beat on the Brat, 1 edit

Snipped a bit

'https://static.wikia.nocookie.net/groundedforlife/images/4/4c/Uncle_eddie.jpg/revision/latest?cb=20['Walt.jpg', 'https://static.wikia.nocookie.net/groundedforlife/images/2/24/Walt.jpg/revision/latest?cb=20130919025829',['Wiki-Preview.png', 'https://static.wikia.nocookie.net/groundedforlife/images/8/8f/Wiki-Preview.png/revision/latest?cb=['Wiki.png', 'https://static.wikia.nocookie.net/groundedforlife/images/b/bc/Wiki.png/revision/latest?cb=20100730161705',Image filenames and URLs saved at... groundedforlifefandomcom-20230924-images.txt
Retrieving images...
Creating "./groundedforlifefandomcom-20230924-wikidump/images" directory
Downloaded 31 images and 31 .desc files.
Downloading index.php (Main Page) as index.html
Downloading Special:Version with extensions and other related info
Downloading site info as siteinfo.json

---> Congratulations! Your dump is complete <---

If you encountered a bug, you can report it on GitHub Issues:
  https://github.com/mediawiki-client-tools/mediawiki-dump-generator/issues

If you need any other help, you can reach out on GitHub Discussions:
  https://github.com/orgs/mediawiki-client-tools/discussions

If this is a public wiki, please, consider publishing this dump.
Do it yourself as explained in:
  https://github.com/WikiTeam/wikiteam/wiki/Tutorial#Publishing_the_dump

Good luck! Bye!

[deedeenelson994]

After a lot of experimenting and having no idea what I'm doing, I got it to work by following the official instructions in a miniconda Python 3.8 environment as opposed to just following the official instructions as written. Here's a rundown of what I did. If anyone might be able to explain what I did right, and what may have gone wrong before, that would be appreciated. The official instructions aren't necessarily very accessible to people who want to contribute but have no background in working with Linux or Python. Most notably, the "poetry update && poetry install && poetry build" command gives me an error even now ("The lock file is not compatible with the current version of Poetry. Upgrade Poetry to be able to read the lock file or, alternatively, regenerate the lock file with the poetry lock command.")

Also, could someone tell me if there's a reason to use Wikiteam3 over the original Wikiteam if one got both to work as intended?

Anyway, here's what I did to get it to work:

I open Terminal in a brand new Ubuntu 23.04 virtual machine and execute the following commands:

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
exec bash
conda update -n base -c defaults conda
conda config --set auto_activate_base false
conda create -n wikiteam3_env
conda activate -n wikiteam3_env
conda install python=3.8
pip install --upgrade -r requirements.txt
sudo apt install git
git clone https://github.com/mediawiki-client-tools/mediawiki-dump-generator
cd mediawiki-dump-generator/
sudo apt install python3-poetry
poetry build
pip install --force-reinstall dist/*.whl
dumpgenerator https://groundedforlife.fandom.com --xml --images --curonly

[robkam]

The previous errors look something like a '301 redirect'. I don't understand the workings of dumpgenerator well enough to be able to explain what's causing this.

Having no experience with Miniconda, I can't comment on this part.

The last six or seven lines follow the instructions in the readme.

The line
poetry update && poetry install && poetry build
is three commands joined together with &&
Poetry is telling you do either
poetry update
or
poetry lock
to fix the error.

Python these days is version 3.whatever. To run the original Wikiteam you'd have to especially install the older 2.7 version of Python.

[deedeenelson994]

Thanks for your reply. Last night I got the original Wikiteam to work in a Python 2.7 Miniconda environment as per the instructions in:

https://groups.google.com/g/wikiteam-discuss/c/JyaqOPE_kUQ, https://github.com/WikiTeam/wikiteam/blob/f265849087b1f717d0a5e3d61e786cf0b72ab10b/README.md and https://docs.conda.io/projects/miniconda/en/latest/

I figured I'd try to get Wikiteam3 to work by replicating this process except with the instructions in the readme, and it does work. I'm happy it works but I'm confused as to why it gives me an error ("Please, provide a correct path to index.php or use --xmlrevisions or --xmlapiexport. Terminating.") when I do the exact same things outside of the Python 3.8 Miniconda environment. Also even now both poetry update and poetry lock result in errors:

(wikiteam3_env) ubuntu@ubuntu-VirtualBox:~/mediawiki-dump-generator$ poetry update
RuntimeError

The lock file is not compatible with the current version of Poetry.

Upgrade Poetry to be able to read the lock file or, alternatively, regenerate the lock file with the poetry lock command.

at /usr/lib/python3/dist-packages/poetry/packages/locker.py:481 in _get_lock_data

  477│                 "Upgrade Poetry to ensure the lock file is read properly or, alternatively, "

  478│                 "regenerate the lock file with the `poetry lock` command."

  479│             )

  480│         elif not lock_version_allowed:

→ 481│             raise RuntimeError(

  482│                 "The lock file is not compatible with the current version of Poetry.\n"

  483│                 "Upgrade Poetry to be able to read the lock file or, alternatively, "

  484│                 "regenerate the lock file with the `poetry lock` command."

  485│             )

Again I'm really happy it works but a better understanding of why some things work and some don't might help me prevent things from breaking in the future.

[yzqzss]

add --index-check-threshold 0

[robkam]

dumpgenerator: error: unrecognized arguments: --index-check-threshold