feat: document the changes

Author: galargh

Dockerfile

@@ -1,5 +1,13 @@
-# docker build . -f Dockerfile -t distributed-wikipedia-mirror
-# docker run --rm -v $(pwd)/snapshots:/github/workspace/snapshots -v $(pwd)/tmp:/github/workspace/tmp distributed-wikipedia-mirror <mirrorzim.sh arguments>
+# This Dockerfile creates a self-contained image in which mirrorzim.sh can be executed
+#
+# You can build the image as follows (remember to use this repo as context for the build):
+#  docker build . -f Dockerfile -t distributed-wikipedia-mirror
+#
+# You can then run the container anywhere as follows
+#  docker run --rm -v $(pwd)/snapshots:/github/workspace/snapshots -v $(pwd)/tmp:/github/workspace/tmp distributed-wikipedia-mirror <mirrorzim.sh arguments>
+#  NOTE(s):
+#   - volume attached at /github/workspace/snapshots will contain downloaded zim files after the run
+#   - volume attached at /github/workspace/tmp will contain created website directories after the run
 
 FROM openzim/zim-tools:3.1.0 AS openzim
 

README.md

@@ -136,7 +136,7 @@ This step won't be necessary when automatic sharding lands in go-ipfs (wip).
 
 ### Step 3: Download the latest snapshot from kiwix.org
 
-Source of ZIM files is at https://download.kiwix.org/zim/wikipedia/  
+Source of ZIM files is at https://download.kiwix.org/zim/wikipedia/
 Make sure you download `_all_maxi_` snapshots, as those include images.
 
 To automate this, you can also use the `getzim.sh` script:
@@ -164,8 +164,8 @@ $ zimdump dump ./snapshots/wikipedia_tr_all_maxi_2021-01.zim --dir ./tmp/wikiped
 
 > ### ℹ️ ZIM's main page
 >
-> Each ZIM file has "main page" attribute which defines the landing page set for the ZIM archive.  
-> It is often different than the "main page" of upstream Wikipedia.  
+> Each ZIM file has "main page" attribute which defines the landing page set for the ZIM archive.
+> It is often different than the "main page" of upstream Wikipedia.
 > Kiwix Main page needs to be passed in the next step, so until there is an automated way to determine "main page" of ZIM, you need to open ZIM in Kiwix reader and eyeball the name of the landing page.
 
 ### Step 5: Convert the unpacked zim directory to a website with mirror info
@@ -242,7 +242,7 @@ Make sure at least two full reliable copies exist before updating DNSLink.
 
 ## mirrorzim.sh
 
-It is possible to automate steps 3-6 via a wrapper script named `mirrorzim.sh`.  
+It is possible to automate steps 3-6 via a wrapper script named `mirrorzim.sh`.
 It will download the latest snapshot of specified language (if needed), unpack it, and add it to IPFS.
 
 To see how the script behaves try running it on one of the smallest wikis, such as `cu`:
@@ -253,9 +253,9 @@ $ ./mirrorzim.sh --languagecode=cu --wikitype=wikipedia --hostingdnsdomain=cu.wi
 
 ## Docker build
 
-A `Dockerfile` with all the software requirements is provided.  
+A `Dockerfile` with all the software requirements is provided.
 For now it is only a handy container for running the process on non-Linux
-systems or if you don't want to pollute your system with all the dependencies.  
+systems or if you don't want to pollute your system with all the dependencies.
 In the future it will be end-to-end blackbox that takes ZIM and spits out CID
 and repo.
 
@@ -340,3 +340,7 @@ We are working on improving deduplication between snapshots, but for now YMMV.
 ## Code
 
 If you would like to contribute more to this effort, look at the [issues](https://github.com/ipfs/distributed-wikipedia-mirror/issues) in this github repo. Especially check for [issues marked with the "wishlist" label](https://github.com/ipfs/distributed-wikipedia-mirror/labels/wishlist) and issues marked ["help wanted"](https://github.com/ipfs/distributed-wikipedia-mirror/labels/help%20wanted).
+
+## GitHub Actions Workflow
+
+The GitHub Actions workflow that is available in this repository takes information about the wiki website that you want to mirror, downloads its' zim, unpacks it, converts it to a website and uploads it to S3 as a tar.gz package which is publicly accessible.

packer/README.md

@@ -0,0 +1,3 @@
+Packer configuration that resides here creates AMI in which:
+- ipfs service is started on machine boot
+- `publish_website_from_s3.sh` is available

terraform/README.md

@@ -0,0 +1,8 @@
+Terraform configuration that resides here creates:
+- S3 bucket where website packages can be uploaded
+- EC2 instance which runs ipfs where `publish_website_from_s3.sh` can be run to publish mirrors
+
+To run `terraform` here you have to export:
+- `TF_VAR_public_key` - public key which will be used to give you SSH access to EC2
+- `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` - creds to AWS account that have enough permissions to create the resources
+- `AWS_REGION` - the name of the region where S3 and EC2 should be created

tools/add_website_to_ipfs.sh

@@ -2,6 +2,9 @@
 
 set -euo pipefail
 
+# This script adds website that was created at <unpacked zim dir>
+#  from <zim file name> to ipfs
+
 usage() {
 	echo "USAGE:"
 	echo "	$0 <zim file name> <unpacked zim dir> [<extra ipfs add flags>]";

tools/publish_website_from_s3.sh

@@ -2,6 +2,9 @@
 
 set -euo pipefail
 
+# This scripts downloads <website name> from s3://wikipedia-on-ipfs,
+#  unpacks it and adds it to ipfs
+
 usage() {
 	echo "USAGE:"
 	echo "	$0 <website name>";

tools/start_ipfs.sh

@@ -2,6 +2,9 @@
 
 set -euo pipefail
 
+# This script starts ipfs daemon
+# If ipfs was not initialised before, this script also initialises ipfs
+
 if ! ipfs repo stat; then
   ipfs init -p server,local-discovery,flatfs,randomports --empty-repo
   ipfs config --json Experimental.AcceleratedDHTClient true