diff --git a/.gitignore b/.gitignore index ebae71a4436..e8a31b06e44 100644 --- a/.gitignore +++ b/.gitignore @@ -13,7 +13,6 @@ config/environments/*.local.yml config/settings.local.yml config/settings/*.local.yml coverage -doc log node_modules public/assets diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1fb14413d27..999f7141d5a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -22,7 +22,7 @@ This guide covers our development workflow, coding standards, and how to get you > [!NOTE] > We don't assign issues to individual contributors. You are welcome to work on any issue, and there's no need to ask first. -> For more details see [our FAQ](FAQ.md) +> For more details see [our FAQ](doc/FAQ.md) We welcome the community to contribute to this repository in any form: @@ -81,7 +81,7 @@ Here's the typical contribution workflow: 1. **Find an Issue**: Browse our [issues](https://github.com/openstreetmap/openstreetmap-website/issues) or identify a bug/feature you'd like to work on 2. **Fork, Clone & Branch**: Fork the repository, clone it to your local machine, and create a new branch for your work. Avoid working directly on the `master` branch. -3. **Set Up**: Follow the [installation guide](INSTALL.md) to set up your development environment +3. **Set Up**: Follow the [installation guide](doc/INSTALL.md) to set up your development environment 4. **Develop**: Make your changes following our [code quality guidelines](#code-quality-guidelines) 5. **Test**: Write tests for your changes and ensure all existing tests pass 6. **Commit**: Write clear commit messages following our [guidelines](#committing) diff --git a/README.md b/README.md index b953abb2873..b743e6a595d 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ a copy of which can be found in the [LICENSE](LICENSE) file. # Installation `openstreetmap-website` is a Ruby on Rails application that uses PostgreSQL as its database, and has a large -number of dependencies for installation. For full details please see [INSTALL.md](INSTALL.md). +number of dependencies for installation. For full details please see [INSTALL.md](doc/INSTALL.md). # Development diff --git a/config/devcontainer.database.yml b/config/devcontainer.database.yml index f65469580c1..8009f089dd3 100644 --- a/config/devcontainer.database.yml +++ b/config/devcontainer.database.yml @@ -1,4 +1,4 @@ -# This configuration is tailored for use with devcontainers. See DEVCONTAINER.md for more information. +# This configuration is tailored for use with devcontainers. See doc/DEVCONTAINER.md for more information. development: adapter: postgresql diff --git a/config/docker.database.yml b/config/docker.database.yml index b0e79b0f878..0b554dda1ee 100644 --- a/config/docker.database.yml +++ b/config/docker.database.yml @@ -1,4 +1,4 @@ -# This configuration is tailored for use with docker compose. See DOCKER.md for more information. +# This configuration is tailored for use with docker compose. See doc/DOCKER.md for more information. development: adapter: postgresql diff --git a/CONFIGURE.md b/doc/CONFIGURE.md similarity index 99% rename from CONFIGURE.md rename to doc/CONFIGURE.md index 24bb096f7c8..0987d5e7d9a 100644 --- a/CONFIGURE.md +++ b/doc/CONFIGURE.md @@ -264,4 +264,4 @@ If you need additional assistance: ## Contributing -For information on contributing changes to the code, see [CONTRIBUTING.md](CONTRIBUTING.md) +For information on contributing changes to the code, see [CONTRIBUTING.md](../CONTRIBUTING.md) diff --git a/DEVCONTAINER.md b/doc/DEVCONTAINER.md similarity index 82% rename from DEVCONTAINER.md rename to doc/DEVCONTAINER.md index 10b02513342..6e5a69291a3 100644 --- a/DEVCONTAINER.md +++ b/doc/DEVCONTAINER.md @@ -84,17 +84,17 @@ If you want to add in the full history later on, perhaps to run `git blame` or ` Start by opening the project with VS Code. Within it, you will need to install the extension _Dev Containers_, which can be done from the _Extensions_ section, reachable via the sidebar icons. Or VS Code may show a popup recommending this extension, with a button to install it directly. -![VS Code: panel to install extensions](./docs/assets/vscode-devcontainers-extension.png) +![VS Code: panel to install extensions](./assets/vscode-devcontainers-extension.png) ## Open the project in a container If everything is correct, this should make a few new commands available to you within VS Code. Find and run the command "Dev Containers: Reopen in Container". -![VS Code: command to open in a devcontainer](./docs/assets/vscode-dev-reopen.png) +![VS Code: command to open in a devcontainer](./assets/vscode-dev-reopen.png) The first time you do this, it will go into a bit of a process. It will create the devcontainer, pull the Docker images, install the dependencies, etc. Go drink some water while this runs. -![VS Code: notification that VS Code is connecting to the Dev Container](./docs/assets/vscode-connecting-to-devcontainer.png) +![VS Code: notification that VS Code is connecting to the Dev Container](./assets/vscode-connecting-to-devcontainer.png) Eventually this will present you with a development environment ready to go. In subsequent occasions this should be much faster. @@ -102,21 +102,30 @@ Eventually this will present you with a development environment ready to go. In If everything went well, you are done! For example, now you can open a shell in this environment using the VS Studio command "Create New Terminal (With Profile)": -![VS Code: command to open a terminal](./docs/assets/vscode-create-terminal.png) +![VS Code: command to open a terminal](./assets/vscode-create-terminal.png) From this terminal, you can run the test suite with `bundle exec rails test:all`: -![Running the test suite in the terminal](./docs/assets/vscode-rails-test-all.png) +![Running the test suite in the terminal](./assets/vscode-rails-test-all.png) Hopefully it should be all green? 🤞 You can also start a development server with `bundle exec rails s`: -![Running the dev server in the terminal](./docs/assets/vscode-rails-server.png) +![Running the dev server in the terminal](./assets/vscode-rails-server.png) It will take a moment to start, after which you will be able to take a browser to http://localhost:3000 and visit your own local version of the site. -## Other useful configuration +> [!NOTE] +> The OSM map tiles you see aren't created from your local database - they are the production map tiles, served from a separate service over the Internet. -See [`CONFIGURE.md`](CONFIGURE.md) for information on how to manage users and enable OAuth for iD, JOSM etc. +## What's next? + +🎉 **Congratulations!** You have successfully installed the OpenStreetMap website. + +**Next steps:** +* **Configuration:** See [CONFIGURE.md](CONFIGURE.md) for populating the database with data, creating users, setting up OAuth, and other configuration tasks. +* **Contributing:** Check out [CONTRIBUTING.md](../CONTRIBUTING.md) for coding style guidelines, testing procedures, and how to submit your contributions. + +Any terminal commands must be run like above, in the terminal provided by VS Code to run within the devcontainer. ## Other tools diff --git a/DOCKER.md b/doc/DOCKER.md similarity index 54% rename from DOCKER.md rename to doc/DOCKER.md index 41994d06dc0..60819afa49c 100644 --- a/DOCKER.md +++ b/doc/DOCKER.md @@ -79,105 +79,65 @@ This will launch one Docker container for each 'service' specified in `docker-co - You can tail logs of a running container with a command like this: `docker compose logs -f web` or `docker compose logs -f db`. - Instead of running the containers in the background with the `-d` flag, you can launch the containers in the foreground with `docker compose up`. The downside of this is that the logs of all the 'services' defined in `docker-compose.yml` will be intermingled. If you don't want this you can mix and match - for example, you can run the database in background with `docker compose up -d db` and then run the Rails app in the foreground via `docker compose up web`. -### Migrations +## Running commands -Run the Rails database migrations: +At this point, the Docker container can be used although there are a couple of steps missing to complete the install. + +From now on, any commands should be run within the container. To do this, first you need to open a shell within it: ```bash -docker compose run --rm web bundle exec rails db:migrate +docker compose run --rm web bash ``` -### Tests +This will open a shell where you can enter commands. These commands will run within the context of the container, without affecting your own machine outside your working directory. -Prepare the test database: +> [!IMPORTANT] +> Unless otherwise stated, make sure that you are in this shell when following any other instructions. -```bash -docker compose run --rm web bundle exec rails db:test:prepare -``` +### Create the databases -Run the test suite: +To create all databases and set up all databases, run: ```bash -docker compose run --rm web bundle exec rails test:all +bundle exec rails db:create ``` -> [!TIP] -> If you encounter errors about missing assets, ensure your asset pipeline is correctly configured for your environment. -> -> In production, assets must be precompiled for better performance: -> -> ```bash -> docker compose run --rm web bundle exec rake assets:precompile -> ``` -> ->In development, missing assets usually indicate a configuration or dependency issue. -> Precompiling assets in *development* will disable dynamic compilation and make debugging harder due to fingerprinted assets. -> To clean the assets you can run: -> -> ```bash -> docker compose run --rm web bundle exec rake assets:clobber -> ``` - -### Loading an OSM extract - -This installation comes with no geographic data loaded. You can either create new data using one of the editors (Potlatch 2, iD, JOSM etc) or by loading an OSM extract. Here an example for loading an OSM extract into your Docker-based OSM instance. - -For example, let's download the District of Columbia from Geofabrik or [any other region](https://download.geofabrik.de): +## Validate Your Installation -```bash -wget https://download.geofabrik.de/north-america/us/district-of-columbia-latest.osm.pbf -``` +Hopefully that's it? Let's check that things are working properly. -You can now use Docker to load this extract into your local Docker-based OSM instance: +### Run the tests -```bash -docker compose run --rm web osmosis \ - -verbose \ - --read-pbf district-of-columbia-latest.osm.pbf \ - --log-progress \ - --write-apidb \ - host="db" \ - database="openstreetmap" \ - user="openstreetmap" \ - validateSchemaVersion="no" -``` +Run the test suite: -**Windows users:** Powershell uses `` ` `` and CMD uses `^` at the end of each line, e.g.: - -```powershell -docker compose run --rm web osmosis ` - -verbose ` - --read-pbf district-of-columbia-latest.osm.pbf ` - --log-progress ` - --write-apidb ` - host="db" ` - database="openstreetmap" ` - user="openstreetmap" ` - validateSchemaVersion="no" +```bash +bundle exec rails test:all ``` -Once you have data loaded for Washington, DC you should be able to navigate to [`http://localhost:3000/#map=12/38.8938/-77.0146`](http://localhost:3000/#map=12/38.8938/-77.0146) to begin working with your local instance. - -### Additional Configuration +This test will take a few minutes, reporting tests run, assertions, and any errors. If you receive no errors, then your installation was successful. On occasion some tests may fail randomly and will pass if run again. We are working towards avoiding this, but it can still happen. -See [`CONFIGURE.md`](CONFIGURE.md) for information on how to manage users and enable OAuth for iD, JOSM etc. +> [!NOTE] +> The unit tests may output parser errors related to "Attribute lat redefined." These can be ignored. -### Bash +### Start the development server -If you want to get into a web container and run specific commands you can fire up a throwaway container to run bash in via: +Rails comes with a built-in webserver, so that you can test on your own machine without needing a server. Run: ```bash -docker compose run --rm web bash +bundle exec rails server ``` -Alternatively, if you want to use the already-running `web` container then you can `exec` into it via: +You can now view the site in your favourite web browser at [http://localhost:3000/](http://localhost:3000/) -```bash -docker compose exec web bash -``` +> [!NOTE] +> The OSM map tiles you see aren't created from your local database - they are the production map tiles, served from a separate service over the Internet. -Similarly, if you want to `exec` in the db container use: +## What's next? -```bash -docker compose exec db bash -``` +🎉 **Congratulations!** You have successfully installed the OpenStreetMap website. + +**Next steps:** +* **Configuration:** See [CONFIGURE.md](CONFIGURE.md) for populating the database with data, creating users, setting up OAuth, and other configuration tasks. +* **Contributing:** Check out [CONTRIBUTING.md](../CONTRIBUTING.md) for coding style guidelines, testing procedures, and how to submit your contributions. + +Don't forget to **run any commands in a shell within the container**, as instructed above under "Running commands". diff --git a/doc/INSTALL.md b/doc/INSTALL.md new file mode 100644 index 00000000000..f3c2c93eea4 --- /dev/null +++ b/doc/INSTALL.md @@ -0,0 +1,41 @@ +# Installation + +These instructions are designed for setting up `openstreetmap-website` development environment. If you want to deploy the software for your own project, then see the [Production Deployment Notes](#production-deployment-notes). + +## Installation Options + +There is more than one way to set up a development environment. + +### Containerized Installation + +We offer containerized environments with Docker which may avoid installation difficulties: + +- To use Docker manually, see [DOCKER.md](DOCKER.md). +- To use Docker via [Development Containers](https://containers.dev) (devcontainers), see [DEVCONTAINER.md](DEVCONTAINER.md). + +### Manual Installation + +This option involves manually installing dependencies directly on your machine. This gives you the most control and is often preferred by experienced developers on Linux systems. + +> [!WARNING] +> **Windows Note:** We don't recommend using this approach on Windows, as some Ruby gems may not be supported. If you are using Windows, we recommend a containerized setup as mentioned above. + +To install manually, see [MANUAL_INSTALL.md](MANUAL_INSTALL.md). + +## Production Deployment Notes + +> [!WARNING] +> Production deployment requires careful configuration and is significantly different from development setup. + +If you want to deploy `openstreetmap-website` for production use, you'll need to make a few changes: + +> [!IMPORTANT] +> It's not recommended to use `rails server` in production. Our recommended approach is to use [Phusion Passenger](https://www.phusionpassenger.com/). Instructions are available for [setting it up with most web servers](https://www.phusionpassenger.com/documentation_and_support#documentation). + +* Passenger will, by design, use the Production environment and therefore the production database - make sure it contains the appropriate data and user accounts. + +> [!TIP] +> The included version of the map call is quite slow and eats a lot of memory. You should consider using [CGIMap](https://github.com/zerebubuth/openstreetmap-cgimap) instead. + +* Make sure you generate the i18n files and precompile the production assets: `RAILS_ENV=production bundle exec i18n export; bundle exec rails assets:precompile` +* Make sure the web server user as well as the rails user can read, write and create directories in `tmp/`. diff --git a/INSTALL.md b/doc/MANUAL_INSTALL.md similarity index 75% rename from INSTALL.md rename to doc/MANUAL_INSTALL.md index 7dd34e3543b..a58fe186d9f 100644 --- a/INSTALL.md +++ b/doc/MANUAL_INSTALL.md @@ -1,30 +1,8 @@ -# Installation - -These instructions are designed for setting up `openstreetmap-website` development environment. If you want to deploy the software for your own project, then see the [Production Deployment Notes](#production-deployment-notes). - -## Installation Options - -There is more than one way to set up a development environment. - -### Containerized Installation - -We offer containerized environments with Docker which may avoid installation difficulties: - -- To use Docker manually, see [DOCKER](DOCKER.md). -- To use Docker via [Development Containers](https://containers.dev) (devcontainers), see [DEVCONTAINER.md](DEVCONTAINER.md). - -### Manual Installation - -Install dependencies directly on your machine (traditional approach, covered in this guide). This gives you the most control and is often preferred by experienced developers on Linux systems. - -> [!WARNING] -> **Windows Note:** We don't recommend using this approach on Windows, as some Ruby gems may not be supported. If you are using Windows, we recommend a containerized setup as mentioned above. - -## Manual Installation Guide +# Manual Installation Guide These instructions are based on Ubuntu 24.04 LTS, though the OSMF servers are currently running Debian 12. The instructions also work, with only minor amendments, for all other current Ubuntu releases, Fedora and macOS. -### Prerequisites +## Prerequisites Many of the dependencies are managed through the standard Ruby on Rails mechanisms - i.e. Ruby gems specified in the Gemfile and installed using Bundler. Some system packages are also required before you can get the various gems installed. @@ -34,9 +12,9 @@ Many of the dependencies are managed through the standard Ruby on Rails mechanis * Bundler (see note below about [developer Ruby setup](#ruby-version-manager-optional)) * JavaScript Runtime -### Step 1: Install System Dependencies +## Step 1: Install System Dependencies -#### Ubuntu 24.04 LTS +### Ubuntu 24.04 LTS ```bash sudo apt-get update @@ -56,7 +34,7 @@ sudo npm install --global yarn > sudo systemctl start postgresql.service > ``` -#### Fedora +### Fedora ```bash sudo dnf install ruby ruby-devel rubygem-rdoc rubygem-bundler \ @@ -81,7 +59,7 @@ Optionally set PostgreSQL to start on boot: sudo systemctl enable postgresql.service ``` -#### macOS +### macOS For macOS, you will need [Xcode Command Line Tools](https://mac.install.guide/commandlinetools/); macOS 14 (Sonoma) or later; and some familiarity with Unix development via the Terminal. @@ -129,7 +107,7 @@ brew install geckodriver > [!NOTE] > OS X does not have a /home directory by default, so if you are using the GPX functions, you will need to change the directories specified in config/application.yml. -### Step 2: Clone the Repository +## Step 2: Clone the Repository The repository is reasonably large (~560MB) and it's unlikely that you'll need the full history. Therefore you can probably do with a shallow clone (~100MB): ```bash @@ -144,9 +122,9 @@ If you want to add in the full history later on, perhaps to run `git blame` or ` > git clone https://github.com/openstreetmap/openstreetmap-website.git > ``` -### Step 3: Install Application Dependencies +## Step 3: Install Application Dependencies -#### Ruby gems +### Ruby gems We use [Bundler](https://bundler.io/) to manage the rubygems required for the project. @@ -155,7 +133,7 @@ cd openstreetmap-website bundle install ``` -#### Node.js modules +### Node.js modules We use [Yarn](https://yarnpkg.com/) to manage the Node.js modules required for the project. @@ -163,9 +141,9 @@ We use [Yarn](https://yarnpkg.com/) to manage the Node.js modules required for t bundle exec bin/yarn install ``` -### Step 4: Prepare Configuration Files +## Step 4: Prepare Configuration Files -#### Local settings file +### Local settings file > [!NOTE] > This is a workaround. [See issues/2185 for details](https://github.com/openstreetmap/openstreetmap-website/issues/2185#issuecomment-508676026). @@ -174,7 +152,7 @@ bundle exec bin/yarn install touch config/settings.local.yml ``` -#### Storage setup +### Storage setup `openstreetmap-website` needs to be configured with an object storage facility - for development and testing purposes you can use the example configuration: @@ -182,7 +160,7 @@ touch config/settings.local.yml cp config/example.storage.yml config/storage.yml ``` -### Step 5: Database Setup +## Step 5: Database Setup `openstreetmap-website` uses three databases - one for development, one for testing, and one for production. The database-specific configuration options are stored in `config/database.yml`, which we need to create from the example template. @@ -193,7 +171,7 @@ cp config/example.database.yml config/database.yml > [!IMPORTANT] > PostgreSQL is configured to, by default, accept local connections without requiring a username or password. This is fine for development. If you wish to set up your database differently, then you should change the values found in the `config/database.yml` file, and amend the instructions below as appropriate. -#### PostgreSQL account setup +### PostgreSQL account setup We need to create a PostgreSQL role (i.e. user account) for your current user, and it needs to be a superuser so that we can create more databases. @@ -203,7 +181,7 @@ createuser -s exit ``` -#### Create the databases +### Create the databases To create the three databases - for development, testing and production - run: @@ -211,7 +189,7 @@ To create the three databases - for development, testing and production - run: bundle exec rails db:create ``` -#### Database structure +### Database structure To create all the tables, indexes and constraints, run: @@ -253,7 +231,7 @@ You can now view the site in your favourite web-browser at [http://localhost:300 **Next steps:** * **Configuration:** See [CONFIGURE.md](CONFIGURE.md) for populating the database with data, creating users, setting up OAuth, and other configuration tasks. -* **Contributing:** Check out [CONTRIBUTING.md](CONTRIBUTING.md) for coding style guidelines, testing procedures, and how to submit your contributions. +* **Contributing:** Check out [CONTRIBUTING.md](../CONTRIBUTING.md) for coding style guidelines, testing procedures, and how to submit your contributions. ## Ruby Version Manager (Optional) @@ -283,21 +261,3 @@ gem install bundler ``` You should now be able to proceed with the rest of the installation. If you're on macOS, make sure you set up the [config override for the libxml2 location](#macos-click-to-expand) _after_ installing bundler. - -## Production Deployment Notes - -> [!WARNING] -> Production deployment requires careful configuration and is significantly different from development setup. - -If you want to deploy `openstreetmap-website` for production use, you'll need to make a few changes: - -> [!IMPORTANT] -> It's not recommended to use `rails server` in production. Our recommended approach is to use [Phusion Passenger](https://www.phusionpassenger.com/). Instructions are available for [setting it up with most web servers](https://www.phusionpassenger.com/documentation_and_support#documentation). - -* Passenger will, by design, use the Production environment and therefore the production database - make sure it contains the appropriate data and user accounts. - -> [!TIP] -> The included version of the map call is quite slow and eats a lot of memory. You should consider using [CGIMap](https://github.com/zerebubuth/openstreetmap-cgimap) instead. - -* Make sure you generate the i18n files and precompile the production assets: `RAILS_ENV=production bundle exec i18n export; bundle exec rails assets:precompile` -* Make sure the web server user as well as the rails user can read, write and create directories in `tmp/`. diff --git a/docs/assets/vscode-connecting-to-devcontainer.png b/doc/assets/vscode-connecting-to-devcontainer.png similarity index 100% rename from docs/assets/vscode-connecting-to-devcontainer.png rename to doc/assets/vscode-connecting-to-devcontainer.png diff --git a/docs/assets/vscode-create-terminal.png b/doc/assets/vscode-create-terminal.png similarity index 100% rename from docs/assets/vscode-create-terminal.png rename to doc/assets/vscode-create-terminal.png diff --git a/docs/assets/vscode-dev-reopen.png b/doc/assets/vscode-dev-reopen.png similarity index 100% rename from docs/assets/vscode-dev-reopen.png rename to doc/assets/vscode-dev-reopen.png diff --git a/docs/assets/vscode-devcontainers-extension.png b/doc/assets/vscode-devcontainers-extension.png similarity index 100% rename from docs/assets/vscode-devcontainers-extension.png rename to doc/assets/vscode-devcontainers-extension.png diff --git a/docs/assets/vscode-rails-server.png b/doc/assets/vscode-rails-server.png similarity index 100% rename from docs/assets/vscode-rails-server.png rename to doc/assets/vscode-rails-server.png diff --git a/docs/assets/vscode-rails-test-all.png b/doc/assets/vscode-rails-test-all.png similarity index 100% rename from docs/assets/vscode-rails-test-all.png rename to doc/assets/vscode-rails-test-all.png diff --git a/docs/assets/vscode-reopen-devcontainer.png b/doc/assets/vscode-reopen-devcontainer.png similarity index 100% rename from docs/assets/vscode-reopen-devcontainer.png rename to doc/assets/vscode-reopen-devcontainer.png diff --git a/docs/assets/vscode-terminal-ready.png b/doc/assets/vscode-terminal-ready.png similarity index 100% rename from docs/assets/vscode-terminal-ready.png rename to doc/assets/vscode-terminal-ready.png