Merge pull request #1267 from CodeForPhilly/staging

Staging

Author: nlebovits

.gitattributes

@@ -0,0 +1 @@
+backup_data/ filter=lfs diff=lfs merge=lfs -text

.github/CODEOWNERS

@@ -1 +1 @@
-@nlebovits @brandonfcohen1 @marvieqa @bacitracin @paulhchoi @CodeWritingCow @SofiaFasullo @cfreedman
+@nlebovits @cfreedman

.gitignore

@@ -65,3 +65,6 @@ data/src/app/service-account-key.json
 
 # Cached and temporary data files from pipeline
 storage/
+
+# cursor
+.cursor/

.pre-commit-config.yaml

@@ -23,7 +23,15 @@ repos:
           echo "All MI grades are B or above."'
         language: system
         files: \.py$
-        pass_filenames: true
+        pass_filenames:
+          true
+          # Code similarity detection
+      - id: pylint-similarities
+        name: Code similarity detection (non-blocking)
+        entry: bash -c 'echo "Checking for code similarities in data/src/..."; pylint --disable=all --enable=similarities --score=no data/src/ || true'
+        language: system
+        files: ^data/src/.*\.py$
+        pass_filenames: false
   - repo: https://github.com/jendrikseipp/vulture
     rev: 'v2.3'
     hooks:

README.md

@@ -1,83 +1,57 @@
 # Clean & Green Philly
 
-Philadelphia has a gun violence problem. This dashboard aims to help solve it by empowering community groups to carry out cleaning and greening interventions in the vacant properties where they can have the biggest impact.
-
-## Motivation
-
-Philadelphia has a gun violence problem. Homicides have been on the rise since 2013. The past three years—2020, 2021, and 2022—have been the deadliest on record, with a high of 562 homicides in 2021. Community members need solutions, but many city-run initiatives are frustratingly slow or inadequate. Nearly 80% of the city’s anti-violence spending focuses on long-term violence reduction without any clear, immediate impact.
-
-Research shows that greening and cleaning vacant and abandoned parcels is one of the most impactful, cost-effective interventions available to reduce gun violence in a neighborhood. Drs. Eugenia South and Charles Branas have led several studies that [demonstrate that greening vacant lots in Philadelphia reduced gun violence by as much as 29% in the surrounding area](https://www.pnas.org/doi/10.1073/pnas.1718503115). Similarly, cleaning and lightly repairing vacant houses led a 13% drop in gun assaults compared to nearby blocks. These “greening and cleaning” interventions not only reduce gun violence but also provide other benefits, such as reducing the urban heat island effect, lowering residents’ stress levels, and contributing to lower levels of depression among residents.
-
-There are roughly 40,000 vaccant properties in Philadelphia. Identifying the highest-priority vacant properties will allow community groups to invest their limited resources where they will have the biggest impact. Combining various public data, this dashboard helps users identify the properties that are ideal for them to intervene in. It also offers additional information about each property to facilitate interventions and indicates the best possible route to cleaning up the property.
-
-## Current Draft
-
-You can view [the current build of the website here](https://www.cleanandgreenphilly.org/). We are building based off of a prototype which looks like this:
-
-![User prototype map](/docs/map.png)
-
-An older proptotype is also [available here](https://nlebovits.github.io/dashboard_demo_website/more_info.html), although it is now significantly out of date.
-
-Likewise, you can view the [very first iteration as an ArcGIS Storymap here](https://storymaps.arcgis.com/stories/551f77d85a584705b97c41db7711ba1b).
-
-## Project Updates
+[Contributor Guide](/docs/CONTRIBUTING.md) | [Code of Conduct](/docs/CODE_OF_CONDUCT.md) | [Setup Instructions](/docs/SETUP/) | [Roadmap](/docs/ROADMAP.md)
 
-**May 1, 2024:** Clean & Green Philly won second place in the [36th annual NJ DEP Mapping Contest's](https://dep.nj.gov/gis/36th-mapping-contest/) Dashboard category!
+🚨 Project Status
 
-**April 26, 2024:** On behalf of Clean & Green Philly, Nissim gave testimony to a Philadelphia City Council hearing on the uses of data and tech to reduce gun violence through place-based interventions. His statement is [available here](/docs/PROJECT_BACKGROUND/City%20Council%20Testimony%20Final,%20April%2026,%202024.pdf).
+_Note: Clean & Green Philly ceased active development and maintainence in July of 2025. For more information, please see our [letter to stakeholders](/docs/PROJECT_BACKGROUND/C&GP%20Shutdown%20Letter,%20June%205,%202025.pdf) from July 2025. The website is still available at [cleanandgreenphilly.org](https://www.cleanandgreenphilly.org/) and the site, codebase, and data will remain available and open source for the foreseeable future._
 
-**April 21, 2024:** We have officially launched v1.0.0! The website is now live at [www.cleanandgreenphilly.org](https://www.cleanandgreenphilly.org/).
+Philadelphia has a gun violence problem. This dashboard aims to help solve it by empowering community groups to carry out cleaning and greening interventions in the vacant properties where they can have the biggest impact.
 
-**Feb. 1, 2024:** We have soft launched a minimum viable product! The website is functional at [www.cleanandgreenphilly.org](https://www.cleanandgreenphilly.org/).
+For more information on the research and data behind the project, please see the background information in [our docs](docs/).
 
-**Nov. 15, 2023:** The Center for Philadelphia's Urban Future has officially voted to be a fiscal sponsor for our project and maintain it long-term. We are working with them to assemble a steering committee that will help inform future development of the web tool.
+## Data Access
 
-**Oct. 2, 2023:** Met with the Philadelphia District Attorney's Office data analytics team. They are very excited about the dashboard and are connecting us with folks in the Managing Director's Office and the Civic Coalition to Save Lives.
+All of our final output data, including final backups of the last reasonably valid vacant properties data, are available in [our `data` subdirectory](/data/backup_data/). There, you will find the backup vacant properties data, along with a GeoParquet file containing our last data outputs from July 2025 and a PMtiles dataset that we use for visualizing the vacant properties on the website. A README in the folder gives more context on the data.
 
-**Sept. 29, 2023** Got our first funding from a donor! He's writing us a check to cover costs associated with user testing, hosting, domain name, etc., and will connet us with other folks who may be willing to give us more.
+Anyone interested in generating new data can simply install the codebase and run the pipeline via Docker, provided they bear in mind that everything _but_ the vacant properties data will be up to date.
 
-**Sept. 27, 2023** Nissim spoke with a reporter from WHYY who may be including our project in a longer piece about Philadelphians working on solutions to the gun violence crisis.
+## Project History
 
-## Current Tasks and Project Roadmap
+### 2025
 
-At the moment, we are working on:
+**July 2025:** Clean & Green Philly ceases active development due to a lack of available data on vacant properties in Philadelphia.
 
-- Organizing a steering committee of stakeholders to guide future project development
-- Improving the accessibility and responsiveness of our MVP
-- Improving the content and design of our Take Action pages
-- Laying groundwork to apply for a grant to support our work
+**July 2025:** Clean & Green Philly publisheds v2.0, which incorporates data on all ~580,000 properties in Philadelphia.
 
-Current tasks for the are visible under the [Issues](https://github.com/CodeForPhilly/vacant-lots-proj/issues) section.
+**June 25, 2025:** Nissim and Amanda publish an op ed in _The Philadelphia Inquirer_ highlighting [the lack of official vacancy data](www.inquirer.com/opinion/commentary/mayor-parker-housing-plan-missing-data-20250625.html) and encourage the City to resume data collection.
 
-For a detailed project roadmap, please see our [ROADMAP.md](/docs/ROADMAP.md) document.
+### 2024
 
-## Looking For
+**October 22, 2024:** Nissim and Amanda publish an op ed in _The Philadelphia Citizen_ encouraging the City to [adopt our approach to strategically targeting vacant properties](https://thephiladelphiacitizen.org/guest-commentary-clean-and-green-philly-where-its-most-needed/).
 
-- An experienced developer to lead our front-end work and coordinate with our UX team
-- A graphic designer to work on content for the website
-- Someone with grant-writing expertise
+**September 20, 2024:** Clean & Green Philly wins [HopePHL's "Great Idea" pitch competition](https://www.instagram.com/p/DAJg9SZprim/).
 
-## Languages/Frameworks
+**June 2024:** The City of Philadelphia ceases to provide accurate vacancy data via the public API.
 
-- React
-- NextJS
-- Tailwind CSS
-- Python (data engineering)
+**May 1, 2024:** Clean & Green Philly won second place in the [36th annual NJ DEP Mapping Contest's](https://dep.nj.gov/gis/36th-mapping-contest/) Dashboard category!
 
-## Codebase
+**April 26, 2024:** On behalf of Clean & Green Philly, Nissim gave testimony to a Philadelphia City Council hearing on the uses of data and tech to reduce gun violence through place-based interventions. His statement is [available here](/docs/PROJECT_BACKGROUND/City%20Council%20Testimony%20Final,%20April%2026,%202024.pdf).
 
-This is a single repository for both the React app and the Python pipeline to ETL the data.
+**April 21, 2024:** We have officially launched v1.0.0! The website is now live at [www.cleanandgreenphilly.org](https://www.cleanandgreenphilly.org/).
 
-## Contribution
+**Feb. 1, 2024:** We have soft launched a minimum viable product! The website is functional at [www.cleanandgreenphilly.org](https://www.cleanandgreenphilly.org/).
 
-Please see the documents in our [docs](/docs) folder for background on the project and instructions on how to contribute. At a minimum, please read:
+### 2023
 
-1. The [guide to contributing](/docs/CONTRIBUTING.md)
-2. The [code of conduct](/docs/CODE_OF_CONDUCT.md)
-3. The appropriate installation setup instructions ([front end](/docs/SETUP/FRONT_END.md) and/or [back end](/docs/SETUP/BACK_END.md))
+**Nov. 15, 2023:** The Center for Philadelphia's Urban Future becomes our fiscal sponsor.
 
-### Point of Contact
+**June 2023:** Clean & Green Philly is launched as a Code for Philly project.
 
 ## License
 
 MIT © Code for Philly
+
+---
+
+_This project was developed to address gun violence in Philadelphia through data-driven community interventions. While no longer actively maintained, we hope the code and data continue to benefit researchers and community organizations working on similar initiatives._

data/.env.example

@@ -0,0 +1,10 @@
+# Environment variables for Clean and Green Philly backend
+# Copy this file to data/.env and fill in your actual values
+
+# Google Cloud Platform credentials
+CLEAN_GREEN_GOOGLE_KEY=your-api-key-here
+GOOGLE_CLOUD_BUCKET_NAME=your-bucket-name-here
+GOOGLE_CLOUD_PROJECT=your-project-id
+
+# Slack integration for diff reporting
+CAGP_SLACK_API_TOKEN=your-slack-token 

data/backup_data/README.md

@@ -0,0 +1,15 @@
+## Backup Data
+
+This folder contains 1) backup vacant properties data from June 2024, the last time these data were reasonably accurate; and 2) the final outputs of our pipeline before project shutdown in July of 2025.
+
+### Vacancy Data
+
+The land backup file contains vacant land data sent to us by the Department of Licenses and Inspections, corresponding to the last reasonably complete dataset on vacant land prior to the City [no longer collecting vacany data](https://www.inquirer.com/opinion/commentary/mayor-parker-housing-plan-missing-data-20250625.html).
+
+The buildings backup are data that we collected ourselves in June of 2024. They are likely missing about a thousand or more buildings, as we hadn't realized at the time that the buildings dataset was corrupted, too, but they are the best data we have available under the circumstances.
+
+Combined, these represent about 34,000 properties. The pipeline is configured to run using these backup data unless the City's APIs suddenly start returning data above the expected threshold again.
+
+### Pipeline Outputs
+
+As it's currently confiured, the pipeline will return new data for everything _except_ the vacant properties themselves, for which it uses our June of 2024 backups. This means that all associated data are the currently-available numbers from their corresponding services, but we have no way to update the vacant properties data themselves. We have stored these here in both GeoParquet format (representing all 580,000+ properties in Philadelphia) and the PMtiles that we use to visualize vacant properties on the website (representing ~34,000 vacant properties from June of 2024, with the rest of the data from July of 2025).

data/backup_data/buildings_backup_2024_06_24.parquet

@@ -13,6 +13,7 @@ services:
       - CAGP_SLACK_API_TOKEN
     volumes:
       - ./src:/app/src
+      - ./backup_data:/app/backup_data
       - ~/.config/gcloud/application_default_credentials.json:/app/service-account-key.json
       - /etc/timezone:/etc/timezone:ro
       - /etc/localtime:/etc/localtime:ro