Skip to content

Data #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
s2t2 wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Data #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0b8920f
Prep tests, organize mocks, change page validation strategy
s2t2 Feb 4, 2018
2f7411e
Write dummy data to file
s2t2 Feb 4, 2018
de91539
Write real data to file
s2t2 Feb 4, 2018
2ac7132
WIP - test CSV data [--skip-ci]
s2t2 Feb 4, 2018
4963a0b
Test JSON data
s2t2 Feb 5, 2018
438e4ba
WIP - early attempts to parse multiple years worth of data [--skip-ci]
s2t2 Feb 5, 2018
b6224e2
Don't write empty files
s2t2 Feb 11, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,15 @@ bin/console

### Updating Rankings Data

1. Download HTML files into the [archives repo](https://github.com/data-creative/us-news-rankings-archive).
1. This repo uses a Git submodule to store raw HTML files. When appropriate, download new HTML files into the [archives repo](https://github.com/data-creative/us-news-rankings-archive).

2. Transform source HTML into table elements:
2. Then transform source HTML into table elements (to be stored in the submodule):

```sh
bundle exec rake extract_tables
```

2. Transform source HTML into machine-readable data in JSON and CSV format:
2. Transform source HTML into machine-readable data in JSON and CSV format (to be stored in this repo):

```sh
bundle exec rake transform_tables
Expand Down
13 changes: 13 additions & 0 deletions Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,26 @@ PATH
remote: .
specs:
us_news_rankings (0.1.0)
activesupport
nokogiri

GEM
remote: https://rubygems.org/
specs:
activesupport (5.1.4)
concurrent-ruby (~> 1.0, >= 1.0.2)
i18n (~> 0.7)
minitest (~> 5.1)
tzinfo (~> 1.1)
coderay (1.1.2)
concurrent-ruby (1.0.5)
diff-lcs (1.3)
github-markup (1.6.2)
i18n (0.9.3)
concurrent-ruby (~> 1.0)
method_source (0.9.0)
mini_portile2 (2.3.0)
minitest (5.11.3)
nokogiri (1.8.2)
mini_portile2 (~> 2.3.0)
pry (0.11.3)
Expand All @@ -32,6 +42,9 @@ GEM
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.7.0)
rspec-support (3.7.0)
thread_safe (0.3.6)
tzinfo (1.2.5)
thread_safe (~> 0.1)
yard (0.9.12)

PLATFORMS
Expand Down
25 changes: 21 additions & 4 deletions Rakefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,29 @@ task :extract_tables do

html_files.each do |html_filepath|
puts html_filepath
page = UsNewsRankings::Page.new(html_filepath)
puts "...skipping" unless page.parse
end
end

task :transform_tables do
#table_files = Dir.glob("web/rankings/education/**/*table*.html")
#puts "EXTRACTING RANKINGS DATA FROM #{table_files.count} HTML TABLES"

#dirs = table_files.map{|filepath|
# filepath.split("/")[0..5].join("/") #> "web/rankings/education/colleges/business/2017"
#}.uniq

dirs = (2012..2017).to_a.map{|rankings_year| "web/rankings/education/grad-schools/law-part-time/#{rankings_year}" }

puts "EXTRACTING RANKINGS DATA FROM #{dirs.count} DIRECTORIES"

dirs.each do |tables_dir|
parser = UsNewsRankings::TableParser.new(tables_dir)
begin
page = UsNewsRankings::Page.new(html_filepath)
page.parse
rescue UsNewsRankings::Page::InvalidTableError
puts "...skipping"
parser.perform
rescue => e
puts "... SKIPPING (#{e})"
end
end
end
65 changes: 65 additions & 0 deletions data/education/grad-schools/law-part-time/2015.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
rankings_list,rankings_year,school_name,school_city,rank,tie,score,peer_score,lsat_25th,lsat_75th,acceptance_rate,enrollment
education/grad-schools/law-part-time,2015,Georgetown University,"Washington, DC",1,false,100,4.0,160,168,0.049,247
education/grad-schools/law-part-time,2015,George Washington University,"Washington, DC",2,false,86,3.3,150,165,0.293,270
education/grad-schools/law-part-time,2015,Fordham University,"New York, NY",3,false,82,3.2,157,163,0.242,203
education/grad-schools/law-part-time,2015,George Mason University,"Arlington, VA",4,false,71,2.7,158,163,0.164,150
education/grad-schools/law-part-time,2015,University of Maryland (Carey),"Baltimore, MD",5,false,69,2.8,156,162,0.243,150
education/grad-schools/law-part-time,2015,University of Connecticut,"Hartford, CT",6,false,68,2.8,154,160,0.361,115
education/grad-schools/law-part-time,2015,Southern Methodist University (Dedman),"Dallas, TX",7,true,65,2.6,153,161,0.186,228
education/grad-schools/law-part-time,2015,Temple University (Beasley),"Philadelphia, PA",7,true,65,2.7,155,161,0.313,160
education/grad-schools/law-part-time,2015,Lewis & Clark College (Northwestern),"Portland, OR",9,false,63,2.4,153,161,0.495,188
education/grad-schools/law-part-time,2015,University of Denver (Sturm),"Denver, CO",10,false,62,2.9,151,160,0.496,138
education/grad-schools/law-part-time,2015,Loyola Marymount University,"Los Angeles, CA",11,true,60,2.5,154,160,0.169,194
education/grad-schools/law-part-time,2015,University of Houston,"Houston, TX",11,true,60,2.6,151,159,0.213,117
education/grad-schools/law-part-time,2015,University of San Diego,"San Diego, CA",13,false,59,2.6,154,162,0.391,108
education/grad-schools/law-part-time,2015,Georgia State University,"Atlanta, GA",14,true,57,2.4,153,159,0.259,192
education/grad-schools/law-part-time,2015,Illinois Institute of Technology (Chicago-Kent),"Chicago, IL",14,true,57,2.6,152,160,0.423,107
education/grad-schools/law-part-time,2015,American University (Washington),"Washington, DC",16,false,55,2.8,152,157,0.436,249
education/grad-schools/law-part-time,2015,Loyola University Chicago,"Chicago, IL",17,true,53,2.5,150,159,0.376,68
education/grad-schools/law-part-time,2015,Marquette University,"Milwaukee, WI",17,true,53,2.4,150,161,0.39,93
education/grad-schools/law-part-time,2015,Brooklyn Law School,"Brooklyn, NY",19,true,52,2.6,152,159,0.323,238
education/grad-schools/law-part-time,2015,"Rutgers, The State University of New JerseyNewark","Newark, NJ",19,true,52,2.5,150,157,0.324,171
education/grad-schools/law-part-time,2015,William Mitchell College of Law,"St. Paul, MN",21,false,51,2.1,148,159,0.34,210
education/grad-schools/law-part-time,2015,University of NevadaLas Vegas,"Las Vegas, NV",22,false,50,2.3,152,157,0.291,106
education/grad-schools/law-part-time,2015,"Rutgers, The State University of New JerseyCamden","Camden, NJ",23,true,49,2.4,152,156,0.22,85
education/grad-schools/law-part-time,2015,Seattle University,"Seattle, WA",23,true,49,2.4,150,158,0.524,163
education/grad-schools/law-part-time,2015,University of Baltimore,"Baltimore, MD",23,true,49,2.2,149,154,0.45,297
education/grad-schools/law-part-time,2015,Indiana UniversityIndianapolis (McKinney),"Indianapolis, IN",26,true,48,2.4,147,156,0.735,326
education/grad-schools/law-part-time,2015,Seton Hall University,"Newark, NJ",26,true,48,2.3,148,156,0.356,190
education/grad-schools/law-part-time,2015,The Catholic University of America,"Washington, DC",26,true,48,2.2,150,155,0.438,164
education/grad-schools/law-part-time,2015,Stetson University,"Gulfport, FL",29,false,47,2.1,151,158,0.337,198
education/grad-schools/law-part-time,2015,University of HawaiiManoa (Richardson),"Honolulu, HI",30,false,46,2.3,150,159,0.438,68
education/grad-schools/law-part-time,2015,St. Louis University,"St. Louis, MO",31,true,45,2.3,150,155,0.451,72
education/grad-schools/law-part-time,2015,Texas A&M; University,"Fort Worth, TX",31,true,45,2.0,151,155,0.269,244
education/grad-schools/law-part-time,2015,New York Law School,"New York, NY",33,false,44,2.0,147,152,0.441,308
education/grad-schools/law-part-time,2015,Southwestern Law School,"Los Angeles, CA",34,false,43,2.0,148,153,0.523,361
education/grad-schools/law-part-time,2015,DePaul University,"Chicago, IL",35,true,42,2.3,148,155,0.477,120
education/grad-schools/law-part-time,2015,Santa Clara University,"Santa Clara, CA",35,true,42,2.5,155,160,0.412,165
education/grad-schools/law-part-time,2015,St. John's University,"Jamaica, NY",35,true,42,2.1,151,158,0.289,122
education/grad-schools/law-part-time,2015,University of San Francisco,"San Francisco, CA",35,true,42,2.1,152,158,0.459,102
education/grad-schools/law-part-time,2015,University of the Pacific (McGeorge),"Sacramento, CA",35,true,42,2.1,147,154,0.726,137
education/grad-schools/law-part-time,2015,Wayne State University,"Detroit, MI",35,true,42,2.0,153,161,0.368,123
education/grad-schools/law-part-time,2015,Hofstra University (Deane),"Hempstead, NY",41,true,41,2.1,144,156,0.346,24
education/grad-schools/law-part-time,2015,Loyola University New Orleans,"New Orleans, LA",41,true,41,2.1,150,157,0.435,99
education/grad-schools/law-part-time,2015,Suffolk University,"Boston, MA",41,true,41,2.1,143,154,0.829,461
education/grad-schools/law-part-time,2015,Widener University,"Wilmington, DE",44,true,40,1.9,146,152,0.472,255
education/grad-schools/law-part-time,2015,Yeshiva University (Cardozo),"New York, NY",44,true,40,2.6,153,156,0.53,75
education/grad-schools/law-part-time,2015,Hamline University,"St. Paul, MN",46,true,38,1.9,149,157,0.631,102
education/grad-schools/law-part-time,2015,University of ArkansasLittle Rock (Bowen),"Little Rock, AR",46,true,38,2.1,146,152,0.175,118
education/grad-schools/law-part-time,2015,Duquesne University,"Pittsburgh, PA",48,true,37,1.9,149,154,0.522,114
education/grad-schools/law-part-time,2015,Michigan State University,"East Lansing, MI",48,true,37,2.3,144,148,0.521,127
education/grad-schools/law-part-time,2015,Quinnipiac University,"Hamden, CT",48,true,37,2.0,150,153,0.463,59
education/grad-schools/law-part-time,2015,University of Akron,"Akron, OH",48,true,37,1.8,148,155,0.685,170
education/grad-schools/law-part-time,2015,Florida International University,"Miami, FL",52,false,36,1.6,150,158,0.177,131
education/grad-schools/law-part-time,2015,Chapman University (Fowler),"Orange, CA",53,false,35,1.7,152,156,0.23,29
education/grad-schools/law-part-time,2015,Cleveland State University (Cleveland-Marshall),"Cleveland, OH",54,false,33,1.8,150,156,0.467,91
education/grad-schools/law-part-time,2015,Pace University,"White Plains, NY",55,true,31,1.9,148,152,0.397,64
education/grad-schools/law-part-time,2015,The John Marshall Law School,"Chicago, IL",55,true,31,1.8,145,152,0.592,303
education/grad-schools/law-part-time,2015,University of Toledo,"Toledo, OH",55,true,31,1.8,148,152,0.545,66
education/grad-schools/law-part-time,2015,Golden Gate University,"San Francisco, CA",58,false,30,1.7,145,151,0.538,148
education/grad-schools/law-part-time,2015,Oklahoma City University,"Oklahoma City, OK",59,false,29,1.7,145,150,0.737,54
education/grad-schools/law-part-time,2015,Northern Kentucky University (Chase),"Highland Heights, KY",60,false,28,1.6,145,153,0.645,136
education/grad-schools/law-part-time,2015,Capital University,"Columbus, OH",61,true,27,1.6,144,151,0.696,184
education/grad-schools/law-part-time,2015,New England Law Boston,"Boston, MA",61,true,27,1.5,148,153,0.683,255
education/grad-schools/law-part-time,2015,South Texas College of Law,"Houston, TX",61,true,27,1.6,147,152,0.511,266
education/grad-schools/law-part-time,2015,University of Detroit Mercy,"Detroit, MI",61,true,27,1.6,143,152,0.521,92
Loading