Skip to content

Commit 513ad5c

Browse files
committed
Docsplit 0.5.2
1 parent 6fbb1b3 commit 513ad5c

File tree

9 files changed

+76
-40
lines changed

9 files changed

+76
-40
lines changed

docsplit.gemspec

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
Gem::Specification.new do |s|
22
s.name = 'docsplit'
3-
s.version = '0.5.1' # Keep version in sync with docsplit.rb
4-
s.date = '2010-04-26'
3+
s.version = '0.5.2' # Keep version in sync with docsplit.rb
4+
s.date = '2011-05-13'
55

66
s.homepage = "http://documentcloud.github.com/docsplit/"
77
s.summary = "Break Apart Documents into Images, Text, Pages and PDFs"
@@ -15,7 +15,6 @@ Gem::Specification.new do |s|
1515
s.authors = ['Jeremy Ashkenas', 'Samuel Clay']
1616
s.email = '[email protected]'
1717
s.rubyforge_project = 'docsplit'
18-
s.has_rdoc = false
1918

2019
s.require_paths = ['lib']
2120
s.executables = ['docsplit']

index.html

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -281,6 +281,13 @@ <h2 id="internals">Internals</h2>
281281

282282
<h2 id="changes">Change Log</h2>
283283

284+
<p>
285+
<b class="header">0.5.2</b><small> &ndash; May 13, 2011</small><br />
286+
For transparent conversion to PDF, made Docsplit prefer GraphicsMagick
287+
over OpenOffice, when the file format is one that GraphicsMagick is able
288+
to read: (png, gif, jpg, jpeg, tif, tiff, bmp, pnm, ppm, svg, eps).
289+
</p>
290+
284291
<p>
285292
<b class="header">0.5.1</b><small> &ndash; April 26, 2011</small><br />
286293
Minor tweaks to the <tt>TextCleaner</tt> to be more lenient about acryonms

lib/docsplit.rb

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# The Docsplit module delegates to the Java PDF extractors.
22
module Docsplit
33

4-
VERSION = '0.5.1' # Keep in sync with gemspec.
4+
VERSION = '0.5.2' # Keep in sync with gemspec.
55

66
ROOT = File.expand_path(File.dirname(__FILE__) + '/..')
77

@@ -14,6 +14,8 @@ module Docsplit
1414
OFFICE = RUBY_PLATFORM.match(/darwin/i) ? '' : '-Doffice.home=/usr/lib/openoffice'
1515

1616
METADATA_KEYS = [:author, :date, :creator, :keywords, :producer, :subject, :title, :length]
17+
18+
GM_FORMATS = [:png, :gif, :jpg, :jpeg, :tif, :tiff, :bmp, :pnm, :ppm, :svg, :eps]
1719

1820
DEPENDENCIES = {:java => false, :gm => false, :pdftotext => false, :pdftk => false, :tesseract => false}
1921

@@ -53,11 +55,19 @@ def self.extract_images(pdfs, opts={})
5355
end
5456

5557
# Use JODCConverter to extract the documents as PDFs.
58+
# If the document is in an image format, use GraphicsMagick to extract the PDF.
5659
def self.extract_pdf(docs, opts={})
60+
out = opts[:output] || '.'
61+
FileUtils.mkdir_p out unless File.exists?(out)
5762
[docs].flatten.each do |doc|
58-
basename = File.basename(doc, File.extname(doc))
59-
options = "-jar #{ROOT}/vendor/jodconverter/jodconverter-core-3.0-beta-3.jar -r #{ROOT}/vendor/conf/document-formats.js"
60-
run "#{options} \"#{doc}\" \"#{opts[:output] || '.'}/#{basename}.pdf\"", [], {}
63+
ext = File.extname(doc)
64+
basename = File.basename(doc, ext)
65+
if GM_FORMATS.include?(ext.sub(/^\./, '').downcase.to_sym)
66+
`gm convert "#{doc}" "#{out}/#{basename}.pdf"`
67+
else
68+
options = "-jar #{ROOT}/vendor/jodconverter/jodconverter-core-3.0-beta-3.jar -r #{ROOT}/vendor/conf/document-formats.js"
69+
run "#{options} \"#{doc}\" \"#{out}/#{basename}.pdf\"", [], {}
70+
end
6171
end
6272
end
6373

lib/docsplit/text_cleaner.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
require 'iconv'
21
require 'strscan'
32

43
module Docsplit
@@ -36,6 +35,7 @@ class TextCleaner
3635
# For the time being, `clean` uses the regular StringScanner, and not the
3736
# multibyte-aware version, coercing to ASCII first.
3837
def clean(text)
38+
require 'iconv' unless defined?(Iconv)
3939
text = Iconv.iconv('ascii//translit//ignore', 'utf-8', text).first
4040
scanner = StringScanner.new(text)
4141
cleaned = []

test/fixtures/corrosion/corrosion_1.txt

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,25 @@
1-
2-
U.S. Deponmem 901 rachel Street, suns 452
1+
(R)
2+
U.S. DSDCIITITISDT 901 Locust Street, Suite 462
33
of Tronsportotion Kansas City, Mo 64106-2641
4-
Pipeline una
4+
Pipeline and
55
Hazardous Materials Safety
66
Administration
77
WARNING LETTER
8-
CERTIFIED MAIL RETURN RECEIPT QUESTED
8+
CERTIFIED MAIL - RETURN RECEIPT REQUESTED
99
January 21, 2010
1010
Mr. Terry McGill, President
11-
Enbridge Energy Paitners,
11+
Enbridge Energy Partners,
1212
1100 Louisiana, Suite 3300
1313
Houston, Texas 77002
14-
. CPF
14+
CPF 3-2010-S002W
1515
Dear Mr. McGill:
1616
On October 6-8, 2008, October 28, 2008, and January 21-22, 2009, a representative of the
1717
Pipeline and Hazardous Materials Safety Administration (PHMSA) pursuant to Chapter 601 of
1818
49 United States Code inspected your facilities associated with the Griffith Unit in Griffith,
19-
Indiana, and surrounding locations
19+
Indiana, and surrounding locations.
2020
As a result ofthe inspection, it appears that you have committed a probable violation of the
2121
Pipeline Safety Regulations, Title 49, Code of Federal Regulations. The items inspected and
22-
the probable violation(s) are:
22+
the probable violation(s) are:
2323
1. 195.579 What must I do to mitigate internal corrosion?
2424
Inhibitors. If you use corrosion inhibitors to mitigate internal corrosion, you
2525
must--

test/fixtures/corrosion/corrosion_2.txt

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,28 +5,28 @@ the inhibitors in mitigating internal corrosion; and
55
(3) Examine the coupons or other monitoring equipment at least twice each
66
calendar year, but with intervals not exceeding 7 1/2 months.
77
Internal corrosion monitoring was discontinued on the five hydrogen permeation monitors
8-
(Beta oils) installed on Line 6B. Two monitors were discontinued in
9-
May 2006. One remotely-interro gated monitor was discontinued in January 2006, and the
8+
(Beta oils) installed on Line Two manually-interrogated monitors were discontinued in
9+
May 2006. One remotely-interro gated monitor was discontinued in Januaiy 2006, and the
1010
other two remotely-interro gated monitors were discontinued in October 2007. Enbridge
1111
representatives stated the monitoring was discontinued due to
1212
"communication/instrumentation problems."
1313
Enbridge is in the process of implementing an alternative method of internal corrosion
1414
monitoring on Line 6B utilizing a technology referred to as Electrical Resistance Tomography
15-
(FSMPIT), however, it is not expected to be implemented on Line 6B until sometime during
16-
the first half of 2010. In the interim, Enbridge provided the following information as
15+
(FSM-IT), however, it is not expected to be implemented on Line 6B until sometime during
16+
the iirst half of 2010. ln the interim, Enbridge provided the following information as
1717
demonstration that the internal corrosion threat is being properly managed:
18-
a comprehensive report related to the internal corrosion mitigation and
18+
0 a comprehensive report related to the internal corrosion mitigation and
1919
monitoring program for their heavy oil pipeline system
20-
repair sleeve installations (which require circumferential non-destructive
20+
^0 repair sleeve installations (which require circumferential non-destructive
2121
testing)
22-
inspection ofthe Line 6B Pig Sending Trap at Griffith Station (which included
22+
inspection of the Line 6B Pig Sending Trap at Griffith Station (which included
2323
ultrasonic inspection of the trap floor between the 5:00 and 7:00 positions)
24-
detailed pipe examinations at in-line inspection indications
24+
9 detailed pipe examinations at in-line inspection indications
2525
records for a weight loss coupon at the Stockbridge Ptunping Station (Line 17),
2626
which sees only fluid flow from Line 6B
2727
The information provided does not demonstrate compliance with the above regulation. Line
28-
6B has been subject to a batch chemical treatment program to inhibit internal corrosion for
29-
several years. As required by Line 6B must have coupons or other monitoring
28+
6B has been subject to a batch chemical treatment program to inhibit internal corrosion for
29+
several years, As required by Line 6B must have coupons or other monitoring
3030
equipment to determine the effectiveness of the inhibitor program, and the coupons or other
3131
monitoring equipment nlust be examined at least twice each calendar year, at intervals not to
3232
exceed 7-l/2 months. PHMSA acknowledges the positive steps being taken to improve

test/fixtures/corrosion/corrosion_3.txt

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,28 @@
1-
Promote Cultural Diplomacy: American artists, performers and thinkers – representing our values and ideals – can inspire people both at home and all over the world. Through efforts like that of the United States Information Agency, America’s cultural leaders were deployed around the world during the Cold War as artistic ambassadors and helped win the war of ideas by demonstrating to the world the promise of America. Artists can be utilized again to help us win the war of ideas against Islamic extremism. Unfortunately, our resources for cultural diplomacy are at their lowest level in a decade. Barack Obama will work to reverse this trend and improve and expand public-private partnerships to expand cultural and arts exchanges throughout the world. Attract Foreign Talent: The flipside to promoting American arts and culture abroad is welcoming members of the foreign arts community to America. Opening America’s doors to students and professional artists provides the kind of two-way cultural understanding that can break down the barriers that feed hatred and fear. As America tightened visa restrictions after 9/11, the world’s most talented students and artists, who used to come here, went elsewhere. Barack Obama will streamline the visa process to return America to its rightful place as the world’s top destination for artists and art students. Provide Health Care to Artists: Finding affordable health coverage has often been one of the most vexing obstacles for artists and those in the creative community. Since many artists work independently or have nontraditional employment relationships, employer-based coverage is unavailable and individual policies are financially out of reach. Barack Obama’s plan will provide all Americans with quality, affordable health care. His plan includes the creation of a new public program that will allow individuals and small businesses to buy affordable health care similar to that available to federal employees. His plan also creates a National Health Insurance Exchange to reform the private insurance market and allow Americans to enroll in participating private plans, which would have to provide comprehensive benefits, issue every applicant a policy, and charge fair and stable premiums. For those who still cannot afford coverage, the government will provide a subsidy. His health plan will lower costs for the typical American family by up to $2,500 per year. Ensure Tax Fairness for Artists: Barack Obama supports the Artist-Museum Partnership Act, introduced by Senator Patrick Leahy (D-VT). The Act amends the Internal Revenue Code to allow artists to deduct the fair market value of their work, rather than just the costs of the materials, when they make charitable contributions.
1+
Promote Cultural Diplomacy: American artists, performers and thinkers – representing our values and ideals
2+
– can inspire people both at home and all over the world. Through efforts like that of the United States
3+
Information Agency, America’s cultural leaders were deployed around the world during the Cold War as artistic
4+
ambassadors and helped win the war of ideas by demonstrating to the world the promise of America. Artists
5+
can be utilized again to help us win the war of ideas against Islamic extremism. Unfortunately, our resources
6+
for cultural diplomacy are at their lowest level in a decade. Barack Obama will work to reverse this trend and
7+
improve and expand public-private partnerships to expand cultural and arts exchanges throughout the world.
8+
Attract Foreign Talent: The flipside to promoting American arts and culture abroad is welcoming members of
9+
the foreign arts community to America. Opening America’s doors to students and professional artists provides
10+
the kind of two-way cultural understanding that can break down the barriers that feed hatred and fear. As
11+
America tightened visa restrictions after 9/11, the world’s most talented students and artists, who used to come
12+
here, went elsewhere. Barack Obama will streamline the visa process to return America to its rightful place as
13+
the world’s top destination for artists and art students.
14+
Provide Health Care to Artists: Finding affordable health coverage has often been one of the most vexing
15+
obstacles for artists and those in the creative community. Since many artists work independently or have nontraditional employment relationships, employer-based coverage is unavailable and individual policies are
16+
financially out of reach. Barack Obama’s plan will provide all Americans with quality, affordable health care.
17+
His plan includes the creation of a new public program that will allow individuals and small businesses to buy
18+
affordable health care similar to that available to federal employees. His plan also creates a National Health
19+
Insurance Exchange to reform the private insurance market and allow Americans to enroll in participating
20+
private plans, which would have to provide comprehensive benefits, issue every applicant a policy, and charge
21+
fair and stable premiums. For those who still cannot afford coverage, the government will provide a subsidy.
22+
His health plan will lower costs for the typical American family by up to $2,500 per year.
23+
Ensure Tax Fairness for Artists: Barack Obama supports the Artist-Museum Partnership Act, introduced by
24+
Senator Patrick Leahy (D-VT). The Act amends the Internal Revenue Code to allow artists to deduct the fair
25+
market value of their work, rather than just the costs of the materials, when they make charitable contributions.
226

327
Paid for by Obama for America
428

test/fixtures/corrosion/corrosion_4.txt

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,15 @@
11
action or penalty assessment proceedings at this time. We advise you to correct the item
22
identified in this letter. Failure to do so will result in Enbridge being subject to additional
33
enforcement action.
4-
No reply to this letter is required. If` you choose to reply, in your correspondence please refer
5-
to CPF Be advised that all material you submit in response to this
6-
enforcement action is subject to being made publicly available. If` you believe that any portion
7-
of your responsive material qualifies for confidential treatment under 5 U.S.C, 552(b), along
8-
with the complete original document you must provide a second copy of the doctunent with the
9-
portions you believe qualify for confidential treatment redacted and an explanation of why you
10-
believe the redacted information qualifies for confidential treatment under 5 U.S.C, 552(b).
4+
No reply to this letter is required. If you choose to reply, in your correspondence please refer
5+
to CPF 3-2010-500ZW. Be advised that all material you submit in response to this
6+
enforcement action is subject to being made publicly available. If you believe that any portion
7+
of your responsive material qualifies for confidential treatment under 5 U.S.C, 5 52(b), along
8+
with the complete original document you must provide a second copy of the doctuneut With the
9+
portions you believe qualify for confidential treatment redacted and an explanation you
10+
believe the redacted information qualifies for confidential treatment under 5 U.S.C. 552(b).
1111
Sincerely,
12+
'Jawa
1213
Ivan A. Huntoon
1314
Director, Central Region
1415
Pipeline and Hazardous Materials Safety Administration

test/unit/test_extract_text.rb

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,10 @@
33

44
class ExtractTextTest < Test::Unit::TestCase
55

6-
FULL_TEXT = <<-EOS
7-
BARACK OBAMA: A CHAMPION FOR ARTS AND CULTURE
8-
Our nation’s creativity has filled the world’s libraries, museums, recital halls, movie houses, and marketplaces with works of genius. The arts embody the American spirit of self-definition. As the author of two best-selling books – Dreams from My Father and The Audacity of Hope – Barack Obama uniquely appreciates the role and value of creative expression. A PLATFORM IN SUPPORT OF THE ARTS Reinvest in Arts Education: To remain competitive in the global economy, America needs to reinvigorate the kind of creativity and innovation that has made this country great. To do so, we must nourish our children’s creative skills. In addition to giving our children the science and math skills they need to compete in the new global context, we should also encourage the ability to think creatively that comes from a meaningful arts education. Unfortunately, many school districts are cutting instructional time for art and music education. Barack Obama believes that the arts should be a central part of effective teaching and learning. The Chairman of the National Endowment for the Arts recently said “The purpose of arts education is not to produce more artists, though that is a byproduct. The real purpose of arts education is to create complete human beings capable of leading successful and productive lives in a free society.” To support greater arts education, Obama will: Expand Public/Private Partnerships Between Schools and Arts Organizations: Barack Obama will increase resources for the U.S. Department of Education’s Arts Education Model Development and Dissemination Grants, which develop public/private partnerships between schools and arts organizations. Obama will also engage the foundation and corporate community to increase support for public/private partnerships. Create an Artist Corps: Barack Obama supports the creation of an “Artists Corps” of young artists trained to work in low-income schools and their communities. Studies in Chicago have demonstrated that test scores improved faster for students enrolled in low-income schools that link arts across the curriculum than scores for students in schools lacking such programs. Publicly Champion the Importance of Arts Education: As president, Barack Obama will use the bully pulpit and the example he will set in the White House to promote the importance of arts and arts education in America. Not only is arts education indispensable for success in a rapidly changing, high skill, information economy, but studies show that arts education raises test scores in other subject areas as well. Support Increased Funding for the NEA: Over the last 15 years, government funding for the National Endowment for the Arts has been slashed from $175 million annually in 1992 to $125 million today. Barack Obama supports increased funding for the NEA, the support of which enriches schools and neighborhoods all across the nation and helps to promote the economic development of countless communities. Paid for by Obama for America
9-
EOS
10-
116
def test_paged_extraction
127
Docsplit.extract_text('test/fixtures/obama_arts.pdf', :pages => 'all', :output => OUTPUT)
138
assert Dir["#{OUTPUT}/*.txt"].length == 2
14-
assert File.read("#{OUTPUT}/obama_arts_1.txt").strip == FULL_TEXT.strip
9+
assert File.read("#{OUTPUT}/obama_arts_1.txt").match("Paid for by Obama for America")
1510
end
1611

1712
def test_page_only_extraction

0 commit comments

Comments
 (0)