How to link to pages with keywords

[rskrisel]

Hi there!

This is a follow up question to a response you shared a few years ago: #693

I'm wondering if there's a way to add a page to the PDF with links to every page where the keyword exists. So for example if I searched for the word "rbg" and I got the response that it was available on pg 1, 5, and 8. I'd like to add a page to the front of the PDF that said "rbg: 1, 5, and 8" with links to pg 1, 5, and 8. Is there a way to do this?

Hope this makes sense! Thanks for your help!

Rebecca

[JorjMcKie]

if there's a way to add a page to the PDF with links to every page where the keyword exists.

First of all, you need to design the layout of that "index page". Could look like

keyword1: 1, 5, 8
keyword2: 2, 4, 6, 20
...

Then make a new page via doc.new_page(). This will initially be appended to the document's end, but never mind, we will correct this alter.
Then insert a line for each key word like above.
So far so good. Now make links which have rectangles for each of the page numbers. Here comes some more tricky stuff:
You could search for the strings of those page numbers inside the text lines you just inserted.

for pno in page_numbers:
    pno_rects = page.search_for(str(pno))
    for rect in pno_rects:
        link = {"kind": fitz.LINK_GOTO, "from": tuple(rect), "to": (50, 50)}
        page.insert_link(link)

The above will point to position (50, 50) on a page where the keyword can be found. Probably not very satisfying.
If you have a list of page numbers plus respective rectangle of the keyword on that page, you can search for e.g. "keyword1" on your index page, then take that rectangle and search for the page numbers within the line of keyword1:

kw_rect = page.search_for("keyword1")[0]  # position on the index page
# extend that rect to include the full line:
kw_rect.x1 = page.rect.width
pno_rect = page.search_for(str(pno), clip=kw_rect)
# look up the rectangle of keyword1 on page pno and use that rectangle's top-left as "to" in the link

I hope you still are with me at that point ...

Once the index page is finished, move it to the desired position inside the document using doc.move_page().
Don't worry about page numbers may change by this: they do change of course, but PDF links do not use them.
So every link will still point to the right place ...

Please do not hesitate to ask if something is unclear.

[rskrisel]

Thanks for getting back to me so quickly. I will play around with these suggestions and ask follow up questions if they arise!

[rskrisel]

I designed the index page and merged it at the end of my larger pdf.

From there, I'm not sure how to get started with the code you suggested. Do I need to define page_numbers? what about pno?

How do I apply the code to the final page in the PDF?

I think I am missing a few steps!

Thanks so much for your help!

So far I have:

`import fitz

page_numbers = ("2", "67", "420", "102", "7", "8", "10", "107", "108", "461", "464", "305", "18", "19", "22",
"24", "92", "95")

doc = fitz.open("NYC_DC_public_testimony_OCR_no_maps_final_index.pdf")

for pno in page_numbers:
pno_rects = page.search_for(str(pno))
for rect in pno_rects:
link = {"kind": fitz.LINK_GOTO, "from": tuple(rect), "to": (50, 50)}
page.insert_link(link)

kw_rect = page.search_for("keyword")[0] # position on the index page

extend that rect to include the full line:

kw_rect.x1 = page.rect.width
pno_rect = page.search_for(str(pno), clip=kw_rect)

look up the rectangle of keyword1 on page pno and use that rectangle's top-left as "to" in the link`

[JorjMcKie]

In principle this is the approach.

• let's say you have a dictionary kw_dict={"kw1": [(1, rect1), (5, rect2), ...], "kw2": [...], ...}.
• So kw1 appears on page 1 within rectangle rect1, on page 5 within rect2, etc. You get the point?

page = doc.new_page()  # this will be our index page
insertion_point = fitz.Point(100,72) # start inserting lines here
for kw in kw_dict.keys():
    pno_string = ", ".join([str(v[0]) for v in kw_dict[kw]])  # look like "1, 5, ..."
    page.insert_text(insertion_point, f"{kw} {pno_string}")
    insertion_point += (0,20)  # insertion of next line here
# text lines inserted, now search for page numbers for each key word
for kw in kw_dict.keys():
    kw_rect = page.search_for(kw)[0]  # kw text is within here
    kwrect.x1 = page.width  # al resp. page number are with this extended rect!
    for pno, rect in kw_dict[kw]:
        pno_rect = page.search_for(str(pno))[0]  # this is the rect for "1", "5", etc.
        link = {"kind":fitz.LINK_GOTO, "page": pno, "from":tuple(pno_rect), "to": tuple(rect.tl)}
        page.insert_link(link)
# index page finished with all links etc.
doc.move_page(page.number, 0)  # move index page to front of document

[rskrisel]

Thanks for this! I'm getting a message that says:

'Page' object has no attribute 'width' in reference to "kwrect.x1 = page.width" I looked in the documentation for pymupdf but couldn't find info on page.width.

Also, for the kw_dict, does it make sense to add rect1 as a string 'rect1'? Otherwise, I get a message that rect1 is not defined.

Sorry to bug you with all of this and thanks for your help!

import fitz

doc = fitz.open("NYC_DC_public_testimony_OCR_no_maps_final.pdf")

kw_dict={"keyword1": [(1, 'rect1'), (5, 'rect2')]}

page = doc.new_page() # this will be our index page
insertion_point = fitz.Point(100,72) # start inserting lines here
for kw in kw_dict.keys(): pno_string = ", ".join([str(v[0]) for v in kw_dict[kw]]) # look like "1, 5, ..." page.insert_text(insertion_point, f"{kw} {pno_string}") insertion_point += (0,20) # insertion of next line here
# text lines inserted, now search for page numbers for each key word
for kw in kw_dict.keys(): kw_rect = page.search_for(kw)[0] # kw text is within here kwrect.x1 = page.width # al resp. page number are with this extended rect! for pno, rect in kw_dict[kw]: pno_rect = page.search_for(str(pno))[0] # this is the rect for "1", "5", etc. link = {"kind":fitz.LINK_GOTO, "page": pno, "from":tuple(pno_rect), "to": tuple(rect.tl)} page.insert_link(link)
# index page finished with all links etc.
doc.move_page(page.number, 0) # move index page to front of document

[JorjMcKie]

oops, a typo: it should read page.rect.width!

[JorjMcKie]

make sense to add rect1 as a string 'rect1'?

No, a misconception here! You are locating key words across all pages, aren't you. When you do this (presumably via the .search_for() method) you certainly get the rectangle wrapping the key word on a specific page.
This rectangle is what I am referring to. It is not a string, but a fitz.Rect.

[rskrisel]

Ok! That makes a lot of sense! Is there a way to automatically create my keyword dictionary from the results of my keyword search? I tried piecing it together below. I see the list of keywords and pages in my output, but not saved in the PDF. Any suggestions?

Thanks again for all your help!

import fitz

search_words = ("jewish", "puerto", "jamaican", "black", "chinese", 'filipino', 'indian', 'vietnamese', 'korean', 'japanese', 
                'hawaiian', 'samoan', 'chamorro', 'islander', 'african american', 'haitian', 'nigerian', 'ethiopian', 'somali',
               'american indian', 'alaska', 'german', 'russian', 'italian', 'irish', 'lebanese', 'egyptian', 
               'hispanic', 'latino', 'spanish', 'mexic', 'cuban', 'mexican', 'mexico')
results = {}
doc = fitz.open("NYC_DC_public_testimony_OCR_no_maps_final.pdf")
for page in doc:
    words = [w[4].lower() for w in page.getText("words")]
    for sword in search_words:
        for word in words:
            if sword in word:  # search word is *part* of a word on the page
                pages = results.get(sword, set())
                pages.add(page.number)
                results[sword] = pages
kw_dict = {}
for word in results:
    result = list(map(str, results[word]))
    page_list = ", ".join(result)
    print("Word '%s' occurs on pages %s." % (word, page_list))
    
page = doc.new_page()  # this will be our index page
insertion_point = fitz.Point(100,72) # start inserting lines here
for kw in kw_dict.keys():
    pno_string = ", ".join([str(v[0]) for v in kw_dict[kw]])  # look like "1, 5, ..."
    page.insert_text(insertion_point, f"{kw} {pno_string}")
    insertion_point += (0,20)  # insertion of next line here
# text lines inserted, now search for page numbers for each key word
for kw in kw_dict.keys():
    kw_rect = page.search_for(kw)[0]  # kw text is within here
    kwrect.x1 = page.rect.width  # al resp. page number are with this extended rect!
    for pno, rect in kw_dict[kw]:
        pno_rect = page.search_for(str(pno))[0]  # this is the rect for "1", "5", etc.
        link = {"kind":fitz.LINK_GOTO, "page": pno, "from":tuple(pno_rect), "to": tuple(rect.tl)}
        page.insert_link(link)
# index page finished with all links etc.
doc.move_page(page.number, 0)  # move index page to front of document

[JorjMcKie]

Method page.getText("words") (please use the new method name get_text instead!) delivers an item like that (x0, y0, x1, y1, "word", ...) for each word. The first 4 sub-items represent the rectangle fitz.Rect(x0, y0, x1, y1) which contains the string "word".
So this is what you must use ... i.e. store together with the word text itself, when making that dictionary.
Also probably watch out if a key word occurs multiple times on the same page. Maybe you only store the rectangle of the first occurrence.

... but not saved in the PDF. Any suggestions?

You forgot to save the document 😎?

[JorjMcKie]

The code for making the dictionary could look like that:

kw_dict = {}
# when encountering a keyword "kw" on page "pno" and its rectangle "rect", do this
pagelist = kw_dict.get(kw, [])  # takes care of kw not yet in dict
if pno not in sorted([n[0] for for n in pagelist]):  this page is not yet stored for kw
    pagelist.append((pno, rect))
    kw_dict[kw] = pagelist

[rskrisel]

Okay, thanks again. Really appreciate all your time and help. I tried making adjustments to the code, and I'm getting a message that "rect" is not defined. I tried adding something like rect= fitz.Rect but that didn't work.

import fitz

kw = ("jewish", "puerto", "jamaican", "black", "chinese", 'filipino', 'indian', 'vietnamese', 'korean', 'japanese', 
                'hawaiian', 'samoan', 'chamorro', 'islander', 'african american', 'haitian', 'nigerian', 'ethiopian', 'somali',
               'american indian', 'alaska', 'german', 'russian', 'italian', 'irish', 'lebanese', 'egyptian', 
               'hispanic', 'latino', 'spanish', 'mexic', 'cuban', 'mexican', 'mexico')
results = {}
doc = fitz.open("NYC_DC_public_testimony_OCR_no_maps_final.pdf")
for page in doc:
    words = [w[4].lower() for w in page.get_text("words")]
    for sword in kw:
        for word in words:
            if sword in word:  # search word is *part* of a word on the page
                pages = results.get(sword, set())
                pages.add(page.number)
                results[sword] = pages

for word in results:
    result = list(map(str, results[word]))
    pno = ", ".join(result)
    print("Word '%s' occurs on pages %s." % (word, pno))
    
kw_dict = {}
# when encountering a keyword "kw" on page "pno" and its rectangle "rect", do this
pagelist = kw_dict.get(kw, [])  # takes care of kw not yet in dict
if pno not in sorted([n[0] for n in pagelist]):  #this page is not yet stored for kw
    pagelist.append((pno, rect))
    kw_dict[kw] = pagelist

    
page = doc.new_page()  # this will be our index page
insertion_point = fitz.Point(100,72) # start inserting lines here
for kw in kw_dict.keys():
    pno_string = ", ".join([str(v[0]) for v in kw_dict[kw]])  # look like "1, 5, ..."
    page.insert_text(insertion_point, f"{kw} {pno_string}")
    insertion_point += (0,20)  # insertion of next line here
# text lines inserted, now search for page numbers for each key word
for kw in kw_dict.keys():
    kw_rect = page.search_for(kw)[0]  # kw text is within here
    kwrect.x1 = page.rect.width  # al resp. page number are with this extended rect!
    for pno, rect in kw_dict[kw]:
        pno_rect = page.search_for(str(pno))[0]  # this is the rect for "1", "5", etc.
        link = {"kind":fitz.LINK_GOTO, "page": pno, "from":tuple(pno_rect), "to": tuple(rect.tl)}
        page.insert_link(link)
# index page finished with all links etc.
doc.move_page(page.number, 0)  # move index page to front of document

doc.save("NYC_DC_public_testimony_OCR_no_maps_final_with_index.pdf")

[JorjMcKie]

OMG another misconception!
kw_dict must of course be filled while walking through the pages! So before anything else define it kw_dict = {}.
Then right after for page in doc: define the full full_words = page.get_text("words").
Then build sublists words = [w[4] for w in words] and rects = [fitz.Rect[w[:4]] for w in words].
Then look for your keyword match and compute the index of the matching word in words. Best you use for i, word in enumerate(words): instead of for word in words:.
Then use that index (take the i if a match) to find the corresponding rectangle in rects.
Then you have all information to be stored in kw_dict.

[rskrisel]

Thanks so much for your patience. I'm a Python newbie and some of this is a bit confusing.

This is the code I have now. I think I'm probably mixing commands. Right now when I run this I get the following error: "'type' object is not subscriptable" in reference to rects = [fitz.Rect[w[:4]] for w in words]

I'm trying to figure this out but I'm having trouble. Thanks again!

import fitz

kw = ("jewish", "puerto", "jamaican", "black", "chinese", 'filipino', 'indian', 'vietnamese', 'korean', 'japanese', 
                'hawaiian', 'samoan', 'chamorro', 'islander', 'african american', 'haitian', 'nigerian', 'ethiopian', 'somali',
               'american indian', 'alaska', 'german', 'russian', 'italian', 'irish', 'lebanese', 'egyptian', 
               'hispanic', 'latino', 'spanish', 'mexic', 'cuban', 'mexican', 'mexico')
results = {}
doc = fitz.open("NYC_DC_public_testimony_OCR_no_maps_final.pdf")


kw_dict = {}
for page in doc: 
    full_words = page.get_text("words")
    words = [w[4].lower() for w in page.get_text("words")] 
    rects = [fitz.Rect[w[:4]] for w in words]
    for sword in kw:
        for i, word in enumerate(words):
            if sword in word:  # search word is *part* of a word on the page
                pages = results.get(sword, set())
                pages.add(page.number)
                results[sword] = pages
                
# when encountering a keyword "kw" on page "pno" and its rectangle "rect", do this
pagelist = kw_dict.get(kw, [])  # takes care of kw not yet in dict
if pno not in sorted([n[0] for n in pagelist]):  #this page is not yet stored for kw
    pagelist.append((pno, rect))
    kw_dict[kw] = pagelist
    

for word in results:
    result = list(map(str, results[word]))
    pno = ", ".join(result)
    print("Word '%s' occurs on pages %s." % (word, pno))
    
    
page = doc.new_page()  # this will be our index page
insertion_point = fitz.Point(100,72) # start inserting lines here
for kw in kw_dict.keys():
    pno_string = ", ".join([str(v[0]) for v in kw_dict[kw]])  # look like "1, 5, ..."
    page.insert_text(insertion_point, f"{kw} {pno_string}")
    insertion_point += (0,20)  # insertion of next line here
# text lines inserted, now search for page numbers for each key word
for kw in kw_dict.keys():
    kw_rect = page.search_for(kw)[0]  # kw text is within here
    kwrect.x1 = page.rect.width  # al resp. page number are with this extended rect!
    for pno, rect in kw_dict[kw]:
        pno_rect = page.search_for(str(pno))[0]  # this is the rect for "1", "5", etc.
        link = {"kind":fitz.LINK_GOTO, "page": pno, "from":tuple(pno_rect), "to": tuple(rect.tl)}
        page.insert_link(link)
# index page finished with all links etc.
doc.move_page(page.number, 0)  # move index page to front of document

doc.save("NYC_DC_public_testimony_OCR_no_maps_final_with_index.pdf")

[JorjMcKie]

another type: fitz.Rect(w[:4])

[rskrisel]

Ok thanks!

I'm not willing to give up because I think I'm almost there, but my script is still not working.

I am getting the following error: ValueError: could not convert string to float: 'a' I looked up why this error typically occurs, but I'm not clear on how to fix it in the script.

I'm sure I've taken up far too much of your time. Any chance you can take one final look at my sript and see if anything jumps out as incorrect?

import fitz

kw = ("jewish", "puerto", "jamaican", "black", "chinese", 'filipino', 'indian', 'vietnamese', 'korean', 'japanese', 
                'hawaiian', 'samoan', 'chamorro', 'islander', 'african american', 'haitian', 'nigerian', 'ethiopian', 'somali',
               'american indian', 'alaska', 'german', 'russian', 'italian', 'irish', 'lebanese', 'egyptian', 
               'hispanic', 'latino', 'spanish', 'mexic', 'cuban', 'mexican', 'mexico')
results = {}
doc = fitz.open("NYC_DC_public_testimony_OCR_no_maps_final.pdf")


kw_dict = {}
for page in doc: 
    full_words = page.get_text("words")
    words = [w[4].lower() for w in page.get_text("words")] 
    rects = [fitz.Rect(w[:4]) for w in words] 
    for sword in kw:
        for i, word in enumerate(words):
            if sword in word:  # search word is *part* of a word on the page
                pages = results.get(sword, set())
                pages.add(page.number)
                results[sword] = pages
                
# when encountering a keyword "kw" on page "pno" and its rectangle "rect", do this
pagelist = kw_dict.get(kw, [])  # takes care of kw not yet in dict
if pno not in sorted([n[0] for n in pagelist]):  #this page is not yet stored for kw
    pagelist.append((pno, rect))
    kw_dict[kw] = pagelist
    

for word in results:
    result = list(map(str, results[word]))
    pno = ", ".join(result)
    print("Word '%s' occurs on pages %s." % (word, pno))
    
    
page = doc.new_page()  # this will be our index page
insertion_point = fitz.Point(100,72) # start inserting lines here
for kw in kw_dict.keys():
    pno_string = ", ".join([str(v[0]) for v in kw_dict[kw]])  # look like "1, 5, ..."
    page.insert_text(insertion_point, f"{kw} {pno_string}")
    insertion_point += (0,20)  # insertion of next line here
# text lines inserted, now search for page numbers for each key word
for kw in kw_dict.keys():
    kw_rect = page.search_for(kw)[0]  # kw text is within here
    kwrect.x1 = page.rect.width  # al resp. page number are with this extended rect!
    for pno, rect in kw_dict[kw]:
        pno_rect = page.search_for(str(pno))[0]  # this is the rect for "1", "5", etc.
        link = {"kind":fitz.LINK_GOTO, "page": pno, "from":tuple(pno_rect), "to": tuple(rect.tl)}
        page.insert_link(link)
# index page finished with all links etc.
doc.move_page(page.number, 0)  # move index page to front of document

doc.save("NYC_DC_public_testimony_OCR_no_maps_final_with_index.pdf") 

[JorjMcKie]

I have taken the liberty to update your script. Seemed more efficient than the other way. Good luck!
test.zip

[rskrisel]

Thank you! I really appreciate it!

It is still getting an error. Any ideas?

[JorjMcKie]

some small changes to link creation statements
test (2).zip

[rskrisel]

it worked!!!! THANK YOU!!!

[rskrisel]

Hi...thanks again for your held and sorry to bother you with a few more questions.

When using page.insert_text is there a way to specify the end of the line as well? Right now my text is getting cut-off. I've tried page.insert_textbox, but no matter the size of the rectangle I put in, the text ends up wanting to be on one line. Any suggestions?

I'm also wondering if there's a way to make sure all instances of pages are linked not just the first time that page gets a link?

Finally, it seems as though Python considers page 1 in PDF as page 0. How do I make my results reflect the PDF page number?

Thank you!

import fitz

kw = (
    "jewish",
    "puerto",
    "jamaican",
    "black",
    "chinese",
    "filipino",
    "indian",
    "vietnamese",
    "korean",
    "japanese",
    "hawaiian",
    "samoan",
    "chamorro",
    "islander",
    "african american",
    "haitian",
    "nigerian",
    "ethiopian",
    "somali",
    "american indian",
    "alaska",
    "german",
    "russian",
    "italian",
    "irish",
    "lebanese",
    "egyptian",
    "hispanic",
    "latino",
    "spanish",
    "mexic",
    "cuban",
    "chelsea",
    "lower east side",
    "lower manhattan"
    "gramercy park", 
    "murray hill",
    "greenwich village", 
    "soho",
    "upper east side",
    "upper west side",
    "central harlem",
    "east harlem",
    "inwood",
    "washington heights", 
    "west side", 
    "tribeca", 
    "stapleton", 
    "st. george", 
    "port richmond", 
    "south shore", 
    "mid-island", 
    "high bridge",
    "morrisania", 
    "central bronx", 
    "hunts point", 
    "mott haven", 
    "bronx park", 
    "fordham", 
    "southeast bronx", 
    "kingsbridge", 
    "riverdale", 
    "northeast bronx", 
    "southeast queens", 
    "northwest queens", 
    "long island city", 
    "northwest brooklyn", 
    "flatbush",
    "borough park",
    "bushwick", 
    "williamsburg",
    "east new york", 
    "new lots",
    "southwest brooklyn",
    "greenpoint",
    "central brooklyn",
    "sunset park",
    "southern brooklyn",
    "canarsie", 
    "flatlands",
    "north queens",
    "northeast queens",
    "central queens",
    "west queens",
    "west central queens",
    "jamaica",
    "southwest queens",
    "rockaways",
)

doc = fitz.open("NYC_DC_public_testimony_OCR_no_maps_final.pdf")


results = {}
kw_dict = {}
for page in doc:
    full_words = page.get_text("words")
    words = [w[4].lower() for w in full_words]
    rects = [fitz.Rect(w[:4]) for w in full_words]
    for sword in kw:
        for i, word in enumerate(words):
            if sword in word:  # search word is *part* of a word on the page
                pages = results.get(sword, set())
                pages.add(page.number + 1)
                results[sword] = pages
                # update the keyword dictionary HERE!
                pagelist = kw_dict.get(sword, [])
                if page.number not in sorted([n[0] for n in pagelist]):
                    pagelist.append((page.number, rects[i]))
                    kw_dict[sword] = pagelist
                    


for word in results:
    result = list(map(str, results[word]))
    pno = ", ".join(result)
    print("Word '%s' occurs on pages %s." % (word, pno))


page = doc.new_page()  # this will be our index page
insertion_point = fitz.Point(20, 20)  # start inserting lines here
rect = fitz.Rect(20, 200, 600, 800)
for kw in kw_dict.keys():
    pno_string = ", ".join([str(v[0]) for v in kw_dict[kw]])  # look like "1, 5, ..."
    #page.insert_text(insertion_point, f"{kw} {pno_string}", fontname='helv')
    page.insert_textbox(rect, f"{kw} {pno_string}", fontname="Times-Roman")
    insertion_point += (0, 20)  # insertion of next line here
# text lines inserted, now search for page numbers for each key word
for kw in kw_dict.keys():
    kw_rect = page.search_for(kw)[0]  # kw text is within here
    kw_rect.x1 = page.rect.width  # al resp. page number are with this extended rect!
    for pno, rect in kw_dict[kw]:
        pno_rect = page.search_for(str(pno))[0]  # this is the rect for "1", "5", etc.
        link = {
            "kind": fitz.LINK_GOTO,
            "page": pno,
            "from": pno_rect,
            "to": rect.tl,
        }
        page.insert_link(link)
# index page finished with all links etc.
doc.move_page(page.number, 0)  # move index page to front of document

doc.save("NYC_DC_public_testimony_OCR_no_maps_final_index.pdf")

[JorjMcKie]

You can check the length that a text will take when inserted.
This obviously depends on the cosen font and font size. If you choose one of the Base-14 standard fonts (Helvetica, Times-Roman, Courier), use fitz.get_text_length(). For arbitrary fonts use the Font class, make a font object and then use font.text_length().

Instead of just inserting a text line, you can take a rectangle and use page.insert_textbox()`. This method automatically breaks the text into line (at word boundaries). Its return code is a float which informs about the success. A negative number means there was not enough room, other values mean the unused space at the rectangle's bottom.

[rskrisel]

Thank you!! That was very helpful!