children of children seem to disappear?

[randomgambit]

Hello there,

I think I may have found a bug (if not, I apologize in advance!) that prevents me from accessing children of children.

Please consider the following example where I am trying to extract VPs by iterating over the VERB/AUX nodes, and checking whether its direct children (or the children of its children) contain the lemma spacy.

As you can see from the output, it looks like the children of the children are not accessible anymore. Is this a bug?

import spacy

# Load the small English model
nlp = spacy.load("en_core_web_lg")

text = '''
spacy is growing more and more since the beginning,
but spacy expectations have been even greater!
'''
doc = nlp(text)


mylist = []

for token in doc:
    if token.pos_ == 'VERB' or token.pos_ == 'AUX':
        print(token.text)
        #get children of verb/aux
        nodechild = token.children
        getchild1 = []
        #iterate over the children
        for child in nodechild:
            getchild1.append(child)
        getchild2 = []
        #get children of children
        for child in nodechild:
            for grandchild in child.children:
                getchild2.append(grandchild)
        print('children are ' + str(getchild1)) 
        print('grandchildren are' + str(getchild2))
        #test if the possible VP contain spacy in the children and children of children
        test1 = [tok.lemma_ == 'spacy' for tok in getchild1]
        test2 = [tok.lemma_ == 'spacy' for tok in getchild2]
        if any(test1) or any(test2):
            fulltok = token.text
            myiter = token
            #extract the VP by checking if the verb is followed by another verb or adv or adj
            while (myiter.nbor().pos_ == 'VERB') or (myiter.nbor().pos_ == 'ADV') or (myiter.nbor().pos_ == 'ADJ') or (myiter.nbor().pos_ == 'ADP'):
                fulltok = fulltok + ' '+ myiter.nbor().text
                myiter = myiter.nbor()
            mylist.append(fulltok)


is
children are []
grandchildren are[]
growing
children are [spacy, is, more, since, ,, but, been]
grandchildren are[]
have
children are []
grandchildren are[]
been
children are [expectations, have, greater, !]
grandchildren are[]            

[adrianeboyd]

No, this isn't a bug. Token.children returns a generator, not a list, so when you try to iterate over it the second time it's empty.

Most of the Doc and Token properties that can potentially end up iterating over a lot of data return generators rather than lists. But to be honest some of the details aren't quite 100% consistent in the API (Doc.ents returns a tuple while Doc.sents returns a generator), so it can be a good idea to check the docs to be sure: https://spacy.io/api/token#children