How to match keys by partial name when location isn't known?

[steve-nay-sage]

I'm working with some OpenAPI definitions that have x- properties at various locations throughout the files. For example, there could be an x-mappedTo property as a top-level property, and/or there could be other x-whatever properties in other places in the file at other levels of nesting. I need to remove those properties/entries. So I need to find all entries whose key value starts with x- regardless of where they are in the file.

• There is no way to know the property hierarchy where these will be found.
• There is no way to know the full property name (key); it's just going to start with x-.

I've attached the top part of an OpenAPI file that contains a few of these properties. It's YAML, even though the file name is .txt.

I hack around at programming and write API documentation, but I haven't been able to figure out how to do this using yq. I've tried some of the examples to get my feet wet, and some work and some don't. One that doesn't work is

yq eval 'with_entries(select(.value | has("b")))' sample.yml

Just returns an empty set when my sample.yaml file definitely contains values that have "b" in them. So I would greatly appreciate any help or insights that anyone can shar
x-test.txt
e.

[mikefarah]

You're on the right track with that, but first you have to tell it to recurse though the whole structure. So you will need to do somthing like:

yq e '.. | select(tag == "!!map") | with_entries(select(.key | match("x-.*")))' examples/data1.yaml

Explanation:

1. .. tells yq to recursively match all the nodes
2. select(tag == "!!map") we only select maps (as arrays wont have x- keys anyway
3. with_entries(select(.key | match("x-.*"))) we filter out the maps to only include the keys that start with x-

[steve-nay-sage]

Wow, thank you for that amazingly fast answer. Your answer looks so elegant, too! I was trying to make this much harder that it is. I'll try to put this to good use tomorrow.

[mikefarah]

You caught me just as I was checking to see if there was any yq action :)

[steve-nay-sage]

@mikefarah I didn't get far today. In this case I need to get values rather than delete something, so I removed the del() part of the request. That works fine on the little test file that I sent you, but when I apply it to something more like the real data that I'm working with it returns this error:

> yq '.. | select(key | test("^x-"))' file.yaml

Error: cannot match with !!int, can only match strings.

I don't see anything that looks like an int in the file, but I'm sure that I'm not reading it right. An example of the real data that I'm working with is this, and I want to be able to find x-permissions and work with what's under it:

paths:
  /objects/class:
    get:
      summary: List classes
      description: >
        Returns up to 100 object references from the collection with a key, ID,
        and link for each object. 
      x-permissions:
        subscription: Company
        userPermissions:
          - userType: Business user with admin privileges
            permissions: List, View, Add, Edit, Delete
          - userType: Employee
            permissions: List, View
          - userType: Approval
            permissions: List, View
      tags:
        - Classes

That pattern could occur several times in each file that I need to process. To work with each one separately, do I need to use a bash for loop? Of course, before getting to that part, I first need to find x-permissions in the file. This is what I have, but is it the best way?

yq '.. | select(has("x-permissions")) | .x-permissions' file.yaml

[mikefarah]

Oh my bad - forgot about arrays (which have int keys:

yq '.. | select(( key | tag == "!!str" ) and  (key | test("^x-")))' file.yaml

Yeah a bash for loop or find with xargs 👍🏼

[steve-nay-sage]

Thank you @mikefarah -- I have my script working, and I wouldn't have been able to do it without your help.