Does OPA support fast loading and dynamic updating

[doaspx]

My current scenario is to do an authentication function

The data source has a large amount of data and is continuously updated (including the addition, deletion and modification of permissions);

I hope that the data source can be loaded into OPA quickly when OPA is started; And it can keep updating with the data source efficiently

I think of two ways now, but I don't think they are very perfect

1. Use bundle to load tar at one time GZ file; However, it can only update through polling. This time, there is a delay, which makes the process very inefficient; I expect to update the loaded data through the restapi, but I'm sorry to prompt "path bindings is owned by bundle " authz \ "

2. When OPA is started, 5g data is loaded into OPA through restapi; Subsequent updates through API; This disadvantage feels that a large amount of data needs to be written through the API. I don't know how the performance is; Of course, I can deploy multiple OPAs by business; The feeling is not very perfect;

I don't know if there is a good solution to load quickly and update the data in it

[anderseknert]

Hi @doaspx 👋

Delta bundles is a recently introduced feature in OPA that builds on the well-known bundle format, and extends it to work with data updates in the form of deltas, meaning you won't need to download the entire dataset each time a change is registered, but only patches of the data changed.

OPAL is another project that aims to solve this by using an additional sidecar to feed updates to OPA via its REST API rather than bundles.

[haiyunhzhang]

Hi @anderseknert, May I ask whether there is a sidecar that monitors the OPA lifecycle? For example, when OPA pod is started by Kubernetes, the sidecar reports its initial status to a policy admin server for initial document creation?

Thanks

[anderseknert]

That sounds like what you'd get from the Status API: https://www.openpolicyagent.org/docs/latest/management-status/

[anderseknert]

May I ask does it include the case when OPA started without Bundles, to just let control plain know that it is ready for REST API to configure it.?

I believe so, yes, but probably easiest to try it out in order to verify :)

My next question is "Does OPA support multiple login user account, e.g. one for read only user and the other for admin user, that is the only user to change policy data and policy rules?" Or even some finer access control to OPA service itself in addition to provide RBAC control to client services?

Yes! See https://www.openpolicyagent.org/docs/latest/security/#authentication-and-authorization

You could for example have a policy that allows a user that identifies with token X to do some operations, and users that identify with token Y other operations.

[haiyunhzhang]

Thank you and it helps

[doaspx]

Thank you very much for your reply

I have a little problem

On my side, I prefer to use API. In my own reality, sidecar is called through rest API;

Does restapi support batch? What is the problem with multiple goroutine writes? What is the maximum limit? What is the maximum write data of a single restapi?

---

非常感谢您的回复

我还有个小问题

我这面还是比较倾向使用API，自己现实，sidecar 通过 REST API 调用；

RestAPI 支持批量吗？如果多goroutine 写入会有什么问题，最大限制是多少呢？单个RestAPI 最大写入数据是多少呢？

[anderseknert]

The REST API patch operation on the data API is described in the OPA docs.

Patches are sent in JSON patch format, and as such may contain as many patches as you want, but I'm not sure what you have in mind when you say "batch". JSON patch is serial by design, so patches will be applied from top to bottom of the list. For that reason, I'm not sure how concurrent writes would work, as you could end up with potentially conflictting operations, i.e. where one write says "delete"and another write says "add"... and you'd have no way of knowing which one was first unless they were ordered.

[doaspx]

Thank you very much for your reply

I understand that sidecar can call API to update OPA by directly subscribing to publication;

If we adopt patches, we need to maintain more "*. Tar. GZ" files;

It feels less flexible than calling API

[doaspx]

If the data is loaded through the bundle and can be modified through the API, it can meet my demand scenario,

I'd like to ask why we prohibit API from modifying bundle data？？

ps：

{
"code": "invalid_parameter",
"message": "all paths owned by bundle "authz""
}

[anderseknert]

Hi again! Some discussion on that topic here: open-policy-agent/opa#3138

[doaspx]

Hi，
Can concurrent calls "Push Data" ? Can the data put method be invoked concurrently? Will concurrency occur when data is written to OPA?

Option 4: Push Data
Another way to replicate external data in its entirety into OPA is to use OPA’s API for injecting arbitrary JSON data. You can build a replicator that pulls information out of the external data source and pushes that information in OPA through its API. This approach is similar in most respects to the bundle API, except it lets you optimize for update latency and network traffic.

for example：

--header 'Content-Type: application/json' \
--data-raw '{
    "user_roles": {
        "alice": [
            "admin"
        ],
        "bob": [
            "employee",
            "billing"
        ],
        "eve": [
            "customer"
        ]
    },
    "role_grants": {
        "customer": [
            {
                "action": "read",
                "type": "dog"
            },
            {
                "action": "read",
                "type": "cat"
            },
            {
                "action": "adopt",
                "type": "dog"
            },
            {
                "action": "adopt",
                "type": "cat"
            }
        ],
        "employee": [
            {
                "action": "read",
                "type": "dog"
            },
            {
                "action": "read",
                "type": "cat"
            },
            {
                "action": "update",
                "type": "dog"
            },
            {
                "action": "update",
                "type": "cat"
            }
        ],
        "billing": [
            {
                "action": "read",
                "type": "finance"
            },
            {
                "action": "update",
                "type": "finance"
            }
        ]
    }
}'```

[srenatus]

Can concurrent calls "Push Data" ? Can the data put method be invoked concurrently? Will concurrency occur when data is written to OPA?

Yes, concurrently pushing data is OK. The server will only process one write transaction at a time; and the write will block until there are no ongoing read transactions to ensure consistency. Any reads will then be blocked until the write has finished.

There's no MVCC or anything like it built into OPA, however. If two "concurrent" PUT requests end up writing the same piece of data, the writes will be executed sequentially and the last one "wins".

[anderseknert]

Moved from issue to the OPA discussion board.