Looking for advice syncing multiple installs via API.

[allenderj]

I host Netbox in a public cloud and query it to pull out tenant data associated with specific prefixes to encrich log data in our SIEM. Even with a local in mem key=value cache this creates a lot of requests to my cloud hosted netbox install. I though about having a local copy of netbox that is a copy of my cloud version to query against instead. This would get rid of the WAN network latency per request and keep me from using all my burst credits in AWS.

The problems comes with keeping the local copy in sync with the cloud copy. I initially thought replicate the DB but I would have to make a large amount of network changes to get this to work. Then I though maybe I can use pynetbox to sync the two across each instances api.

My thought process is query the cloud version to get.all() per endpoint and run it through dict() and somehow load that data into a bulk create or update against the local copy. I am probably overlooking so much but I want to see if the is a recommended approach before I start spinning by wheels or if someone generally just has a suggestion on how to accomplish this.

[markkuleinio]

I'm curious about your setup as you mentioned that your app is already using in-mem cache but you say that doesn't really help. Can you improve that part of your setup?

I'm having similar situation where the cloud-hosted NetBox is about 10 ms away, and I don't want to hammer it with single requests (if avoidable). For now I've had good enough solution where the app logic fetches more than one object at a time (for example, all prefixes of a single tenant, or pre-fetches a longer list of interfaces instead of requesting interfaces one-by-one) and keep that data in memory.

My performance-sensitive apps are generally batch runs, so they can build their own in-memory tables during their runs, and use that data if already fetched from NetBox. The runs are also short enough so that I don't have to care about the NetBox data possibly changing during the runs.

[allenderj]

The in-mem cache is built into the SIEM. That only parameters I can adjust are the cache retention length and number of entries it can hold. This has a potential to explode in size because the key is an ip address and the value is the ternate associated with the subnet that address is part of. This can't be batched out because the logs are streamed in 24hours a day since its reflecting live network data. I imagine I can come up with something that queries the data unfiltered from NETBOX on a schedule and create some simple routes with flask or something to parse through the returned data in memory, but I am not looking for another internal tool to manage right now.

I did some more digging and its looking like if I do postgres replication I won't actually have to make any network changes as I initially thought so that might be what I do. I would then set the local copy to maintenance mode so its readonly and point it at the readonly replica of postgres.

[candlerb]

I think Postgres-level replication is the way to go here, even if it means manually shipping the WALs, or finding some postgres tool which can upload the WALs into an S3 bucket or suchlike. (Googling turns up wal-e, which describes itself as 'obsolete' but links to other more modern tools)

Netbox's circular dependencies make it very hard to replicate any other way. For example, a Device points to two IP address objects for primary_ip4 and primary_ip6, but each IP address references an Interface, and the Interface references the Device. The REST API does not provide a way to create all these objects simultaneously within one transaction, so it's necessary to create them partially and then patch them up afterwards. More importantly, the REST API doesn't let you choose the ID of a new object - it's always allocated from a sequence - which forces you to do SQL-level replication, as there is no other unique ID for every object.

An alternative approach would be to work out what data you want to query, do some periodic queries in the cloud (e.g. CSV or JSON exports) which cover the required fields, download those files and use them. But then you can't query it in the same way as you do the live instance.

[allenderj]

I think your right with setting up a read replica. Didn't think about the circular dependencies because my use case is just tenants, prefixes and ip addresses. So I would have just loaded then in order and I think that would have worked, maybe. I did notice that I couldn't specify the ID when creating an object via the API so I knew that would be a hurdle. Would have to scrub the data , fix any id issues, deal with deltas, etc. Seems like more pain than gain.

I do want to try to keep the local copy pretty close to live so WALs might be the way to go.

Currently using postgres on RDS with HA across different AZs. I am not a DBA so excuse the simple questions but is there a way to set up an additional replication target natively in postgres? I don't want to use the AWS DB migration service if I can help it. If I failover the rds pair will the new writable node continue replication with the on prem node?

[candlerb]

Sorry, I have no idea whether RDS gives you access to the WALs, or can replicate to a non-RDS instance of postgres.