Support parsing HyperlocalExperiment #276

Mygod · 2025-03-30T05:39:28Z

Fixes #272.

jfberry · 2025-03-30T09:58:08Z

Thanks for your contribution. I've got some questions about this that I hope you can help me with before any formal review

What is the purpose of storing this detail in golbat? Are the spawns captured conventionally and this is just "for information"? How would the information usefully be consumed or displayed in third party tools?
This looks like it is per map cell, what range do you get to see this in a GMO (eg only when you are on top of it, or from distance?) This will impact the number of times these values are seen and with no cache in front of it, the number of round trips to the database
Is the experiment id re-used for different activations? Do the timings get updated? How far in advance of being activated does this get seen?
Is there sense in keeping the data long term? (I note that there is no expiry on data retention)

Mygod · 2025-03-30T19:57:00Z

What is the purpose of storing this detail in golbat? Are the spawns captured conventionally and this is just "for information"? How would the information usefully be consumed or displayed in third party tools?

Spawns are captured conventionally but will be impacted by whether they are in the region or outside. In the past (during elite/t4 raid bonus), extra spawnpoints are activated inside the region and the spawn is drawn from a different pool. All the changes are reverted after the timer expires. It's like a mini-event but only for spawns (/players?) inside the circle.

ReactMap could display these circles and Poracle could probably alert the player that a tracked spawn is inside the bonus region and is subject to change after the region expires/reactivates.

This looks like it is per map cell, what range do you get to see this in a GMO (eg only when you are on top of it, or from distance?) This will impact the number of times these values are seen and with no cache in front of it, the number of round trips to the database

This could be in-memory only like Pokemon since they are all short lived. No idea about distance.

Is the experiment id re-used for different activations? Do the timings get updated? How far in advance of being activated does this get seen?

ID (3578) + challenge key (spawn_wild_bug_types) + event radius (100m) is the same across all rows for this event and I suspect that this would continue to be the case. I imagine experiment ID is actually a global value that is also used for global event toggles as well and challenge key is probably the key for the in-game bonus text displayed to the player.

Is there sense in keeping the data long term? (I note that there is no expiry on data retention)

No. This should be treated like Pokemon table.

jfberry · 2025-03-30T20:04:36Z

On the first point, I can see it being useful if we can determine the change time (either it's going to change, or it's going to change back). The "going to change" might be difficult for us to determine although it's likely s2 related rather than exactly circular. I wonder if there is anything on the spawn that indicates it is associated with the experiment that would allow correlation

Mygod · 2025-03-30T20:12:27Z

I don't think there is but I haven't checked, and I believe it could be exactly circular given the small radius (I think the biggest radius was 300m?).

Mygod · 2025-07-08T05:39:13Z

Can we merge this now that we have initial ReactMap support? :)

WatWowMap/ReactMap#1116

Mygod · 2025-07-13T19:24:52Z

Now that we have caching and cleanup, can we merge? 👀

jfberry · 2025-07-13T19:26:13Z

I will review tomorrow and give comments. It will need community testing and verification also before merge

Mygod · 2025-07-14T08:18:17Z

Unfortunately, testing will be difficult though. There are no future announced events (yet) that use this feature, and the last one was three months ago.

jfberry · 2025-07-14T08:19:04Z

It can wait until then. If there are no more it isn’t relevant anyway :-)

jfberry · 2025-07-15T10:34:45Z

decoder/hyperlocal.go

+	Lon               float64 `db:"lon" json:"lon"`
+	RadiusM           float64 `db:"radius_m" json:"radius_m"`
+	ChallengeBonusKey string  `db:"challenge_bonus_key" json:"challenge_bonus_key"`
+	UpdatedMs         int64   `db:"updated_ms" json:"updated_ms"`


Interested in others thoughts about moving updated to a ms field (where others are not). Consistency might be more important than other concerns (I'd be willing to consider a PR moving to a virtual updated and every other table being updated to ms as the per second resolution annoys me too)

I think it is fine to just do this instead to keep compatibility. I imagine this would break compatibility for a lot of other things if we introduce this to older tables though?

You would use a virtual column (as a calculated field) to maintain the old name. Anyway, for the time being we should change this to updated

Why? This is a new table and RM PR is already implemented for this. Do you mean we should add a virtual updated column for this instead?

Also since the column explicitly gives the unit (which is actually a convenient consequence of following the same naming convention as proto), I think it's fine to just keep this as is.

jfberry · 2025-07-15T10:36:56Z

decoder/hyperlocal.go

+func (h *Hyperlocal) getKey() HyperlocalKey {
+	return HyperlocalKey{
+		ExperimentId: h.ExperimentId,
+		Lat:          h.Lat,


Urgh. I understand there may be no unique identifier (unusual for niantic though this is!) but I wonder if lat/lon as a float works reliably given the round-trip to database (I doubt it will be guaranteed byte-exact). We may be forced for the key to move to an integer representing a fixed precision - but I haven't seen the resolution of the data that comes from niantic for this - how many decimal places is it?

I think it should be exact. From my data gathered three months ago, these fields are always truncated to 1e-6 (ignoring floating point rounding), e.g. 20.681438,-156.442944.

Full row: 3578,1743375172270,1743376072270,20.681438,-156.442944,100.00000000000001,spawn_wild_bug_types,1743376070600

I guess if testing turns out to be problematic, we can store (int)(1e6*lat) instead. (It didn't seem problematic in the data I collected.)

jfberry · 2025-07-15T10:37:51Z

decoder/hyperlocal.go

+		old.EndMs != new.EndMs ||
+		old.RadiusM != new.RadiusM ||
+		old.ChallengeBonusKey != new.ChallengeBonusKey ||
+		old.UpdatedMs < new.UpdatedMs


this is not considering lat/lon - there is a method for nearby precision comparison of floats
updated wouldn't normally be considered here I don't think

They aren't considered because they are primary keys. :)

jfberry · 2025-07-15T10:40:05Z

decoder/main.go

+			Lon:          raw.Data.GetLngDegrees(),
+		}
+
+		hyperlocalMutex, _ := hyperlocalStripedMutex.GetLock(uint64(key.ExperimentId) ^ math.Float64bits(key.Lat) ^ math.Float64bits(key.Lon))


I wonder if the hash would be better as a method on the key - ... I think the stripedmutex just calls the standard hashing interface from memory

stripedmutex takes a string key so this is more performant.

jfberry · 2025-07-15T10:40:21Z

decoder/pokestop.go

 				infoData["pokemon_id"] = int(info.GetPokemonId())
 			}
-			if info.ShinyProbability > 0.0 {
+			if info.ShinyProbability != 0.0 {


why? equality is generally best avoided with floats (though 0 generally works as is a special float)

These are unlikely to be a result of a floating point math calculation so I think precise comparison is fine. Also the old code is comparing with 0 as well instead of something like > 1e-12.

And yes 0.0 is a special float in proto, indicating default value.

jfberry · 2025-07-15T10:44:36Z

sql/48_hyperlocal.up.sql

+ PRIMARY KEY(`experiment_id`,`lat`,`lon`),
+ KEY `ix_end_ms` (`end_ms`),
+ KEY `ix_updated_ms` (`updated_ms`)
+) ENGINE = InnoDB


We don't normally force engine (some people use alternates), charset or collate (forcing and not using database default gives problems on a join)
Suggest that you probably need an index on lat,lon,end_ms for the standard map type query
and don't need ix_updated_ms as this won't be used anywhere
the end_ms is likely used to expire

I think I copied this ENGINE part from a similar sql migration script.

Just following existing codebase.
Updated keys/indices as suggested.

I will PR the removal of these, this is a mistake

Support parsing HyperlocalExperiment

6f1c351

Mygod mentioned this pull request Jul 8, 2025

feat: bonus regions via vibe coding WatWowMap/ReactMap#1116

Open

Mygod added 3 commits July 11, 2025 18:09

Merge branch 'main' into hyperlocal

899f539

Rename sql

0f9ad78

Implement cache and expiry

8096afc

jfberry reviewed Jul 15, 2025

View reviewed changes

Update indexes

9423feb

Support parsing HyperlocalExperiment #276

Are you sure you want to change the base?

Support parsing HyperlocalExperiment #276

Uh oh!

Conversation

Mygod commented Mar 30, 2025

Uh oh!

jfberry commented Mar 30, 2025

Uh oh!

Mygod commented Mar 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfberry commented Mar 30, 2025

Uh oh!

Mygod commented Mar 30, 2025

Uh oh!

Mygod commented Jul 8, 2025

Uh oh!

Mygod commented Jul 13, 2025

Uh oh!

jfberry commented Jul 13, 2025

Uh oh!

Mygod commented Jul 14, 2025

Uh oh!

jfberry commented Jul 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mygod Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Mygod commented Mar 30, 2025 •

edited

Loading

Mygod Jul 15, 2025 •

edited

Loading