Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
Version 0.9.0 (2026-02-06)
--------------------------
Extract ASN fields from ISP database (#183)
Add support for ASN database (#184)
Add anonymousIp documentation to README (#184)

Version 0.8.2 (2025-11-03)
--------------------------
Fix CI (#182)
Expand Down
84 changes: 69 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,28 @@
This is a Scala wrapper for the MaxMind [Java Geo-IP2][java-lib] library. The main benefits of using
this wrapper over directly calling the Java library from Scala are:

1. **Provides a common interface to four MaxMind databases** - it works with MaxMind's databases for
looking up geographic location, ISP, domain, and connection type from an IP address
1. **Provides a common interface to six MaxMind databases** - it works with MaxMind's databases for
looking up geographic location, ISP, ASN, domain, connection type, and anonymous IP from an IP address
2. **Better type safety** - the MaxMind Java library is somewhat null-happy. This wrapper uses
Option-boxing wherever possible
3. **Better performance** - as well as or instead of using MaxMind's own caching (`CHMCache`), you
can also configure an LRU (Least Recently Used) cache of variable size

## Installation

The latest version of scala-maxmind-iplookups is **0.8.2** and is compatible with Scala 2.13.
The latest version of scala-maxmind-iplookups is **0.9.0** and is compatible with Scala 2.13.

Add this to your SBT config:

```scala
val maxmindIpLookups = "com.snowplowanalytics" %% "scala-maxmind-iplookups" % "0.8.2"
val maxmindIpLookups = "com.snowplowanalytics" %% "scala-maxmind-iplookups" % "0.9.0"
```

Retrieve the `GeoLite2-City.mmdb` file from the [MaxMind downloads page][maxmind-downloads]
([direct link][geolitecity-dat]).

MaxMind also has databases for looking up [ISPs][maxmind-isp], [domain names][maxmind-domain], and
[connection types][maxmind-connection-type] from IP addresses. Scala MaxMind IP Lookups supports all
MaxMind also has databases for looking up [ISPs][maxmind-isp], [ASN][maxmind-asn], [domain names][maxmind-domain],
[connection types][maxmind-connection-type], and [anonymous IPs][maxmind-anonymous] from IP addresses. Scala MaxMind IP Lookups supports all
of these.

## Usage
Expand Down Expand Up @@ -102,6 +102,8 @@ final case class IpLookups(
ispFile: Option[File],
domainFile: Option[File],
connectionTypeFile: Option[File],
anonymousFile: Option[File],
asnFile: Option[File],
memCache: Boolean = true,
lruCache: Int = 10000
)
Expand All @@ -116,14 +118,16 @@ def createFromFilenames(
ispFile: Option[String],
domainFile: Option[String],
connectionTypeFile: Option[String],
anonymousFile: Option[String],
asnFile: Option[String],
memCache: Boolean = true,
lruCache: Int = 10000
)
```

The first four arguments are the MaxMind databases from which the lookup should be performed.
`geoFile`, `ispFile`, `domainFile`, and `connectionTypeFile` refer respectively to MaxMind's
databases for looking up location, ISP, domain, and connection type based on an IP address. They are
The first six arguments are the MaxMind databases from which the lookup should be performed.
`geoFile`, `ispFile`, `domainFile`, `connectionTypeFile`, `anonymousFile`, and `asnFile` refer respectively to MaxMind's
databases for looking up location, ISP, domain, connection type, anonymous IP, and ASN based on an IP address. They are
all wrapped in `Option`, so if you don't have access to all of them, just pass in `None` as in the
example above.

Expand All @@ -143,18 +147,24 @@ final case class IpLookupResult(
ipLocation: Option[Either[Throwable, IpLocation]],
isp: Option[Either[Throwable, String]],
organization: Option[Either[Throwable, String]],
asn: Option[Either[Throwable, Asn]],
domain: Option[Either[Throwable, String]],
connectionType: Option[Either[Throwable, String]]
connectionType: Option[Either[Throwable, String]],
anonymousIp: Option[Either[Throwable, AnonymousIp]]
)
```

The first element is the result of the geographic location lookup. It is either `None` (if no
geographic lookup database was provided) or `Some(ipLocation)`, where `ipLocation` is an instance of
the `IpLocation` case class described below. The other three elements in the tuple are `Option`s
wrapping the results of the other four possible lookups: ISP, organization, domain, and connection
type.
the `IpLocation` case class described below. The other elements are `Option`s wrapping the results
of the other possible lookups: ISP, organization, ASN, domain, connection type, and anonymous IP.

Note that enabling providing an ISP database will return an `organization` in addition to an `isp`.
Note that providing an ISP database will return `organization` and `asn` in addition to `isp`.
The `asn` field contains an `Asn` case class with `autonomousSystemNumber` and
`autonomousSystemOrganization` information. ASN information can be extracted from either the ISP
database or a dedicated ASN database. If both databases are provided, the ISP database takes
precedence. If the ISP database is provided but doesn't contain ASN information for a given IP,
the lookup will fall back to the ASN database (if provided).

### IpLocation case class

Expand All @@ -178,9 +188,41 @@ final case class IpLocation(
)
```

### Asn case class

The ASN lookup (extracted from either the ISP or ASN database) returns an `Asn` case class instance with the
following structure:

```scala
final case class Asn(
autonomousSystemNumber: Long,
autonomousSystemOrganization: Option[String]
)
```

Note that `autonomousSystemNumber` is a required field. If an ASN database entry doesn't contain a valid
ASN number, the lookup will return an error instead of an `Asn` instance.

### AnonymousIp case class

The anonymous IP lookup returns an `AnonymousIp` case class instance with the following structure:

```scala
final case class AnonymousIp(
ipAddress: String,
isAnonymous: Boolean,
isAnonymousVpn: Boolean,
isHostingProvider: Boolean,
isPublicProxy: Boolean,
isTorExitNode: Boolean
)
```

This lookup provides information about whether an IP address is associated with various types of anonymous or proxy services.

### An example using multiple databases

This example shows how to do a lookup using all four databases.
This example shows how to do a lookup using all six databases.

```scala
import com.snowplowanalytics.maxmind.iplookups.IpLookups
Expand All @@ -191,6 +233,8 @@ val lookupResult = (for {
ispFile = Some("/opt/maxmind/GeoIP2-ISP.mmdb"),
domainFile = Some("/opt/maxmind/GeoIP2-Domain.mmdb"),
connectionType = Some("/opt/maxmind/GeoIP2-Connection-Type.mmdb"),
anonymousFile = Some("/opt/maxmind/GeoIP2-Anonymous-IP.mmdb"),
asnFile = Some("/opt/maxmind/GeoLite2-ASN.mmdb"),
memCache = false,
lruCache = 10000
)
Expand All @@ -207,11 +251,19 @@ println(lookupResult.isp) // => Some(Right("FDN Communications"))
// Organization lookup
println(lookupResult.organization) // => Some(Right("DSLAM WAN Allocation"))

// ASN lookup
println(lookupResult.asn.map(_.autonomousSystemNumber)) // => Some(Right(123))
println(lookupResult.asn.map(_.autonomousSystemOrganization)) // => Some(Right(Some("Example ISP")))

// Domain lookup
println(lookupResult.domain) // => Some(Right("nuvox.net"))

// Connection type lookup
println(lookupResult.connectionType) // => Some(Right("Dialup"))

// Anonymous IP lookup
println(lookupResult.anonymousIp.map(_.isAnonymous)) // => Some(Right(false))
println(lookupResult.anonymousIp.map(_.isTorExitNode)) // => Some(Right(false))
```

### LRU cache
Expand Down Expand Up @@ -260,8 +312,10 @@ limitations under the License.

[maxmind-downloads]: https://dev.maxmind.com/geoip/geoip2/downloadable/#MaxMind_APIs
[maxmind-isp]: https://www.maxmind.com/en/geoip2-isp-database
[maxmind-asn]: https://dev.maxmind.com/geoip/docs/databases/asn
[maxmind-domain]: https://www.maxmind.com/en/geoip2-domain-name-database
[maxmind-connection-type]: https://www.maxmind.com/en/geoip2-connection-type-database
[maxmind-anonymous]: https://www.maxmind.com/en/geoip2-anonymous-ip-database
[geolitecity-dat]: http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz

[license]: http://www.apache.org/licenses/LICENSE-2.0
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ sealed trait CreateIpLookups[F[_]] {
* @param domainFile Domain lookup database file
* @param connectionTypeFile Connection type lookup database file
* @param anonymousFile Anonymous lookup database file
* @param asnFile ASN lookup database file
* @param memCache Whether to use MaxMind's CHMCache
* @param lruCacheSize Maximum size of LruMap cache
*/
Expand All @@ -49,6 +50,7 @@ sealed trait CreateIpLookups[F[_]] {
domainFile: Option[File] = None,
connectionTypeFile: Option[File] = None,
anonymousFile: Option[File] = None,
asnFile: Option[File] = None,
memCache: Boolean = true,
lruCacheSize: Int = 10000
): F[IpLookups[F]]
Expand All @@ -60,6 +62,7 @@ sealed trait CreateIpLookups[F[_]] {
* @param domainFile Domain lookup database filepath
* @param connectionTypeFile Connection type lookup database filepath
* @param anonymousFile Anonymous lookup database filepath
* @param asnFile ASN lookup database filepath
* @param memCache Whether to use MaxMind's CHMCache
* @param lruCacheSize Maximum size of LruMap cache
*/
Expand All @@ -69,6 +72,7 @@ sealed trait CreateIpLookups[F[_]] {
domainFile: Option[String] = None,
connectionTypeFile: Option[String] = None,
anonymousFile: Option[String] = None,
asnFile: Option[String] = None,
memCache: Boolean = true,
lruCacheSize: Int = 10000
): F[IpLookups[F]] = createFromFiles(
Expand All @@ -77,6 +81,7 @@ sealed trait CreateIpLookups[F[_]] {
domainFile.map(new File(_)),
connectionTypeFile.map(new File(_)),
anonymousFile.map(new File(_)),
asnFile.map(new File(_)),
memCache,
lruCacheSize
)
Expand All @@ -94,6 +99,7 @@ object CreateIpLookups {
domainFile: Option[File] = None,
connectionTypeFile: Option[File] = None,
anonymousFile: Option[File] = None,
asnFile: Option[File] = None,
memCache: Boolean = true,
lruCacheSize: Int = 10000
): F[IpLookups[F]] =
Expand All @@ -112,6 +118,7 @@ object CreateIpLookups {
domainFile,
connectionTypeFile,
anonymousFile,
asnFile,
memCache,
lruCache
)
Expand All @@ -128,6 +135,7 @@ object CreateIpLookups {
domainFile: Option[File] = None,
connectionTypeFile: Option[File] = None,
anonymousFile: Option[File] = None,
asnFile: Option[File] = None,
memCache: Boolean = true,
lruCacheSize: Int = 10000
): Eval[IpLookups[Eval]] =
Expand All @@ -145,6 +153,7 @@ object CreateIpLookups {
domainFile,
connectionTypeFile,
anonymousFile,
asnFile,
memCache,
lruCache
)
Expand All @@ -161,6 +170,7 @@ object CreateIpLookups {
domainFile: Option[File] = None,
connectionTypeFile: Option[File] = None,
anonymousFile: Option[File] = None,
asnFile: Option[File] = None,
memCache: Boolean = true,
lruCacheSize: Int = 10000
): Id[IpLookups[Id]] = {
Expand All @@ -176,6 +186,7 @@ object CreateIpLookups {
domainFile,
connectionTypeFile,
anonymousFile,
asnFile,
memCache,
lruCache
)
Expand All @@ -199,17 +210,18 @@ class IpLookups[F[_]: Monad] private[iplookups] (
domainFile: Option[File],
connectionTypeFile: Option[File],
anonymousFile: Option[File],
asnFile: Option[File],
memCache: Boolean,
lru: Option[LruMap[F, String, IpLookupResult]]
)(implicit SR: SpecializedReader[F], IAR: IpAddressResolver[F]) {
// Configure the lookup services
private val geoService = getService(geoFile)
private val ispService = getService(ispFile).map((_, ReaderFunctions.isp))
private val orgService = getService(ispFile).map((_, ReaderFunctions.org))
private val ispService = getService(ispFile)
private val domainService = getService(domainFile).map((_, ReaderFunctions.domain))
private val connectionTypeService =
getService(connectionTypeFile).map((_, ReaderFunctions.connectionType))
private val anonymousService = getService(anonymousFile)
private val asnService = getService(asnFile)

/**
* Get a LookupService from a database file
Expand All @@ -235,7 +247,7 @@ class IpLookups[F[_]: Monad] private[iplookups] (
*/
private def getLookup(
ipAddress: Either[Throwable, InetAddress],
service: Option[(DatabaseReader, ReaderFunction)]
service: Option[(DatabaseReader, ReaderFunction[String])]
): F[Option[Either[Throwable, String]]] =
(ipAddress, service) match {
case (Right(ipA), Some((db, f))) =>
Expand All @@ -260,7 +272,7 @@ class IpLookups[F[_]: Monad] private[iplookups] (
ipAddress: Either[Throwable, InetAddress]
): F[Option[Either[Throwable, IpLocation]]] = (ipAddress, geoService) match {
case (Right(ipA), Some(gs)) =>
SR.getCityValue(gs, ipA)
SR.getValue(ReaderFunctions.city, gs, ipA)
.map(loc => loc.map(IpLocation(_)).some)
case (Left(f), _) => Monad[F].pure(Some(Left(f)))
case _ => Monad[F].pure(None)
Expand All @@ -270,12 +282,40 @@ class IpLookups[F[_]: Monad] private[iplookups] (
ipAddress: Either[Throwable, InetAddress]
): F[Option[Either[Throwable, AnonymousIp]]] = (ipAddress, anonymousService) match {
case (Right(ipA), Some(gs)) =>
SR.getAnonymousValue(gs, ipA)
SR.getValue(ReaderFunctions.anonymousIp, gs, ipA)
.map(loc => loc.map(AnonymousIp(_)).some)
case (Left(f), _) => Monad[F].pure(Some(Left(f)))
case _ => Monad[F].pure(None)
}

private def getIspLookup(
ipAddress: Either[Throwable, InetAddress]
): F[Option[Either[Throwable, Isp]]] = (ipAddress, ispService) match {
case (Right(ipA), Some(gs)) =>
SR.getValue(ReaderFunctions.isp, gs, ipA)
.map(ispResponse => ispResponse.map(Isp(_)).some)
case (Left(f), _) => Monad[F].pure(Some(Left(f)))
case _ => Monad[F].pure(None)
}

private def getAsnLookup(
ipAddress: Either[Throwable, InetAddress],
ispResponse: Option[Either[Throwable, Isp]]
): F[Option[Either[Throwable, Asn]]] = {
val asnFromIsp = ispResponse.map(_.flatMap(_.asn))
asnFromIsp.flatMap(_.toOption) match {
case Some(_) => Monad[F].pure(asnFromIsp)
case None =>
(ipAddress, asnService) match {
case (Right(ipA), Some(gs)) =>
SR.getValue(ReaderFunctions.asn, gs, ipA)
.map(asnResponse => asnResponse.flatMap(Asn.create).some)
case (Left(f), _) => Monad[F].pure(Some(Left(f)))
case _ => Monad[F].pure(asnFromIsp)
}
}
}

/**
* This version does not use the LRU cache.
* Concurrently looks up information
Expand All @@ -288,15 +328,16 @@ class IpLookups[F[_]: Monad] private[iplookups] (
*/
private def performLookupsWithoutLruCache(ip: String): F[IpLookupResult] =
for {
ipAddress <- IAR.resolve(ip)

ipLocation <- getLocationLookup(ipAddress)
isp <- getLookup(ipAddress, ispService)
org <- getLookup(ipAddress, orgService)
ipAddress <- IAR.resolve(ip)
ipLocation <- getLocationLookup(ipAddress)
ispResponse <- getIspLookup(ipAddress)
ispName = ispResponse.map(_.map(_.name))
org = ispResponse.map(_.map(_.organization))
asn <- getAsnLookup(ipAddress, ispResponse)
domain <- getLookup(ipAddress, domainService)
connectionType <- getLookup(ipAddress, connectionTypeService)
anonymous <- getAnonymousIpLookup(ipAddress)
} yield IpLookupResult(ipLocation, isp, org, domain, connectionType, anonymous)
} yield IpLookupResult(ipLocation, ispName, org, asn, domain, connectionType, anonymous)

/**
* Returns the MaxMind location for this IP address
Expand Down
Loading
Loading