Add support for ASN database

README.md

@@ -10,8 +10,8 @@
 This is a Scala wrapper for the MaxMind [Java Geo-IP2][java-lib] library. The main benefits of using
 this wrapper over directly calling the Java library from Scala are:
 
-1. **Provides a common interface to four MaxMind databases** - it works with MaxMind's databases for
-looking up geographic location, ISP, domain, and connection type from an IP address
+1. **Provides a common interface to six MaxMind databases** - it works with MaxMind's databases for
+looking up geographic location, ISP, ASN, domain, connection type, and anonymous IP from an IP address
 2. **Better type safety** - the MaxMind Java library is somewhat null-happy. This wrapper uses
 Option-boxing wherever possible
 3. **Better performance** - as well as or instead of using MaxMind's own caching (`CHMCache`), you
@@ -30,8 +30,8 @@ val maxmindIpLookups = "com.snowplowanalytics" %% "scala-maxmind-iplookups" % "0
 Retrieve the `GeoLite2-City.mmdb` file from the [MaxMind downloads page][maxmind-downloads]
 ([direct link][geolitecity-dat]).
 
-MaxMind also has databases for looking up [ISPs][maxmind-isp], [domain names][maxmind-domain], and
-[connection types][maxmind-connection-type] from IP addresses. Scala MaxMind IP Lookups supports all
+MaxMind also has databases for looking up [ISPs][maxmind-isp], [ASN][maxmind-asn], [domain names][maxmind-domain],
+[connection types][maxmind-connection-type], and [anonymous IPs][maxmind-anonymous] from IP addresses. Scala MaxMind IP Lookups supports all
 of these.
 
 ## Usage
@@ -102,6 +102,8 @@ final case class IpLookups(
   ispFile: Option[File],
   domainFile: Option[File],
   connectionTypeFile: Option[File],
+  anonymousFile: Option[File],
+  asnFile: Option[File],
   memCache: Boolean = true,
   lruCache: Int = 10000
 )
@@ -116,14 +118,16 @@ def createFromFilenames(
   ispFile: Option[String],
   domainFile: Option[String],
   connectionTypeFile: Option[String],
+  anonymousFile: Option[String],
+  asnFile: Option[String],
   memCache: Boolean = true,
   lruCache: Int = 10000
 )
 ```
 
-The first four arguments are the MaxMind databases from which the lookup should be performed.
-`geoFile`, `ispFile`, `domainFile`, and `connectionTypeFile` refer respectively to MaxMind's
-databases for looking up location, ISP, domain, and connection type based on an IP address. They are
+The first six arguments are the MaxMind databases from which the lookup should be performed.
+`geoFile`, `ispFile`, `domainFile`, `connectionTypeFile`, `anonymousFile`, and `asnFile` refer respectively to MaxMind's
+databases for looking up location, ISP, domain, connection type, anonymous IP, and ASN based on an IP address. They are
 all wrapped in `Option`, so if you don't have access to all of them, just pass in `None` as in the
 example above.
 
@@ -145,18 +149,22 @@ final case class IpLookupResult(
   organization: Option[Either[Throwable, String]],
   asn: Option[Either[Throwable, Asn]],
   domain: Option[Either[Throwable, String]],
-  connectionType: Option[Either[Throwable, String]]
+  connectionType: Option[Either[Throwable, String]],
+  anonymousIp: Option[Either[Throwable, AnonymousIp]]
 )
 ```
 
 The first element is the result of the geographic location lookup. It is either `None` (if no
 geographic lookup database was provided) or `Some(ipLocation)`, where `ipLocation` is an instance of
 the `IpLocation` case class described below. The other elements are `Option`s wrapping the results
-of the other possible lookups: ISP, organization, ASN, domain, and connection type.
+of the other possible lookups: ISP, organization, ASN, domain, connection type, and anonymous IP.
 
 Note that providing an ISP database will return `organization` and `asn` in addition to `isp`.
 The `asn` field contains an `Asn` case class with `autonomousSystemNumber` and
-`autonomousSystemOrganization` information extracted from the ISP database.
+`autonomousSystemOrganization` information. ASN information can be extracted from either the ISP
+database or a dedicated ASN database. If both databases are provided, the ISP database takes
+precedence. If the ISP database is provided but doesn't contain ASN information for a given IP,
+the lookup will fall back to the ASN database (if provided).
 
 ### IpLocation case class
 
@@ -182,19 +190,39 @@ final case class IpLocation(
 
 ### Asn case class
 
-The ASN lookup (extracted from the ISP database) returns an `Asn` case class instance with the
+The ASN lookup (extracted from either the ISP or ASN database) returns an `Asn` case class instance with the
 following structure:
 
 ```scala
 final case class Asn(
-  autonomousSystemNumber: Option[Long],
+  autonomousSystemNumber: Long,
   autonomousSystemOrganization: Option[String]
 )
 ```
 
+Note that `autonomousSystemNumber` is a required field. If an ASN database entry doesn't contain a valid
+ASN number, the lookup will return an error instead of an `Asn` instance.
+
+### AnonymousIp case class
+
+The anonymous IP lookup returns an `AnonymousIp` case class instance with the following structure:
+
+```scala
+final case class AnonymousIp(
+  ipAddress: String,
+  isAnonymous: Boolean,
+  isAnonymousVpn: Boolean,
+  isHostingProvider: Boolean,
+  isPublicProxy: Boolean,
+  isTorExitNode: Boolean
+)
+```
+
+This lookup provides information about whether an IP address is associated with various types of anonymous or proxy services.
+
 ### An example using multiple databases
 
-This example shows how to do a lookup using all four databases.
+This example shows how to do a lookup using all six databases.
 
 ```scala
 import com.snowplowanalytics.maxmind.iplookups.IpLookups
@@ -205,6 +233,8 @@ val lookupResult = (for {
     ispFile = Some("/opt/maxmind/GeoIP2-ISP.mmdb"),
     domainFile = Some("/opt/maxmind/GeoIP2-Domain.mmdb"),
     connectionType = Some("/opt/maxmind/GeoIP2-Connection-Type.mmdb"),
+    anonymousFile = Some("/opt/maxmind/GeoIP2-Anonymous-IP.mmdb"),
+    asnFile = Some("/opt/maxmind/GeoLite2-ASN.mmdb"),
     memCache = false,
     lruCache = 10000
   )
@@ -222,14 +252,18 @@ println(lookupResult.isp) // => Some(Right("FDN Communications"))
 println(lookupResult.organization) // => Some(Right("DSLAM WAN Allocation"))
 
 // ASN lookup
-println(lookupResult.asn.map(_.autonomousSystemNumber)) // => Some(Right(Some(123)))
+println(lookupResult.asn.map(_.autonomousSystemNumber)) // => Some(Right(123))
 println(lookupResult.asn.map(_.autonomousSystemOrganization)) // => Some(Right(Some("Example ISP")))
 
 // Domain lookup
 println(lookupResult.domain) // => Some(Right("nuvox.net"))
 
 // Connection type lookup
 println(lookupResult.connectionType) // => Some(Right("Dialup"))
+
+// Anonymous IP lookup
+println(lookupResult.anonymousIp.map(_.isAnonymous)) // => Some(Right(false))
+println(lookupResult.anonymousIp.map(_.isTorExitNode)) // => Some(Right(false))
 ```
 
 ### LRU cache
@@ -278,8 +312,10 @@ limitations under the License.
 
 [maxmind-downloads]: https://dev.maxmind.com/geoip/geoip2/downloadable/#MaxMind_APIs
 [maxmind-isp]: https://www.maxmind.com/en/geoip2-isp-database
+[maxmind-asn]: https://dev.maxmind.com/geoip/docs/databases/asn
 [maxmind-domain]: https://www.maxmind.com/en/geoip2-domain-name-database
 [maxmind-connection-type]: https://www.maxmind.com/en/geoip2-connection-type-database
+[maxmind-anonymous]: https://www.maxmind.com/en/geoip2-anonymous-ip-database
 [geolitecity-dat]: http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
 
 [license]: http://www.apache.org/licenses/LICENSE-2.0

src/main/scala/com.snowplowanalytics.maxmind.iplookups/IpLookups.scala

@@ -40,6 +40,7 @@ sealed trait CreateIpLookups[F[_]] {
    * @param domainFile Domain lookup database file
    * @param connectionTypeFile Connection type lookup database file
    * @param anonymousFile Anonymous lookup database file
+   * @param asnFile ASN lookup database file
    * @param memCache Whether to use MaxMind's CHMCache
    * @param lruCacheSize Maximum size of LruMap cache
    */
@@ -49,6 +50,7 @@ sealed trait CreateIpLookups[F[_]] {
     domainFile: Option[File] = None,
     connectionTypeFile: Option[File] = None,
     anonymousFile: Option[File] = None,
+    asnFile: Option[File] = None,
     memCache: Boolean = true,
     lruCacheSize: Int = 10000
   ): F[IpLookups[F]]
@@ -60,6 +62,7 @@ sealed trait CreateIpLookups[F[_]] {
    * @param domainFile Domain lookup database filepath
    * @param connectionTypeFile Connection type lookup database filepath
    * @param anonymousFile Anonymous lookup database filepath
+   * @param asnFile ASN lookup database filepath
    * @param memCache Whether to use MaxMind's CHMCache
    * @param lruCacheSize Maximum size of LruMap cache
    */
@@ -69,6 +72,7 @@ sealed trait CreateIpLookups[F[_]] {
     domainFile: Option[String] = None,
     connectionTypeFile: Option[String] = None,
     anonymousFile: Option[String] = None,
+    asnFile: Option[String] = None,
     memCache: Boolean = true,
     lruCacheSize: Int = 10000
   ): F[IpLookups[F]] = createFromFiles(
@@ -77,6 +81,7 @@ sealed trait CreateIpLookups[F[_]] {
     domainFile.map(new File(_)),
     connectionTypeFile.map(new File(_)),
     anonymousFile.map(new File(_)),
+    asnFile.map(new File(_)),
     memCache,
     lruCacheSize
   )
@@ -94,6 +99,7 @@ object CreateIpLookups {
       domainFile: Option[File] = None,
       connectionTypeFile: Option[File] = None,
       anonymousFile: Option[File] = None,
+      asnFile: Option[File] = None,
       memCache: Boolean = true,
       lruCacheSize: Int = 10000
     ): F[IpLookups[F]] =
@@ -112,6 +118,7 @@ object CreateIpLookups {
             domainFile,
             connectionTypeFile,
             anonymousFile,
+            asnFile,
             memCache,
             lruCache
           )
@@ -128,6 +135,7 @@ object CreateIpLookups {
       domainFile: Option[File] = None,
       connectionTypeFile: Option[File] = None,
       anonymousFile: Option[File] = None,
+      asnFile: Option[File] = None,
       memCache: Boolean = true,
       lruCacheSize: Int = 10000
     ): Eval[IpLookups[Eval]] =
@@ -145,6 +153,7 @@ object CreateIpLookups {
             domainFile,
             connectionTypeFile,
             anonymousFile,
+            asnFile,
             memCache,
             lruCache
           )
@@ -161,6 +170,7 @@ object CreateIpLookups {
       domainFile: Option[File] = None,
       connectionTypeFile: Option[File] = None,
       anonymousFile: Option[File] = None,
+      asnFile: Option[File] = None,
       memCache: Boolean = true,
       lruCacheSize: Int = 10000
     ): Id[IpLookups[Id]] = {
@@ -176,6 +186,7 @@ object CreateIpLookups {
         domainFile,
         connectionTypeFile,
         anonymousFile,
+        asnFile,
         memCache,
         lruCache
       )
@@ -199,6 +210,7 @@ class IpLookups[F[_]: Monad] private[iplookups] (
   domainFile: Option[File],
   connectionTypeFile: Option[File],
   anonymousFile: Option[File],
+  asnFile: Option[File],
   memCache: Boolean,
   lru: Option[LruMap[F, String, IpLookupResult]]
 )(implicit SR: SpecializedReader[F], IAR: IpAddressResolver[F]) {
@@ -209,6 +221,7 @@ class IpLookups[F[_]: Monad] private[iplookups] (
   private val connectionTypeService =
     getService(connectionTypeFile).map((_, ReaderFunctions.connectionType))
   private val anonymousService = getService(anonymousFile)
+  private val asnService       = getService(asnFile)
 
   /**
    * Get a LookupService from a database file
@@ -285,6 +298,24 @@ class IpLookups[F[_]: Monad] private[iplookups] (
     case _            => Monad[F].pure(None)
   }
 
+  private def getAsnLookup(
+    ipAddress: Either[Throwable, InetAddress],
+    ispResponse: Option[Either[Throwable, Isp]]
+  ): F[Option[Either[Throwable, Asn]]] = {
+    val asnFromIsp = ispResponse.map(_.flatMap(_.asn))
+    asnFromIsp.flatMap(_.toOption) match {
+      case Some(_) => Monad[F].pure(asnFromIsp)
+      case None =>
+        (ipAddress, asnService) match {
+          case (Right(ipA), Some(gs)) =>
+            SR.getValue(ReaderFunctions.asn, gs, ipA)
+              .map(asnResponse => asnResponse.flatMap(Asn.create).some)
+          case (Left(f), _) => Monad[F].pure(Some(Left(f)))
+          case _            => Monad[F].pure(asnFromIsp)
+        }
+    }
+  }
+
   /**
    * This version does not use the LRU cache.
    * Concurrently looks up information
@@ -297,13 +328,12 @@ class IpLookups[F[_]: Monad] private[iplookups] (
    */
   private def performLookupsWithoutLruCache(ip: String): F[IpLookupResult] =
     for {
-      ipAddress <- IAR.resolve(ip)
-
+      ipAddress   <- IAR.resolve(ip)
       ipLocation  <- getLocationLookup(ipAddress)
       ispResponse <- getIspLookup(ipAddress)
       ispName = ispResponse.map(_.map(_.name))
       org     = ispResponse.map(_.map(_.organization))
-      asn     = ispResponse.map(_.map(_.asn))
+      asn            <- getAsnLookup(ipAddress, ispResponse)
       domain         <- getLookup(ipAddress, domainService)
       connectionType <- getLookup(ipAddress, connectionTypeService)
       anonymous      <- getAnonymousIpLookup(ipAddress)

src/main/scala/com.snowplowanalytics.maxmind.iplookups/SpecializedReader.scala

@@ -66,4 +66,5 @@ object ReaderFunctions {
     db.connectionType(ip).getConnectionType.toString
   val city        = (db: DatabaseReader, ip: InetAddress) => db.city(ip)
   val anonymousIp = (db: DatabaseReader, ip: InetAddress) => db.anonymousIp(ip)
+  val asn         = (db: DatabaseReader, ip: InetAddress) => db.asn(ip)
 }

src/main/scala/com.snowplowanalytics.maxmind.iplookups/model.scala

@@ -23,6 +23,7 @@ import cats.instances.either._
 
 import com.maxmind.geoip2.DatabaseReader
 import com.maxmind.geoip2.model.{AnonymousIpResponse, AsnResponse, CityResponse, IspResponse}
+import com.maxmind.geoip2.exception.AddressNotFoundException
 
 object model {
   type ReaderFunction[A] = (DatabaseReader, InetAddress) => A
@@ -58,15 +59,15 @@ object model {
 
   /** A case class wrapper around the MaxMind AsnResponse class. */
   final case class Asn(
-    autonomousSystemNumber: Option[Long],
+    autonomousSystemNumber: Long,
     autonomousSystemOrganization: Option[String]
   )
 
   /** A case class wrapper around the MaxMind IspResponse class. */
   final case class Isp(
     name: String,
     organization: String,
-    asn: Asn
+    asn: Either[Throwable, Asn]
   )
 
   /** Companion class contains a constructor which takes a MaxMind CityResponse. */
@@ -125,19 +126,28 @@ object model {
 
   }
 
-  /** Companion class contains a constructor which takes a MaxMind AsnResponse. */
+  /** Companion class contains a helper method which takes a MaxMind AsnResponse. */
   object Asn {
 
     /**
      * Constructs an Asn instance from a MaxMind AsnResponse instance.
      * @param asnResponse MaxMind AsnResponse object
      * @return Asn
      */
-    def apply(asnResponse: AsnResponse): Asn =
-      Asn(
-        autonomousSystemNumber = Option(asnResponse.getAutonomousSystemNumber).map(_.toLong),
-        autonomousSystemOrganization = Option(asnResponse.getAutonomousSystemOrganization)
-      )
+    def create(asnResponse: AsnResponse): Either[Throwable, Asn] =
+      Option(asnResponse.getAutonomousSystemNumber)
+        .map(_.toLong)
+        .map(asn =>
+          Asn(
+            autonomousSystemNumber = asn,
+            autonomousSystemOrganization = Option(asnResponse.getAutonomousSystemOrganization)
+          )
+        )
+        .toRight(
+          new AddressNotFoundException(
+            s"The address ${asnResponse.getIpAddress} is not in the database."
+          )
+        )
   }
 
   /** Companion class contains a constructor which takes a MaxMind IspResponse. */
@@ -152,7 +162,7 @@ object model {
       Isp(
         name = ispResponse.getIsp,
         organization = ispResponse.getOrganization,
-        asn = Asn(ispResponse)
+        asn = Asn.create(ispResponse)
       )
   }