diff --git a/CHANGELOG b/CHANGELOG index 1440c39..9bece5f 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,3 +1,9 @@ +Version 0.9.0 (2026-02-06) +-------------------------- +Extract ASN fields from ISP database (#183) +Add support for ASN database (#184) +Add anonymousIp documentation to README (#184) + Version 0.8.2 (2025-11-03) -------------------------- Fix CI (#182) diff --git a/README.md b/README.md index 870f55e..16f4b4d 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,8 @@ This is a Scala wrapper for the MaxMind [Java Geo-IP2][java-lib] library. The main benefits of using this wrapper over directly calling the Java library from Scala are: -1. **Provides a common interface to four MaxMind databases** - it works with MaxMind's databases for -looking up geographic location, ISP, domain, and connection type from an IP address +1. **Provides a common interface to six MaxMind databases** - it works with MaxMind's databases for +looking up geographic location, ISP, ASN, domain, connection type, and anonymous IP from an IP address 2. **Better type safety** - the MaxMind Java library is somewhat null-happy. This wrapper uses Option-boxing wherever possible 3. **Better performance** - as well as or instead of using MaxMind's own caching (`CHMCache`), you @@ -19,19 +19,19 @@ can also configure an LRU (Least Recently Used) cache of variable size ## Installation -The latest version of scala-maxmind-iplookups is **0.8.2** and is compatible with Scala 2.13. +The latest version of scala-maxmind-iplookups is **0.9.0** and is compatible with Scala 2.13. Add this to your SBT config: ```scala -val maxmindIpLookups = "com.snowplowanalytics" %% "scala-maxmind-iplookups" % "0.8.2" +val maxmindIpLookups = "com.snowplowanalytics" %% "scala-maxmind-iplookups" % "0.9.0" ``` Retrieve the `GeoLite2-City.mmdb` file from the [MaxMind downloads page][maxmind-downloads] ([direct link][geolitecity-dat]). -MaxMind also has databases for looking up [ISPs][maxmind-isp], [domain names][maxmind-domain], and -[connection types][maxmind-connection-type] from IP addresses. Scala MaxMind IP Lookups supports all +MaxMind also has databases for looking up [ISPs][maxmind-isp], [ASN][maxmind-asn], [domain names][maxmind-domain], +[connection types][maxmind-connection-type], and [anonymous IPs][maxmind-anonymous] from IP addresses. Scala MaxMind IP Lookups supports all of these. ## Usage @@ -102,6 +102,8 @@ final case class IpLookups( ispFile: Option[File], domainFile: Option[File], connectionTypeFile: Option[File], + anonymousFile: Option[File], + asnFile: Option[File], memCache: Boolean = true, lruCache: Int = 10000 ) @@ -116,14 +118,16 @@ def createFromFilenames( ispFile: Option[String], domainFile: Option[String], connectionTypeFile: Option[String], + anonymousFile: Option[String], + asnFile: Option[String], memCache: Boolean = true, lruCache: Int = 10000 ) ``` -The first four arguments are the MaxMind databases from which the lookup should be performed. -`geoFile`, `ispFile`, `domainFile`, and `connectionTypeFile` refer respectively to MaxMind's -databases for looking up location, ISP, domain, and connection type based on an IP address. They are +The first six arguments are the MaxMind databases from which the lookup should be performed. +`geoFile`, `ispFile`, `domainFile`, `connectionTypeFile`, `anonymousFile`, and `asnFile` refer respectively to MaxMind's +databases for looking up location, ISP, domain, connection type, anonymous IP, and ASN based on an IP address. They are all wrapped in `Option`, so if you don't have access to all of them, just pass in `None` as in the example above. @@ -143,18 +147,24 @@ final case class IpLookupResult( ipLocation: Option[Either[Throwable, IpLocation]], isp: Option[Either[Throwable, String]], organization: Option[Either[Throwable, String]], + asn: Option[Either[Throwable, Asn]], domain: Option[Either[Throwable, String]], - connectionType: Option[Either[Throwable, String]] + connectionType: Option[Either[Throwable, String]], + anonymousIp: Option[Either[Throwable, AnonymousIp]] ) ``` The first element is the result of the geographic location lookup. It is either `None` (if no geographic lookup database was provided) or `Some(ipLocation)`, where `ipLocation` is an instance of -the `IpLocation` case class described below. The other three elements in the tuple are `Option`s -wrapping the results of the other four possible lookups: ISP, organization, domain, and connection -type. +the `IpLocation` case class described below. The other elements are `Option`s wrapping the results +of the other possible lookups: ISP, organization, ASN, domain, connection type, and anonymous IP. -Note that enabling providing an ISP database will return an `organization` in addition to an `isp`. +Note that providing an ISP database will return `organization` and `asn` in addition to `isp`. +The `asn` field contains an `Asn` case class with `autonomousSystemNumber` and +`autonomousSystemOrganization` information. ASN information can be extracted from either the ISP +database or a dedicated ASN database. If both databases are provided, the ISP database takes +precedence. If the ISP database is provided but doesn't contain ASN information for a given IP, +the lookup will fall back to the ASN database (if provided). ### IpLocation case class @@ -178,9 +188,41 @@ final case class IpLocation( ) ``` +### Asn case class + +The ASN lookup (extracted from either the ISP or ASN database) returns an `Asn` case class instance with the +following structure: + +```scala +final case class Asn( + autonomousSystemNumber: Long, + autonomousSystemOrganization: Option[String] +) +``` + +Note that `autonomousSystemNumber` is a required field. If an ASN database entry doesn't contain a valid +ASN number, the lookup will return an error instead of an `Asn` instance. + +### AnonymousIp case class + +The anonymous IP lookup returns an `AnonymousIp` case class instance with the following structure: + +```scala +final case class AnonymousIp( + ipAddress: String, + isAnonymous: Boolean, + isAnonymousVpn: Boolean, + isHostingProvider: Boolean, + isPublicProxy: Boolean, + isTorExitNode: Boolean +) +``` + +This lookup provides information about whether an IP address is associated with various types of anonymous or proxy services. + ### An example using multiple databases -This example shows how to do a lookup using all four databases. +This example shows how to do a lookup using all six databases. ```scala import com.snowplowanalytics.maxmind.iplookups.IpLookups @@ -191,6 +233,8 @@ val lookupResult = (for { ispFile = Some("/opt/maxmind/GeoIP2-ISP.mmdb"), domainFile = Some("/opt/maxmind/GeoIP2-Domain.mmdb"), connectionType = Some("/opt/maxmind/GeoIP2-Connection-Type.mmdb"), + anonymousFile = Some("/opt/maxmind/GeoIP2-Anonymous-IP.mmdb"), + asnFile = Some("/opt/maxmind/GeoLite2-ASN.mmdb"), memCache = false, lruCache = 10000 ) @@ -207,11 +251,19 @@ println(lookupResult.isp) // => Some(Right("FDN Communications")) // Organization lookup println(lookupResult.organization) // => Some(Right("DSLAM WAN Allocation")) +// ASN lookup +println(lookupResult.asn.map(_.autonomousSystemNumber)) // => Some(Right(123)) +println(lookupResult.asn.map(_.autonomousSystemOrganization)) // => Some(Right(Some("Example ISP"))) + // Domain lookup println(lookupResult.domain) // => Some(Right("nuvox.net")) // Connection type lookup println(lookupResult.connectionType) // => Some(Right("Dialup")) + +// Anonymous IP lookup +println(lookupResult.anonymousIp.map(_.isAnonymous)) // => Some(Right(false)) +println(lookupResult.anonymousIp.map(_.isTorExitNode)) // => Some(Right(false)) ``` ### LRU cache @@ -260,8 +312,10 @@ limitations under the License. [maxmind-downloads]: https://dev.maxmind.com/geoip/geoip2/downloadable/#MaxMind_APIs [maxmind-isp]: https://www.maxmind.com/en/geoip2-isp-database +[maxmind-asn]: https://dev.maxmind.com/geoip/docs/databases/asn [maxmind-domain]: https://www.maxmind.com/en/geoip2-domain-name-database [maxmind-connection-type]: https://www.maxmind.com/en/geoip2-connection-type-database +[maxmind-anonymous]: https://www.maxmind.com/en/geoip2-anonymous-ip-database [geolitecity-dat]: http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz [license]: http://www.apache.org/licenses/LICENSE-2.0 diff --git a/src/main/scala/com.snowplowanalytics.maxmind.iplookups/IpLookups.scala b/src/main/scala/com.snowplowanalytics.maxmind.iplookups/IpLookups.scala index d35e0a5..3c3ecb1 100644 --- a/src/main/scala/com.snowplowanalytics.maxmind.iplookups/IpLookups.scala +++ b/src/main/scala/com.snowplowanalytics.maxmind.iplookups/IpLookups.scala @@ -40,6 +40,7 @@ sealed trait CreateIpLookups[F[_]] { * @param domainFile Domain lookup database file * @param connectionTypeFile Connection type lookup database file * @param anonymousFile Anonymous lookup database file + * @param asnFile ASN lookup database file * @param memCache Whether to use MaxMind's CHMCache * @param lruCacheSize Maximum size of LruMap cache */ @@ -49,6 +50,7 @@ sealed trait CreateIpLookups[F[_]] { domainFile: Option[File] = None, connectionTypeFile: Option[File] = None, anonymousFile: Option[File] = None, + asnFile: Option[File] = None, memCache: Boolean = true, lruCacheSize: Int = 10000 ): F[IpLookups[F]] @@ -60,6 +62,7 @@ sealed trait CreateIpLookups[F[_]] { * @param domainFile Domain lookup database filepath * @param connectionTypeFile Connection type lookup database filepath * @param anonymousFile Anonymous lookup database filepath + * @param asnFile ASN lookup database filepath * @param memCache Whether to use MaxMind's CHMCache * @param lruCacheSize Maximum size of LruMap cache */ @@ -69,6 +72,7 @@ sealed trait CreateIpLookups[F[_]] { domainFile: Option[String] = None, connectionTypeFile: Option[String] = None, anonymousFile: Option[String] = None, + asnFile: Option[String] = None, memCache: Boolean = true, lruCacheSize: Int = 10000 ): F[IpLookups[F]] = createFromFiles( @@ -77,6 +81,7 @@ sealed trait CreateIpLookups[F[_]] { domainFile.map(new File(_)), connectionTypeFile.map(new File(_)), anonymousFile.map(new File(_)), + asnFile.map(new File(_)), memCache, lruCacheSize ) @@ -94,6 +99,7 @@ object CreateIpLookups { domainFile: Option[File] = None, connectionTypeFile: Option[File] = None, anonymousFile: Option[File] = None, + asnFile: Option[File] = None, memCache: Boolean = true, lruCacheSize: Int = 10000 ): F[IpLookups[F]] = @@ -112,6 +118,7 @@ object CreateIpLookups { domainFile, connectionTypeFile, anonymousFile, + asnFile, memCache, lruCache ) @@ -128,6 +135,7 @@ object CreateIpLookups { domainFile: Option[File] = None, connectionTypeFile: Option[File] = None, anonymousFile: Option[File] = None, + asnFile: Option[File] = None, memCache: Boolean = true, lruCacheSize: Int = 10000 ): Eval[IpLookups[Eval]] = @@ -145,6 +153,7 @@ object CreateIpLookups { domainFile, connectionTypeFile, anonymousFile, + asnFile, memCache, lruCache ) @@ -161,6 +170,7 @@ object CreateIpLookups { domainFile: Option[File] = None, connectionTypeFile: Option[File] = None, anonymousFile: Option[File] = None, + asnFile: Option[File] = None, memCache: Boolean = true, lruCacheSize: Int = 10000 ): Id[IpLookups[Id]] = { @@ -176,6 +186,7 @@ object CreateIpLookups { domainFile, connectionTypeFile, anonymousFile, + asnFile, memCache, lruCache ) @@ -199,17 +210,18 @@ class IpLookups[F[_]: Monad] private[iplookups] ( domainFile: Option[File], connectionTypeFile: Option[File], anonymousFile: Option[File], + asnFile: Option[File], memCache: Boolean, lru: Option[LruMap[F, String, IpLookupResult]] )(implicit SR: SpecializedReader[F], IAR: IpAddressResolver[F]) { // Configure the lookup services private val geoService = getService(geoFile) - private val ispService = getService(ispFile).map((_, ReaderFunctions.isp)) - private val orgService = getService(ispFile).map((_, ReaderFunctions.org)) + private val ispService = getService(ispFile) private val domainService = getService(domainFile).map((_, ReaderFunctions.domain)) private val connectionTypeService = getService(connectionTypeFile).map((_, ReaderFunctions.connectionType)) private val anonymousService = getService(anonymousFile) + private val asnService = getService(asnFile) /** * Get a LookupService from a database file @@ -235,7 +247,7 @@ class IpLookups[F[_]: Monad] private[iplookups] ( */ private def getLookup( ipAddress: Either[Throwable, InetAddress], - service: Option[(DatabaseReader, ReaderFunction)] + service: Option[(DatabaseReader, ReaderFunction[String])] ): F[Option[Either[Throwable, String]]] = (ipAddress, service) match { case (Right(ipA), Some((db, f))) => @@ -260,7 +272,7 @@ class IpLookups[F[_]: Monad] private[iplookups] ( ipAddress: Either[Throwable, InetAddress] ): F[Option[Either[Throwable, IpLocation]]] = (ipAddress, geoService) match { case (Right(ipA), Some(gs)) => - SR.getCityValue(gs, ipA) + SR.getValue(ReaderFunctions.city, gs, ipA) .map(loc => loc.map(IpLocation(_)).some) case (Left(f), _) => Monad[F].pure(Some(Left(f))) case _ => Monad[F].pure(None) @@ -270,12 +282,40 @@ class IpLookups[F[_]: Monad] private[iplookups] ( ipAddress: Either[Throwable, InetAddress] ): F[Option[Either[Throwable, AnonymousIp]]] = (ipAddress, anonymousService) match { case (Right(ipA), Some(gs)) => - SR.getAnonymousValue(gs, ipA) + SR.getValue(ReaderFunctions.anonymousIp, gs, ipA) .map(loc => loc.map(AnonymousIp(_)).some) case (Left(f), _) => Monad[F].pure(Some(Left(f))) case _ => Monad[F].pure(None) } + private def getIspLookup( + ipAddress: Either[Throwable, InetAddress] + ): F[Option[Either[Throwable, Isp]]] = (ipAddress, ispService) match { + case (Right(ipA), Some(gs)) => + SR.getValue(ReaderFunctions.isp, gs, ipA) + .map(ispResponse => ispResponse.map(Isp(_)).some) + case (Left(f), _) => Monad[F].pure(Some(Left(f))) + case _ => Monad[F].pure(None) + } + + private def getAsnLookup( + ipAddress: Either[Throwable, InetAddress], + ispResponse: Option[Either[Throwable, Isp]] + ): F[Option[Either[Throwable, Asn]]] = { + val asnFromIsp = ispResponse.map(_.flatMap(_.asn)) + asnFromIsp.flatMap(_.toOption) match { + case Some(_) => Monad[F].pure(asnFromIsp) + case None => + (ipAddress, asnService) match { + case (Right(ipA), Some(gs)) => + SR.getValue(ReaderFunctions.asn, gs, ipA) + .map(asnResponse => asnResponse.flatMap(Asn.create).some) + case (Left(f), _) => Monad[F].pure(Some(Left(f))) + case _ => Monad[F].pure(asnFromIsp) + } + } + } + /** * This version does not use the LRU cache. * Concurrently looks up information @@ -288,15 +328,16 @@ class IpLookups[F[_]: Monad] private[iplookups] ( */ private def performLookupsWithoutLruCache(ip: String): F[IpLookupResult] = for { - ipAddress <- IAR.resolve(ip) - - ipLocation <- getLocationLookup(ipAddress) - isp <- getLookup(ipAddress, ispService) - org <- getLookup(ipAddress, orgService) + ipAddress <- IAR.resolve(ip) + ipLocation <- getLocationLookup(ipAddress) + ispResponse <- getIspLookup(ipAddress) + ispName = ispResponse.map(_.map(_.name)) + org = ispResponse.map(_.map(_.organization)) + asn <- getAsnLookup(ipAddress, ispResponse) domain <- getLookup(ipAddress, domainService) connectionType <- getLookup(ipAddress, connectionTypeService) anonymous <- getAnonymousIpLookup(ipAddress) - } yield IpLookupResult(ipLocation, isp, org, domain, connectionType, anonymous) + } yield IpLookupResult(ipLocation, ispName, org, asn, domain, connectionType, anonymous) /** * Returns the MaxMind location for this IP address diff --git a/src/main/scala/com.snowplowanalytics.maxmind.iplookups/SpecializedReader.scala b/src/main/scala/com.snowplowanalytics.maxmind.iplookups/SpecializedReader.scala index 0e5f656..962a41f 100644 --- a/src/main/scala/com.snowplowanalytics.maxmind.iplookups/SpecializedReader.scala +++ b/src/main/scala/com.snowplowanalytics.maxmind.iplookups/SpecializedReader.scala @@ -18,102 +18,53 @@ import cats.{Eval, Id} import cats.effect.Sync import cats.syntax.either._ import com.maxmind.geoip2.DatabaseReader -import com.maxmind.geoip2.model.CityResponse -import com.maxmind.geoip2.model.AnonymousIpResponse import model._ /** Data type letting you read data in maxmind's DatabaseReader. */ sealed trait SpecializedReader[F[_]] { - def getValue( - f: ReaderFunction, + def getValue[A]( + f: ReaderFunction[A], db: DatabaseReader, ip: InetAddress - ): F[Either[Throwable, String]] - - def getCityValue( - db: DatabaseReader, - ip: InetAddress - ): F[Either[Throwable, CityResponse]] - - def getAnonymousValue( - db: DatabaseReader, - ip: InetAddress - ): F[Either[Throwable, AnonymousIpResponse]] - + ): F[Either[Throwable, A]] } object SpecializedReader { implicit def syncSpecializedReader[F[_]: Sync]: SpecializedReader[F] = new SpecializedReader[F] { - def getValue( - f: ReaderFunction, + def getValue[A]( + f: ReaderFunction[A], db: DatabaseReader, ip: InetAddress - ): F[Either[Throwable, String]] = + ): F[Either[Throwable, A]] = Sync[F].delay(Either.catchNonFatal(f(db, ip))) - - def getCityValue( - db: DatabaseReader, - ip: InetAddress - ): F[Either[Throwable, CityResponse]] = - Sync[F].delay(Either.catchNonFatal(db.city(ip))) - - def getAnonymousValue( - db: DatabaseReader, - ip: InetAddress - ): F[Either[Throwable, AnonymousIpResponse]] = - Sync[F].delay(Either.catchNonFatal(db.anonymousIp(ip))) - } implicit def evalSpecializedReader: SpecializedReader[Eval] = new SpecializedReader[Eval] { - def getValue( - f: ReaderFunction, + def getValue[A]( + f: ReaderFunction[A], db: DatabaseReader, ip: InetAddress - ): Eval[Either[Throwable, String]] = + ): Eval[Either[Throwable, A]] = Eval.later(Either.catchNonFatal(f(db, ip))) - - def getCityValue( - db: DatabaseReader, - ip: InetAddress - ): Eval[Either[Throwable, CityResponse]] = - Eval.later(Either.catchNonFatal(db.city(ip))) - - def getAnonymousValue( - db: DatabaseReader, - ip: InetAddress - ): Eval[Either[Throwable, AnonymousIpResponse]] = - Eval.later(Either.catchNonFatal(db.anonymousIp(ip))) - } implicit def idSpecializedReader: SpecializedReader[Id] = new SpecializedReader[Id] { - def getValue( - f: ReaderFunction, + def getValue[A]( + f: ReaderFunction[A], db: DatabaseReader, ip: InetAddress - ): Id[Either[Throwable, String]] = + ): Id[Either[Throwable, A]] = Either.catchNonFatal(f(db, ip)) - - def getCityValue( - db: DatabaseReader, - ip: InetAddress - ): Id[Either[Throwable, CityResponse]] = - Either.catchNonFatal(db.city(ip)) - - def getAnonymousValue( - db: DatabaseReader, - ip: InetAddress - ): Id[Either[Throwable, AnonymousIpResponse]] = - Either.catchNonFatal(db.anonymousIp(ip)) } } object ReaderFunctions { - val isp = (db: DatabaseReader, ip: InetAddress) => db.isp(ip).getIsp - val org = (db: DatabaseReader, ip: InetAddress) => db.isp(ip).getOrganization + val isp = (db: DatabaseReader, ip: InetAddress) => db.isp(ip) val domain = (db: DatabaseReader, ip: InetAddress) => db.domain(ip).getDomain val connectionType = (db: DatabaseReader, ip: InetAddress) => db.connectionType(ip).getConnectionType.toString + val city = (db: DatabaseReader, ip: InetAddress) => db.city(ip) + val anonymousIp = (db: DatabaseReader, ip: InetAddress) => db.anonymousIp(ip) + val asn = (db: DatabaseReader, ip: InetAddress) => db.asn(ip) } diff --git a/src/main/scala/com.snowplowanalytics.maxmind.iplookups/model.scala b/src/main/scala/com.snowplowanalytics.maxmind.iplookups/model.scala index da8244a..254af93 100644 --- a/src/main/scala/com.snowplowanalytics.maxmind.iplookups/model.scala +++ b/src/main/scala/com.snowplowanalytics.maxmind.iplookups/model.scala @@ -22,11 +22,11 @@ import cats.instances.option._ import cats.instances.either._ import com.maxmind.geoip2.DatabaseReader -import com.maxmind.geoip2.model.CityResponse -import com.maxmind.geoip2.model.AnonymousIpResponse +import com.maxmind.geoip2.model.{AnonymousIpResponse, AsnResponse, CityResponse, IspResponse} +import com.maxmind.geoip2.exception.AddressNotFoundException object model { - type ReaderFunction = (DatabaseReader, InetAddress) => String + type ReaderFunction[A] = (DatabaseReader, InetAddress) => A type Error[A] = Either[Throwable, A] @@ -57,6 +57,19 @@ object model { isTorExitNode: Boolean ) + /** A case class wrapper around the MaxMind AsnResponse class. */ + final case class Asn( + autonomousSystemNumber: Long, + autonomousSystemOrganization: Option[String] + ) + + /** A case class wrapper around the MaxMind IspResponse class. */ + final case class Isp( + name: String, + organization: String, + asn: Either[Throwable, Asn] + ) + /** Companion class contains a constructor which takes a MaxMind CityResponse. */ object IpLocation { @@ -113,11 +126,52 @@ object model { } + /** Companion class contains a helper method which takes a MaxMind AsnResponse. */ + object Asn { + + /** + * Constructs an Asn instance from a MaxMind AsnResponse instance. + * @param asnResponse MaxMind AsnResponse object + * @return Asn + */ + def create(asnResponse: AsnResponse): Either[Throwable, Asn] = + Option(asnResponse.getAutonomousSystemNumber) + .map(_.toLong) + .map(asn => + Asn( + autonomousSystemNumber = asn, + autonomousSystemOrganization = Option(asnResponse.getAutonomousSystemOrganization) + ) + ) + .toRight( + new AddressNotFoundException( + s"The address ${asnResponse.getIpAddress} is not in the database." + ) + ) + } + + /** Companion class contains a constructor which takes a MaxMind IspResponse. */ + object Isp { + + /** + * Constructs an Isp instance from a MaxMind IspResponse instance. + * @param ispResponse MaxMind IspResponse object + * @return Isp + */ + def apply(ispResponse: IspResponse): Isp = + Isp( + name = ispResponse.getIsp, + organization = ispResponse.getOrganization, + asn = Asn.create(ispResponse) + ) + } + /** Result of MaxMind lookups */ final case class IpLookupResult( ipLocation: Option[Either[Throwable, IpLocation]], isp: Option[Either[Throwable, String]], organization: Option[Either[Throwable, String]], + asn: Option[Either[Throwable, Asn]], domain: Option[Either[Throwable, String]], connectionType: Option[Either[Throwable, String]], anonymousIp: Option[Either[Throwable, AnonymousIp]] @@ -129,6 +183,7 @@ object model { Option[IpLocation], Option[String], Option[String], + Option[Asn], Option[String], Option[String], Option[AnonymousIp] @@ -137,11 +192,12 @@ object model { val location = ipLocation.sequence[Error, IpLocation].toValidatedNel val provider = isp.sequence[Error, String].toValidatedNel val org = organization.sequence[Error, String].toValidatedNel + val asnRes = asn.sequence[Error, Asn].toValidatedNel val dom = domain.sequence[Error, String].toValidatedNel val connection = connectionType.sequence[Error, String].toValidatedNel val anonymous = anonymousIp.sequence[Error, AnonymousIp].toValidatedNel - (location, provider, org, dom, connection, anonymous).tupled + (location, provider, org, asnRes, dom, connection, anonymous).tupled } } } diff --git a/src/test/resources/com/snowplowanalytics/maxmind/iplookups/GeoLite2-ASN-Test.mmdb b/src/test/resources/com/snowplowanalytics/maxmind/iplookups/GeoLite2-ASN-Test.mmdb new file mode 100644 index 0000000..6013680 Binary files /dev/null and b/src/test/resources/com/snowplowanalytics/maxmind/iplookups/GeoLite2-ASN-Test.mmdb differ diff --git a/src/test/scala/com.snowplowanalytics.maxmind.iplookups/IpLookupsTest.scala b/src/test/scala/com.snowplowanalytics.maxmind.iplookups/IpLookupsTest.scala index eb171e1..20b7906 100644 --- a/src/test/scala/com.snowplowanalytics.maxmind.iplookups/IpLookupsTest.scala +++ b/src/test/scala/com.snowplowanalytics.maxmind.iplookups/IpLookupsTest.scala @@ -17,7 +17,6 @@ import cats.effect.testing.specs2.CatsEffect import cats.implicits._ import cats.{Id, Monad} import com.maxmind.geoip2.exception.AddressNotFoundException -import com.snowplowanalytics.maxmind.iplookups.IpLookupsTest.ipLookupsFromFiles import com.snowplowanalytics.maxmind.iplookups.model._ import org.specs2.mutable.Specification import org.specs2.specification.Tables @@ -30,23 +29,13 @@ object IpLookupsTest { val domainFile = getClass.getResource("GeoIP2-Domain-Test.mmdb").getFile val connectionTypeFile = getClass.getResource("GeoIP2-Connection-Type-Test.mmdb").getFile val anonymousFile = getClass.getResource("GeoIP2-Anonymous-IP-Test.mmdb").getFile - - def ipLookupsFromFiles[F[_]: CreateIpLookups](memCache: Boolean, lruCache: Int): F[IpLookups[F]] = - CreateIpLookups[F] - .createFromFilenames( - Some(geoFile), - Some(ispFile), - Some(domainFile), - Some(connectionTypeFile), - Some(anonymousFile), - memCache, - lruCache - ) + val asnFile = getClass.getResource("GeoLite2-ASN-Test.mmdb").getFile def failedLookupCauseUnknownHost(host: String): IpLookupResult = IpLookupResult( ipLocation = unknownHostException(host), isp = unknownHostException(host), organization = unknownHostException(host), + asn = unknownHostException(host), domain = unknownHostException(host), connectionType = unknownHostException(host), anonymousIp = unknownHostException(host) @@ -75,6 +64,7 @@ object IpLookupsTest { new AddressNotFoundException("The address 175.16.199.0 is not in the database.").asLeft.some, new AddressNotFoundException("The address 175.16.199.0 is not in the database.").asLeft.some, new AddressNotFoundException("The address 175.16.199.0 is not in the database.").asLeft.some, + new AddressNotFoundException("The address 175.16.199.0 is not in the database.").asLeft.some, "Dialup".asRight.some, AnonymousIp( ipAddress = "175.16.199.0", @@ -103,6 +93,10 @@ object IpLookupsTest { ).asRight.some, "Century Link".asRight.some, "Lariat Software".asRight.some, + Asn( + autonomousSystemNumber = 209L, + autonomousSystemOrganization = None + ).asRight.some, new AddressNotFoundException("The address 216.160.83.56 is not in the database.").asLeft.some, new AddressNotFoundException("The address 216.160.83.56 is not in the database.").asLeft.some, AnonymousIp( @@ -132,6 +126,10 @@ object IpLookupsTest { ).asRight.some, "Loud Packet".asRight.some, "zudoarichikito_".asRight.some, + Asn( + autonomousSystemNumber = 35908L, + autonomousSystemOrganization = None + ).asRight.some, "shoesfin.NET".asRight.some, new AddressNotFoundException("The address 67.43.156.0 is not in the database.").asLeft.some, AnonymousIp( @@ -147,6 +145,7 @@ object IpLookupsTest { new AddressNotFoundException("The address 81.2.69.11 is not in the database.").asLeft.some, new AddressNotFoundException("The address 81.2.69.11 is not in the database.").asLeft.some, new AddressNotFoundException("The address 81.2.69.11 is not in the database.").asLeft.some, + new AddressNotFoundException("The address 81.2.69.11 is not in the database.").asLeft.some, "in-addr.arpa".asRight.some, new AddressNotFoundException("The address 81.2.69.11 is not in the database.").asLeft.some, AnonymousIp( @@ -166,15 +165,51 @@ object IpLookupsTest { new AddressNotFoundException("The address 192.0.2.0 is not in the database.").asLeft.some, new AddressNotFoundException("The address 192.0.2.0 is not in the database.").asLeft.some, new AddressNotFoundException("The address 192.0.2.0 is not in the database.").asLeft.some, + new AddressNotFoundException("The address 192.0.2.0 is not in the database.").asLeft.some, new AddressNotFoundException("The address 192.0.2.0 is not in the database.").asLeft.some + ), + "18.11.120.0" -> IpLookupResult( + new AddressNotFoundException("The address 18.11.120.0 is not in the database.").asLeft.some, + "Massachusetts Institute of Technology".asRight.some, + "Massachusetts Institute of Technology".asRight.some, + Asn( + autonomousSystemNumber = 3L, + autonomousSystemOrganization = Some("Massachusetts Institute of Technology") + ).asRight.some, + new AddressNotFoundException("The address 18.11.120.0 is not in the database.").asLeft.some, + new AddressNotFoundException("The address 18.11.120.0 is not in the database.").asLeft.some, + AnonymousIp( + ipAddress = "18.11.120.0", + isAnonymous = false, + isAnonymousVpn = false, + isHostingProvider = false, + isPublicProxy = false, + isTorExitNode = false + ).asRight.some + ), + "8.33.20.1" -> IpLookupResult( + new AddressNotFoundException("The address 8.33.20.1 is not in the database.").asLeft.some, + "Level 3 Communications".asRight.some, + "Level 3 Communications".asRight.some, + new AddressNotFoundException("The address 8.33.20.1 is not in the database.").asLeft.some, + new AddressNotFoundException("The address 8.33.20.1 is not in the database.").asLeft.some, + new AddressNotFoundException("The address 8.33.20.1 is not in the database.").asLeft.some, + AnonymousIp( + ipAddress = "8.33.20.1", + isAnonymous = false, + isAnonymousVpn = false, + isHostingProvider = false, + isPublicProxy = false, + isTorExitNode = false + ).asRight.some ) ) } class IpLookupsTest extends Specification with Tables with CatsEffect { + import IpLookupsTest._ - "Looking up some IP address locations should match their expected locations" should { - import IpLookupsTest._ + "Looking up some IP address" should { for { memCache <- Seq(true, false) @@ -188,6 +223,22 @@ class IpLookupsTest extends Specification with Tables with CatsEffect { "work for Id monad" in { assertWithFiles[Id](memCache, lruCache, ip, expected) } + "return correct ASN when both ISP and ASN database provided" in { + assertWithFiles[IO](memCache, lruCache, ip, expected, Some(asnFile), Some(ispFile)) + } + "return correct ASN when only ISP database provided" in { + assertWithFiles[IO](memCache, lruCache, ip, expected, None, Some(ispFile)) + } + "return correct ASN when only ASN database provided" in { + assertWithFiles[IO]( + memCache, + lruCache, + ip, + expected.copy(isp = None, organization = None), + Some(asnFile), + None + ) + } } } } @@ -231,9 +282,21 @@ class IpLookupsTest extends Specification with Tables with CatsEffect { memCache: Boolean, lruCache: Int, ip: String, - expected: IpLookupResult + expected: IpLookupResult, + asnFileParam: Option[String] = Some(asnFile), + ispFileParam: Option[String] = Some(ispFile) ) = { - ipLookupsFromFiles[F](memCache, lruCache) + CreateIpLookups[F] + .createFromFilenames( + Some(geoFile), + ispFileParam, + Some(domainFile), + Some(connectionTypeFile), + Some(anonymousFile), + asnFileParam, + memCache, + lruCache + ) .flatMap(_.performLookups(ip)) .map(r => matchIpLookupResult(r, expected)) } @@ -248,7 +311,7 @@ class IpLookupsTest extends Specification with Tables with CatsEffect { memCache = true, lruCacheSize = 0 ) - val expected = IpLookupResult(None, None, None, None, None, None) + val expected = IpLookupResult(None, None, None, None, None, None, None) noFilesLookup .flatMap(_.performLookups("67.43.156.0")) @@ -260,6 +323,7 @@ class IpLookupsTest extends Specification with Tables with CatsEffect { "iplocation" ! expected.ipLocation ! actual.ipLocation | "isp" ! expected.isp ! actual.isp | "organization" ! expected.organization ! actual.organization | + "asn" ! expected.asn ! actual.asn | "domain" ! expected.domain ! actual.domain | "connection type" ! expected.connectionType ! actual.connectionType | "anonymous" ! expected.anonymousIp ! actual.anonymousIp | { (_, e, a) =>