Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE]Add iplocation function to PPL for IP address geolocation #672

Closed
YANG-DB opened this issue Sep 16, 2024 · 4 comments
Closed

[FEATURE]Add iplocation function to PPL for IP address geolocation #672

YANG-DB opened this issue Sep 16, 2024 · 4 comments
Labels
0.6 enhancement New feature or request Lang:PPL Pipe Processing Language support

Comments

@YANG-DB
Copy link
Member

YANG-DB commented Sep 16, 2024

Description:
We propose adding an geoip function to OpenSearch's Piped Processing Language (PPL) and SQL to provide built-in IP address geolocation capabilities.
This feature would be similar to functionality used in OpenSearch's geospatial feature, enhancing PPL's ability to enrich log data with geographical information based on IP addresses.

Proposed Functionality:

  1. The 'geoip' function should take an IP address as input and return geographical information.
  2. It should support both IPv4 and IPv6 addresses.
  3. The function should return multiple fields including country, region, city, latitude, longitude, and others as available.
  4. It should allow users to specify which geolocation fields to include in the output.
  5. The function should use a regularly updated IP geolocation database for accuracy.

Example Usage:

... | eval geolocation = geoip(ip_field)

This would add a new field 'geolocation' with all available location information for the IP address in 'ip_field'.

... | eval country = geoip(ip_field, "country")
... | eval lat = geoip(ip_field, "lat"), lon = iplocation(ip_field, "lon")

This would add new fields with specific geolocation information.

... | eval location_info = geoip(ip_field, "country,region,city,lat,lon")

This would add a new field 'location_info' with multiple pieces of geolocation data.

Additional considerations

  • Allow for registering a DB table that allows resolving the IP to Geo
  • Adding a generic way to register the IP to Geo location resolving mechanism / service
    • Adding auth tokens for calling such service
@dblock
Copy link
Member

dblock commented Oct 7, 2024

[Catch All Triage - 1, 2, 3, 4]

@YANG-DB YANG-DB added the 0.6 label Oct 9, 2024
@salyh salyh moved this from Todo to Design in PPL Commands Oct 16, 2024
@YANG-DB YANG-DB moved this from Design to In Progress in PPL Commands Oct 17, 2024
@salyh salyh moved this from In Progress to Design in PPL Commands Oct 29, 2024
@anasalkouz anasalkouz moved this from Design to In Progress in PPL Commands Nov 12, 2024
@abdullahdevrel
Copy link

Just passing by - considering the discussion is around GeoIP functionality, would it be possible to add support for our (IPinfo) IP databases as well? We have a bunch of IP databases that come in MMDB data file format, which also include our free IP database (https://ipinfo.io/products/free-ip-database). Our free IP databases include the IP to Country ASN/ISP database that comes with full accuracy, no EULA, and allows commercial re-distribution through only providing an attribute.

@YANG-DB
Copy link
Member Author

YANG-DB commented Dec 4, 2024

@abdullahdevrel thanks for your interest and proposal
Our goal is to provide a vendor agnostic support for any IP datasource eventually - the first iteration would be a just simple table to use as a lookup resolution
the next phase would add additional types of data-sets and you are very welcome to contribute your IP data-source - it would be great !

@YANG-DB YANG-DB moved this from In Progress to InReview in PPL Commands Dec 19, 2024
@YANG-DB YANG-DB moved this from InReview to Done in PPL Commands Dec 19, 2024
@YANG-DB YANG-DB closed this as completed by moving to Done in PPL Commands Dec 19, 2024
@abdullahdevrel
Copy link

Thank you very much @YANG-DB. Please keep me in the loop. We have the best IP databases out there. So, we will be super happy to bring our data to the community.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.6 enhancement New feature or request Lang:PPL Pipe Processing Language support
Projects
Status: Done
Development

No branches or pull requests

3 participants