Let's say we have a collection of documents where each document represents a network subnet like this:
{'network':'10.0.0.0/24,'someattr':'aaa'}
{'network':'192.168.0.0/16','someattr':'bbb'}
{'network':'172.16.0.0/16','someattr':'ccc'}
Is there any way I can lookup a single ip address (e.g. '10.0.100.50') in the MongoDB collection and identify the subnet/document it belongs to?
Remember that IPs are just ints and subnet masks are just ranges of ints - they're displayed as they are so that humans can read them (among other reasons). What I would do is add two additional fields to my MongoDB schema: startAddr and endAddr. Represent them as ints and they will be the starting and ending addresses for your subnet.
Next, you can convert the IP to an int in your application. See here for information on converting an IP to an integer, you didn't mention what language you're writing your application in so can't help you there.
Finally, query MongoDB on the following
db.collection.find({startAddr: {$lte: <ip>}, {endAddr: {$gte: <ip>}}})
You can also combine this new schema with some application level code to easily do things like find documents that have overlapping subnets if you need to do that.
Related
I'd like to store ip2location database in a postgres database.
Guide on ip2location website suggests to store IP addresses that specify the range (ip_from and ip_to) in columns of type DECIMAL(39, 0).
I would have expected this to be of type INET.
Is there any advantage (e.g. in terms of speed, size, ...) to use DECIMAL(39, 0) instead of INET? Other than the ip2location database format contains IPs converted to integers already and one would have to convert that back to IP addresses obviously.
The advantage of using the inet data type are
it takes up only 4 bytes to store an IPv4 address
you can make use of all the useful built-in functions and operators for IP addresses and CIDRs
There is no advantage in using NUMERIC(39), except that it seems to be required by the software you want to use.
I'm trying to create an API that I can send an IP address to and the response will contain the subnet that the IP belongs to (if it belongs to any in the table).
I have a list of subnets all stored in a table in DynamoDB like such:
subnet
45.221.27.0/24
102.215.216.0/23
192.168.0.0/16
etc...
I can't seem to figure out how I could efficiently query the table to determine which subnet an IP belongs to. I am using a Lambda to make the request so I am trying to avoid reading all the subnets in because that will use a lot of memory. I'm also trying to avoid scanning the table rather than querying because that can become too expensive.
I've been thinking about different ways of storing the subnets in the table such that it becomes possible to get more granular with queries but I also feel like I'm overcomplicating something that shouldn't be so complex.
How funny, I'm actually writing a blog on this. I'll add the link once it's published. There's a lot of interesting scaling topics related to this problem for how to load and query with max efficiency. Here's the simplest approach:
Use a singular Partition Key value (that is, the same for all items). Use the range start IP address as the Sort Key. But make it the 32-bit numeric value of the IP address not the string value, because we need to sort by it and sorting by the string value is problematic. (All IP addresses are really just 32-bit numbers underneath.) The other attributes will be the metadata you want to retrieve.
The lookup then is to issue a Query where the PK is the singular value and the SK is <= the lookup IP address (in numeric form).
The one caveat is you need to make sure that any gaps in the IP address range data set need to be filled during the load with marker items saying "gap here", otherwise a lookup that hits the gap will return the range ahead of the gap.
I am using the freely available geonames data locally to do autocomplete searches during the sign up stage on one of my websites.
I am having trouble working out the best way to make the form more user friendly by auto selecting a geoname based on their IP address and also be able to lookup a geoname based on the postcode data.
The problem is that I can't see a way to easily link an IP range or a postal code to a geoname. So what is the best practice here? Do I just run a separate query to lookup the nearest geoname by long/lat against the postcode or IP address?
You don't mention how you are geolocating the IP address, but the MaxMind GeoIP2 and GeoLite2 databases provide the geoname_id of the location. See, e.g., the CSV docs. The binary databases provide this same information.
I have three different collections.
The first collection is User, (userId, name, address.. etc)
the second collection is service, (serviceId, name, title)
the third collection is service2User(serviceId, and recipientUserId)
(I know i could use some array inside the service instead of the service2User
this is done because the serviceRegister2User contain much more then 2 fields, and can be very big.)
I need to find a collection of users which don't have current service (i.e service=10)
(the solution can be done by: linq or directly through the c# mongo driver)
to my best understand this is a two process action
First: I need to search the serviceRegister2User collection and find all recipientUserId which have already serviceId=10.
Second: I need to find all users which are different from the users found in my first query.
those users are users which did not register to serviceId=10
The collection found after the second process is the wanted result.
Can someone tell me how to do it in both way?
- linq or directly through the c# mongo driver
if it is done by Linq driver, then it need to return MongoCollection.
Thank you.
I have a Java/mongo object Node that can contain another Node, etc.
So my structure in mongo is:
like Document->Node->Node->...
A Node has a name attribute, and I want to find all Documents that have a node (including any nested Nodes) that contains a certain name.
I was using dot notation to do something like:
query.field("document.node.name").equal(name)
but that only works if the parent node has a matching name.. what I need is some kind of wildcard to search for any name (document.node.node....name etc) that is in a Node object.
Thanks for the help!
There is no wildcard search in MongoDB.
You'd need to store the Nodes somehow flattened to perform that query. You could for example store the hierarchy (the parent chain) in each Node so that you could recreate the hierarchy with your client application code.
The most commonly used structures are very well documented here.