[–] reincoder 2 days ago
[–] cowlevel 9 hours ago
Your CTO or CEO was saying they always trust geofeed and an ISP with wrong info just needs to update their geofeed, then he said you actually never trust geofeeds, but don't worry just update the geofeed and it'll be trusted, then when pointed out the correct information was already in the geofeed but not on IPinfo, he said IPinfo uses data from the geofeed so it's correct... Something like that, it made no sense but it made it clear you guys don't know what your product does.
[–] BLKNSLVR 3 days ago
[–] cowlevel 3 days ago
[–] BLKNSLVR 3 days ago
I'd be very interested in AI capabilities in detecting anomalous activity given that it should be very effective at pattern matching across large data sets. What would be required, however, is significant training with industry/application specific 'normal' browsing traffic patterns to determine acceptable tolerances, and then design escalating deterrence if and when tolerances are breached.
How is the scraper process stepping through the site? It must be programmatic, even if it's programmed to be random. What does that look like in traffic data in terms of timing and content requests?
What hazards can be put in place that a normal user would never go near, but a scraper is likely to trip over?
If scraping is being performed from multiple locations in parallel, what might that look like? ie. joining multiple sessions together that may show a more 'scraper' kind of behaviour.
Under IPv6 wouldn't detection start burning a lot of addresses? Yes, the pool is large, but that just means it's a scaling issue from IPv4 to IPv6, rather than a new problem requiring new thinking to solve? (assuming ways of detection were successful).
I'm being somewhat devil's advocate in my blocking rhetoric. The internet is in a weird confluence of wild-west and surveillance state. I like that there are ways and means of being anon-/pseudon-ymous; I like a bit of anarchaos, that's where the spirit of hacking lives. But I also don't want it sticking to my boots when I walk in the house. I'm also not fundamentally against scraping; there's definitely good scraping. There is an irony, though, in protecting scraped data or services that are built on scraped data (and I'm not accusing you, personally, just my general logical ideology of scraping). On residential proxies, I think they're 99% run by erstwhile criminals leveraging what were, until recently, referred to as botnets into a service they can launder their botnets through as 'I am legitimate business man'. I think most, basically all, residential proxies are run involuntarily or unwittingly and that blocking them would provide a greater society benefit than it would a drawback. Anything that promotes the sanitisation of end user devices can only be a good thing. The whole ecosystem is in a terrible state and regulation doesn't even seem to be on the distant horizon.
Maybe I'm hypocritical, or I'm trying to maintain mutually exclusive positions. This is where I'm at though.
IP scoring is such a difficult product to do reliably. When someone says we provide IP scoring, you have to ask them based on what and what is your methodology. Now, IP reputation scoring can be good if both the vendor and the customer know what they are looking for very specifically.
For example, there are abusive IP databases. Now, some abusive IP databases run global honeypots, and they can say these IP addresses tried to exploit the CVE patterns. Sounds good, so when you buy their product, you can say the scoring they are providing can help with blocking IP addresses that run automated CVE exploits that match this pattern. Those services act as beacons, rather than a flat, generic, and everlasting database of bad IP addresses.
Now, do you take that score and try to use it as a baseline for protecting your login portal or credit card fraud? Of course not! You have to go to specialized vendors who provide specialized scoring.
We do not do IP reputation because we provide IP metadata that we can verify, and we can discuss methodologies publicly. If you ask us to do IP reputation scoring, we can easily do that, as we run a global honeypot ourselves as well. But that will involve us providing explanations and education to folks that the score you see fits a very specific pattern of attack that is not generic.
For example, we sell residential proxy detection data, and the database contains 100 million IP addresses. Are all 100 million IP addresses bad? Of course not. But this is not our call to make; our customers have to see the metadata we provide as well, such as percent days seen, last seen, etc. Now, if an IP address is flagged for being 100 percent seen and the last seen date is fairly new, it fits a certain threat model which can be categorized as concerning.