Submission rate / throttle ?

[maaaaz]

Hello there,

I can't find this information elsewhere, so I am asking here: what is the maximum rate / throttle of user submissions to the public instance of Lookyloo ?

How many max submissions during an amount of time ? Identified by the IP address of the submitter ?

Cheers!

[Rafiot]

There is no max rate/throttle on the public instance (mostly because it has never been an issue). The two things to keep in mind are the following:

1. if you submit the same capture within a certain interval (300s by default), you receive the UUID of the submission you did before (https://github.com/ail-project/LacusCore/blob/main/lacuscore/lacuscore.py#L334)
2. if the same IP submits a lot of captures, the priority of their capture is gradually reduced (https://github.com/Lookyloo/lookyloo/blob/main/lookyloo/lookyloo.py#L596) - the main reason is that this way, if someone sends 100 captures and others send 1, the ones sending 1 capture will be done first.

But you can submit lots of captures, it might take some time to get them all done, but it will eventually happen.

More generally, I'd strongly recommend to setup an internal instance if you want to keep the results for a long time or have anything vaguely private. Anything you submit on the public instance can disappear at any moment without a warning, and/or it might be shared with third parties if we find it interesting. We will probably not share publicly something that is flagged as non-listed on the index, but everything is considered TLP:GREEN at most (unless you get in touch with us for a specific critical capture).

[maaaaz]

Hello @Rafiot,

Yes it's me ! Sorry for the inconvenience caused !

• These scans come from daily new registered domain names (https://github.com/maaaaz/dnsdumps/)
• The today amount of 20k is unusual (due to the integration of a brand new perimeter to the dnsdumps repo, AXFR scans), most of the time it is rather 1 or 2K a day.
• I don't want these scans to be private neither not-shared to third-party: on the contrary, I want them to be TLP:GREEN, I want them to be used by anyone, in anti-phishing/malicious websites detection [or any other kind of workflow] and I think, just like urlscan.io, Lookyloo could be useful [and more importantly, be a kind of an alternative/backup to urlscan.io which remains a platform from a private/commercial enterprise]. So clearly, I don't need a private Lookyloo instance.

All in all, tell me:

• If 1 or 2 K per day, with sometimes (few times a year) an exception burst to like 20 K, is okay ?
• If you want me to put a sleep(X seconds) between each URL submission: I use the free plan of GitHub actions, where a job can last at most 6 consecutive hours
• Or if you would me to stop using your public instance of Lookyloo for this usage. No worry!

Cheers!

[Rafiot]

It is in this case more an issue with the disk space 😭.

1-2k URLs/day is fine, we need to increase the disk space on the machine anyway. I mostly want to avoid 20K URLs daily from a single source.

Don't bother with a sleep: it is handled by lacus, which will not capture more than 25 URLs in parallel.

I'm wondering if it would make sense to setup a dedicated instance, we'll discuss it. Don't change anything for now, it is all good.

[maaaaz]

Thanks @Rafiot, don't hesitate to tell me if some changes are needed on my side !

[Rafiot]

Quick update: I saw another 15K captures today (no problem, do not change anything, the URLs I've been looking at are great), and it is taking a very long time to get them all done because... the whois query to .lk domains is broken and waits for a timeout (instead of failing quick).

I'll implement a fix for that soon, but maybe not over the weekend.

[maaaaz]

Hello @Rafiot,

Sorry for the inconvenience again, but yeah I just get to see that the amount of today scan is abnormally high...because of something weird with the .lk zone: a increase of 20K (22%) FQDN since yesterday. (it might be linked with the WHOIS error you are experiencing on that same zone).

Unfortunately for me and for you, the scans results are quite unpredictable: most of the time, it is only between 1 and 3 K entries, but sometimes, when top-level DNS are going mad, it can be multiple-dozens of K entries.

Again, feel free to tell me if and how could I reduce the burden.

Cheers!