Broken after botguard integration

[sn0opy]

Niantic introduced Google's botguard protection to the Intel map. This prevents ingress-exporter to work after a few requests.

IITCs current approach is to eval the code that's being sent with every request.

[breezewish]

I have little time to work on this issue currently. Seems that botguard uses key and mouse events to calculate the data.

[Granor]

It is still working fine for me. However, if an account gets banned it is still posible to get a few requests done before the server denies any further requests. I can not tell if this is because of botguard.

[sn0opy]

@Granor it is because of Botguard. You can do ~5 requests before you get temp banned for a few minutes. Those few requests aren't enough to get any good data from the Intel.

[Granor]

@sn0opy I enqounter the same behavior on certain accounts. I just requested 13516 portals with details without any Problems. Have you tried it with another (new) account?
This is my request config, but this should not matter anyway.

Request:
    UserAgent:     'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36'
    MaxParallel:   15
    MinIntervalMS: 100  # Too fast requesting would get IP banned

[danizwam]

I startet the script last sunday and it worked fine. Today i get an error wit the MungeDetector:

js-bson: Failed to load c++ bson extension, using pure JS version
2014-10-18T16:59:38.813Z - info: [MungeDetector] Trying to use internal munge data.
2014-10-18T16:59:39.459Z - warn: [MungeDetector] Failed.
2014-10-18T16:59:39.460Z - info: [MungeDetector] Trying to extract munge data from ingress.com/intel.
2014-10-18T16:59:40.646Z - warn: [MungeDetector] Failed.
2014-10-18T16:59:40.650Z - error: [MungeDetector] Could not detect munge data. Tasks are terminated.

Botguard? ...or my fault?

[sn0opy]

Botguard. either fix it, or create another account
On Oct 18, 2014 7:04 PM, "danizwam" [email protected] wrote:

I startet the script last sunday and it worked fine. Today i get an error
wit the MungeDetector:

js-bson: Failed to load c++ bson extension, using pure JS version
2014-10-18T16:59:38.813Z - info: [MungeDetector] Trying to use internal
munge data.
2014-10-18T16:59:39.459Z - warn: [MungeDetector] Failed.
2014-10-18T16:59:39.460Z - info: [MungeDetector] Trying to extract munge
data from ingress.com/intel.
2014-10-18T16:59:40.646Z - warn: [MungeDetector] Failed.
2014-10-18T16:59:40.650Z - error: [MungeDetector] Could not detect munge
data. Tasks are terminated.

Botguard? ...or my fault?

—
Reply to this email directly or view it on GitHub
#11 (comment)
.

[danizwam]

I checked the account in my browser and everything works flawless. I can see portals and the log. Are you sure, a new account would fix this problem?

[sn0opy]

It works on the Intel map, because you "answering" Botguards questions
correctly.

That's a very basic answer.

Try to create a new one, but it won't survive a full day, believe me.
On Oct 18, 2014 7:14 PM, "danizwam" [email protected] wrote:

I checked the account in my browser and everything works flawless. I can
see portals and the log. Are you sure, a new account would fix this problem?

—
Reply to this email directly or view it on GitHub
#11 (comment)
.

[danizwam]

I forgot to mention, that i'm using IITC. Works flawless... Did they find the solution to the Botguard?

[sn0opy]

They're using mostly the same methods the standard Intel uses which works
quite good. Porting these changes to ingress-exporter unfortunately isn't
that easy I think.
On Oct 18, 2014 7:53 PM, "danizwam" [email protected] wrote:

I forgot to mention, that i'm using IITC. Works flawless... Did they find
the solution to the Botguard?

—
Reply to this email directly or view it on GitHub
#11 (comment)
.

[Granor]

I got the Problem. Its not connectet to the Botguard, but to the change of the data format for the gamescore. Therefore, munge data testing fails, but it is still valid. As a quick fix I simply removed the testing part and it works fine. The canged function in src/lib/mungedetector.coffee:

extractMunge = (callback) ->

    request.get '/jsc/gen_dashboard.js', (error, response, body) ->

        if error
            callback 'fail'
            return

        body = body.toString()

        try
            result = Utils.extractIntelData body
        catch err
            console.log err
            callback 'fail'
            return

        Munges.Data      = [result]
        Munges.ActiveSet = 0
        callback()
        # test it
        ###
        tryMungeSet (err) ->

            if not err?
                callback()
                return

            callback 'fail'
###

The complete file: http://pastebin.com/DxGg8efM
(Dont forget to run grunt again after you changed the file)

[danizwam]

@Granor: Thx! It works ;)

Let's see if i run into the Botguard ;)

Edit: Even IITC has problems showing the MU controlled by E and R. NaN ;)

[Camisa]

@Granor

Your quick fix does work and using it, there was no problem fetching all portals and details for a given cell. Using zoom level 15, all portals and details for a given cell were fetched in ~8.75 minutes.

Unfortunately, invoking the exporter for all public messages within the same grid ( cell ) triggers the botguard after 31.5 minutes.

[Granor]

@Camisa It seems to take a pretty long time for that, but that could be due to your internet connection, an slow intel map or the portal density in the area.
As for the public messages: if you just started using the tool and don't use tracedays to limit requests, it will request 30 days of chat, which will result in millions of lines in an active area. To prevent getting banned I circulate through some accounts using the cookie flag.

[sn0opy]

@Camisa To understand how Botguard works, I recommend reading this: https://plus.google.com/u/0/+GavinTreadgold/posts/gtDEL2PAuuM

Keep in mind, that this project won't receive any updates by @breeswish anymore and you may encounter way more issues in the future.

[sasha2002]

node build/app.js

2015-01-19T08:47:25.544Z - info: [MungeDetector] Trying to extract munge data from ingress.com/intel.
2015-01-19T08:47:26.291Z - error: [Auth] Authentication failed. Please update the cookie.

But the cookie is OK

[sn0opy]

@sasha2002 you can stop using ingress-exporter after the last update. It's completely broken now, except you use a lot of stuff from IITC.

Regarding your issue: There's one check, which counts the felds in the returned entity object. Due to the introduction of botguard and especially the last update, the object now returns more than 2 elements. Thus it returns you that error. If you want to fix ingress-exporter, have fun porting this stuff: iitc-project/ingress-intel-total-conversion@8d910a6

[danizwam]

The issue with the new format has been discussed and solved in this thread: #19

Unfortunately, i tried the script a minute ago and i get now a js error:

"Response" is not defined...

/home/walter/ingress-exporter/build/lib/entity.js:84
if (response.capturedTime != null) {
^
ReferenceError: response is not defined
at Object. (/home/walter/ingress-exporter/build/lib/entity.js:84:7)
at Object. (/home/walter/ingress-exporter/build/lib/entity.js:225:4)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (/home/walter/ingress-exporter/build/app.js:37:3)
at Object. (/home/walter/ingress-exporter/build/app.js:112:4)

Did i messed up my scripts by myself, or is it a new issue. can someone please test it?

[sn0opy]

@danizwam thanks, didn't see that issue actually.

Regarding your error: looks like you used "Response" with a capital "R" instead of a lowercase one.

[danizwam]

@sn0opy My fault ;) I kinda messed something up... rebuild from git source and everything works fine ;)

[sasha2002]

Sorry for bothersome but can somebody put this file into project or let me know where i can download it

[danizwam]

@sasha2002 Please look into the issue #19, there is all you need.

[danizwam]

@sasha2002 After altering my config.cson i run into the same error. Cookie was definetely correct. I probably messed something up with Notepad++. I delete my cson and recreated it with nano. That fixed it!

[sasha2002]

Broken again, new botguard modify