Multiple Agents #25
Comments
|
Yeah this prompted me to do some design thinking.. how do we deal with
multiple bot versions playing bot vs bot?
For sure though everything should be governed by the Actions proto message.
Whatever fits in there should just be blindly executed by the respective
bot.
…On Wed, Dec 19, 2018, 14:30 Nostrademous ***@***.*** wrote:
Since we are moving to multi-agent design I wanted to raise the question
of how many agents do we actually want.
At start there are two possibilities:
1. One Agent Per Hero (5 agents per Team)
2. One Master Agent that controls 5 heroes
*3) One Agent Per Hero and Per Controllable Unit (meaning summons
(Nerconomicon, Warlock Ultimate, Beastmaster Hawk & Boar, etc.), illusions
(Manta, Illusion Rune, Wall of Replica, Disruption, etc.), clones (Meepo),
doubles (Arc Warden Ultimate) or dominated (i.e., Helm of Dominator,
Enchantress, Chen, etc.) units) - typically these are controlled by the
same agent that is controlling the owning Hero that caused the
summon/domination to occur, but doesn't have to be.
Each approach has its pros and cons in terms of flexibility (co-play with
humans, etc.), agent action synchronization (via shared world state model
versus independent world models), training time, and so forth. OpenAI went
with #1 <#1> above, but we
do not necessarily have to, although I think I would prefer #1
<#1> above as well.
Next, there are a few other things that could be their own agents:
a) Courier - there is one courier per team - it can be controlled by any
Hero Agent and even in real games it often gets abused by numerous players
at once. We could make it its own agent.
b) Glyph Usage - anyone on the team can use the Glyph (when available)
c) Minimap Scan - anyone on the team can use the Minimap Scan function
*d) One Team Agent to control a), b) and c) as a single entity
Just wanted to throw this out there to think about and post your thoughts.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#25>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRPBRxT2Nq2TzivU41yhfL-ybpo0Zks5u6j-QgaJpZM4ZaP5G>
.
|
|
By the way, I see that you are using "auto-generated files" as a means to communicate between the bots and the agent. This assumes they always exist on the same hardware, basically disallowing the possibility of dotaservice running on one physical host while the agent code runs on a different one (well, short of doing network file transfers between the two). I previously implemented the comms method of using CreateHTTPRequest() API (and I believe there is also a CreateRemoteHTTPRequest() API - could be wrong - Valve kept flip-flopping between allowing this and not): While not necessarily "reliable", it is WAY faster than using files as a comms method. Currently I'm seeing the Think() function take 18-20ms for Nevermore, whereas I believe the method I had above was about 10x less time. I believe this has to do with "loadfile" being rather slow, but I could be wrong. |
|
The http thing is unreliable. The lua wrapper sucks too, the callback only
comes in after 2 frames, and the callback isn't purely async: you can't
wait for it. So: you cannot do anything synchronously. The lua files are
the way things keep synchronous. Notice I am running at 8x host_timescale
on average, and I play smoothly with the bots to far. 10~15 ms is fine. The
way you want to roll something like this out to the public is into a lua
file that has the weights, or some cpp library, not through dotaserivce.
You need to be synchronous to keep sanity or you will go bananas. The http
callback is also slow, because you're guaranteed you have to wait for 2
ticks to get your callback executed, so that's guaranteed to be at least
2/30 seconds latency.
…On Wed, Dec 19, 2018 at 3:08 PM Nostrademous ***@***.***> wrote:
By the way, I see that you are using "auto-generated files" as a means to
communicate between the bots and the agent. This assumes they always exist
on the same hardware, basically disallowing the possibility of dotaservice
running on one physical host while the agent code runs on a different one
(well, short of doing network file transfers between the two).
I previously implemented the comms method of using CreateHTTPRequest() API
(and I believe there is also a CreateRemoteHTTPRequest() API - could be
wrong - Valve kept flip-flopping between allowing this and not):
https://github.com/Nostrademous/Dota2-WebAI/blob/master/webserver_out.lua#L443-L467
While not necessarily "reliable", it is WAY faster than using files as a
comms method. Currently I'm seeing the Think() function take 18-20ms for
Nevermore, whereas I believe the method I had above was about 10x less
time. I believe this has to do with "loadfile" being rather slow, but I
could be wrong.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRJWyK6WFkBQk1iDw6RQ1q0B1ilHHks5u6khzgaJpZM4ZaP5G>
.
|
|
I'm fine with file passing providing it's "fast enough" which I believe it is. Back to the initial question though - as we slowly transition to 5 agents and self-play (10 agents) - what about courier/glyph/scan functions - are they part of each agent or their own entities? |
|
The Actions protobuf allows each actor to do this, so that's what makes
sense. When when we know the C++ API, we can use that immediatelly and
scrap half of the dotaservice's code.
https://github.com/TimZaman/dotaservice/tree/master/botcpp
…On Wed, Dec 19, 2018 at 5:28 PM Nostrademous ***@***.***> wrote:
I'm fine with file passing providing it's "fast enough" which I believe it
is.
Back to the initial question though - as we slowly transition to 5 agents
and self-play (10 agents) - what about courier/glyph/scan functions - are
they part of each agent or their own entities?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSREMYfNFZmkOqjkLFVbN1un1c80Yoks5u6mkhgaJpZM4ZaP5G>
.
|
|
which library/executable did the "Init" reside in? I guess I'm asking if you know if the functions I am to find the signatures for exist in the main dota2 binary or a specific library that gets loaded for handling bots |
|
libserver.dylib calls them. There are exactly 4 symbols expected, but we
have zero knowledge of the _full_ signature.
…On Wed, Dec 19, 2018 at 8:44 PM Nostrademous ***@***.***> wrote:
which library/executable did the "Init" reside in?
I guess I'm asking if you know if the functions I am to find the
signatures for exist in the main dota2 binary or a specific library that
gets loaded for handling bots
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRPuiP8af3GE4ReWPrynikd7uWctFks5u6pcGgaJpZM4ZaP5G>
.
|
|
Observe(size_t teamID, void *, void *) First arg is team ID. 2 for Radiant, 3 for Dire. Busy with kids right now but I believe 2nd will be pointer to protobuf. |
|
Yep
void Observe(int team_id, const CMsgBotWorldState& ws)
…On Thu, Dec 20, 2018, 01:07 Nostrademous ***@***.*** wrote:
Observe(size_t teamID, void *, void *)
First arg is team ID. 2 for Radiant, 3 for Dire.
Busy with kids right now but I believe 2nd will be pointer to protobuf.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRLIKfukALe-3DaBKQDRfcJiIyVKMks5u6tS0gaJpZM4ZaP5G>
.
|
That's the callstack to get to our Observe call |
|
The only open q is: what should Act return? |
|
I agree, makes the most sense, but wouldn't it have to be wrapped in a CMsgBotWorldState anyways? |
|
How? What do you mean? |
|
Well, isn't the Msg Actions protobuf embedded inside the CMsgBotWorldState? |
|
It's just a submessage of it, not a field though.
…On Thu, Dec 20, 2018 at 5:30 AM Nostrademous ***@***.***> wrote:
Well, isn't the Msg Actions protobuf embedded inside the CMsgBotWorldState?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRKiYRpZwhmQN7cdKwlzgSIk2ywkAks5u6xJUgaJpZM4ZaP5G>
.
|
|
Okay, at least reading the Observe() is done. Example Code: Example Output: |
|
yep. again, the only remaining question is: what does act return?
…On Thu, Dec 20, 2018 at 5:33 AM Nostrademous ***@***.***> wrote:
Okay, at least reading the Observe() is done.
Example Code:
extern "C" void Observe(int teamID, const CMsgBotWorldState &ws) {
// The intention of this script is most probably to setup a connection with the
// worldstate sockets. Kinda weird though, because the dota game would be the server
// and then this script would be the client of that..
// Observe is called every tick that corresponds to the worldstate server's ticks,
// Exactly before the Act() is called.
cout << "Observe::cout" << endl;
cout << "TeamID: " << ws.team_id() << endl;
cout << "DotaTime: " << ws.dota_time() << endl;
cout << "GameState: " << ws.game_state() << endl;
for (CMsgBotWorldState_Player player : ws.players() ) {
cout << "\tPlayerID: " << player.player_id() << endl;
cout << "\tHeroID: " << player.hero_id() << endl;
}
}
Example Output:
Observe::cout
TeamID: 2
DotaTime: -77.2335
GameState: 4
PlayerID: 0
HeroID: 11
PlayerID: 1
HeroID: 35
PlayerID: 2
HeroID: 35
PlayerID: 3
HeroID: 35
PlayerID: 4
HeroID: 35
PlayerID: 5
HeroID: 35
PlayerID: 6
HeroID: 35
PlayerID: 7
HeroID: 35
PlayerID: 8
HeroID: 35
PlayerID: 9
HeroID: 35
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRFPvhsdK3NpFSoZ2AkeH_LMBfwi0ks5u6xMZgaJpZM4ZaP5G>
.
|
|
Well, we could do a guessing game, but how will we know when we are right? |
|
When we see some movement! It's 99.9% the Actions protobuf object.
…On Thu, Dec 20, 2018 at 5:36 AM Nostrademous ***@***.***> wrote:
Well, we could do a guessing game, but how will we know when we are right?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRChqrYkmXA0kcHdUnsX_7Wdpgxiqks5u6xPpgaJpZM4ZaP5G>
.
|
|
Well I agree... so easy to test. |
|
So it's (probably) const CMsgBotWorldState::Actions& Act(int team_id)But the action is: what are the strictly required values? Or possibly, we might need to return something in Init (or Observe) that toggles some functionality. And still: why is it that we need to keep the worldstate sockets open? |
|
So I assume we need to open the socket in Init() just to trigger pumping the world state through the ports. |
|
could it not be: I'm thinking that way we don't have to allocate the protobuf and keep track of the memory. Perhaps an INOUT variable. |
|
Tried a bunch of stuff, can't say i was successful on Act() yet. |
|
I think maybe init should register which bots it is going to control or
something.
…On Thu, Dec 20, 2018 at 6:47 AM Nostrademous ***@***.***> wrote:
Tried a bunch of stuff, can't say i was successful on Act() yet.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRP3_aHJS6z9ux1Agvr3x1g1723m0ks5u6yRngaJpZM4ZaP5G>
.
|
|
Will check back tomorrow and work some more on it. 1am here and I have to get up at 6am. Calling it a night for now. |
|
I'm thinking there is a 2nd arg to Act() (only valid after game starts, it's NULL during hero selection. I believe it might be the ArenaAllocator(). Currently this compiles and doesn't crash but still doesn't move. BTW, as expected, the first arg of Init() is int teamID |
|
Notes for self: |
|
It's also very much possible the 2nd argument is actually a pointer to an already created [empty] Actions protobuf... |
|
Function that calls our botcpp_*.so 0x8931f3f3 is where Act() is called |
I thought that too, but when I tried dumping it to string I would get a crash. |
|
I believe Line #153 is call to Observe() |
|
|
|
Regarding the first IDA image I posted (2 images above) I believe Line #153 is call to Observe() |
|
By the way... it looks like the intention is for us to be able to just call functions to take actions directly inside the Act(), and not use any protobuf. It just happens that in the Observe call the 2nd arg points the internal CMsgBot CLASS to the right offset so we sit at the WorldState implementation, but the class itself is full of virtual functions and other information. When they pass the same data to Act() its probably for purposes of orientation but the intent might be to directly invoke using function pointers the appropriate actions using the provided class. |
|
The code is something like this: Our DLL sits inside the callDynamicallyLoadedLibrary Function |
|
That would mean you need headers? And that wouldnt make much sense since
its a dlopen..
…On Fri, Dec 21, 2018, 16:55 Nostrademous ***@***.*** wrote:
By the way... it looks like the intention is for us to be able to just
call functions to take actions directly inside the Act(), and not use any
protobuf. It just happens that in the Observe call the 2nd arg points the
internal CMsgBot CLASS to the right offset so we sit at the WorldState
implementation, but the class itself is full of virtual functions and other
information.
When they pass the same data to Act() its probably for purposes of
orientation but the intent might be to directly invoke using function
pointers the appropriate actions using the provided class.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRAo1dWJ3tknEpJhOZFEpE2TLI4hNks5u7QRmgaJpZM4ZaP5G>
.
|
|
Inside the callDynamicallyLoadedLibrary functions is a stub that eventually does: |
You don't need headers, you just extern anything you want to call that exists somewhere else; and honestly, in reality most of those functions would be access via function pointers available inside the main CMsgBot class that's passed to us (we just don't know what offsets points to what yet). |
|
Btw, OpenAI are doing all of this with `LD_PRELOAD`. But i am entirely
unsure which calls they are overloading.
On Fri, Dec 21, 2018 at 8:38 AM Nostrademous <[email protected]>
wrote:
… That would mean you need headers? And that wouldnt make much sense since
its a dlopen..
… <#m_1243224858975479198_>
On Fri, Dec 21, 2018, 16:55 Nostrademous ***@***.*** wrote: By the way...
it looks like the intention is for us to be able to just call functions to
take actions directly inside the Act(), and not use any protobuf. It just
happens that in the Observe call the 2nd arg points the internal CMsgBot
CLASS to the right offset so we sit at the WorldState implementation, but
the class itself is full of virtual functions and other information. When
they pass the same data to Act() its probably for purposes of orientation
but the intent might be to directly invoke using function pointers the
appropriate actions using the provided class. — You are receiving this
because you commented. Reply to this email directly, view it on GitHub <#25
(comment)
<#25 (comment)>>,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHXSRAo1dWJ3tknEpJhOZFEpE2TLI4hNks5u7QRmgaJpZM4ZaP5G
.
You don't need headers, you just extern anything you want to call that
exists somewhere else; and honestly, in reality most of those functions
would be access via function pointers available inside the main CMsgBot
class that's passed to us (we just don't know what offsets points to what
yet).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#25 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHXSRJ5n4EKrPMGhRVboiXsIpzUhV1oaks5u7Q6EgaJpZM4ZaP5G>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Since we are moving to multi-agent design I wanted to raise the question of how many agents do we actually want.
At start there are two possibilities:
*3) One Agent Per Hero and Per Controllable Unit (meaning summons (Nerconomicon, Warlock Ultimate, Beastmaster Hawk & Boar, etc.), illusions (Manta, Illusion Rune, Wall of Replica, Disruption, etc.), clones (Meepo), doubles (Arc Warden Ultimate) or dominated (i.e., Helm of Dominator, Enchantress, Chen, etc.) units) - typically these are controlled by the same agent that is controlling the owning Hero that caused the summon/domination to occur, but doesn't have to be.
Each approach has its pros and cons in terms of flexibility (co-play with humans, etc.), agent action synchronization (via shared world state model versus independent world models), training time, and so forth. OpenAI went with #1 above, but we do not necessarily have to, although I think I would prefer #1 above as well.
Next, there are a few other things that could be their own agents:
a) Courier - there is one courier per team - it can be controlled by any Hero Agent and even in real games it often gets abused by numerous players at once. We could make it its own agent.
b) Glyph Usage - anyone on the team can use the Glyph (when available)
c) Minimap Scan - anyone on the team can use the Minimap Scan function
*d) One Team Agent to control a), b) and c) as a single entity
Just wanted to throw this out there to think about and post your thoughts.
The text was updated successfully, but these errors were encountered: