align is not working #6

DLAcoding · 2022-04-25T17:12:47Z

I made the configurations said in the Readme but something is missing I think...

When aling is called nothing happens, but if I do the aling manually then is recognised (isAligned =1). I tried to modify the code but cant get the logic behind this point...

Weird thing happens (or maybe not): I have noticed that when isAligned = 1, targetX and targetY usually stay at =-1. Infact, to se values different than -1 I have to not center the navPointImg.

It may be anything related to graphics (like shades, etc?)

Matrixchung · 2022-04-29T12:06:04Z

isAligned comes from checkAlignWithTemplate() which detects the destCircle at the center of the game screen, and it works well on yours.

targetX and targetY come from getNavPointsByCompass() which is the main logic of alignment.
Usually, you dont have to manually set them, and if isAligned then the target navpoint may be undecectable, because it shows at the same position with compass center. In other words, if it is not aligned and targetX/Y !=-1, it seems that the detecting process works fine.

To get inside, you can make the targetX/Y displayed on status screen, or you can modify the sharedMemory array size in imageProcessing and uncomment some lines to get the navPoints image return.

DLAcoding · 2022-04-29T17:58:40Z

I think I understand what you mean: If isAligned (which I can confirm works well) usually the navpoint is not detected so targetX and targetY are =-1. The thing is that most of the time they stay at -1, even when the ship is far from being Alingned with the destCircle. I only was able to see some values distinct than -1 when I manually tried to center de navpoint. When is almost in the center is when some values distinct thant 1 appear for a while.

I made some screenshots of your processed images in a case when is not center and the ship stays totally freeze. ¿Are they seen as they suppose?

Example of not nav point detected

In the meanwhile I've been trying to learn a bit of cv2 in orther to make it work for me. I have not implemented anything yet but was able to achieve this:

Unfortunately when the objective is behind is also detected...I am a bit stucked

Based on this mask:

With this code:

compassImgSmall = screenshot2[int(gameResolution[1] / 1.36):int(gameResolution[1] / 1.25),
                int(gameResolution[0] / 2.76):int(gameResolution[0] / 2.51)]
  compassImgSmall2 = compassImgSmall.copy()
  shv = cv2.cvtColor(compassImgSmall, cv2.COLOR_BGR2HSV)
  #     # Blue color
  lower_bound = np.array([219, 78, 0])
  upper_bound = np.array([255, 255, 184])

  mask = cv2.inRange(compassImgSmall, lower_bound, upper_bound)
  frame = cv2.bitwise_and(compassImgSmall, compassImgSmall, mask=mask)

contours, hierarchy = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    for contour in contours:
        area = cv2.contourArea(contour)

        if area > 10 and area < 50:
            M = cv2.moments(contour)
            if M['m00'] != 0:
                cx = int(M['m10'] / M['m00'])
                cy = int(M['m01'] / M['m00'])
                cv2.circle(compassImgSmall, (cx, cy), 3, (0, 0,255 ), -1)

And last but not least, I also noticed that the screenCapture grabs the EliteDangerous screen with the top border and 8 pixels each side (dont know if it will be appreciated in this capure, look at the left side, some letters are seen from a window at the back...). Does this happens too when you executed?

I modified (deleted both sides and the tittle of the window). and ding ding ding!! offset_button_mission_back which was not clicking well before now it does!

Had to modify this function:

def getWindowRectByHwnd(windowHwnd):
    ctypes.windll.shcore.SetProcessDpiAwareness(2)
    left, top, right, bottom = win32gui.GetWindowRect(windowHwnd)
    w = right - left
    h = bottom - top
    # account for the window border and titlebar and cut them off
    border_pixels = 8
    titlebar_pixels = 30
    w = w - (border_pixels * 2)
    h = h - titlebar_pixels - border_pixels
    left = left + border_pixels
    top = top + titlebar_pixels
    return (left,top,w,h)

Matrixchung · 2022-04-30T07:46:50Z

For the compass detecting :

The method I use is cv2.SimpleBlobDetector, which is better than cv2.matchTemplate in my test.
Here it seems that you use cv2.findContours, and it might be good from your test result but I haven't tried it before. You can compare those two methods, if yours is better I think we can switch to it just with a small bit of codes changed (mainly in getNavPointsByCompass which is modular and just expect targetX/Y and navCenter returned, so it doesn't affect any other part in theory).
for the object at the back of ship (not straight back) , there's no need to worry about it. All you have to do is keeping turning according to point of the navCompass. The direction is constant from the ship's current look whether front or behind.
If the object is straight behind, I thought it can be dealt with since the alignment function need isAligned = 1. In the straight behind situation, the navPoint may be occluded, causing targetX/Y == -1, but isAligned is still not true (it can't find the destCircle). In that case, we may need just a small jitter to the ship trying to change the navpoint, then the problem will be solved. However, that case is completely a "Black swan" and I haven't face it yet :)

For the screenCapture :

I have noticed the grabbing issue, and it might come from win32gui.GetWindowRect() itself.
The reason why I didn't tweak it is that I think changing the base (Left, Top) coordinate may cause misclick in PyAutoGui previously. But now that you solved some clicking issues, it might be necessary to unify all origin coordinate in screenCapture buttonOffset and mouseClick, because I used some relative offsets to prevent misclick in multi-monitor systems, etc.

DLAcoding · 2022-04-30T20:03:38Z

I think I managed to implement my findContours but I noticed that the color changes depending on the system I guess. So first aling worked but in the second the color of the nav change to kind of purple:

Nav in Robigo

Nav in 'Wredguia TH-U c16-18

That breaks my function as I was using the bluemask I configurated, and now its shown totally black as there are no blues on the image...I am about to kill my self if can't get this working xD

Matrixchung · 2022-05-01T02:18:31Z

Maybe try to use grayscale image for processing, there's no need to worry about the color.
Btw, did you test your method's correct rate and robustness? If it is good you can make a PR and merge it

DLAcoding · 2022-05-01T06:38:46Z

Sadly I was not able to find contours in the grayscale image (not sure if there is more image processing to do before triying to find the contours...).

About my method to be honest it only worked once in the first aling. I had to modify the +10 of navCenter = compassRadius+10.0 to another value as I noticed that no matter if the navpoint was up-left, up-right,bottom-right or botton-left, the function always left the NavCenter at the same distance but not totally centered. playing with the value I was able to make it center but didn't understad the logic of adding values to compassRadius. Second try did not worked with the same value so I assume I am doing something wrong.

I am not good at algebra so also dont understad how with the radius of the compass and the coords of the navpoint you are able to calculate the distance and direction you must move the ship...

I modified getNavPointsByCompass, keyPointDetector2 and imageProcessing

def imageProcessing(coordShrName):
    isAligned = 0
    windowHwnd = win32gui.FindWindow(None,globalWindowName)
    while True:
        try: # already has windowHwnd
            gameCoord = getWindowRectByHwnd(windowHwnd)
        except: # gameHwnd changed
            gameCoord,windowHwnd = getWindowRectByName(globalWindowName)
        try:
            startTime = time.time()
            isFocused = isForegroundWindow(globalWindowName,windowHwnd)
            img = pyautogui.screenshot(region=gameCoord)
            # img2 = pyautogui.screenshot(region=gameCoord)
            screenshot = cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR)
            # img_copy = img.copy()  # -> Not sure if needed

            gameResolution = gameCoord[2],gameCoord[3]
            # gameCenterActual = gameCoord[0]+gameCoord[2]/2,gameCoord[1]+gameCoord[3]/2 # 绝对中点 用于鼠标操作
            gameCenterRel = gameCoord[2]/2,gameCoord[3]/2 # 相对中点

            # outsideOffsetY = (gameCoord[3]/3)*2
            cv2OriginImg = cv2.cvtColor(np.asarray(img),cv2.COLOR_RGB2BGR)
            # cv2ShowImg = cv2OriginImg.copy() # ShowImg for Overlay
            cv2GrayImg = cv2.cvtColor(cv2OriginImg,cv2.COLOR_BGR2GRAY)

            centerImg = cv2GrayImg[int(gameCenterRel[1]-180):int(gameCenterRel[1]+180),int(gameCenterRel[0]-220):int(gameCenterRel[0]+220)]
            compassImg = cv2GrayImg[int(gameResolution[1]/1.63):int(gameResolution[1]/1.06),int(gameResolution[0]/4.04):int(gameResolution[0]/2.02)] # Magic Number: size for compass img
        
            compassOriginImg = cv2OriginImg[int(gameResolution[1]/1.63):int(gameResolution[1]/1.06),int(gameResolution[0]/4.04):int(gameResolution[0]/2.02)]
            compassHsv = cv2.cvtColor(compassOriginImg,cv2.COLOR_BGR2HSV)
            # compassShowImg = compassOriginImg.copy() # screen overlay

            compassImgSmall = screenshot[int(gameResolution[1] / 1.36):int(gameResolution[1] / 1.25),
                              int(gameResolution[0] / 2.76):int(gameResolution[0] / 2.51)]  # -> use img_copy just
            # print("Hola")
            if checkAlignWithTemplate(centerImg,destCircleImg) is True: isAligned = 1
            else: isAligned = 0

            # (targetX,targetY),navCenter,compassShowImg,navShowImg = getNavPointsByCompass(compassImg,compassShowImg,compassHsv) # NO IMAGE RETURN NEEDED
            # (targetX,targetY),navCenter = getNavPointsByCompass(compassImg,compassHsv)
            (targetX,targetY),navCenter = getNavPointsByCompass2(compassImg,compassHsv,compassImgSmall)

            shr_coord = shared_memory.SharedMemory(name=coordShrName)
            coordArray = np.ndarray(shape=8,dtype=np.float64,buffer=shr_coord.buf)
            elapsedTime = time.time()-startTime
            coordArray[:] = [targetX,targetY,navCenter,isAligned,isFocused,elapsedTime,gameCoord[0],gameCoord[1]]  
        except Exception as e:
            pass
            print(e)

def getNavPointsByCompass2(compassImg,compassHsv,compassImgSmall): # NO IMAGE RETURN NEEDED
    """Mi versión"""
    global navPointsPrevX,navPointsPrevY
    try:
        # print("GettingNavPoints")
        shv = cv2.cvtColor(compassImgSmall, cv2.COLOR_BGR2HSV)
        #     # Blue color
        lower_bound = np.array([219, 78, 0])
        upper_bound = np.array([255, 255, 184])

        mask = cv2.inRange(compassImgSmall, lower_bound, upper_bound)


        binary = cv2.GaussianBlur(mask,(3,3),0)
        circles = cv2.HoughCircles(binary, method=cv2.HOUGH_GRADIENT,dp=1,minDist=200,param1=50,param2=48,minRadius=20,maxRadius=30)
        if circles is not None:

            circles = circles[0,:]
            compassX,compassY,compassRadius=circles[0]

            if compassRadius !=0 :

                navCenter = compassRadius +20

                # fin mi parte?


                navPoints = keyPointDetector2(mask)

                if navPoints is not None:

                    targetX = navPoints[0]
                    targetY = navPoints[1] # change to float
                    print(targetX,targetY)
                    if navPointsPrevX == -1.0 or navPointsPrevY == -1.0: # initialize
                        navPointsPrevX = targetX
                        navPointsPrevY = targetY
                    elif abs(navPointsPrevX-targetX)>=40 or abs(navPointsPrevY-targetY)>=40: # 滤波
                        targetX = navPointsPrevX
                        targetY = navPointsPrevY
                    else:
                        navPointsPrevX = targetX
                        navPointsPrevY = targetY
                    
                else:

                    targetX=targetY=-1.0
        else:
            targetX=targetY=navCenter=-1.0
            # navShowImg = None
    except Exception as e:
        print('Error in getNavPointsByCompass()',e)
        # exc_type, exc_obj, exc_tb = sys.exc_info()
        # fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        # print(exc_type, fname, exc_tb.tb_lineno)
        # traceback.print_exc()
        targetX=targetY=navCenter=-1.0
        # navShowImg = None
    # return (targetX,targetY),navCenter,compassShowImg,navShowImg
    return (targetX,targetY),navCenter # NO IMAGE RETURN NEEDED

def keyPointDetector2(mask):
    contours, hierarchy = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    for contour in contours:
        area = cv2.contourArea(contour)

        if area > 10 and area < 50:
            M = cv2.moments(contour)
            if M['m00'] != 0:
                cx = int(M['m10'] / M['m00'])
                cy = int(M['m01'] / M['m00'])
                print(cx,cy)
                return cx,cy
    return None

Matrixchung · 2022-05-01T13:02:04Z

reimplemented the image showing part.
everything goes ok here
I think findContours might be designed for a more complex pattern instead of a small dot, when SimpleBlobDetector could be much easier to find it.

Matrixchung · 2022-05-01T13:16:45Z

alright, see 5c4ed08

DLAcoding · 2022-05-02T17:15:38Z

Hi again! I have modified my graphic settings and now aling is working much better!! There must be something about the configuration...That said, I couldn't finish a whole run yet because when the objective is near the Nav edge it cant found it:

For example, here is not detected:

Do you think is factible to have a kind of timer that if target X is not foundin 10 seconds it gives a random thrust up?

Following the route throuht Wredguia TH-U c16-18 found this situation several times in the route.

DLAcoding · 2022-05-02T19:27:45Z

Another weird bug: Dont know what may be detecting, cant see any other bright light

Matrixchung · 2022-05-03T03:40:04Z

Update hsv filters bdd3967
I wonder if I have to change the UI color because it is unrecognizable when facing sunlight
and there's a UI system & auto keybindings on the way

DLAcoding · 2022-05-03T18:52:24Z

Now its working much better but still some times gets stuck:

Matrixchung · 2022-05-04T02:02:08Z

mistakes are normal sometimes. They usually wont interfere the result, just taking more time

…

________________________________ From: DLAcoding ***@***.***> Sent: Wednesday, May 4, 2022 2:52:34 AM To: Matrixchung/EDAutopilot ***@***.***> Cc: Matr1x. ***@***.***>; Comment ***@***.***> Subject: Re: [Matrixchung/EDAutopilot] align is not working (Issue #6) Now its working much better but still some times gets stuck: [image]<https://user-images.githubusercontent.com/28142896/166522563-98c41b06-8ca4-4a58-b1ba-c9baea4058da.png> [image]<https://user-images.githubusercontent.com/28142896/166522617-96dd5e94-3e5d-4889-a4f7-f25b5010f0ad.png> [image]<https://user-images.githubusercontent.com/28142896/166522653-186c2d6b-4a32-49a4-8e47-a877fc26d656.png> [image]<https://user-images.githubusercontent.com/28142896/166522676-beec6629-3e89-43d1-ac02-00f341c09e38.png> — Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AE4OVWR6GN62FOFGENYQEQ3VIFYXFANCNFSM5UJHFHYQ>. You are receiving this because you commented.Message ID: ***@***.***>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

align is not working #6

align is not working #6

DLAcoding commented Apr 25, 2022

Matrixchung commented Apr 29, 2022

DLAcoding commented Apr 29, 2022 •

edited

Loading

Matrixchung commented Apr 30, 2022

DLAcoding commented Apr 30, 2022

Matrixchung commented May 1, 2022

DLAcoding commented May 1, 2022

Matrixchung commented May 1, 2022

Matrixchung commented May 1, 2022

DLAcoding commented May 2, 2022

DLAcoding commented May 2, 2022

Matrixchung commented May 3, 2022

DLAcoding commented May 3, 2022

Matrixchung commented May 4, 2022 via email

align is not working #6

align is not working #6

Comments

DLAcoding commented Apr 25, 2022

Matrixchung commented Apr 29, 2022

DLAcoding commented Apr 29, 2022 • edited Loading

Matrixchung commented Apr 30, 2022

For the compass detecting :

For the screenCapture :

DLAcoding commented Apr 30, 2022

Matrixchung commented May 1, 2022

DLAcoding commented May 1, 2022

Matrixchung commented May 1, 2022

Matrixchung commented May 1, 2022

DLAcoding commented May 2, 2022

DLAcoding commented May 2, 2022

Matrixchung commented May 3, 2022

DLAcoding commented May 3, 2022

Matrixchung commented May 4, 2022 via email

DLAcoding commented Apr 29, 2022 •

edited

Loading