如何使用AccessibilityService在Android上执行拖动(基于X,Y鼠标坐标)?

2019年8月1日 31点热度 0条评论

我想知道如何在基于X,Y鼠标坐标的android上执行拖动吗?考虑两个简单的示例,Team Viewer / QuickSupport分别在远程智能手机和Windows Paint Pen上绘制“密码模式”。

我所能做的就是simulate touch(带有dispatchGesture()AccessibilityNodeInfo.ACTION_CLICK)。

我找到了这些相关链接,但不知道它们是否有用:

  • Perform swipe on screen using AccessibilityService
  • Example 1
  • Continued gestures
  • 以下是我的工作代码,该代码用于将鼠标坐标(在
    PictureBox控件内部)发送到远程电话并模拟触摸。

    Windows窗体应用程序:

    private void pictureBox1_MouseDown(object sender, MouseEventArgs e)
    {
        foreach (ListViewItem item in lvConnections.SelectedItems)
        {
            // Remote screen resolution
            string[] tokens = item.SubItems[5].Text.Split('x'); // Ex: 1080x1920
    
            int xClick = (e.X * int.Parse(tokens[0].ToString())) / (pictureBox1.Size.Width);
            int yClick = (e.Y * int.Parse(tokens[1].ToString())) / (pictureBox1.Size.Height);
    
            Client client = (Client)item.Tag;
    
            if (e.Button == MouseButtons.Left)
                client.sock.Send(Encoding.UTF8.GetBytes("TOUCH" + xClick + "<|>" + yClick + Environment.NewLine));
        }
    }
    


    编辑:

    我的最后一次尝试是分别使用鼠标坐标(“C#Windows窗体应用程序”)和自定义android例程(引用上面链接的“滑动屏幕”的代码)进行“滑动屏幕”:

    private Point mdownPoint = new Point();
    
    private void pictureBox1_MouseDown(object sender, MouseEventArgs e)
    {
        foreach (ListViewItem item in lvConnections.SelectedItems)
        {
            // Remote screen resolution
            string[] tokens = item.SubItems[5].Text.Split('x'); // Ex: 1080x1920
    
            Client client = (Client)item.Tag;
    
            if (e.Button == MouseButtons.Left)
            {
                xClick = (e.X * int.Parse(tokens[0].ToString())) / (pictureBox1.Size.Width); 
                yClick = (e.Y * int.Parse(tokens[1].ToString())) / (pictureBox1.Size.Height);
    
                // Saving start position:
    
                mdownPoint.X = xClick; 
                mdownPoint.Y = yClick; 
    
                client.sock.Send(Encoding.UTF8.GetBytes("TOUCH" + xClick + "<|>" + yClick + Environment.NewLine));
            }
        }
    }
    
    private void PictureBox1_MouseMove(object sender, MouseEventArgs e)
    {
        foreach (ListViewItem item in lvConnections.SelectedItems)
        {
            // Remote screen resolution
            string[] tokens = item.SubItems[5].Text.Split('x'); // Ex: 1080x1920
    
            Client client = (Client)item.Tag;
    
            if (e.Button == MouseButtons.Left)
            {
                xClick = (e.X * int.Parse(tokens[0].ToString())) / (pictureBox1.Size.Width);
                yClick = (e.Y * int.Parse(tokens[1].ToString())) / (pictureBox1.Size.Height);
    
                client.sock.Send(Encoding.UTF8.GetBytes("MOUSESWIPESCREEN" + mdownPoint.X + "<|>" + mdownPoint.Y + "<|>" + xClick + "<|>" + yClick + Environment.NewLine));
            }
        }
    }
    

    android AccessibilityService:

    public void Swipe(int x1, int y1, int x2, int y2, int time) {
    
    if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.N) {
        System.out.println(" ======= Swipe =======");
    
        GestureDescription.Builder gestureBuilder = new GestureDescription.Builder();
        Path path = new Path();
        path.moveTo(x1, y1);
        path.lineTo(x2, y2);
    
        gestureBuilder.addStroke(new GestureDescription.StrokeDescription(path, 100, time));
        dispatchGesture(gestureBuilder.build(), new GestureResultCallback() {
            @Override
            public void onCompleted(GestureDescription gestureDescription) {
                System.out.println("SWIPE Gesture Completed :D");
                super.onCompleted(gestureDescription);
            }
        }, null);
    }
    
    }
    

    会产生以下结果(但仍然无法绘制“图案密码”,例如TeamViewer)。但是,就像在下面的评论中说的那样,我认为可以使用
    Continued gestures来实现类似的方法。欢迎向这个方向提出任何建议。

    编辑2:

    肯定地,解决方案是
    continued gestures,就像之前的
    上所说的那样编辑

  • Simulating joystick movement using AccessibilityService
  • Why the continueStroke function is not work
  • 下面是我发现的
    here =>的假定固定代码

    android AccessibilityService:

    // Simulates an L-shaped drag path: 200 pixels right, then 200 pixels down.
    Path path = new Path();
    path.moveTo(200,200);
    path.lineTo(400,200);
    
    final GestureDescription.StrokeDescription sd = new GestureDescription.StrokeDescription(path, 0, 500, true);
    
    // The starting point of the second path must match
    // the ending point of the first path.
    Path path2 = new Path();
    path2.moveTo(400,200);
    path2.lineTo(400,400);
    
    final GestureDescription.StrokeDescription sd2 = sd.continueStroke(path2, 0, 500, false); // 0.5 second
    
    HongBaoService.mService.dispatchGesture(new GestureDescription.Builder().addStroke(sd).build(), new AccessibilityService.GestureResultCallback(){
    
    @Override
    public void onCompleted(GestureDescription gestureDescription){
    super.onCompleted(gestureDescription);
    HongBaoService.mService.dispatchGesture(new GestureDescription.Builder().addStroke(sd2).build(),null,null);
    }
    
    @Override
    public void onCancelled(GestureDescription gestureDescription){
    super.onCancelled(gestureDescription);
    }
    },null);
    

    然后,我的疑问是:
    如何为上面的代码正确发送鼠标坐标,从而可以向任何方向执行拖动? 有什么主意吗?

    编辑3:

    我发现了两个用于执行拖动的例程,但是它们使用的是
    UiAutomation +
    injectInputEvent() 。 AFAIK,事件注入(inject)仅在诸如
    here
    here之类的系统应用程序中有效,我不想要它。

    这是发现的例程:

  • public boolean swipe(int downX, int downY, int upX, int upY, int steps, boolean drag)
  • public boolean swipe(Point[] segments, int segmentSteps)
  • 然后要实现我的目标,我认为第二个例程更适合与
    上显示的代码一起使用(遵循逻辑,不包括事件注入(inject))编辑2 并分别发送
    pictureBox1_MouseDown
    pictureBox1_MouseMove(C#Windows Forms Application)的所有点来填充
    Point[]动态地发送,并在
    pictureBox1_MouseUp上发送cmd以执行例程并使用此数组填充。如果您对第一个例程有想法,请告诉我:D。

    如果在阅读完此
    之后,可以解决此问题,请在答案中告诉我,而我将尝试测试该想法。

    解决方案如下:

    这是基于编辑3 问题的解决方案示例。

    C#Windows Froms应用程序“formMain.cs”:

    using System.Net.Sockets;
    
    private List<Point> lstPoints;
    
    private void pictureBox1_MouseDown(object sender, MouseEventArgs e) 
    {
        if (e.Button == MouseButtons.Left)
        {
            lstPoints = new List<Point>();
            lstPoints.Add(new Point(e.X, e.Y));
        }
    }
    
    private void PictureBox1_MouseMove(object sender, MouseEventArgs e)
    {
        if (e.Button == MouseButtons.Left)
        {
            lstPoints.Add(new Point(e.X, e.Y));
        }
    }
    
    private void PictureBox1_MouseUp(object sender, MouseEventArgs e)
    {
        lstPoints.Add(new Point(e.X, e.Y));
    
        StringBuilder sb = new StringBuilder();
    
        foreach (Point obj in lstPoints)
        {
            sb.Append(Convert.ToString(obj) + ":");
        }
    
        serverSocket.Send("MDRAWEVENT" + sb.ToString() + Environment.NewLine);
    }
    

    android服务“SocketBackground.java”:

    import java.net.Socket;
    
    String xline;
    
    while (clientSocket.isConnected()) {
    
        BufferedReader xreader = new BufferedReader(new InputStreamReader(clientSocket.getInputStream(), StandardCharsets.UTF_8));
    
        if (xreader.ready()) {
    
            while ((xline = xreader.readLine()) != null) {
                    xline = xline.trim();
    
                if (xline != null && !xline.trim().isEmpty()) {
    
                    if (xline.contains("MDRAWEVENT")) {
    
                        String coordinates = xline.replace("MDRAWEVENT", "");
                        String[] tokens = coordinates.split(Pattern.quote(":"));
                        Point[] moviments = new Point[tokens.length];
    
                        for (int i = 0; i < tokens.length; i++) {
                           String[] coordinates = tokens[i].replace("{", "").replace("}", "").split(",");
    
                           int x = Integer.parseInt(coordinates[0].split("=")[1]);
                           int y = Integer.parseInt(coordinates[1].split("=")[1]);
    
                           moviments[i] = new Point(x, y);
                        }
    
                        MyAccessibilityService.instance.mouseDraw(moviments, 2000);
                    }
                }
            }
        }
    }
    

    android AccessibilityService “MyAccessibilityService.java”:

    public void mouseDraw(Point[] segments, int time) {
        if (android.os.Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
    
            Path path = new Path();
            path.moveTo(segments[0].x, segments[0].y);
    
            for (int i = 1; i < segments.length; i++) {
    
                path.lineTo(segments[i].x, segments[i].y);
    
                GestureDescription.StrokeDescription sd = new GestureDescription.StrokeDescription(path, 0, time);
    
                dispatchGesture(new GestureDescription.Builder().addStroke(sd).build(), new AccessibilityService.GestureResultCallback() {
    
                    @Override
                    public void onCompleted(GestureDescription gestureDescription) {
                        super.onCompleted(gestureDescription);
                    }
    
                    @Override
                    public void onCancelled(GestureDescription gestureDescription) {
                        super.onCancelled(gestureDescription);
                    }
                }, null);
            }
        }
    }