Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Linux support to qc_handle_{accept,bind,connect} functions. #23

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

MCApollo
Copy link

@MCApollo MCApollo commented Aug 6, 2020

Handles issue #22, open to changes.

EDIT

Help Wanted:
tunnel out:ip:port returns a deadlock error:
accept failed: 35 - I can't get a breakpoint to hit.

@sickcodes
Copy link

sickcodes commented Aug 7, 2020

Builds on both Linux and OSX
On the first try, it worked, it locked up the machine but the port was definitely open.

Then I built the new tcp tunnel on OSX with the new qemu and I couldn't SSH in on OSX.

Brought the hfs back into linux and SSH tunnel fails, but fails to connect. It doesn't give the 255 error like before.

I'll try again shortly

but stalls without adding the new tcp-tunnel from xnu-qemu-arm64-tools.

export XNU_SOURCES=$PWD/darwin-xnu
export KERNEL_SYMBOLS_FILE=$PWD/symbols.nm
export QEMU_DIR=$PWD/xnu-qemu-arm64/
export NUM_BLOCK_DEVS=2
export KERNEL_CACHE=$PWD/kernelcache.release.n66.out
export DTB_FIRMWARE=$PWD/Firmware/all_flash/DeviceTree.n66ap.im4p.out
export DRIVER_FILENAME=$PWD/aleph_bdev_drv.bin
export IOS_DIR=$PWD
export QEMU_DIR=$PWD/xnu-qemu-arm64/
export HFS_MAIN=$PWD/hfs.main
export HFS_SEC=$PWD/hfs.sec

cd xnu-qemu-arm64

# guest cross-compilers enabled: aarch64-linux-gnu-gcc
./configure \
--target-list=aarch64-softmmu \
--disable-capstone \
--disable-werror \
--disable-pie \

xnu-qemu-arm64/aarch64-softmmu/qemu-system-aarch64 \
-M iPhone6splus-n66-s8000,kernel-filename=${KERNEL_CACHE},dtb-filename=${DTB_FIRMWARE},driver-filename=${DRIVER_FILENAME},qc-file-0-filename=${HFS_MAIN},qc-file-1-filename=${HFS_SEC},kern-cmd-args="debug=0x8 kextlog=0xfff cpus=1 rd=disk0 serial=2",xnu-ramfb=off \
-cpu max \
-m 6G \
-serial mon:stdio \
-vga std \




export PATH=$PATH:/iosbinpack64/usr/bin:/iosbinpack64/bin:/iosbinpack64/usr/sbin:/iosbinpack64/sbin:/sbin:/bin:/usr/sbin:/usr/bin

@MCApollo
Copy link
Author

MCApollo commented Aug 7, 2020

In the tcp-tunnel’s Makefile, uncomment the part about FLAG += -DLINUX; Make sure the filesystem is non-dirty before transferring or use hfsprogs to check the filesystem.

No real easy way to have a ‘universal’ binary, EDEADLK is 11 on sys/errno.h on Darwin- replacing EAGAIN.

If you’ve verified issues on macos with lldb, let me know. Nothing was changed for macos except making structs for sockaddr a alias they were originally were.

@sickcodes
Copy link

sickcodes commented Aug 7, 2020

Alright, will try now.

If I use the non-patched tcp-tunnel, I can almost fully connect:

$ ssh -vvvvv localhost -p2222
OpenSSH_8.3p1, OpenSSL 1.1.1g  21 Apr 2020
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "localhost" port 2222
debug2: ssh_connect_direct
debug1: Connecting to localhost [::1] port 2222.
debug1: connect to address ::1 port 2222: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 2222.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 0
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.3
kex_exchange_identification: Connection closed by remote host
Connection closed by 127.0.0.1 port 2222

# unpatched version btw

Trying your tips now

My HFS is always dirty on Linux, I'll check it out, does it get dirty if I boot it on OSX first?

@sickcodes
Copy link

FYI, trying your steps now

### patched tcp-tunnel
$ fsck.hfsplus hfs_new.main
** hfs_new.main
   Executing fsck_hfs (version 540.1-Linux).
** Checking non-journaled HFS Plus Volume.
** Detected a case-sensitive volume.
   The volume name is PeaceB16B92.arm64UpdateRamDisk
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
** Checking volume information.
** The volume PeaceB16B92.arm64UpdateRamDisk appears to be OK.
### upatched tcp-tunnel
$ fsck.hfsplus hfs.main
** hfs.main
   Executing fsck_hfs (version 540.1-Linux).
** Checking non-journaled HFS Plus Volume.
** Detected a case-sensitive volume.
   The volume name is PeaceB16B92.arm64UpdateRamDisk
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
** Checking volume information.
** The volume PeaceB16B92.arm64UpdateRamDisk appears to be OK.

@sickcodes
Copy link

sickcodes commented Aug 7, 2020

Okay I think it got dirty when I rsynced it from the KVM guest to host (compressed), about to test!

@sickcodes
Copy link

sickcodes commented Aug 7, 2020

@MCApollo amazing dude!

-bash-4.4# uname -a
Darwin localhost 18.2.0 Darwin Kernel Version 18.2.0: Tue Oct 16 21:02:23 PDT 2018; root:xnu-4903.222.5~1/RELEASE_ARM64_S8000 iPhone8,2

Will test the other way to make sure

@sickcodes
Copy link

Handles issue #22, open to changes.

EDIT

Help Wanted:

tunnel out:ip:port returns a deadlock error:

accept failed: 35 - I can't get a breakpoint to hit.

I get the same error, unless I tunnel in. I tried to tunnel in

sshpass -p "alpine" ssh -N [email protected] -p 2222 5900:127.0.0.1:5900

Was trying to run VNC, signed screendumpd launch daemon with jtool and I can reach the VNC server and get a response via loop back tunneling except something about the VNC doesn't work.

I cant tunnel out I also get error 35, I can only have one connection at a time.

The VNC drops and the error is BigEndian = 0 or something. Can't you tunnel everything out on the one port, like a VPN?

Whenever a connection drops, I can't use that host port again

I opened
tunnel 59001:0.0.0.0:5900

Everything works and I can get thru ssh tunneled in, and then i get error 61, which I assume is Refused Connection in the console

@aronsky
Copy link
Collaborator

aronsky commented Aug 9, 2020

Hi @MCApollo

First of all, thanks for the PR! It's great to see new features such as Linux support being added to the project! However, the extensive use of ifdefs and define wrappers around various structs really hurts readability, IMO. I suggest refactoring this by wrapping the cpu_memory_rw_debug to access the struct fields correctly. This will minimize the number of changes required, and make them easier to review. We basically need something similar to ntohs/htons, but for sockaddr in place of short, and Linux/Darwin in place of host/network byte order. I'll try to come up with a suggested implementation of such a wrapper soon.

@MCApollo
Copy link
Author

MCApollo commented Aug 9, 2020

That’s a actually a better idea and was hoping someone would help with; Feel free to continue to use this PR, I know this code is god-awful and is a straight up bandaid until someone/I has the time.

I can continue to work on this, been busy with some other things.

There’s really two issues that need a checkbox

  • Darwin uses sin_len and may slightly deviant from the network standard. addrlen is the same by standard.
  • Darwin uses 35 as a EAGAIN, maybe it would be better to edit the code on the host side on accept to report EAGAIN as 35.

Some notes are:

  • SOCK_NONBLOCK is needed on Linux or any connection hangs on accept. I’m unsure if this is the default on Darwin or I’m missing something.
  • Not giving addrlen a value in accept causes it fail.

Comment on lines 49 to 55
#ifdef __APPLE__
# define ADDR addr
# undef NETWORK_HACK
#else /* Linux, etc. */
# define ADDR darwin_addr
# define NETWORK_HACK 1
#endif
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no reason for the NETWORK_HACK and ADDR defines, we can condition the functions based on __APPLE__ only.

#endif

static int convert_error(int err) {
#ifndef NETWORK_HACK
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#ifndef NETWORK_HACK
#ifdef __APPLE__

# define NETWORK_HACK 1
#endif

static int convert_error(int err) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest a function name that makes the platform clear, such as darwin_error.

#endif
}

static struct sockaddr_in convert_sockaddr_to(S_SOCKADDR_IN from) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that you only call this function when NETWORK_HACK is defined, which means the following:

  1. The #ifndef inside the function is redundant
  2. The name of the function can be more explicit, such as convert_sockaddr_darwin_to_gnu
  3. It should receive darwin_sockaddr_in explicitly (and the S_SOCKADDR_IN define can be dropped)
Suggested change
static struct sockaddr_in convert_sockaddr_to(S_SOCKADDR_IN from) {
static struct sockaddr_in convert_sockaddr_darwin_to_gnu(darwin_sockaddr_in from) {

Comment on lines 82 to 84
#ifndef NETWORK_HACK
return from;
#else
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant...

Comment on lines 154 to 158
#ifdef NETWORK_HACK
S_SOCKADDR_IN ADDR = convert_sockaddr_from(addr);
#endif
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &ADDR,
sizeof(ADDR), 1);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#ifdef NETWORK_HACK
S_SOCKADDR_IN ADDR = convert_sockaddr_from(addr);
#endif
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &ADDR,
sizeof(ADDR), 1);
#ifndef __APPLE__
darwin_sockaddr_in darwin_addr = convert_sockaddr_gnu_to_darwin(addr);
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &darwin_addr,
sizeof(darwin_addr), 1);
#else
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
sizeof(addr), 1);
#endif

Comment on lines 170 to 172
#ifdef NETWORK_HACK
S_SOCKADDR_IN ADDR;
#endif
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will necessarily be darwin_sockaddr_in darwin_addr, if __APPLE__ is defined...

Comment on lines 181 to 186
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &ADDR,
sizeof(ADDR), 0);

#ifdef NETWORK_HACK
addr = convert_sockaddr_to(ADDR);
#endif
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to the change in accept:

Suggested change
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &ADDR,
sizeof(ADDR), 0);
#ifdef NETWORK_HACK
addr = convert_sockaddr_to(ADDR);
#endif
#ifndef __APPLE__
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &darwin_addr,
sizeof(darwin_addr), 0);
addr = convert_sockaddr_to(darwin_addr);
#else
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
sizeof(addr), 0);
#endif

Comment on lines 191 to 195
#ifdef NETWORK_HACK
ADDR = convert_sockaddr_from(addr);
#endif
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &ADDR,
sizeof(ADDR), 1);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#ifdef NETWORK_HACK
ADDR = convert_sockaddr_from(addr);
#endif
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &ADDR,
sizeof(ADDR), 1);
#ifndef __APPLE__
darwin_addr = convert_sockaddr_from(addr);
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
sizeof(addr), 1);
#else
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
sizeof(addr), 1);
#endif

}
}

return retval;
}

int32_t qc_handle_connect(CPUState *cpu, int32_t sckt, struct sockaddr *g_addr,
socklen_t addrlen)
int32_t qc_handle_connect(CPUState *cpu, int32_t sckt, S_SOCKADDR *g_addr,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to bind...

@MCApollo
Copy link
Author

MCApollo commented Aug 11, 2020

Sounds about right, thanks for being better than me at C- I’ll work on finishing so we can finally fix this.

(TIL about compound literals for easier reading, 👍🏼)

As far as accept goes, I’m not sure if Darwin acts differently vs Linux (see github.com/dotnet/runtime/issues/25069#ref-commit-e36b3b7 -> Darwin inheriting flags from fd).

Correct me if I’m wrong, but my understanding is that this should set nonblock on the host side right? here:
out->fcntl(out->fd, F_SETFL, flags | O_NONBLOCK) < 0)
Without using accept4 to set nonblock, qemu obviously hangs waiting for a connection so your insight would be helpful.

@aronsky
Copy link
Collaborator

aronsky commented Aug 11, 2020

Darwin should be inheriting flags from the parent socket upon accept, similar to BSD, and contrary to Linux. We can explicitly set the flags on the newly accepted socket - this will be redundant on macOS, but it's not an expensive operation and will make sure the new socket is non-blocking as well, without conditionally calling accept on macOS and accept4 on Linux.

accept itself shouldn't be the blocking call, since we set the O_NONBLOCK flag beforehand here. The file descriptor it returns, however, is stored inside the in socket - while the only socket we set to be non-blocking inside handle_incoming_connection is the out socket (that's the line you linked in your comment). So what happens is that the accepted socket is non-blocking on macOS (because it inherits the flags we set on the listening socket), and everything works great, but on Linux, the socket is blocking, and blocks on recv...

I'd like to avoid conditional code as much as possible, which would leave the accept call here the same across Linux and macOS, and require the tunnel binary to explicitly set the non-blocking flag on the socket returned by accept. Alternatively, if we want to properly emulate the behavior of accept, and abstract away the inconsistencies between Linux and macOS (after all, the tcp-tunnel binary shouldn't really care what host it runs on), we can add an #ifdefed block that would copy the flags from the parent socket to the socket returned by accept (note, that if we go the second route - we should copy the flags instead of using accept4 with a hardcoded O_NONBLOCK, since we're emulating the behavior of Darwin, which simply inherits all the flags from the parent socket).

@MCApollo
Copy link
Author

MCApollo commented Aug 11, 2020

Yeah, the accept4 was a bandaid fix until I can confirm if it was a bug/“feature” since no documentation mentions it without digging through man pages.

I think forcing non_blocking breaks out anyway, so that’s next-up.

Copy link
Collaborator

@aronsky aronsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, looks good. We're close to the final revision. I still wish we could streamline the part where we copy the memory to/from the guest, but with pointers involved, I don't see a neat solution there.

hw/arm/guest-socket.c Outdated Show resolved Hide resolved
Comment on lines 124 to 142
} else if ((guest_svcs_fds[retval] =
#ifndef __APPLE__
accept4(guest_svcs_fds[sckt],
(struct sockaddr *) &addr,
&addrlen, SOCK_NONBLOCK)) < 0)
#else
accept(guest_svcs_fds[sckt],
(struct sockaddr *) &addr,
&addrlen)) < 0) {
&addrlen)) < 0)
#endif
{
retval = -1;
guest_svcs_errno = errno;
guest_svcs_errno = darwin_error(errno);
} else {
#ifndef __APPLE__
struct darwin_sockaddr_in darwin_addr = convert_sockaddr_to_darwin(addr);
cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &darwin_addr,
sizeof(darwin_addr), 1);
#endif
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use just accept, and copy the flags in the #ifndef __APPLE__ section.

Comment on lines +36 to +37
#define SOCKADDR sockaddr
#define SOCKADDR_IN sockaddr_in
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for these defines. You can use darwin_sockaddr/darwin_sockaddr_in directly. If you don't want to redefine the structs in Darwin, you can use #define darwin_sockaddr sockaddr and #define darwin_sockaddr_in sockaddr_inwhen __APPLE__ is defined.

@MCApollo
Copy link
Author

  • accept4:
    Working on it- probably finish tomorrow. Do suggest a place to pull/set flags off the listening socket.
  • sockaddr macro:
    I feel like that affects readability for future contributors, but can be still changed if wished.

That previous ADDR macro was there to avoid reductant/repeating cpu_rw code (if that’s a problem).

@aronsky
Copy link
Collaborator

aronsky commented Aug 12, 2020

  • you can set the flags right in the next block (under else), inside the #ifndef __APPLE__ block
  • I think if you use darwin_sockaddr in all the relevant spots, it's quite readable. It's clear that it's a sockaddr structure, and also that it's Darwin-specific.

@aronsky
Copy link
Collaborator

aronsky commented Aug 26, 2020

@MCApollo any progress with this?

@Maroc-OS
Copy link

bind is not working even on macOS after the latest changes, error 45, can't work on this no time, but just to let you know

@MCApollo
Copy link
Author

Run it under lldb for me, this PR shouldn’t affect any macos parts at all.

@Maroc-OS
Copy link

Maroc-OS commented Aug 28, 2020

Run it under lldb for me, this PR shouldn’t affect any macos parts at all.

i mean when kvm was implemented the tunneling stoped working so.
read from ''' Implementation-defined registers and user-space ''' here :

https://alephsecurity.com/2020/07/19/xnu-qemu-kvm/

@aronsky
Copy link
Collaborator

aronsky commented Aug 30, 2020

@Maroc-OS are you running on ARM with KVM, or is there an issue when using regular QEMU emulation? Because the KVM changes should not affect the latter.

@Maroc-OS
Copy link

Maroc-OS commented Sep 1, 2020

@Maroc-OS are you running on ARM with KVM, or is there an issue when using regular QEMU emulation? Because the KVM changes should not affect the latter.

@aronsky yep regular QEMU emulation.

After updating to kvm commits it stopped working, and no ssh is there.

here is a little script am using adapted from @MCApollo :

#!/bin/sh

export XNU_SOURCES=$PWD/darwin-xnu
export KERNEL_SYMBOLS_FILE=$PWD/symbols.nm
export QEMU_DIR=$PWD/xnu-qemu-arm64/
export QEMU_TOOLS_DIR=$PWD/xnu-qemu-arm64-tools/
export NUM_BLOCK_DEVS=2
export KERNEL_CACHE=$PWD/kernelcache.release.n66.out
export DTB_FIRMWARE=$PWD/Firmware/all_flash/DeviceTree.n66ap.im4p.out
export DRIVER_FILENAME=$PWD/aleph_bdev_drv.bin
export IOS_DIR=$PWD
export HFS_MAIN=$PWD/hfs.main
export HFS_SEC=$PWD/hfs.sec


# Because there is no graceful shutdown of the system. mount & remount disk image
hdiutil attach -imagekey diskimage-class=CRawDiskImage ${HFS_MAIN}
hdiutil detach /Volumes/PeaceB16B92.arm64UpdateRamDisk
hdiutil attach -imagekey diskimage-class=CRawDiskImage ${HFS_SEC}
hdiutil detach /Volumes/PeaceB16B92.arm64UpdateRamDisk

# Update tree & Build the Custom Block Device Driver
cd ${QEMU_TOOLS_DIR}
git pull
cd ${IOS_DIR}

make -C ${QEMU_TOOLS_DIR}/aleph_bdev_drv
cp ${QEMU_TOOLS_DIR}/aleph_bdev_drv/bin/aleph_bdev_drv.bin ${DRIVER_FILENAME}

# Update tree & Build XNU QEMU for iOS
cd ${QEMU_DIR}
git pull

./configure \
--target-list=aarch64-softmmu \
--disable-capstone \
--disable-slirp \
--disable-pie \

export nproc=`sysctl -n hw.ncpu`
make -j${nproc}

cd ${IOS_DIR}

# Launch the kernel :)
xnu-qemu-arm64/aarch64-softmmu/qemu-system-aarch64 \
-M iPhone6splus-n66-s8000,kernel-filename=${KERNEL_CACHE},dtb-filename=${DTB_FIRMWARE},driver-filename=${DRIVER_FILENAME},qc-file-0-filename=${HFS_MAIN},qc-file-1-filename=${HFS_SEC},kern-cmd-args="debug=0x8 kextlog=0xfff cpus=1 rd=disk0 serial=2",xnu-ramfb=off \
-cpu max \
-m 6G \
-serial mon:stdio \
-vga std \

resulting crash of "MyUnixSocket" which is responsible of loading the (signed/unsigned both tested) tcp-tunnel binary, trying to launch it manually resulting "error 45" at binding

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants