Let's analyze system calls

DIFF.	CHALLENGE	AUTHORS	SOLVES
Loading challenges...

Let's analyze x86-64 Linux system calls!

Beginner Hint 1: The Calling Convention for System Calls and GCC's Extended Inline Assembly

The calling convention for system calls on x86-64 Linux is described in the A.2.1 Calling Conventions section of the System V Application Binary Interface AMD64 Architecture Processor Supplement which is available at https://gitlab.com/x86-psABIs/x86-64-ABI, and https://man7.org/linux/man-pages/man2/syscall.2.html.
- To execute a system call, we need to specify a system call number to the rax register, and up to six arguments to the rdi, rsi, rdx, r10, r8, and r9 registers sequentially, and then execute a syscall instruction.
- The result of the system call execution is stored in the rax register.
GCC's extended inline assembly syntax is described at https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html and https://gcc.gnu.org/onlinedocs/gcc/Machine-Constraints.html
- The extended inline assembly used for the first syscall instruction in this challenge specifies the following:
  - A OutputOperands parameter is unspecified.
  - A InputOperands parameter set 318 in the rax register, &result_1 in the rdi register, sizeof(result_1) in the rsi register, and 0 in the rdx register, before executing the extended inline assembly.
  - A Clobbers parameter specifies that the rcx register, r11 register, and memory will be modified during the execution of the extended inline assembly.
    - Note that a syscall instruction itself modifies the rcx and r11 registers.
- The extended inline assembly used for the second syscall instruction in this challenge specifies the following:
  - A OutputOperands parameter specifies that a content of the rax register are assigned to result_2 after the extended inline assembly executes.
  - A InputOperands parameter set 102 in the rax register before executing the extended inline assembly.
  - A Clobbers parameter specifies that the rcx register, r11 register will be modified during the execution of the extended inline assembly.

Beginner Hint 2: Approach to Solving the Challenge

There are several ways to approach this challenge.
One ways is to look up the system call numbers for x86-64 Linux and check the arguments and return values for the corresponding system call.
Another way is to use a decompiler.
- Some decompilers can perfectly decompile syscall calls from the start, while others produce poor decompiled results initially but can be perfectly decompiled with additional operations.

lets-analyze-system-calls.tar.gz