Mastering x64 Windows Assembly: Syntax, Instructions, and Memory Operations

Learning the Language

Engineer Mirrai provides a deep dive into x64 Windows Assembly syntax. The guide details how instructions like XOR are used to avoid null bytes in shellcode encoding.

Why This Matters

Understanding low-level assembly is critical because high-level abstractions hide the reality of memory management and CPU state. In security contexts, failure to understand the stack—specifically how ‘call’ and ‘ret’ manipulate the return address—leaves systems vulnerable to buffer overflow exploitation where execution flow is intentionally corrupted.

Key Insights

• Memory dereferencing via square brackets allows reading/writing values at an address, similar to C pointers (Mirrai, 2026).
• The XOR idiom ‘xor reg, reg’ is preferred over ‘mov reg, 0’ in shellcode to reduce encoding size and eliminate null bytes that terminate string functions.
• Conditional branching relies on the FLAGS register (ZF, SF, CF, OF), which is updated automatically by arithmetic or comparison instructions like ‘cmp’.
• Stack interaction via ‘push’ and ‘pop’ directly modifies the RSP register in 8-byte increments.

Working Examples

An implementation of a loop that calls MessageBoxA twice using R12 as a non-volatile counter.

BITS 64
default rel
global main
extern ExitProcess
extern MessageBoxA
section .data
text_1 db "Hello World", 0
text_2 db "Hello from Mirrai", 0
section .text
main:
sub rsp, 40 ; shadow space + alignment
xor r12, r12 ; Set r12 to zero. Our counter register
loop_start:
cmp r12, 2 ; check if r12 == 2
je loop_end ; if so, exit loop
xor rcx, rcx ; hWnd = NULL
lea rdx, [text_1] ; lpText
lea r8, [text_2] ; lpCaption
mov r9, 1 ; uType = MB_OKCANCEL
call MessageBoxA
inc r12 ; increments r12 by 1
jmp loop_start
loop_end:
xor rcx, rcx
call ExitProcess

Practical Applications

• )Use case: Shellcode development utilizing ‘xor rcx, rcx’ to bypass null byte restrictions in functions like strcpy.
• )Pitfall: Attempting a direct memory-to-memory move (‘mov [rax], [rbx]’), which results in assembler rejection as a register must be used as an intermediary.

References:

• https://dev.to/mirrai/x64-windows-assembly-fundamentals-part-2-learning-the-language-c9c