nasm on OpenBSD


Recently I decided to study for the SLAE64 course from Pentester Academy to work on my assembly knowledge, specifically on x86_64. Through the course does focus on Linux I want to apply the knowledge to OpenBSD/amd64 too and thus I installed NASM and looked at what I needed to adjust on my Linux samples to get it working on OpenBSD. Turns out, not that much actually!

Both operating systems use same calling convention, namely the System V AMD64 ABI. Wikipedia has a fairly good article on it, but it basically means that most UNIX-like operating systems use the same method for passing arguments to function and where they expect return values to be stored. For the SLAE64 course the most relevant are the registers in which to pass arguments, which are in order: RDI, RSI, RDX, RCX, R8, R9. Return values up to 64 bits are stored in RAX.

The key difference between OpenBSD and Linux when coding the assembly are the syscalls and their numbers.

So let's get started and start even simpler than "Hello world". Let's just invoke a simple syscall. When it comes to syscalls there's they don't come much simpler than exit. Simply put the syscall number in RAX, the return code in the RDI register and make the call.

global _start

section .text

_start:
    mov rax, 0x1
    mov rdi, 0x2a
    syscall

Assemble and link: nasm -f elf64 exit.nasm -o exit.o && ld exit exit.o; that worked, so let's run it:

$ ./exit
zsh: exec format error: ./exit
$

Turns out we need an extra ELF NOTE section for OpenBSD called .note.openbsd.ident which which we can define with:

%ifdef OpenBSD
section .note.openbsd.ident
        align   2
        dd      8,4,1
        db      "OpenBSD",0
        dd      0
        align   2
%endif

and then assemble with -D OpenBSD. I have taken this defintion from src/lib/csu/os-note-elf.h.

However this still resulted in the exec format error...on an OpenBSD discussion channel it was hinted that I might need -nopie for my static binary; nope still not.

Then I tried it with the GNU linker (ld.bfd) rather than the default linker (ld.lld really) and that worked!

$ nasm -D OpenBSD -f elf64 exit.nasm -o exit.o && \
    ld.bfd -static -e _start exit.o -o exit && \
    ./exit ; echo $?
42
$

But surely lld cannot be broken? I tried the equivalent of the above code with the GNU assembler to determine if the issue is the linker or the assembler:

.section ".note.openbsd.ident", "a"
  .align 2
  .long 8
  .long 4
  .long 1
  .asciz "OpenBSD"
  .long 0
  .align 2

.section .text

.global _start

_start:
  mov $0x1,  %rax
  mov $0x2a, %rdi
  syscall

Assemble and link with: as -o exit_gas.o exit_gas.s && ld -o exit_gas exit_gas.o (and also -static) and that works fine.

It seems there's something present (or absent) in the object file emitted by NASM that lld isn't capable of handling.

For now I'll be able to use NASM with ld.bfd to write my OpenBSD-equivalent code for SLAE64, though I'd like to figure out at some point what's going wrong with lld, or what I'm missing here.

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification.
Student ID: SLAE64-1614