5.1. Linux
5.1.1. Linking to GCC
This is the preferred way if you are developing mixed C-asm project. Check GCC docs and examples from Linux kernel .S files that go through gas (not those that go through as86).
32-bit arguments are pushed down stack in reverse syntactic order (hence accessed/popped in the right order), above the 32-bit near return address. %ebp, %esi, %edi, %ebx are callee-saved, other registers are caller-saved; %eax is to hold the result, or %edx:%eax for 64-bit results.
FP stack: I'm not sure, but I think result is in st(0), whole stack caller-saved.
Note that GCC has options to modify the calling conventions by reserving registers, having arguments in registers, not assuming the FPU, etc. Check the i386 .info pages.
Beware that you must then declare the cdecl or regparm(0) attribute for a function that will follow standard GCC calling conventions. See C Extensions::Extended Asm:: section from the GCC info pages. See also how Linux defines its asmlinkage macro.
5.1.2. ELF vs a.out problems
Some C compilers prepend an underscore before every symbol, while others do not.
Particularly, Linux a.out GCC does such prepending, while Linux ELF GCC does not.
If you need to cope with both behaviors at once, see how existing packages do. For instance, get an old Linux source tree, the Elk, qthreads, or OCaml.
You can also override the implicit C->asm renaming by inserting statements like
void foo asm("bar") (void);
|
Note that the objcopy utility from the binutils package should allow you to transform your a.out objects into ELF objects, and perhaps the contrary too, in some cases. More generally, it will do lots of file format conversions.
5.1.3. Direct Linux syscalls
Often you will be told that using C library (libc) is the only way, and direct system calls are bad. This is true. To some extent. In general, you must know that libc is not sacred, and in most cases it only does some checks, then calls kernel, and then sets errno. You can easily do this in your program as well (if you need to), and your program will be dozen times smaller, and this will result in improved performance as well, just because you're not using shared libraries (static binaries are faster). Using or not using libc in assembly programming is more a question of taste/belief than something practical. Remember, Linux is aiming to be POSIX compliant, so does libc. This means that syntax of almost all libc "system calls" exactly matches syntax of real kernel system calls (and vice versa). Besides, GNU libc(glibc) becomes slower and slower from version to version, and eats more and more memory; and so, cases of using direct system calls become quite usual. However, the main drawback of throwing libc away is that you will possibly need to implement several libc specific functions (that are not just syscall wrappers) on your own (printf() and Co.), and you are ready for that, aren't you? :-)
Here is summary of direct system calls pros and cons.
Pros:
the smallest possible size; squeezing the last byte out of the system
the highest possible speed; squeezing cycles out of your favorite benchmark
full control: you can adapt your program/library to your specific language or memory requirements or whatever
no pollution by libc cruft
no pollution by C calling conventions (if you're developing your own language or environment)
static binaries make you independent from libc upgrades or crashes, or from dangling #! path to an interpreter (and are faster)
just for the fun out of it (don't you get a kick out of assembly programming?)
Cons:
If any other program on your computer uses the libc, then duplicating the libc code will actually wastes memory, not saves it.
Services redundantly implemented in many static binaries are a waste of memory. But you can make your libc replacement a shared library.
Size is much better saved by having some kind of bytecode, wordcode, or structure interpreter than by writing everything in assembly. (the interpreter itself could be written either in C or assembly.) The best way to keep multiple binaries small is to not have multiple binaries, but instead to have an interpreter process files with #! prefix. This is how OCaml works when used in wordcode mode (as opposed to optimized native code mode), and it is compatible with using the libc. This is also how Tom Christiansen's Perl PowerTools reimplementation of unix utilities works. Finally, one last way to keep things small, that doesn't depend on an external file with a hardcoded path, be it library or interpreter, is to have only one binary, and have multiply-named hard or soft links to it: the same binary will provide everything you need in an optimal space, with no redundancy of subroutines or useless binary headers; it will dispatch its specific behavior according to its argv[0]; in case it isn't called with a recognized name, it might default to a shell, and be possibly thus also usable as an interpreter!
You cannot benefit from the many functionalities that libc provides besides mere linux syscalls: that is, functionality described in section 3 of the manual pages, as opposed to section 2, such as malloc, threads, locale, password, high-level network management, etc.
Therefore, you might have to reimplement large parts of libc, from printf() to malloc() and gethostbyname. It's redundant with the libc effort, and can be quite boring sometimes. Note that some people have already reimplemented "light" replacements for parts of the libc - - check them out! (Redhat's minilibc, Rick Hohensee's libsys, Felix von Leitner's dietlibc, asmutils project is working on pure assembly libc)
Static libraries prevent you to benefit from libc upgrades as well as from libc add-ons such as the zlibc package, that does on-the-fly transparent decompression of gzip-compressed files.
The few instructions added by the libc can be a ridiculously small speed overhead as compared to the cost of a system call. If speed is a concern, your main problem is in your usage of system calls, not in their wrapper's implementation.
Using the standard assembly API for system calls is much slower than using the libc API when running in micro-kernel versions of Linux such as L4Linux, that have their own faster calling convention, and pay high convention-translation overhead when using the standard one (L4Linux comes with libc recompiled with their syscall API; of course, you could recompile your code with their API, too).
See previous discussion for general speed optimization issue.
If syscalls are too slow to you, you might want to hack the kernel sources (in C) instead of staying in userland.
If you've pondered the above pros and cons, and still want to use direct syscalls, then here is some advice.
You can easily define your system calling functions in a portable way in C (as opposed to unportable using assembly), by including asm/unistd.h, and using provided macros.
Since you're trying to replace it, go get the sources for the libc, and grok them. (And if you think you can do better, then send feedback to the authors!)
As an example of pure assembly code that does everything you want, examine .
Basically, you issue an int 0x80, with the __NR_syscallname number (from asm/unistd.h) in eax, and parameters (up to six) in ebx, ecx, edx, esi, edi, ebp respectively.
Result is returned in eax, with a negative result being an error, whose opposite is what libc would put into errno. The user-stack is not touched, so you needn't have a valid one when doing a syscall.
 | Passing sixth parameter in ebp appeared in Linux 2.4, previous Linux versions understand only 5 parameters in registers. |
Linux Kernel Internals, and especially How System Calls Are Implemented on i386 Architecture? chapter will give you more robust overview.
As for the invocation arguments passed to a process upon startup, the general principle is that the stack originally contains the number of arguments argc, then the list of pointers that constitute *argv, then a null-terminated sequence of null-terminated variable=value strings for the environment. For more details, do examine , read the sources of C startup code from your libc (crt0.S or crt1.S), or those from the Linux kernel (exec.c and binfmt_*.c in linux/fs/).
5.1.4. Hardware I/O under Linux
If you want to perform direct port I/O under Linux, either it's something very simple that does not need OS arbitration, and you should see the IO-Port-Programming mini-HOWTO; or it needs a kernel device driver, and you should try to learn more about kernel hacking, device driver development, kernel modules, etc, for which there are other excellent HOWTOs and documents from the LDP.
Particularly, if what you want is Graphics programming, then do join one of the GGI or XFree86 projects.
Some people have even done better, writing small and robust XFree86 drivers in an interpreted domain-specific language, GAL, and achieving the efficiency of hand C-written drivers through partial evaluation (drivers not only not in asm, but not even in C!). The problem is that the partial evaluator they used to achieve efficiency is not free software. Any taker for a replacement?
Anyway, in all these cases, you'll be better when using GCC inline assembly with the macros from linux/asm/*.h than writing full assembly source files.
5.1.5. Accessing 16-bit drivers from Linux/i386
Such thing is theoretically possible (proof: see how DOSEMU can selectively grant hardware port access to programs), and I've heard rumors that someone somewhere did actually do it (in the PCI driver? Some VESA access stuff? ISA PnP? dunno). If you have some more precise information on that, you'll be most welcome. Anyway, good places to look for more information are the Linux kernel sources, DOSEMU sources, and sources for various low-level programs under Linux. (perhaps GGI if it supports VESA).
Basically, you must either use 16-bit protected mode or vm86 mode.
The first is simpler to setup, but only works with well-behaved code that won't do any kind of segment arithmetics or absolute segment addressing (particularly addressing segment 0), unless by chance it happens that all segments used can be setup in advance in the LDT.
The later allows for more "compatibility" with vanilla 16-bit environments, but requires more complicated handling.
In both cases, before you can jump to 16-bit code, you must
mmap any absolute address used in the 16-bit code (such as ROM, video buffers, DMA targets, and memory-mapped I/O) from /dev/mem to your process' address space,
setup the LDT and/or vm86 mode monitor.
grab proper I/O permissions from the kernel (see the above section)
Again, carefully read the source for the stuff contributed to the DOSEMU project, particularly these mini-emulators for running ELKS and/or simple .COM programs under Linux/i386.