DavidsBlog

Templates in C

david@davidpriver.com — Sun, 24 Jul 2022 03:48:51 GMT

Templates in C

David Priver, July 22nd, 2022

Sometimes I see people on the internet sharing their generic libraries in C. More often than not, they turn out to use some giant macro that generates a huge amount of code on a single source line. I thought I would show a better way of doing it if you need to write some C and don't want to copy-paste yet another dynamic array or hash table.

None of these ideas are new — I am sure people were doing this in the 80s — but not everyone is aware of them.

Note, _Generic has nothing to do with this (althought you could use it to simulate overloads if you generate a lot of code).

Generics in C

There are a few ways of creating generic data structures or algorithms in C:

Void Pointers

If the data you are working with can be coerced to a void pointer, you can side step the problem by writing a single implementation that only works with void pointers or only with raw bytes. When the user actually uses it they have to cast back to the original type and to be careful not to get confused about which data structure is which.

This is easy to implement, but is error-prone and type-unsafe to use. If you require all data to be pointers the user is encouraged to heap allocate a large number of individual items, which is bad for performance.

Function pointers

You write your library or algorithm to work with opaque types and when you need type-specific functionality, you call a function pointer given by the user.

A good example of this in the standard C library is the qsort function. For moving things it just memcpy's a void pointer, but for the actual comparison it calls the comparison function given by the user.

The advantage of this approach is that it takes up very little code space - there is just one qsort.

The disadvantages are that it is type unsafe — there is no way to check that the user's function pointer is the right one and it is annoying to use. The user has to define a (probably trivial) global function just to sort some data.

Inline Macros

Another approach is to not define a function at all and just do the work in a macro. The classic K&R definition of max is an example of this:

#define max(a, b) ((a) > (b)? (a) : (b))

Ignoring the multiple evaluation problem (a or b is evaluated twice, which can be expensive or surprising if it involves side-effects), this is not adequate for more advanced operations. Sans extensions, you can't really define or use local variables, you can't use recursion, you can't get the normal compiler errors when you pass the wrong type to the "function" and this forcibly inlines the code at every usage site.

It is very easy to use though (it looks like a function).

Code Generating Macros

Another approach is to actually generate a function with the actual types and call that generated function. To avoid having to actually copy-paste, you generate the function via a macro.

#define COMBINE(a, b) a##b
#define MAKE_MAX(T, prefix) \
  static inline \
  T COMBINE(prefix, max)(T a, T b){ \
      if(a > b) return a; \
      return b; \
  }
MAKE_MAX(int, int_)

#include <assert.h>
int main(void){
    int x = int_max(1, 2);
    assert(x == 2);
    return 0;
}

This is type safe and you get the benefits of functions (separating implementation details from usage, local variables, early returns, etc.), but it becomes impossible to debug if you need to step through the generated code — the nature of C macros is everything will be on a single line. Checking the result of the macro expansion is also more difficult.

Additionally, customizing the expanded code is difficult: you either need a large number of arguments to the macro or just forgo customization of things like prefixes to function names.

Finally, just writing it is annoying. You need a \ on every line, most editors give up on syntax highlighting it properly, etc.

Source Code Generation

Another alternative is to generate extra source files that then are compiled into your program. The issue with this is that it greatly complicates the building and compiling of your code. Suddenly you need a build system that is aware of the need to generate the code instead of just leveraging the C compiler that you already have. You step farther away from the ideal of just compiling with cc *.c -o program.

Copy Paste

The manual version of source code generation is copy-pasting the data structure and functions every time you need a new one. This is a nightmare for maintainability — a bug copy-pasted 100 times might never get fixed, especially if that code is then altered and diverges.

Template Headers with Multiple Inclusion

This approach is very similar to Code Generating Macros, but with the benefit of it being easy to check the expanded code, it is possible to debug and customization is much more straightforward thanks to #ifdef. The rest of this article will explain how they work.

Template Headers

The idea of a template header is that you have a file that's meant to be included multiple times. Each time the file is included, it generates a new data structure or new functions, specialized to a given type or types, or really on anything you can imagine.

The easiest way to explain this is via an example:

// Darray.h
// Include this header multiple times to implement a
// simplistic dynamic array.  Before inclusion define at
// least DARRAY_T to the type the dynamic array can hold.
// See DARRAY_NAME, DARRAY_PREFIX and DARRAY_LINKAGE for
// other customization points.
//
// If you define DARRAY_DECLS_ONLY, only the declarations
// of the type and its function will be declared.
//

#ifndef DARRAY_HEADER_H
#define DARRAY_HEADER_H
// Inline functions, #defines and includes that will be
// needed for all instantiations can go up here.
#include <stdlib.h> // realloc, size_t

#define DARRAY_IMPL(word) DARRAY_COMB1(DARRAY_PREFIX,word)
#define DARRAY_COMB1(pre, word) DARRAY_COMB2(pre, word)
#define DARRAY_COMB2(pre, word) pre##word

#endif // DARRAY_HEADER_H

// NOTE: this section is *not* guarded as it is intended
// to be included multiple times.

#ifndef DARRAY_T
#error "DARRAY_T must be defined"
#endif

// The name of the data type to be generated.
// If not given, will expand to something like
// `darray_int` for an `int`.
#ifndef DARRAY_NAME
#define DARRAY_NAME DARRAY_COMB1(DARRAY_COMB1(darray,_), DARRAY_T)
#endif

// Prefix for generated functions.
#ifndef DARRAY_PREFIX
#define DARRAY_PREFIX DARRAY_COMB1(DARRAY_NAME, _)
#endif

// Customize the linkage of the function.
#ifndef DARRAY_LINKAGE
#define DARRAY_LINKAGE static inline
#endif

typedef struct DARRAY_NAME DARRAY_NAME;
struct DARRAY_NAME {
    DARRAY_T* items;
    size_t count;
    size_t capacity;
};

#define DARRAY_push DARRAY_IMPL(push)

#ifdef DARRAY_DECLS_ONLY

DARRAY_LINKAGE
void
DARRAY_push(DARRAY_NAME* array, DARRAY_T item);

#else

DARRAY_LINKAGE
void
DARRAY_push(DARRAY_NAME* array, DARRAY_T item){
    if(array->count >= array->capacity){
        size_t old_cap = array->capacity;
        size_t new_cap = old_cap?old_cap*2:4;
        size_t new_size = new_cap * sizeof(DARRAY_T);
        array->items = realloc(array->items, new_size);
        array->capacity = new_cap;
    }
    array->items[array->count++] = item;
}
#endif

// Cleanup
// These need to be undef'ed so they can be redefined the
// next time you need to instantiate this template.
#undef DARRAY_T
#undef DARRAY_PREFIX
#undef DARRAY_NAME
#undef DARRAY_LINKAGE
#undef DARRAY_push
#ifdef DARRAY_DECLS_ONLY
#undef DARRAY_DECLS_ONLY
#endif

Example Usage

// example.c

#include <assert.h>
#include <stdio.h>

#define DARRAY_T int
#define DARRAY_PREFIX i
#define DARRAY_NAME IntArray
// Must be manually instantiated by #including the file
#include "darray.h"

int main(void){
    IntArray ints = {0};
    ipush(&ints, 1);
    ipush(&ints, 2);
    ipush(&ints, 3);
    ipush(&ints, 4);
    ipush(&ints, 5);
    ipush(&ints, 6);
    assert(ints.count == 6);
    assert(ints.items[0] == 1);
    assert(ints.items[1] == 2);
    assert(ints.items[2] == 3);
    assert(ints.items[3] == 4);
    assert(ints.items[4] == 5);
    assert(ints.items[5] == 6);
    assert(ints.capacity == 8);
    for(size_t i = 0; i < ints.count; i++){
        printf("[%zu] = %d\n", i, ints.items[i]);
    }
    free(ints.items);
    return 0;
}

One reason this approach shines is that it is simple to debug the generation of the code. For example, you can simply see what the code expands to by a command like:

$ cc -DDARRAY_T=int darray.h -E -P

(assuming a gcc or clang-like cc).

If you tried to do a similar thing with a Code Generating Macro, it would be a giant mess on a single line.

The biggest benefit though is that if you have a bug in your generated code and an assertion fails or address-sanitizer complains — you have a real source file with a real location. You can step through the code in a debugger and it is readable and reasonable.

The final benefit is that it is simple to implement customization points: check for a macro to be defined and if not use some default behavior. This can get pretty ugly, but it is better than a huge number of arguments to a Code Generating Macro.

Conclusion

By leveraging the ability of the preprocessor to include files multiple times and the limited amount of introspection it provides, you can generate efficient, specialized code without resorting to generating external files, giving up type safety or copy-pasting code around.

Newer languages than C have better solutions to this problem, but there are many situations you still want to write some C and need things like type safe dynamic arrays, hashtables or even sorting functions (this approach can easily be used to specialize a quicksort).

Copyright

All code in this article is released into the public domain.

run(**vars(args))

david@davidpriver.com — Wed, 27 Jul 2022 23:29:53 GMT

run(**vars(args))

Home

run(**vars(args))

David Priver, July 27th, 2022

Python has the pretty decent argument parsing module argparse as part of the standard library. It offers a procedural API for building up a parser for a program's command line arguments. After calling parse_args, you get back a simple namespace object that contains the values as defined by your parser (which I'll refer to as args.)

# some_script.py
import argparse

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument('N', type=int)
    parser.add_argument('pid', type=int)
    parser.add_argument('--with-candles', action='store_true')

    args = parser.parse_args()
    ...

if __name__ == '__main__':
    main()

There's just one problem: the only way to tell what is in this returned args is by reading the code that builds up the parser.

What I've found to be an effective way to help with that problem is to immediately grab the attributes off of the args and pass them to a function that actually does the work (normally named run). You want to spend as little time as possible with this dynamic, unstructured args.

# some_script.py
import argparse

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument('N', type=int)
    parser.add_argument('pid', type=int)
    parser.add_argument('--with-candles', action='store_true')

    args = parser.parse_args()
    run(args.N, args.pid, args.with_candles)


def run(N:int, pid:int, with_candles:bool=False) -> None:
    ...

if __name__ == '__main__':
    main()

Having a separate run function is nice anyway, as it means you can write scripts that import your script and just call the run function:

# myscript.py
import some_script
import another_script

some_script.run(1, 2, with_candles=True)
another_script.run('big', 1, 'hello.txt')

However, this run function poses its own problem. It's tedious to grab all of the attributes off of args, which tempts you to the dark side of just passing the args object around in your application, making it very hard to understand what's going on.

Luckily, there is a relatively obscure builtin function in python called vars. vars (when given an argument) basically grabs all of the attributes off of an object and puts them into a dict. Coupling that with python's convenient ** for dictionary unpacking, you are saved the trouble of manually peeling the attributes off yourself. Thus the title of this article: you use the incantation run(**vars(args)) immediately after parsing command-line arguments.

# some_script.py
import argparse

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument('N', type=int)
    parser.add_argument('pid', type=int)
    parser.add_argument('--with-candles', action='store_true')

    args = parser.parse_args()
    run(**vars(args)) # Here it is.

def run(N:int, pid:int, with_candles:bool=False) -> None:
    ...

if __name__ == '__main__':
    main()

Relatively simple pattern, but I've found it to be useful. I'm sure there are argument parsing libraries that do nicer things, but with this pattern you can stick to the standard library which lessens the dependency problem.

Copyright

All code in this article is released into the public domain.

_Generic for Type Reification in C

david@davidpriver.com — Fri, 29 Jul 2022 15:33:52 GMT

_Generic for Type Reification in C

Home

_Generic for Type Reification in C

David Priver, July 29th, 2022

Occasionally, you need to turn a type into a value. An example of when you would need to do this is when constructing a tagged union that can hold pointers to different types:

// anypointer.h
#include <stdio.h>

// prefixes and other types elided for brevity
enum AnyPointerTag {
    UNINIT,
    INT,
    CHAR_STAR,
    DOUBLE,
};
typedef struct AnyPointer AnyPointer;
struct AnyPointer {
    enum AnyPointerTag tag;
    union {
        void* pointer;
        int* integer;
        char** charstar;
        double* double_;
    };
};

void
print_any(AnyPointer any){
    switch(any.tag){
        case UNINIT: return;
        case INT:
            printf("%d\n", *any.integer);
            return;
        case CHAR_STAR:
            printf("%s\n", *any.charstar);
            return;
        case DOUBLE:
            printf("%f\n", *any.double_);
            return;
        default: return;
    }
}

// example.c
#include "anypointer.h"

int main(void){
    int x = 3;
    AnyPointer any = {.tag=INT, .integer=&x};
    print_any(any); // prints 3
    return 0;
}

This example is small so it is not that difficult to manually set the tag when constructing the tagged union. However, setting the tag wrong would be disastrous, as you would be misinterpreting what is being pointed to.

// problem.c
#include "anypointer.h"

int main(void){
    int x = 3;
    AnyPointer any = {.tag=CHAR_STAR, .integer=&x};
    print_any(any); // segfaults
}

_Generic

C11 added the rarely used feature _Generic. It somewhat resembles a switch statement, but using types instead of integer constant expressions. Its motivating use case was to simulate overloading in C, but it can select arbitrary expressions, not just functions.

// generic.c
#include <stdio.h>

int main(void){
    // prints: "int: 3"
    printf(
        _Generic(3,
        int: "int: %d\n",
        float: "float: %f\n"), 3);
    return 0;
}

In common use, _Generic is used in a macro.

_Generic for Tagging

We can use the power of _Generic to automatically select the appropriate type tag for our tagged union.

// generic_tag.c
#include "anypointer.h"

#define TAG(x) _Generic(x, \
    int: INT, \
    char*: CHAR_STAR, \
    double: DOUBLE)

int main(void){
    int x = 3;
    AnyPointer any = {.tag=TAG(x), .pointer=&x};
    print_any(any); // prints 3
    return 0;
}

Going even farther, we can completely automate the creation of our tagged union in a macro:

// generic_compound.c
#include "anypointer.h"

// NOTE: we now take a pointer instead of the value
#define TAG(x) _Generic(x, \
    int*: INT, \
    char**: CHAR_STAR, \
    double*: DOUBLE)

#define ANY(x) (AnyPointer){.tag=TAG(x), .pointer=x}

int main(void){
    int x = 3;
    AnyPointer any = ANY(&x);
    print_any(any); // prints 3
    return 0;
}

Now, we can safely and concisely construct our tagged union without the risk of accidentally setting the wrong tag. Additionally, if we try to wrap a value of an unsupported type, we'll get a compilation error as the _Generic will fail to match any of its known types.

Use Cases

In practice, you don't use this technique very often. C code doesn't usually need to use this level of indirection and using too many macros like this can make the code obfuscated. However, there are times when a little "magic" like this is justified in my opinion.

Argument parsing libraries

You can use a type-reifying macro in essentially the same way as in the above examples. When parsing arguments, you would switch on the type tag to determine what type conversion function to call or action to take.

Serialization code

You can use a type-reifying macro like this, along with sizeof and offsetof to create a descriptor for a data type. The serializer/deserializer could then use that descriptor to parse binary data or something like JSON.

Dynamic "objects"

Instead of a type tag, you could have a pointer to a vtable. This would allow you to convert staticly typed objects to an interface.

It's a useful technique to have in your toolbox. Don't overuse it.

Conclusion

C11 added the means to reify types, a capability lacking from the C programming language before then. Using the _Generic construct, we can bridge one of the gaps between compile time information (types) and runtime information (type tags). Doing so allows you to implement things like an Any type, introspecting values for data processing and other use cases. Opportunities to use this technique are not too frequent, but when you do need it the technique can save error-prone manual boilerplate.

Appendix: Other Languages

In C++, you can achieve the same effect by writing a collection of trivial constexpr overloaded functions that return the type tag. This is more verbose, but can yield the same result.

If you are using GCC, but for some reason can't use C11 features, you can use __builtin_types_compatible_p.

In languages with first class types, you can just store the pointer to the type instead.

Copyright

All code in this article is released into the public domain.

Doctests in C

david@davidpriver.com — Fri, 29 Jul 2022 19:02:35 GMT

Doctests in C

Home

Doctests in C

David Priver, July 30th, 2022

Doctests are a technique of testing the examples embedded in documentation. They were first created for the Python language in 1999, but have since spread to other languages like Rust.

Embedding examples in documentation is very useful as it can immediately clarify usage of an API or the meaning of unclear terms. However, they can rot as the API changes, which can be worse than having nothing at all. What are we to do?

Well, as usual in C when you have a problem that can't be solved directly in the language itself, you reach for the preprocessor (this usage isn't that crazy, I promise!). Just include the examples in the doc, but guard them with an #ifdef. You can then write a test program that #includes the documentation file, but with that macro defined so the examples are actual code. Then, run the tests!

An example:

// some_api.h
#include <stddef.h> // size_t
typedef struct SomeApiContext SomeApiContext;

// Visibility modifiers, dllexport, and some documentation
// elided for brevity.

SomeApiContext* some_api_create(void);
// ...

void some_api_store_data(SomeApiContext*, int*, size_t);
// ...

size_t
some_api_get_data(
    SomeApiContext* ctx,
    int* buff, size_t bufflen,
    size_t* cookie);
// Copies the data into a user provided buffer.
//
// Arguments:
// ----------
// ctx:
//      The api context
// buff:
//      The buffer to copy the data into.
// bufflen:
//      The length (in items, not bytes) of buff.
// cookie:
//      A pointer to an opaque value for remembering where
//      in the data this function is. Initialize cookie to 0
//      before calling this function.
// 
// Returns:
// --------
// The number of items copied into buff. If 0 is returned,
// no items were copied into the buff and there are no more
// items to copy.
//
// Example:
// --------
#ifdef SOME_API_EXAMPLE
int sum_some_api_data(SomeApiContext* ctx){
    int result = 0;
    enum {buff_len=32};
    int buff[buff_len];
    size_t n = 0;
    size_t cookie = 0;
    while((n = some_api_get_data(ctx, buff, buff_len, &cookie))){
        for(size_t i = 0; i < n; i++){
            result += buff[i];
        }
    }
    return result;
}
#endif // SOME_API_EXAMPLE

And the test code:

// test_some_api_docs.c
#include <assert.h>
#define SOME_API_EXAMPLE
#include "some_api.h"

int main(void){
    int data[3] = {1, 2, 3};
    SomeApiContext* ctx = some_api_create();
    some_api_store_data(ctx, data, 3);
    assert(sum_some_api_data(ctx) == 6);
    return 0;
}

This isn't as nice as automatically extracting the examples like doctest can do, but when integrated into your test suite you will no longer have regressions in your documentation's examples. Additionally, it is nice to have examples right next to the definitions in the header anyway as that is what you'll jump to when you use your editor's GoToDefinition functionality. Finally, it is easier to write the examples as the examples are written as actual code instead of embedded in comments (so your editor and other tools understand it as code).

You do need a documentation generator which understands that it needs to include these #ifdef'd example blocks. I'm working on such a tool based on libclang, but it is not ready for release yet.

Copyright

All code in this article is released into the public domain.

C in WASM

david@davidpriver.com — Sat, 21 Jan 2023 03:08:21 GMT

C in WASM

Home

C in WASM

David Priver, January 20th, 2023

If you search for information on how to compile C code to webassembly, most sources will tell you to use Emscripten. For various reasons (complication, bloat, etc.), you probably don’t want to use Emscripten. This is partially just for my own reference, but here is how you could compile some simple C code to wasm and access it from the browser.

I am assuming you want to target the browser as I don’t see the point of targeting the wasm runtimes that exist outside of the browser — just generate native code. C is already a portable language and with some care you can target any OS used today. It will be much faster as well.

The Compiler

You'll need Clang. Other C compilers, such as GCC or MSVC, do not support compiling to wasm. Note that “Apple Clang” does not support generating webassembly. Strangely, it does support compiling to webassembly, but lacks the ability to actually generate the bytecode — it will typecheck, semantically analyze etc. just fine and error when it would generate code.

You can get clang from your package manager or just download from their github releases.

The Flags

You'll need to pass several flags to clang:

# Wasm
--target=wasm32

# Don't link to libc or assume there's crt1.o.
--no-standard-libraries

# Not a hosted environment.
--ffreestanding

# Don't add normal c include paths.
-nostdinc

# If you want to provide your own libc headers.
-isystem <your replacement libc headers>

Also, enable the wasm extensions:

-mbulk-memory
-mreference-types
-mmultivalue
-mmutable-globals
-mnontrapping-fptoin
-msign-ext

Basically, you want to use the wasm extensions that your targeted browsers will support.

Additionally, more flags need to be passed to the wasm linker (pass them through Clang using -Wl):

# export all of our non-hidden symbols
-Wl,--export-all

# don't look for _start or _main
-Wl,--no-entry

# allow functions to be defined at runtime
-Wl,--allow-undefined

The Standard Library

Almost nothing from the C standard library will be available out of the box. You'll either need to write them yourself or find suitable versions to adapt. There are a few exceptions though: notably, memmove, memcpy and memset can be provided by the host environment as long as you enable the bulk-memory extension for wasm (which is available basically everywhere).

In my opinion, it is most useful to treat webassembly like an embedded platform — don’t try to provide things like printf, file IO, etc. If you want to pretend you’re not actually targetting wasm, just use Emscripten.

Most of the functions you want to use from libc are not actually that hard to implement. The tough ones are things like scanf that you don’t want to use anyway in a reasonable C lib.

You can also always just import functions from javascript.

malloc

You're going to want at least a primitive version of malloc so that javascript can alloc memory in C land (this will come up for string conversion).

A primitive malloc can be implemented as so:

// primitive-malloc.c

// Use clang extensions to your heart's delight, it's
// your only C compiler.
#pragma clang assume_nonnull begin
typedef typeof(sizeof(1)) size_t;
enum {SIZE_T_SIZE=sizeof(size_t)};

// This will be defined by the instantiating wasm
// code.
extern unsigned char __heap_base[];

static unsigned char*_base_ptr = __heap_base;

// Useful function expose to javascript.
void
reset_memory(void){
    _base_ptr = __heap_base;
}

// The workhorse function.
// Just a bump allocator.
static inline
void*_Nonnull
alloc(size_t size, size_t alignment){
    if(alignment > SIZE_T_SIZE)
        alignment = SIZE_T_SIZE;
    size_t b = (size_t)_base_ptr;
    if(b & (alignment-1)){
        b += alignment - (b & (alignment-1));
        _base_ptr = (unsigned char*)b;
    }
    void* result = _base_ptr;
    _base_ptr += size;
    return result;
}

void*
malloc(size_t size){
    return alloc(size, 8);
}

void*
calloc(size_t n_items, size_t item_size){
    void* result = alloc(n_items*item_size, item_size);
    // Provided by bulk memory extension.
    __builtin_memset(result, 0, n_items*item_size);
    return result;
}

// Just don't do anything.
// You could improve this by maintaining a free list.
void
free(void*_Nullable p){
    (void)p;
}
#pragma clang assume_nonnull end

Strings

You'll probably want to pass strings back and forth between C and JavaScript. Unfortunately, there seems to be no performant way to do this - if you do this too much it will dominate the runtime of your code. I'll show the simplest way to do it.

Basically, to pass JS strings to C, you have javascript malloc a buffer and then use the TextEncoder API to encode their utf-16 strings into utf-8 (this is half of what is slow and I can’t figure out a fast way to just copy the utf-16 string directly as it's not that hard to write your string processing on utf-16 (like utf-8, ascii characters are the same)).

To pass C strings to JS, you call a function provided by your javascript code that takes a pointer + length pair. JavaScript can then decode that memory into a JS String using the TextDecoder API.

Code demonstrating this:

// js-strings.js
let mem;
let memview;
const decoder = new TextDecoder();
const encoder = new TextEncoder();
function wasm_string_to_js(p, len) {
    const sub = mem.subarray(p, p + len);
    const text = decoder.decode(sub);
    return text;
}
function write4(p, val) {
    memview.setInt32(p, val, true);
}
function js_string_to_wasm(s) {
    const encoded = encoder.encode(s);
    const p = malloc(encoded.length + 4);
    write4(p, encoded.length);
    mem.set(encoded, p + 4);
    return p;
}

// js-strings.c

// I've found wrapping APIs this way is easiest
// for me.
typedef struct PrefixedString PrefixedString;
struct PrefixedString {
    size_t length;
    unsigned char data[];
};

static
void some_api(size_t length, const char* txt);

void
some_api_js(PrefixedString* ps){
    some_api(ps->length, (char*)ps->data);
}

Building the Code

You'll end up with a commandline that looks like this (put in a Makefile).

clang mycode_wasm.c -o mycode.wasm \
--target=wasm32 --no-standard-libraries \
-Wl,--export-all -Wl,--no-entry \
-Wl,--allow-undefined -ffreestanding \
-nostdinc -isystem Wasm \
-mbulk-memory -mreference-types \
-mmultivalue -mmutable-globals \
-mnontrapping-fptoint -msign-ext \
-O3

Instantiating the Code

The MDN docs are pretty good, but for completeness, here is an example of how to instantiate a web assembly module in JavaScript.

let wasm_inst; // WebAssembly.Instance
let mem; // Uint8Array
let memview; // DataView
let exports; // WebAssembly.Exports
const decoder = new TextDecoder();
const encoder = new TextEncoder();

function wasm_string_to_js(p, len){
    const sub = mem.subarray(p, p+len);
    const text = decoder.decode(sub);
    return text;
}
function write4(p, val){
    memview.setInt32(p, val, true);
}
function js_string_to_wasm(s){
    const encoded = encoder.encode(s);
    const p = exports.malloc(encoded.length+4);
    write4(p, encoded.length);
    mem.set(encoded, p+4);
    return p;
}
const imports = {
    // JavaScript functions you’re exposing to C go here.
    env:{
    },
};
fetch('some_wasm_path')
  .then(response => response.arrayBuffer())
  .then(bytes => WebAssembly.instantiate(bytes, imports))
  .then(x=>{
      wasm_inst = x.instance;
      exports = wasm_inst.exports;
      const m = exports.memory;
      m.grow(1024);
      mem = new Uint8Array(m.buffer);
      memview = new DataView(mem.buffer);
  });

The above example doesn’t end up doing much, but it will instantiate the C code and have you set up for passing strings back and forth.

Debugging

As far as I can tell, debugging wasm code is basically only supported by Chrome. You can step through the bytecode in Firefox and Safari, but to actually map the instructions to your original source code requires chrome + a chrome extension. You’ll need this extension.

You'll need to add the -g flag to your compiler command line. That instructs clang to include DWARF debug info in the wasm file.

The extension looks up the source files by absolute path. This means that if you compile on one machine and debug on another, you’ll need to have the same file hierarchy on both machines.

Profiling

Profiling in the major browsers are all more or less the same - you can see how much time is spent at the granularity of function calls. This is somewhat unfortunate if you are used to native profiling tools which give you timings of each line or even drop down to the assembly level. So you won't be able to do things like optimize control flow. Hopefully this improves in the future.

I personally prefer the Firefox profiler UI, but they are more or less equivalent.

Demo

Below is a complete example of using WASM.

This does very little, but shows all the necessary pieces.

// demo.js
"use strict";
let wasm_inst; // WebAssembly.Instance
let mem; // Uint8Array
let memview; // DataView
let exports; // WebAssembly.Exports
const decoder = new TextDecoder();
const encoder = new TextEncoder();
function wasm_string_to_js(p, len){
    const sub = mem.subarray(p, p+len);
    const text = decoder.decode(sub);
    return text;
}
function write4(p, val){
    memview.setInt32(p, val, true);
}
function js_string_to_wasm(s){
    const encoded = encoder.encode(s);
    const p = exports.malloc(encoded.length+4);
    write4(p, encoded.length);
    mem.set(encoded, p+4);
    return p;
}
const imports = {
    env:{
      write: (p, len) => {
        const s = wasm_string_to_js(p, len);
        document.getElementById('scratch').innerText+=s;
      },
    },
};
document.addEventListener('DOMContentLoaded', ()=>{
  // prog64 is the base64 version of the wasm file.
  const wasm_buffer = Uint8Array.from(
    atob(prog64),
    c => c.charCodeAt(0)
  ).buffer;
  WebAssembly.compile(wasm_buffer)
    .then(x => WebAssembly.instantiate(x, imports))
    .then(x => {
      wasm_inst = x;
      wasm_inst.exports.memory.grow(24);
      exports = wasm_inst.exports;
      mem = new Uint8Array(exports.memory.buffer);
      memview = new DataView(mem.buffer);
      let i = 0;
      document.getElementById('click-me').onclick=()=>{
        exports.clicked(i++);
      };
    });
});

// example.c
#include "primitive_malloc.c"

extern void write(const char*, size_t);

void clicked(int i){
    char c[2] = {(i &0xf)+ 'a', '\n'};
    write(c, sizeof c);
}

Copyright

All code in this article is released into the public domain.

c-hash-table

david@davidpriver.com — Thu, 04 Jan 2024 04:53:59 GMT

Hashtable in C

Home

Hashtable in C

David Priver, Jan 3rd, 2024

There's many times when the problem you are solving needs something like a map of keys to values with O(1) insertion and retrieval. C doesn't come with one already ready-for-use like more "modern" languages like Python, so you gotta roll up your sleeves and do it yourself.

However, if you search around for "how to implement a hash table in C", you'll often find material that hashes to a fixed number of buckets and then has a linked list of items. This is slower than it needs to be and introduces complicated memory management that would be better avoided.

So we'll implement our own instead. As a bonus, we'll add some extra properties like easy C-style iteration, ordered insertion and realloc-friendly memory usage.

You could easily "templatize" this code.

Basic Reqirements

Your key type needs two basic properties:

Equatible (can compare two keys for equality).
Hashability (can map the key to a well-distributed number).

Your value type can be whatever.

Design Decisions

Limit the number of items in the hashtable to pow(2, 31). This let's us use 32 bit indexes which saves some memory.
Single memory allocation.

Data Layout

You'll (re-)allocate a block of memory such that it is laid out as follows:

|       Items          |        Indexes        |
+---------+------------+-----------------------+
|Occupied | Unoccupied | Indexes into occupied |
+---------+------------+-----------------------+

|- count -|            |----- 2*capacity ------|
|----- capacity -------|

When an item is inserted, the key is hashed and used to start the search in the Indexes array. If it finds a valid index, it uses it to index into the Items array and compare the key there. If it's a match, then return the pointer to the value half of that item. Otherwise, keep going. If an invalid index is found, that means the key is not in the table already. Insert the key after the end of the occupied portion of the items array and increment count. Store this new index into the Indexes array where you found the invalid index.

When the items array is filled, realloc the memory block (the enlarged Items array will now overlap with what was once the Indexes array). Invalidate all indexes in the new Indexes array and regenerate it from the Items array.

In this design, there is just a single allocation, which makes de-allocating the table easier. Additionally, it is re-alloc friendly and if you can re-alloc in place, then no copying of the Items needs to be done, which can be a valuable property if you are allocating from a linear allocator.

Another property is that the table remembers the order in which keys were inserted, which can be a useful property and makes debugging easier.

Finally, you don't need to remember the special way of how to iterate over this table or have some iterator macros or construct. You iterate it like you would an array:

void foo(void){
  for(size_t i = 0; i < table.count; i++){
    Pair* p = &table.data[i];
    // do whatever with p
  }
}

Implementation

#include <stdint.h>
#include <stdlib.h>
#include <string.h>

// Define these as makes sense.
uint32_t hash_key(const KeyType* key);
_Bool key_eq(const KeyType* a, const KeyType* b);

// You'll probably name these something else in your
// implementation to avoid collisions.
typedef struct Pair Pair;
struct Pair {
  KeyType key;
  ValueType value;
};

typedef struct Table Table;
struct Table {
  uint32_t count, capacity;
  Pair* data;
};

enum {TABLE_EMPTY_SLOT=UINT32_MAX};

// Grows the table to the new size.
// Returns 0 on success and non-zero on failure (usually
// allocation failure).
int
grow_table(Table* table, size_t new_size);

// If key is already in the table, returns the pointer to
// the corresponding value.
// Otherwise, inserts the key into the table and returns a
// pointer to the uninitialized value.
// Returns NULL if the table needed to grow and that
// failed.
ValueType*
table_getsert(Table* table, KeyType* key);

// If key is in the table, returns the pointer to the
// corresponding value.
// Otherwise returns NULL.
ValueType*
table_has(const Table* table, const KeyType* key);

// This is faster than modulo while preserving the same
// properties if all the bits are equally well distributed.
// Do *not* use identity hash with this.
uint32_t
fast_reduce32(uint32_t x, uint32_t y){
  return ((uint64_t)x * (uint64_t)y) >> 32;
}

int
grow_table(Table* table, size_t new_cap){
  size_t cap = table->capacity;
  if(new_cap <= cap) return 0;
  if(new_cap > UINT32_MAX/2) return 1;

  // On a 32 bit system, you'll want to check
  // for overflow here.
  size_t new_size = new_cap*(sizeof(Pair)
                    +2*sizeof(uint32_t));
  char* new_data = realloc(table->data, new_size);
  if(!new_data) return 1;
  table->data = (Pair*)new_data;
  table->capacity = (uint32_t)new_cap;

  size_t offset = new_cap * sizeof(Pair);
  uint32_t* indexes = (uint32_t*)(new_data+offset);
  size_t idx_cap = 2*new_cap;
  memset(indexes, 0xff, idx_cap * sizeof *indexes);

  Pair* items = (Pair*)new_data;
  for(size_t i = 0; i < table->count; i++){
    Pair* p = &items[i];
    uint32_t hash = hash_key(&p->key);
    uint32_t idx = fast_reduce32(hash, (uint32_t)idx_cap);
    while(indexes[idx] != TABLE_EMPTY_SLOT){
      idx++;
      if(idx == idx_cap) idx = 0;
    }
    indexes[idx] = i;
  }
  return 0;
}

ValueType*
table_getsert(Table* table, KeyType* key){
  if(table->count == table->capacity){
    size_t new_cap;
    if(table->capacity)
        new_cap = 2*table->capacity;
    else
        new_cap = 32;
    int err = grow_table(table, new_cap);
    if(err) return NULL;
  }
  uint32_t cap = table->capacity;
  uint32_t idx_cap = cap * 2;
  uint32_t hash = hash_key(key);
  uint32_t idx = fast_reduce32(hash, idx_cap);

  size_t offset = cap * sizeof(Pair);
  char* data = (char*)table->data;
  uint32_t* indexes = (uint32_t*)(data + offset);
  Pair* items = table->data;
  for(;;){
    uint32_t i = indexes[idx];
    if(i == TABLE_EMPTY_SLOT){
      indexes[idx] = table->count;
      items[table->count].key = *key;
      return &items[table->count++].value;
    }

    Pair* p = &items[i];
    if(key_eq(&p->key, key))
      return &p->value;

    idx++;
    if(idx == idx_cap) idx = 0;
  }
}

ValueType*
table_has(const Table* table, const KeyType* key){
  if(!table->count) return NULL; // empty table
  uint32_t cap = table->capacity;
  uint32_t idx_cap = cap * 2;
  uint32_t hash = hash_key(key);
  uint32_t idx = fast_reduce32(hash, idx_cap);

  size_t offset = cap * sizeof(Pair);
  char* data = (char*)table->data;
  uint32_t* indexes = (uint32_t*)(data + offset);
  Pair* items = table->data;
  for(;;){
    uint32_t i = indexes[idx];
    if(i == TABLE_EMPTY_SLOT)
      return NULL;

    Pair* p = &items[i];
    if(key_eq(&p->key, key))
      return &p->value;

    idx++;
    if(idx == idx_cap) idx = 0;
  }
}

Deletion

I didn't show to handle deleting items. Funnily enough I haven't found much need to delete things, but you have a few options.

Somehow mark the pair as deleted (using a sentinel for the value part or a book-keeping bitmask or whatever) and adjust the table growing code to shift non-deleted items down. This makes iteration more complicated and growing slower. This retains O(1) deletion.
Eagerly shift the non-deleted items into place and fixup the indexes. This makes deletion O(n).

There's probably other options as well, but you can tune

Alterations

There's some ways you could change the above design.

If hashing is expensive, and you want to avoid the re-hashing on grow, you can store the hash in the Pairs.
You could use 3 different allocations for the keys/values/indexes. There's not really much point to doing this.

Conclusion

It's certainly inconvenient to need to implement your own basic data structures in C, but in doing so you develop the skill to tailor them to your specific problem. And as shown above, you can get a functional hashtable in not very much code at all.

Copyright

All code in this article is released into the public domain.

C-macro-reflection-in-D

david@davidpriver.com — Tue, 30 Jul 2024 20:43:23 GMT

C-macro-reflection-in-D

Home

C-macro-reflection-in-D

David Priver, July 30th, 2024

I was reading Hacker News when I saw the link to this article. I've been playing around with D lately and thought to myself "I bet D could do this as well". D added the ability to import C files (with macros) and has strong reflection capabilities as well.

So here is my attempt at that:

With a C file with macros:

// macros.c
// #include <Windows.h>
// just #define the macros here for simplicity
#define WM_A 0x1
#define WM_B 0x2
#define WM_C 0x3

And a D file to import the C file:

// imports the C file "macros.c"
import macros;
import std.stdio: writeln;
void print_macro(int num){
    switch(num){
        // compile time foreach so we can generate cases
        static foreach(m; __traits(allMembers, macros)){
            static if(m.length > 3 && m[0..3] == "WM_"){
                case __traits(getMember, macros, m):
                    writeln(m, ": ", num);
                    return;
            }
        }
        default:
            writeln("unknown macro: ", num);
            return;
    }
}

void main(){
    print_macro(0x1);
    print_macro(0x2);
    print_macro(0x3);
    print_macro(0x4);
}

Compile and run it:

$ ldc2 mac.d -of mac
$ ./mac
WM_A: 1
WM_B: 2
WM_C: 3
unknown macro: 4

Copyright

All code in this article is released into the public domain.

VIM Macros are Overrated

david@davidpriver.com — Sat, 30 Nov 2024 05:45:25 GMT

VIM Macros are Overrated

Home

VIM Macros are Overrated

David Priver, Nov 29th, 2024

Macros

I'm a big VIM user and enjoy working in it, both professionally and recreationally. One of vim's nice features is the ability to record macros into a register and then replay it. As the macros can include motion, you can set them up so that they can be easily repeated and use a count modifier, like 22@w to apply them multiple times. Do :h complex-repeat to learn more.

However, there are some drawbacks to VIM macros. If you have a long or complicated sequence of actions, it can be error prone to record the exact macro needed to do what you want. They are stored in normal registers, so you can paste them into a buffer, edit them and then yank them back into a register, but that is pretty annoying and they quickly get unreadable.

Another common pitfall is you realize after executing the macro that you need some small thing changed or some additional transform. You then have to either edit the macro or record a new one from scratch. Do this a few times and you'll find you spent more time trying to record the perfect macro than just doing the text operation. And good luck debugging it if your macro isn't quite right.

If only we could write the text transforms as code. Wait a minute...

Filters

Vim also has the feature that you can shell out to arbitrary programs, writing to their stdin and reading from their stdout. You can feed them an entire buffer or a selection of a buffer. The input lines are then replaced with whatever the program wrote to stdout. Vim refers to these as "filters" and invoke them with filter commands. Do :h filter to learn more.

In my experience, a better alternative to recording a complicated macro is to just write a simple python script as the filter program. As we are just writing normal code, this means it can be arbitrarily complicated and you could even do syntactic/semantic analysis for your custom text transforms. Additionally, it's just normal code instead of weird editor commands so you can easily test and debug the transform if it's getting hairy, write functions, break it up into multiple statements, etc.

An Example

Recently I was working with some C code where I needed to generate an X-macro from a list of names. Example input would look like:

Potion of Healing
Wand of Fireball
Magic Sword
...  etc ...

I would like this to be converted to an X-macro, defining various forms of the names:

// #define X(UPPER, Title, snake, string)
#define XNames(X) \
  X(POTION_OF_HEALING, Potion_of_Healing, potion_of_healing, "Potion of Healing") \
  X(WAND_OF_FIREBALL, Wand_of_Fireball, wand_of_fireball, "Wand of Fireball") \
  X(MAGIC_SWORD, Magic_Sword, magic_sword, "Magic Sword") \
  ... etc ...

You can do this with vim commands and macros, but it is not easy. But behold, the simple python script that does this conversion:

import sys
for line in sys.stdin:
    l = line.strip()
    s = l.replace(' ', '_')
    print(f'    X({s.upper()}, {s}, {s.lower()}, "{l}") \\')

Simple, easy to understand code. You can debug it, you can throw examples at it, whatever. I usually name these little scripts f.py in my working directly and invoke them with :'<,'>!python3 f.py.

That's it! When you find yourself trying to setup a complicated macro, stop and think if it would be easier to express it as a little script that processes stdin instead. Most of the time, the answer is yes.

Copyright

All code in this article is released into the public domain.

DavidsBlog

Templates in C

Templates in C

Generics in C

Template Headers

Example Usage

Conclusion

Copyright

run(**vars(args))

run(**vars(args))

Copyright

_Generic for Type Reification in C

_Generic for Type Reification in C

_Generic

_Generic for Tagging

Use Cases

Argument parsing libraries

Serialization code

Dynamic "objects"

Conclusion

Appendix: Other Languages

Copyright

Doctests in C

Doctests in C

Copyright

C in WASM

C in WASM

Contents

The Compiler

The Flags

The Standard Library

malloc

Strings

Building the Code

Instantiating the Code

Debugging

Profiling

Demo

Copyright

c-hash-table

Hashtable in C

Contents

Basic Reqirements

Design Decisions

Data Layout

Implementation

Deletion

Alterations

Conclusion

Copyright

C-macro-reflection-in-D

C-macro-reflection-in-D

Copyright

VIM Macros are Overrated

VIM Macros are Overrated

Macros

Filters

An Example

Copyright