Just look any accessor function in src/dwg_api.c; they lack return statements for the "undefined" case.
Let's take dwg_ent_lwpline_get_elevation
if(lwpline == 0), then the return value of the function is undefined behaviour.
I recommend returning NaN in this case.
The root problem is, people will be lazy and check the error parameter passed in.
I don't even agree with this error pattern but at this point it's no reason to change all your hard work.
There is a lot of great stuff in this project and the current problem with it is not how the API functions are organized, but a lack of attention to the basics (especially the realloc problem...)