Monday 31 August 2015

BSDMag, 10th collaboration

Hi folks,

A new BSDMag was issued, all about the difference between TrueNas and FreeNSD, some Unix fundamentals, an interesting Unix United article and my 10th article about an usage of the famous FreeBSD libprocstat's API in a web context.

Hope you will enjoy !

Labels: , , , , ,

View David Carlier's profile on LinkedIn

Friday 28 August 2015

Lua first contact

Hi folks,

Recently in professional context, I had to deal with Lua which I heard so much but never looked at until now. Memory efficient, portable, flexible, clear C api ...
Indeed, while the next BSDMag is preparing, I finally gave it a try ... So for a start I build a lua module on top of DeviceAtlas C api ... So what I need basically ? Enough Lua functions to cover those needs : Reading the data file, getting the properties, eventually some side informations. Behind this, I need to cleanup all the resources in the end. This is not a problem at all with the Lua C api ...

So let's define an usual C struct to hold on everything we need ...

typedef struct {
   da_atlas_t atlas;
   void *atlasptr;
   ...
   const char *jsonpath;

} dalua_t;

So we now need a function to expose to "instantiate" this Lua object ...

// When you'll declare your functions, via an array of luau_Reg which expects a function's pointer
int (*lua_CFunction)(lua_State *);

static int
dalua_new(lua_State *L)
{
    dalua_t *dl;

    dl = (dalua_t *)lua_newuserdata(L, sizeof(*dl));
    if (dl == 0)
        return (0);
    dl->atlasptr = 0;
    dl->jsonpath = 0;
    ...
    return (0);
}
...
static int
dalua_load_data_from_file(lua_State *L)
{
    dalua_t *dl;
    // The instance from the user data as first argument ...
    dl = (dalua_t *)luaL_checkudata(L, 1, "DAlua");
    ...
    luaL_pushboolean(L, 0);
    // in those functions we return the number of values
    // as we pushed a value on the stack above ...
    return (1);
}
...
static int
dalua_get_properties(lua_State *L)
{
    ...
    // This function can accept a simple string as argument
    if (lua_isstring(L, 2) == 1) {
        evsz ++;
        const char *value = luaL_checkstring(L, 2);
        ...
    // ... or a table
    } else if (lua_istable(L, 2) == 1) {
        // which we iterate through here
        for (lua_pushnil(L); lua_next(L, 2) != 0 && evsz < MAX_PROPS;    lua_pop(L, 1)) {
            // Both keys and values should be strings ...
            if (!lua_isstring(L, -2) || !lua_isstring(L, -1))
                continue;
            const char *key = luaL_checkstring(L, -2);
            const char *value = luaL_checkstring(L, -1);
    ...
    // In return, we want to provide a table so we push one to the stack
    lua_newtable(L);
    ...
    lua_setfield(L, -2, pkey);
    lua_pushstring(L, pvalue);
    ...
    return (1);
}
...
// The cleanup function we mentioned earlier in case we need to
// free some resources ...
static int
dalua_free(lua_State *L)
{
    ...
}
...
// Optionally we can implement a function how to display our
// dalua_t instance via the Lua's print function for example
static int
dalua_tostring(lua_State *L)
{
    ...
    // Preferably using a string buffer, it is pretty straightforward
    luaL_Buffer buf;
    ...
    luaL_buffinit(L, &buf);
    luaL_addstring(&buf, "\n");
    luaL_addstring(&buf, "DeviceAtlas instance (");
    ...
    // In the end we, as usual, push it to the stack
    luaL_pushresult(&buf);

    return (1);
}

// We finally declare our "methods" via a NULL terminated list
static const struct luaL_Reg dalua_methods[] = {
    { "load_data_from_file", dalua_load_data_from_file },
    { "get_properties", dalua_get_properties },
    ...
    { "__tostring", dalua_tostring },
    // As you would understand, once the Garbage collector is in action,
    // this will call our dear friend the cleanup function, do not hesitate
    // to print a message inside as a "proof" ;-)
    { "__gc", dalua_free },
    { NULL, NULL },
};

// And our new "operator"
static const struct luaL_Reg dalua_functions[] = {
    { "new", dalua_new },
    { NULL, NULL },
};

// Finally one last function to declare
int
luaopen_dalua(lua_State *L)
{
    luaL_newmetatable(L, "DAlua");
    lua_pushstring(L, "__index");
    lua_pushvalue(L, -2);
    lua_settable(L, -3);
    // Note this is the 5.1 Api ...
    luaL_openlib(L, 0, dalua_methods, 0);
    luaL_openlib(L, "DAlua", dalua_functions, 0);
    // with 5.2 there are new (better) functions
    luaL_setfuncs(L, dalua_methods, 0);
    luaL_newlib(L, dalua_functions);

    return (1);
}


We can finally compile our module as a shared library. I did everything on OpenBSD and installed Lua via the ports ...
cc -shared -g -O2 -Wall -o dalua.so dalua.c -I/usr/local/include/lua-5.1 -I/usr/local/include -L/usr/local/lib -llua5.1 -lda

Let's try with a simple lua script now !

...
dl = require("dalua")

headers = {}
headers["accept-language"] = "en-US"
headers["user-agent"] = "iPhone"
d = dl.new()
b = d:load_data_from_file("../tests/sample.json")
print(b)
x = os.clock()

for i=0, 5000 do
    t = d:get_properties(headers)
end
y = os.clock()
-- Here we calculate the time in sec for 5000 detections ...
print(y - x)

print(os.date("%x", d:get_jsoncreation()))
print(d)

b = d:load_data_from_file("../tests/samples.json")
print(b)
print(d)
-- If we want to load another data file, it is not a problem ...
-- The module takes care of the tricky C resources release
-- and reallocation underneath
b = d:load_data_from_file("../tests/sample2.json")

-- We had kept the properties from a previous detection let's print them !
print("properties:")
for k, v in pairs(t) do
    print(k, v)
end
...

# lua51 test.lua                                                                                                                                 
true => result of the first data file load, which happened well
0.76 => for 5000 detections, not too bad ...
03/18/15

=> the results of printing our instance
DeviceAtlas instance (0x1c888dcac000)
JSON's loaded in memory: 

[
        path: ../tests/sample.json
        revision: 39728
        creation timestamp: 1426640811
        version: 2.7
]

false => result of the second print(b) after the second data load which attempted to get a non existent file ...
=> Then now our instance is 'nil'
DeviceAtlas instance (0x0)
JSON's loaded in memory: 

[
        path: /
        revision: 0
        creation timestamp: 0
        version: /
]
properties:
...
isChecker       false
isEReader       false
isBrowser       false
isMediaPlayer   false
supportsClientSide      true
...

This is it ... to summarise the few hours I spent on Lua, the C api itself is relatively clear, the trickiest part might be how the Lua stack works. I find it the whole C api pretty well thought and a bit superior than Python which I appreciate as well.

Labels: ,

View David Carlier's profile on LinkedIn

Saturday 1 August 2015

New BSD Mag issue

Hi folks,

For this mid-summer, a new BSD Mag issue, some interesting tips for Python programmers, securing syslog via TLS and my article about making package for pkgsrc

Time for vacations now, a pause about all of that is a good thing too ;-)

Labels: , , ,

View David Carlier's profile on LinkedIn