Monday, 4 March 2019

LLVM 8 import starting

Hi folks, following up the now rc3 tagged of LLVM 8.0.0, Dimitry Andric had decided to import it in the base system already. Indeed, generally the 3rd or so release candidate already reflects the final release ; the changes, at this stage, usually occurs more into the documentations for example, it is pretty safe as is ... Curious as always, I personally already planned to recompile the whole system as I write to benefit all the improvements I mentioned over the last months.

While at LLVM ; like many other "githubers", I go through other people projects and one particularly attracted my attention, if you are interested into LLVM IR generation, there is this Java-7 compatible compiler even though it requires a particular environment to build it all and even if you do not have direct professional or personal usage per se ; it can be somewhat an interesting subject of study. About php, the awaited new feature, which is the JIT support, is still under review as I write, despite it is Dmitry Stogov behind the keyboard it needs to take time to fix and improve as it is tested ... but ... it might be for soon !

Personally, these last days I have been pushing few improvements to radare to report better arm and mips ELF binaries sections or also more specifically adding OpenBSD sections. Also, I have been pushing a tiny update to the NSS library, which "powers" Firefox for example, to support the new getentropy call from FreeBSD 12 which had been merged ; another slightly more important code will be reviewed.The jemalloc project had entered into the release candidate phase thus my little contribution to improve the profiling for FreeBSD won t be into the soon 5.2 release as I pushed a bit too late :-) but that is very understandable ; they need to be careful as it undergoes stress tests and so on.

Labels: , , , , , , ,

View David Carlier's profile on LinkedIn

Thursday, 21 February 2019

Finally it made it !

Hi folks !

Nicely, warmer weather is coming here in Dublin, making pull over wearing unnecessary :-)
These few days I have been updating my FreeBSD machine and was pleasantly surprised to see ASLR support had been added, all the work from HardenedBSD and other individuals in past years came to fruition in a way. Its usefulness had been the subject of controversy over the years, both for possible performance concerns and low outcome in term of security but this mitigation technique among the others is a must have so even if it appears way later than other oses, I am still happy whatsoever :-) So basically you can set it for 32 bits and or 64 bits which is necessary in case old binaries still need to run with fixed addresses. I have personally enabled only for 64 bits since 2 days and noticed no real performance drop while its shuffling shows up here. For example, as a test, I have been running a 64 bits and 32 bits version of the same basic executable which request few memory regions and the results speak by themselves ...

dragonflame% ./a.out 
0x804b7f000-0x804d91000 diff: 18446744073707380736
0x80ae51000-0x80bab1000 diff: 18446744073696575488
0x80c56c000-0x80d008000 diff: 18446744073698426880
0x80d9e2000-0x80e537000 diff: 18446744073697669120
0x80e740000-0x80f01c000 diff: 18446744073700261888
0x80f809000-0x80ff86000 diff: 18446744073701699584
0x810c09000-0x8118c8000 diff: 18446744073696186368
0x811e0f000-0x812c97000 diff: 18446744073694314496
0x812f52000-0x8135b6000 diff: 18446744073702850560
0x813904000-0x814258000 diff: 18446744073699770368
0x8148f0000-0x81491f000 diff: 18446744073709359104
0x81588c000-0x81634f000 diff: 18446744073698267136
0x816442000-0x816607000 diff: 18446744073707696128
0x8171f9000-0x817778000 diff: 18446744073703788544
0x818214000-0x818aee000 diff: 18446744073700270080
0x819882000-0x819999000 diff: 18446744073708408832
dragonflame% ./a.out
0x805315000-0x806129000 diff: 18446744073694789632
0x80caa8000-0x80d352000 diff: 18446744073700466688
0x80de53000-0x80e333000 diff: 18446744073704439808
0x80f302000-0x81008b000 diff: 18446744073695358976
0x8107c3000-0x810a24000 diff: 18446744073707057152
0x811a16000-0x8124e8000 diff: 18446744073698205696
0x8126f8000-0x813452000 diff: 18446744073695551488
0x81345f000-0x813bac000 diff: 18446744073701896192
0x814603000-0x814ac8000 diff: 18446744073704550400
0x815346000-0x81599c000 diff: 18446744073702907904
0x816740000-0x8176c9000 diff: 18446744073693261824
0x817748000-0x817af1000 diff: 18446744073705713664
0x8187d3000-0x819287000 diff: 18446744073698328576
0x819d87000-0x81a968000 diff: 18446744073697095680
0x81b784000-0x81b970000 diff: 18446744073707536384
0x81bd21000-0x81bdcf000 diff: 18446744073708838912
dragonflame% ./a32.out 
0x2064b000-0x2064c000 diff: 4294963200
0x20657000-0x20658000 diff: 4294963200
0x20659000-0x2065a000 diff: 4294963200
0x2065b000-0x2065c000 diff: 4294963200
0x2065d000-0x2065e000 diff: 4294963200
0x2065f000-0x20660000 diff: 4294963200
0x20661000-0x20662000 diff: 4294963200
0x20663000-0x20664000 diff: 4294963200
0x20665000-0x20666000 diff: 4294963200
0x20667000-0x20668000 diff: 4294963200
0x20669000-0x2066a000 diff: 4294963200
0x2066b000-0x2066c000 diff: 4294963200
0x2066d000-0x2066e000 diff: 4294963200
0x2066f000-0x20670000 diff: 4294963200
0x20671000-0x20672000 diff: 4294963200
0x20673000-0x20674000 diff: 4294963200
dragonflame% ./a32.out 
0x2064b000-0x2064c000 diff: 4294963200
0x20657000-0x20658000 diff: 4294963200
0x20659000-0x2065a000 diff: 4294963200
0x2065b000-0x2065c000 diff: 4294963200
0x2065d000-0x2065e000 diff: 4294963200
0x2065f000-0x20660000 diff: 4294963200
0x20661000-0x20662000 diff: 4294963200
0x20663000-0x20664000 diff: 4294963200
0x20665000-0x20666000 diff: 4294963200
0x20667000-0x20668000 diff: 4294963200
0x20669000-0x2066a000 diff: 4294963200
0x2066b000-0x2066c000 diff: 4294963200
0x2066d000-0x2066e000 diff: 4294963200
0x2066f000-0x20670000 diff: 4294963200
0x20671000-0x20672000 diff: 4294963200
0x20673000-0x20674000 diff: 4294963200


There will be probably further work until the 13th branch will be released however at the moment that is promising ... I ve been added this feature support in radare just yesterday.

In a previous post, I mentioned further FreeBSD support in PHP, so the network flags one had been now accepted and merged... Another one ought to be reviewed soon-ish bringing FreeBSD rfork call into that language ... About the video game Stalker, the second batch of changes for FreeBSD had been also accepted and merged, that was one of the most important parts, there should be more to come but not until other folks had merged their clang support basically because at the moment it needs gcc 7 or so but under FreeBSD the code generation crashes.

More important, next events this year ought to be AsiaBSDCon and BSDCan, hopefully will try to be present in one of those two :-)

Labels: , , , ,

View David Carlier's profile on LinkedIn

Monday, 11 February 2019

Android NDK ... how is it ?

Hi folks,

As mentioned in a previous post, I do quite some Android NDK development these days. And since you are accessing the low level of the Android system, you would just expect, more or less, "Linux for smartphones". Which is true ... to some extents ! First, if we are talking only about the kernel, you have somehow to accept it will be only a subset and some of the features are accessible only from a specific Android version and also dependent obviously to the current security level of the smartphone (rooted or not, SELinux activated etc) ... For example while in Linux you can easily bind a thread to a cpu it is less straightforward to do so on Android (we can still bind a process to a cpu) ... The kernel performance measurement api which allows to measure the number of instructions for a given workload, while existing in Android too, barely allow to measure generally a process ; but not per cpu ... Some C function wrappers functions might not exist until late from an api SDK level "point of view" (ie functions to loop through network interfaces for example but fortunately there are possibilities via the lower level interface ;-)) otherwise, for security reasons, you will generally not access most of hardware in a low level manner like the camera device, but only through the official NDK API provided.

In another hand, the libc is completely different, it is not the famous glibc nor the musl-libc but one called "bionic", which takes a lot from the OpenBSD one (feature-wise ... as it is mainly C++) like the memory allocator (a bit different and more fit to Android needs though), the presence of safe strings API like strlcpy/strlcat and so on ... which I appreciate for its practicality and security goals ... and remove a dependency I usually use under Linux which is the famous libbsd ...

Even though Android is not my primary world, I appreciate most of the aspects of the development despite the sometime frustrating but challenging limitations. Make me think I would appreciate a similar "ensemble" and widely available under FreeBSD, for example, which should be just awesome under the arm architectures ;-)

Apart of these, "same old same old" you might say :-) I continue to push little improvements for php and FreeBSD support not yet merged but seems accepted by the main maintainers like giving more FreeBSD specific options when developing with sockets like setting a cookie to be able to trace it from DTrace ... memcached,  the lead developer had promised he will look into shortly into my existing features proposal, so fingers crossed. Also got my first very little FreeBSD contribution ,among the two, merged for this year. Also I mentioned I try to contribute to a video game in order to make it work under FreeBSD. For who follows me on GitHub it is not a secret but otherwise it is xray the engine behind "STALKER the Call of Pripyat". This time I proceed step by step to avoid enormous, "code regression makers" and hard to review changes, one of them is already merged the second still in review. We will see how far it goes :-)

Labels: , , , , ,

View David Carlier's profile on LinkedIn

Tuesday, 5 February 2019

Just in time ...

.. well ... more or less :-)

Indeed, this week, PHP main developers had decided to push (one more time would say some "old timers") a JIT compiler solution, based on DynASM from luajit, which will be present at least for the future 8.x version maybe for 7.4 (as experimental feature). All use cases might not benefit it for sure (and again it is mainly for the x86 architectures anyway) but that might push some devops to consider php to other tasks/contexts than just the dear Web. Here, if you are interested, is the Merge Request still in review when I write for further technical details, I might say that something worthy to look at even though it is "concealed", for now, into the opcache module.

Other than that, these last days I did a sort of indirect contribution for LMMS, more specifically its custom memory allocator named rpmalloc. Just improved a bit the FreeBSD/OpenBSD support (super page support for the former) but alone this project is pretty interesting if, like me, you re used to play with memory management runtime replacements like jemalloc and so on ... To finish I was pleasantly surprised one of my old patches for OpenBSD had been merged (I almost forgot must have been early last year or before :-)) ... just to get the name of the thread ; at least to be equal to NetBSD for example.

Labels: , ,

View David Carlier's profile on LinkedIn

Wednesday, 30 January 2019

Jacket not warm enough ...

... in this end of January, finally the coldness has decided to show up :-) hope you are all fine whatever your plans are. First, sorry for the motivated people willing to go through the last meetup but finally most of the other organisers could not make it. Professional life or little health issues had came in our way in the last moment. We will try to make it happen as soon as we can ...

Since the last time, OpenBSD started, carefully, to import LLVM 7 (in base) while backporting fixes for the code generation. The port version is planned too as you can see here. Also, for the non hardcode followers, OpenBSD default terminal font had been changed/updated (depends how you see it :-)), which frankly quite rejuvenate it, it is definitely more modern looking to my tastes at least.

In a more personal level, I have been doing quite a lot of Android (NDK) development professionally but still had time to contribute to some of the projects I usually overwatch. For instance, php 7.4 will give FreeBSD the possibility to support the "huge" page features for FreeBSD for the opcache module (thus different than the super page support I mentioned a couple of months ago since it resided in the general zend engine memory allocation). I ve had proposed a little update for the network interface lookup functions too which seems to be accepted (basically displaying also if an interface is up or not) but not merged yet.

I will push further new features to radare, one is the possibility to display a binary hashes suite from rabin2 (for now only visible from r2).

Labels: , , ,

View David Carlier's profile on LinkedIn

Wednesday, 16 January 2019

After eight

... Not really mentioning here the famous mint chocolate brand here :-) most likely, for who is interested, LLVM 8 branching had been done just today as scheduled, now time to pass on to the testers people/doc writers/... ! Normally the real release date is still scheduled in a month and half more or less ... Thanks for few FreeBSD people having reviewed few of my changes (Ed Maste, Mark Johnston) ... I'll be watching this test period for a little while if there any concern for sanitisers in BSD for example ... but mainly my mind has already shifted towards the future 9th version. That was my first time following a full release development cycle from the beginning though, so this experience will serve me just fine ...

Having spent most of the time adding api interceptions with Kamil, for next I may spend more time in other sanitiser components I did not look at for a while even though I do not have real precise plans for now. If I can, I may do a bit of static code analysis work again ...


Otherwise got a little "step back" recently, getting stolen laptop and few other things (if it would have been used for proper purposes, it would have been great but I doubt the new owner has such ideas :-)) ... that is just usual events happening in life ... As possible, do not let material possessions define you but what you have up there in your head is much more valuable ;-) now rebuilding slowly but surely my usual unixes environments and all will be fine.


Labels: ,

View David Carlier's profile on LinkedIn

Monday, 14 January 2019

Let's encrypt while we still can !

Hello folks,

A bit of freshness starts to show up in Dublin, at least it reminds us it's January after all :-) But that does not slow down by any mean the creative process. First things ... about BSD Meetups, we wanted to do it in December so apologies we finally did not but that was due to personal schedules (mainly), finally we'll do it next week ... One of the benefits is to see again, after so long, our dear friend Dario ... definitely a very busy CTO ... he'll have so much to say you can bet on it ... So this time, Anton will bring up talks about Rust within FreeBSD (which interest more and more developers, ie mentioned in my previous post the lengthy discussion in freebsd-hackers list). Also I will try not to bore (too much) people about LLVM OpenBSD/FreeBSD support improvements since the 7.0 version ;-) Hopefully Tom Smyth will make it again ; also seemingly some other "old timers" are going back ... We will see !

In the meantime, I took a second cut into botan, the famous C++ cryptography library. I implemented the first grounds for properly sandboxing the client. For OpenBSD/FreeBSD. The main developer will take care of Linux/seccomp support and might extend it for other purposes. Happy my idea was welcomed easily.

Also, finally, my long term change into PHP and its openssl module had been merged. Had been a bit of a battle, with some rebasing in the meantime which shows resilience makes the difference ... That's great, I can finally move on further in my plans ...

Since 2019, in general, I prefer now to step up in term of contributions, in term of importance at least, so I decreased a bit my radare contributions but the changes implement new features, less about just fixing bugs ... We'll see how it will go, in the meantime I wish you all fine !

Labels: , , , ,

View David Carlier's profile on LinkedIn