Software, technology, sysadmin war stories, and more. Feed
Sunday, August 23, 2020

Libraries and dependencies I'm willing to take

I've been on a bit of a tear lately about the perils of adding dependencies to projects and expecting them to work properly. I've also seen that there are some people who try to parse this as a declaration of "nothing is good enough, therefore everything must be replaced". That's obviously not the case, but they get their jollies trying to claim otherwise.

It seems like the easiest way to handle this is to just come out and talk about the libraries that I actually use, or have used recently, and probably will use again.

First up is the Google stuff. There's gmock and gtest, which are both generally available from things like yum (in the case of my RHEL/CentOS flavored machines), apt (for the Ubuntu/Debian ones) and Macports. I got into the habit of using them way back when during my days on the inside, and continue using them now. This isn't *quite* the same situation as a runtime library since this only affects my test code, but they're a third-party dependency so it counts here. They also work well enough, so I keep them around.

There's also protobuf. It's also easily found as an OS-level package, and it's just one of those things I fell into using back in the day, and continue using now. One thing that people don't seem to talk about too much is the fact that protobuf has a canonical ASCII representation, and it makes for a pretty nifty config file format that's type-safe, fully specified, and which turns into a nice little object inside the program. You can even have comments! Yes, really. I use it for both config loading and for actually serializing stuff and blasting it around to other places.

Now let's talk about the software defined radio stuff. My scanner project has been on ice for the past few years, but the code still lives (and still builds, too). It was based around a whole bunch of stuff. The biggest one is clearly GNU Radio. It's a large assortment of software implementations of all kinds of signal processing stuff you'd normally do with components on a board. It's also super heavy in terms of math in places, has a bunch of MMX/SSE type optimizations and probably even some architecture-specific assembly blobs for really hot spots. It's pretty much what you use if you want to turn I/Q samples into usable data, or vice-versa. (The less said about REDHAWK, the better, am I right?)

My initial SDR work was done on Ettus USRP equipment, which has used its own little interface called the UHD library for quite some time now. That becomes a dependency of the bigger scheme, albeit buried inside the global GNU Radio build. There are also the binary blobs for the various FPGAs inside the equipment which come along for the ride when you run the UHD stuff.

I also got into those cute little ~$25 "rtlsdr" sticks, where smart people realized you could turn a DVB television receiver stick into a SDR receiver that wasn't half bad. That means the rtlsdr library itself which acts as a driver for the actual hardware side of things, and gr-osmosdr which then connects that (and a bunch of other things, if you have them) back into the GNU Radio world.

The first big radio system I monitored was based around a system from Motorola called SmartNet, and a library called gr-smartnet came into my life to deal with decoding the data channel. I had to effectively fork this one internally to deal with a number of issues over the years it was in production. Still, none of that would have worked without the signal decoding work done by the author.

The second big radio system was (and is) based around a scheme called APCO Project 25, or just P25, and there's a project called OP25 which decodes that. It's a much bigger problem space than, say, SmartNet, because P25 also implies digital audio channels which then must be decoded. Further complicating matters is that these voice paths use a patented (!) codec from DVSI, and many people are understandably skittish about reverse-engineering it. This leads to a lot of cloak and dagger stuff around tools like "dsd". OP25 actually handles all of them, including the once-unimaginable TDMA-based "phase 2" flavor of things, and somehow doesn't run afoul of the lawyers. How, I don't know.

I actually switched away from using the OP25 codec stuff at one point in favor of an actual device sold primarily to ham radio operators but which was also usable for my situation. It amounts to a USB stick which wraps around a FTDI RS-232 chip which then wraps around the DVSI codec chip. You push voice packets at it, and it pushes PCM audio back at you. It happens over a serial port and isn't the fastest thing in the world, but it does work. I actually coded this part up myself using the specs for the AMBE chip on the board.

Still in the world of scanner and radio stuff here, all of the calls on the actual web site were served out as MP3s. No part of GNU Radio writes MP3s, unsurprisingly. The solution to that was to use LAME as a library, pushing PCM audio in one side and getting MP3-formatted data out the other. (There were also some dark days early on where I shelled out to run the 'lame' CLI tool from inside my receiver, but the less said about that, the better. Oh, prototypes.)

Leaving SDR behind, let's talk about web stuff. Some of my programs sport their own internal web servers for serving up status pages, or (up until earlier this year) handling my own terrible "RPC" type scheme. I did not write my own http server for that. Instead, I use libmicrohttpd which is another GNU library that has made it into the usual yum/apt/Macports type places since I started using it, happily. I no longer have to track it separately since I can get it from the distributor (on the Linux boxes) and it'll work fine.

For the client side of web stuff, there's libcurl, like basically everybody else. This is used for normal web requests, like back in the day when I had a RSS/Atom feed reader and needed to fetch updates from remote servers. It also was used for my personal internal "RPC over HTTP" scheme mentioned above. This also drags in OpenSSL for https support.

[Sidebar: I'm not really happy with the way the web protocols are going of late. The more complicated they get, the more people will pile onto a handful of implementations. Then we will have epic security shit-fests when someone pops the common implementation and EVERYBODY has to patch.

Don't believe me? Look at what happened with SNMP: ASN.1 sucks, so everyone used {cmu,ucd,net}-snmp. It got popped bigtime around 2002, and all kinds of stuff, much of it *embedded* (routers!) had to be patched or just thrown out. It was terrible.

We're approaching that point with the web.]

Incidentally, that feed reader project needed to consume XML, and so libxml entered the picture. Or was it libxml2? One of those things. It's weird when you have a number as part of the name and not just the version. (I'm sure there's some reason for it. I'm just not aware, and really don't care.)

Finally in the web world, I have to emit JSON sometimes when feeding web browsers. It's terrible but it's not like I have a lot of choice in the matter. For a little while I did something awful which you might think of as "JSON via printf", and it was just as horrible as that sounds. But, I found a library called jansson, wrote some wrappers to make it fit better in my C++ world, and started using that instead. Happily, it is also readily available from various OS distribution packaging systems.

Some of my programs need to talk to databases. That means libmysqlclient or libpq (Postgres). Those are pretty much a given. You don't have much choice in the matter when you talk to them. They are relatively "thick", too, in that they do more than just speak over the network to the far end.

Oh, and, it should go without saying, but just for the stickler Meeseeks in the crowd, obviously I (along with everyone else) depend on the whole gamut of libc, libstdc++, ld.so, zlib, and all of the other things that get pulled in at the base levels of a distribution without you having any control over it.

The list above was compiled by going around to my machines and looking for libraries that I've explicitly added to my projects. I've tried to be exhaustive but have probably missed some things which operate very well and just hide in the bottom of the stack doing their jobs. If you're one of those authors and I didn't mention you, sorry about that, but be proud: your code is working and not generating any drama.

That might just be the best situation any of us could hope for.