pLisp Migrated to Autotools Framework

I take back the bad things I've said about autotools. My second shot at making pLisp portable was successful; if not fully in the portability aspects, at least in making it conform to the whole configure/make/make install paradigm (ironic really, since my main objective was to port pLisp to Windows, which has not been achieved yet. Well, it should work for Cygwin anyway).

There is a lot of material out there on autotools, but most of them are outdated or handle only trivial Hello World programs. As with life in general, you need to do your homework to separate the wheat from the chaff and to dig out information that's really relevant to you. David A Wheeler's Howto was the most useful one for me.

At the risk of this post turning into yet another autotools tutorial that adds to the noise, here are some general guidelines and tricks while using autotools (I'll be using pLisp as a case study):

First off, you need to handcraft two files: configure.ac and Makefile.am. The first file tells autotools various things about your project like the package name, version number, which tools (GCC, flex, bison, etc.) it uses, what package dependencies does it have, and so on. You will not spend much time with this file once you put in all this information (actually, there's one gotcha: when mentioning package dependencies, I had to do them for each package separately [a la PKG_CHECK_MODULES([gtk], [gtk+-3.0 >= 3.4.2])]; for some reason, using DEPS_CFLAGS and DEPS_LIBS didn't work for me, and I had to build AM_CFLAGS and AM_LIBS from the individual variables as below:

AM_CFLAGS = ${gtk_CFLAGS} ${gtksourceview_CFLAGS} ${libffi_CFLAGS}
AM_LIBS = ${gtk_LIBS} ${gtksourceview_LIBS} ${libffi_LIBS} ${LEXLIB}

)

In contrast to the short amount of time you devoted to configure.ac, be prepared to really duke it out with Makefile.am. The final version of Makefile.am for pLisp is only about 40 lines long, but each of those lines has its own story of blood, sweat and tears.

The rest of this post will be in the form of inline comments interspersed with Makefile.am lines.

AM_YFLAGS = -d
This line tells autotools that we want to include the -d flag in the call to bison. Needed because we want bison to generate a header file containing all the useful symbols like yyin, yyparse, and so on.

all:    ${bin_PROGRAMS} ${lib_LTLIBRARIES} help.html
This sets up the targets that need to be built: pLisp needs a binary as well as a library (.so) to be built, and also a language reference HTML, which is generated automatically from a JSON data file.

bin_PROGRAMS = plisp
plisp_SOURCES = ...
Here we list the binaries to be built, and for each binary specify the sources. Note that we simply list all the sources--including header files--and let autotools automatically take care of the dependencies. Extremely nifty feature.

noinst_PROGRAMS = docgen
docgen_SOURCES = tools/docgen.c src/util.c src/util.h src/json.h src/jsonlib.c src/json.l src/json_parser.y
The prefix 'noinst' indicates that these targets need to be built but not deployed. docgen is one such internal program; it generates the above-mentioned language reference HTML file, and it's work is done. Forever. The next line lists the sources needed to build docgen.

Since we're talking about source files, this is as good a place as any to say this: do not name your flex source files with a '.lex' extension; autotools will barf on them, i.e. refuse to invoke flex on them. The extension has to be '.l'.

AM_CFLAGS = -DDATADIR=\"$(pkgdatadir)\" ${gtk_CFLAGS} ${gtksourceview_CFLAGS} ${libffi_CFLAGS}
AM_LIBS = ${gtk_LIBS} ${gtksourceview_LIBS} ${libffi_LIBS} ${LEXLIB}
These two lines set the preprocessor and linker flags at the global level. You can also set them at a per-target level, like plisp_CFLAGS = ...

ACLOCAL_AMFLAGS = -I m4 --install
I really don't know much about this line, except that it is required to make use of the m4 directory (courtesy the Wheeler tutorial referred above).

plisp_LDADD = ${AM_LIBS} -ltcc
This line sets the libraries needed for the plisp binary; in addition to the global libraries, we also specify the Tiny C Compiler library. Required because tcc doesn't follow the pkg-config framework.

lib_LTLIBRARIES = libplisp.la
libplisp_la_SOURCES = src/plisp_utils_ffi.c
libplisp_la_LDFLAGS = -version-info 0:0:0
These three lines build the 'libplisp.so' shared library. That's the extent of my knowledge of the matter, my lord.

pkgdata_DATA = data/plisp.lang data/help.json lib/plisp_full_monty_compiler.lisp
dist_doc_DATA = doc/help.html doc/pLisp_User_Manual.pdf
iconsdir = $(pkgdatadir)/icons
icons_DATA  = icons/abort.png ...
Now this is where the power of autotools really shines through: pLisp, like any application worth its salt, has a bunch of resources that are needed for correct operation. These resources range from the plisp.lang file needed by GtkSourceView to do syntax colouring to the PGN files needed for the icons in the application toolbars. These four lines take care of generating the makefile commands that copy these resources from their respective locations in the source directory hierarchy to the user-specified directory (when she runs './configure') without us or the source code being aware of any of these shenanigans. Truly magical. Hyperbole aside, how this is accomplished is much more mundane: if you look at the AM_CFLAGS definition above, you'll notice the -DDATADIR flag; this flag is set to the package data directory (typically /usr/local/share/plisp), and is used in the source code [e.g. gtk_image_new_from_file(DATADIR "/icons/load_file.png")] to abstract away the location of the resource file.

src/json.c:    src/json.l
    $(LEX) --prefix=json -o src/json.c src/json.l
src/json_parser.c:    src/json_parser.y
    $(YACC) -d -v --name-prefix=json src/json_parser.y -o src/json_parser.c
And that brings us to the primary reason for all the blood, sweat and tears alluded to above: the inability to handle multiple lexers/parsers within the same program. pLisp uses flex/bison for two things: a) to parse the Lisp source code entered by the user (or fed from a file) and b) to parse the JSON strings used for representing both pLisp images and the online help content. When I was using a handcrafted makefile, I could keep the flex/bison symbols (yyin, yylval,  yyparse) separate by using different name prefixes in the individual makefile rules. The only way to get the same behaviour in autotools seems to be to specify the same rules manually. Goes against the philosophy, but there doesn't seem to be any other way to do this.

help.html:    docgen
    ./docgen
This use of a manual rule is justified, since this is an application-specific thing to generate the online help file.

And we're done with Makefile.am.

Some odds and ends:

1. You will need to create a bunch of dummy files at the project root: AUTHORS, NEWS, COPYING, ChangeLog, etc. Also create a directory called 'm4' with a dummy file in it (e.g. NOTES, as suggested in the Wheeler tutorial).

2. Create a script called autogen.sh with just a single line 'autoreconf --install || exit 1' in the project root as well, and run this script to process any changes you've made to configure.ac and/or Makefile.am

3. You need to store configure.ac, Makefile.am, and the files mentioned in #1 and #2 above in your version control system.

December 15, 2015

Random thoughts on the recent Chennai rains:

1. Let's get the unpleasant stuff out of the way first: notwithstanding all the talk about Singara Chennai, there is a reason (well, reasons, to be more accurate) that the phrase 'third-world shithole' keeps getting thrown around when it comes to Chennai in particular and India in general. Overcrowding, lack of public hygiene and civic sense, corruption, unregulated and illegal growth, you name it. All this is business as usual, but when something like a once-in-a-hundred-years flood occurs, the rottenness of the system is there for all to see. There are folks who still harbour love for the city in their hearts, of course, but there are two aspects to this: one is the affection and familiarity one feels towards their favourite childhood haunts, the neighbourhood they grew up in, and so on, while the other pertains to the truly beautiful things the city is known for. The first is operative mainly because of the strong sense of nostalgia that glosses over all the squalid bits, while we're fast running out of things in the second category (if you are a regular reader of S. Muthiah's column in The Hindu, you can't help but notice that said beautiful things were all the creation of people who left us at least thirty to fifty years ago).


2. People will be people. In particular, good people who feel sympathy for others' suffering will help them, while not-so-good people will try to take advantage of others' misery. You therefore have folks wading through waist-deep water, risking their own safety to reach out to people who need help, and also those who take this opportunity to loot homes abandoned because of the flooding, and who jack up their auto fares to take advantage of people's predicament (on a related note, the most apt icon/mascot for Singara Chennai would be a snarling autorickshaw driver). While the outpouring of goodwill during the crisis was indeed admirable, expect good old reversion-to-the-mean to kick in shortly, if it hasn't already. By the way, there are Facebook stories doing the rounds in which one hears that motorists' driving behaviour has become more patient and courteous; these are either made-up crap like the Chickenshit for the Soul stories, or are one-off anecdotes.


3. At long last I found some use for social media. Nothing like Twitter to bring to you instantaneous pictures of the flooding around your locality and thereby saving you from an injudicious route choice. Newspapers were practically worthless (if you manage to get hold of them, that is), with their lagging-by-eight-to-twelve-hour coverage. Even their Twitter feeds were not as useful as citizen-journalists', probably because of their need to curate what they put out.


4. If I were made dictator, one of my first acts would be to can the entire meteorological department and replace it with two things: a) a subscription to the high-fidelity Accuweather satellite data and b) a Python script that does ARIMA. Enough said. Oh, by the way, that Ramanan guy is definitely trolling everybody (vittu vittu mazai peyyum, sila idangalil mazai peyyum).


5. Some practical advice for what to do if you're caught in such a situation in the future (notwithstanding the tendency to attribute all such extreme weather conditions to global climate change, I think it's unlikely that this will recur in the next five years):
1. Always keep at least Rs 5000 in cash with you in the house. Cash is king in such times. Also, rotate the money every six months. You never know when the government will get serious about black money and declare that all old currency notes--especially 500 rupee notes--are no longer legal tender.
2. Wean yourself from milk. Unless you're a baby, of course. People complaining about no milk during these times makes me shake my head. You can always stock milk powder.
3. Potatoes and onions don't need refrigeration. In general, keep a stock of provisions to last you at least a week. And don't forget batteries. Lots of them.
4. This is a bit of an overkill, but battery-operated amateur radio is a great way to communicate when the cell towers give out.
5. When you still have your wits/senses about you, make a list of important stuff to take with you when you need to evacuate. Otherwise you will end up carrying out worthless stuff like, I don't know, dosai maavu, while leaving behind a pile of black money for the looters to find when they come calling (true story).
6.  Make a hard copy of all the important phone numbers in your address book and keep it in your wallet. You never know when you might need to call somebody from another person's phone because yours ran out of juice.
7. And last but not the least, learn swimming.