I take back the bad things I've said about autotools. My second shot at making pLisp portable was successful; if not fully in the portability aspects, at least in making it conform to the whole configure/make/make install paradigm (ironic really, since my main objective was to port pLisp to Windows, which has not been achieved yet. Well, it should work for Cygwin anyway).
There is a lot of material out there on autotools, but most of them are outdated or handle only trivial Hello World programs. As with life in general, you need to do your homework to separate the wheat from the chaff and to dig out information that's really relevant to you. David A Wheeler's Howto was the most useful one for me.
At the risk of this post turning into yet another autotools tutorial that adds to the noise, here are some general guidelines and tricks while using autotools (I'll be using pLisp as a case study):
First off, you need to handcraft two files: configure.ac and Makefile.am. The first file tells autotools various things about your project like the package name, version number, which tools (GCC, flex, bison, etc.) it uses, what package dependencies does it have, and so on. You will not spend much time with this file once you put in all this information (actually, there's one gotcha: when mentioning package dependencies, I had to do them for each package separately [a la PKG_CHECK_MODULES([gtk], [gtk+-3.0 >= 3.4.2])]; for some reason, using DEPS_CFLAGS and DEPS_LIBS didn't work for me, and I had to build AM_CFLAGS and AM_LIBS from the individual variables as below:
AM_CFLAGS = ${gtk_CFLAGS} ${gtksourceview_CFLAGS} ${libffi_CFLAGS}
AM_LIBS = ${gtk_LIBS} ${gtksourceview_LIBS} ${libffi_LIBS} ${LEXLIB}
)
In contrast to the short amount of time you devoted to configure.ac, be prepared to really duke it out with Makefile.am. The final version of Makefile.am for pLisp is only about 40 lines long, but each of those lines has its own story of blood, sweat and tears.
The rest of this post will be in the form of inline comments interspersed with Makefile.am lines.
AM_YFLAGS = -d
This line tells autotools that we want to include the -d flag in the call to bison. Needed because we want bison to generate a header file containing all the useful symbols like yyin, yyparse, and so on.
all: ${bin_PROGRAMS} ${lib_LTLIBRARIES} help.html
This sets up the targets that need to be built: pLisp needs a binary as well as a library (.so) to be built, and also a language reference HTML, which is generated automatically from a JSON data file.
bin_PROGRAMS = plisp
plisp_SOURCES = ...
Here we list the binaries to be built, and for each binary specify the sources. Note that we simply list all the sources--including header files--and let autotools automatically take care of the dependencies. Extremely nifty feature.
noinst_PROGRAMS = docgen
docgen_SOURCES = tools/docgen.c src/util.c src/util.h src/json.h src/jsonlib.c src/json.l src/json_parser.y
The prefix 'noinst' indicates that these targets need to be built but not deployed. docgen is one such internal program; it generates the above-mentioned language reference HTML file, and it's work is done. Forever. The next line lists the sources needed to build docgen.
Since we're talking about source files, this is as good a place as any to say this: do not name your flex source files with a '.lex' extension; autotools will barf on them, i.e. refuse to invoke flex on them. The extension has to be '.l'.
AM_CFLAGS = -DDATADIR=\"$(pkgdatadir)\" ${gtk_CFLAGS} ${gtksourceview_CFLAGS} ${libffi_CFLAGS}
AM_LIBS = ${gtk_LIBS} ${gtksourceview_LIBS} ${libffi_LIBS} ${LEXLIB}
These two lines set the preprocessor and linker flags at the global level. You can also set them at a per-target level, like plisp_CFLAGS = ...
ACLOCAL_AMFLAGS = -I m4 --install
I really don't know much about this line, except that it is required to make use of the m4 directory (courtesy the Wheeler tutorial referred above).
plisp_LDADD = ${AM_LIBS} -ltcc
This line sets the libraries needed for the plisp binary; in addition to the global libraries, we also specify the Tiny C Compiler library. Required because tcc doesn't follow the pkg-config framework.
lib_LTLIBRARIES = libplisp.la
libplisp_la_SOURCES = src/plisp_utils_ffi.c
libplisp_la_LDFLAGS = -version-info 0:0:0
These three lines build the 'libplisp.so' shared library. That's the extent of my knowledge of the matter, my lord.
pkgdata_DATA = data/plisp.lang data/help.json lib/plisp_full_monty_compiler.lisp
dist_doc_DATA = doc/help.html doc/pLisp_User_Manual.pdf
iconsdir = $(pkgdatadir)/icons
icons_DATA = icons/abort.png ...
Now this is where the power of autotools really shines through: pLisp, like any application worth its salt, has a bunch of resources that are needed for correct operation. These resources range from the plisp.lang file needed by GtkSourceView to do syntax colouring to the PGN files needed for the icons in the application toolbars. These four lines take care of generating the makefile commands that copy these resources from their respective locations in the source directory hierarchy to the user-specified directory (when she runs './configure') without us or the source code being aware of any of these shenanigans. Truly magical. Hyperbole aside, how this is accomplished is much more mundane: if you look at the AM_CFLAGS definition above, you'll notice the -DDATADIR flag; this flag is set to the package data directory (typically /usr/local/share/plisp), and is used in the source code [e.g. gtk_image_new_from_file(DATADIR "/icons/load_file.png")] to abstract away the location of the resource file.
src/json.c: src/json.l
$(LEX) --prefix=json -o src/json.c src/json.l
src/json_parser.c: src/json_parser.y
$(YACC) -d -v --name-prefix=json src/json_parser.y -o src/json_parser.c
And that brings us to the primary reason for all the blood, sweat and tears alluded to above: the inability to handle multiple lexers/parsers within the same program. pLisp uses flex/bison for two things: a) to parse the Lisp source code entered by the user (or fed from a file) and b) to parse the JSON strings used for representing both pLisp images and the online help content. When I was using a handcrafted makefile, I could keep the flex/bison symbols (yyin, yylval, yyparse) separate by using different name prefixes in the individual makefile rules. The only way to get the same behaviour in autotools seems to be to specify the same rules manually. Goes against the philosophy, but there doesn't seem to be any other way to do this.
help.html: docgen
./docgen
This use of a manual rule is justified, since this is an application-specific thing to generate the online help file.
And we're done with Makefile.am.
Some odds and ends:
1. You will need to create a bunch of dummy files at the project root: AUTHORS, NEWS, COPYING, ChangeLog, etc. Also create a directory called 'm4' with a dummy file in it (e.g. NOTES, as suggested in the Wheeler tutorial).
2. Create a script called autogen.sh with just a single line 'autoreconf --install || exit 1' in the project root as well, and run this script to process any changes you've made to configure.ac and/or Makefile.am
3. You need to store configure.ac, Makefile.am, and the files mentioned in #1 and #2 above in your version control system.
There is a lot of material out there on autotools, but most of them are outdated or handle only trivial Hello World programs. As with life in general, you need to do your homework to separate the wheat from the chaff and to dig out information that's really relevant to you. David A Wheeler's Howto was the most useful one for me.
At the risk of this post turning into yet another autotools tutorial that adds to the noise, here are some general guidelines and tricks while using autotools (I'll be using pLisp as a case study):
First off, you need to handcraft two files: configure.ac and Makefile.am. The first file tells autotools various things about your project like the package name, version number, which tools (GCC, flex, bison, etc.) it uses, what package dependencies does it have, and so on. You will not spend much time with this file once you put in all this information (actually, there's one gotcha: when mentioning package dependencies, I had to do them for each package separately [a la PKG_CHECK_MODULES([gtk], [gtk+-3.0 >= 3.4.2])]; for some reason, using DEPS_CFLAGS and DEPS_LIBS didn't work for me, and I had to build AM_CFLAGS and AM_LIBS from the individual variables as below:
AM_CFLAGS = ${gtk_CFLAGS} ${gtksourceview_CFLAGS} ${libffi_CFLAGS}
AM_LIBS = ${gtk_LIBS} ${gtksourceview_LIBS} ${libffi_LIBS} ${LEXLIB}
)
In contrast to the short amount of time you devoted to configure.ac, be prepared to really duke it out with Makefile.am. The final version of Makefile.am for pLisp is only about 40 lines long, but each of those lines has its own story of blood, sweat and tears.
The rest of this post will be in the form of inline comments interspersed with Makefile.am lines.
AM_YFLAGS = -d
This line tells autotools that we want to include the -d flag in the call to bison. Needed because we want bison to generate a header file containing all the useful symbols like yyin, yyparse, and so on.
all: ${bin_PROGRAMS} ${lib_LTLIBRARIES} help.html
This sets up the targets that need to be built: pLisp needs a binary as well as a library (.so) to be built, and also a language reference HTML, which is generated automatically from a JSON data file.
bin_PROGRAMS = plisp
plisp_SOURCES = ...
Here we list the binaries to be built, and for each binary specify the sources. Note that we simply list all the sources--including header files--and let autotools automatically take care of the dependencies. Extremely nifty feature.
noinst_PROGRAMS = docgen
docgen_SOURCES = tools/docgen.c src/util.c src/util.h src/json.h src/jsonlib.c src/json.l src/json_parser.y
The prefix 'noinst' indicates that these targets need to be built but not deployed. docgen is one such internal program; it generates the above-mentioned language reference HTML file, and it's work is done. Forever. The next line lists the sources needed to build docgen.
Since we're talking about source files, this is as good a place as any to say this: do not name your flex source files with a '.lex' extension; autotools will barf on them, i.e. refuse to invoke flex on them. The extension has to be '.l'.
AM_CFLAGS = -DDATADIR=\"$(pkgdatadir)\" ${gtk_CFLAGS} ${gtksourceview_CFLAGS} ${libffi_CFLAGS}
AM_LIBS = ${gtk_LIBS} ${gtksourceview_LIBS} ${libffi_LIBS} ${LEXLIB}
These two lines set the preprocessor and linker flags at the global level. You can also set them at a per-target level, like plisp_CFLAGS = ...
ACLOCAL_AMFLAGS = -I m4 --install
I really don't know much about this line, except that it is required to make use of the m4 directory (courtesy the Wheeler tutorial referred above).
plisp_LDADD = ${AM_LIBS} -ltcc
This line sets the libraries needed for the plisp binary; in addition to the global libraries, we also specify the Tiny C Compiler library. Required because tcc doesn't follow the pkg-config framework.
lib_LTLIBRARIES = libplisp.la
libplisp_la_SOURCES = src/plisp_utils_ffi.c
libplisp_la_LDFLAGS = -version-info 0:0:0
These three lines build the 'libplisp.so' shared library. That's the extent of my knowledge of the matter, my lord.
pkgdata_DATA = data/plisp.lang data/help.json lib/plisp_full_monty_compiler.lisp
dist_doc_DATA = doc/help.html doc/pLisp_User_Manual.pdf
iconsdir = $(pkgdatadir)/icons
icons_DATA = icons/abort.png ...
Now this is where the power of autotools really shines through: pLisp, like any application worth its salt, has a bunch of resources that are needed for correct operation. These resources range from the plisp.lang file needed by GtkSourceView to do syntax colouring to the PGN files needed for the icons in the application toolbars. These four lines take care of generating the makefile commands that copy these resources from their respective locations in the source directory hierarchy to the user-specified directory (when she runs './configure') without us or the source code being aware of any of these shenanigans. Truly magical. Hyperbole aside, how this is accomplished is much more mundane: if you look at the AM_CFLAGS definition above, you'll notice the -DDATADIR flag; this flag is set to the package data directory (typically /usr/local/share/plisp), and is used in the source code [e.g. gtk_image_new_from_file(DATADIR "/icons/load_file.png")] to abstract away the location of the resource file.
src/json.c: src/json.l
$(LEX) --prefix=json -o src/json.c src/json.l
src/json_parser.c: src/json_parser.y
$(YACC) -d -v --name-prefix=json src/json_parser.y -o src/json_parser.c
And that brings us to the primary reason for all the blood, sweat and tears alluded to above: the inability to handle multiple lexers/parsers within the same program. pLisp uses flex/bison for two things: a) to parse the Lisp source code entered by the user (or fed from a file) and b) to parse the JSON strings used for representing both pLisp images and the online help content. When I was using a handcrafted makefile, I could keep the flex/bison symbols (yyin, yylval, yyparse) separate by using different name prefixes in the individual makefile rules. The only way to get the same behaviour in autotools seems to be to specify the same rules manually. Goes against the philosophy, but there doesn't seem to be any other way to do this.
help.html: docgen
./docgen
This use of a manual rule is justified, since this is an application-specific thing to generate the online help file.
And we're done with Makefile.am.
Some odds and ends:
1. You will need to create a bunch of dummy files at the project root: AUTHORS, NEWS, COPYING, ChangeLog, etc. Also create a directory called 'm4' with a dummy file in it (e.g. NOTES, as suggested in the Wheeler tutorial).
2. Create a script called autogen.sh with just a single line 'autoreconf --install || exit 1' in the project root as well, and run this script to process any changes you've made to configure.ac and/or Makefile.am
3. You need to store configure.ac, Makefile.am, and the files mentioned in #1 and #2 above in your version control system.
