Recent blog updates

Shown here are only posts related to make. You can view all posts here.

Four years later, GNU Make 3.82 is released!

Gnu Make is not one of those products, website of which you frequently check for updates. One of the reasons is it's leisurely development pace: version 3.80 was released in 2002, 3.81—in 2006, and 3.82—in 2010. So each new release is breaking news.

Why so conservative?

Why's that? Well, I can't know for sure, but let me introduce some facts about the product.

First, Make is actually a profound tool. It contains a lot of very basic features, which makes nearly every build scenario you can imagine plausible to be implemented as a Makefile. Make's even Turing-complete. I think that this made the majority of people who develop Make think, that Make is already a perfect tool to do its job.

And then we see CMake, which is something that compiles into Make, just like Make was an assembly language of sorts...

What Make it actually lacks is a "standard library", something that would make using Make more convenient to writing complex stuff. If not that, there's nearly nothing to add to Make. And nearly nothing is added indeed.

But is it really conservativeness? What could characterize a conservative design? Perhaps a pursuit for stability and compatibility among releases?

However, this doesn't seem the case either. Some new changes (I'll comment on them later) introduce backward incompatibility—i.e. some old makefiles wouldn't be processed by Make. Some of them broke the build of one of the basic component of GNU/Linux system, Glibc (C runtime library)! This Glibc bug was filed only five days after the release of Make 3.82. Not before. This means that this backward-incompatible change, while forward-compatible, was not addressed, or checked, or noted, by anybody (and not by Make maintainers in the first place)! Is this called conservative?

An only conclusion I can make is that Make has reached all its capabilities. It's maintained, you know, not by the most stupid people in the world. And even if they can't foster its progress then who could? Perhaps, no one, and that's the point?

So what's new?

But alright, let's observe the changes GNU Make 3.82 brings us. If you go to the official site of GNU Make, it will kindly redirect you to the NEWS file, to get which you should download and unpack the release. But I'll release you from this burden, and quote the file here. Fully. Look:

Version 3.82

A complete list of bugs fixed in this version is available here:

http://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=104&set=custom

* Compiling GNU make now requires a conforming ISO C 1989 compiler and
  standard runtime library.

* WARNING: Future backward-incompatibility!
  Wildcards are not documented as returning sorted values, but up to and
  including this release the results have been sorted and some makefiles are
  apparently depending on that.  In the next release of GNU make, for
  performance reasons, we may remove that sorting.  If your makefiles
  require sorted results from wildcard expansions, use the $(sort ...)
  function to request it explicitly.

* WARNING: Backward-incompatibility!
  The POSIX standard for make was changed in the 2008 version in a
  fundamentally incompatible way: make is required to invoke the shell as if
  the '-e' flag were provided.  Because this would break many makefiles that
  have been written to conform to the original text of the standard, the
  default behavior of GNU make remains to invoke the shell with simply '-c'.
  However, any makefile specifying the .POSIX special target will follow the
  new POSIX standard and pass '-e' to the shell.  See also .SHELLFLAGS
  below.

* WARNING: Backward-incompatibility!
  The '$?' variable now contains all prerequisites that caused the target to
  be considered out of date, even if they do not exist (previously only
  existing targets were provided in $?).

* WARNING: Backward-incompatibility!
  As a result of parser enhancements, three backward-compatibility issues
  exist: first, a prerequisite containing an "=" cannot be escaped with a
  backslash any longer.  You must create a variable containing an "=" and
  use that variable in the prerequisite.  Second, variable names can no
  longer contain whitespace, unless you put the whitespace in a variable and
  use the variable.  Third, in previous versions of make it was sometimes
  not flagged as an error for explicit and pattern targets to appear in the
  same rule.  Now this is always reported as an error.

* WARNING: Backward-incompatibility!
  The pattern-specific variables and pattern rules are now applied in the
  shortest stem first order instead of the definition order (variables
  and rules with the same stem length are still applied in the definition
  order). This produces the usually-desired behavior where more specific
  patterns are preferred. To detect this feature search for 'shortest-stem'
  in the .FEATURES special variable.

* WARNING: Backward-incompatibility!
  The library search behavior has changed to be compatible with the standard
  linker behavior. Prior to this version for prerequisites specified using
  the -lfoo syntax make first searched for libfoo.so in the current
  directory, vpath directories, and system directories. If that didn't yield
  a match, make then searched for libfoo.a in these directories. Starting
  with this version make searches first for libfoo.so and then for libfoo.a
  in each of these directories in order.

* New command line option: --eval=STRING causes STRING to be evaluated as
  makefile syntax (akin to using the $(eval ...) function).  The evaluation
  is performed after all default rules and variables are defined, but before
  any makefiles are read.

* New special variable: .RECIPEPREFIX allows you to reset the recipe
  introduction character from the default (TAB) to something else.  The
  first character of this variable value is the new recipe introduction
  character.  If the variable is set to the empty string, TAB is used again.
  It can be set and reset at will; recipes will use the value active when
  they were first parsed.  To detect this feature check the value of
  $(.RECIPEPREFIX).

* New special variable: .SHELLFLAGS allows you to change the options passed
  to the shell when it invokes recipes.  By default the value will be "-c"
  (or "-ec" if .POSIX is set).

* New special target: .ONESHELL instructs make to invoke a single instance
  of the shell and provide it with the entire recipe, regardless of how many
  lines it contains.  As a special feature to allow more straightforward
  conversion of makefiles to use .ONESHELL, any recipe line control
  characters ('@', '+', or '-') will be removed from the second and
  subsequent recipe lines.  This happens _only_ if the SHELL value is deemed
  to be a standard POSIX-style shell.  If not, then no interior line control
  characters are removed (as they may be part of the scripting language used
  with the alternate SHELL).

* New variable modifier 'private': prefixing a variable assignment with the
  modifier 'private' suppresses inheritance of that variable by
  prerequisites.  This is most useful for target- and pattern-specific
  variables.

* New make directive: 'undefine' allows you to undefine a variable so that
  it appears as if it was never set. Both $(flavor) and $(origin) functions
  will return 'undefined' for such a variable. To detect this feature search
  for 'undefine' in the .FEATURES special variable.

* The parser for variable assignments has been enhanced to allow multiple
  modifiers ('export', 'override', 'private') on the same line as variables,
  including define/endef variables, and in any order.  Also, it is possible
  to create variables and targets named as these modifiers.

* The 'define' make directive now allows a variable assignment operator
  after the variable name, to allow for simple, conditional, or appending
  multi-line variable assignment.

Oh... My... God. Four years we've been waiting for this??? Three changes a year??

But let's observe the most important changes.

A lot of backward incompatibilities

Gentoo is a Linux distribution. I use it on my home machine, and find it highly convenient for a Linux used who likes to see how its system works. The pleasure of having such a system is of the same kind as watching clock with a transparent dial ticking, or keeping your PC without a cover.

One of its peculiarities is that by default every software is builtat the user's site. Package manager downloads sources and invokes ./configure; make; make install or whatever is relevant for the package. Since most of the Linux software uses Make to build, incompatible changes in Make's behavior is painful for such a system. That's why I refer to Gentoo bugzilla as to a good reference to bugs introduced by these changes.

By the way, you can get Gentoo at its homepage.

GNU naming convention is betrayed, a +1 to the minor version "81" breaks backwards compatibility. One issue with Glibc was noted above. Is there more? Sure. Here's a Gentoo bug that comprises all 3.82 regressions. Currently there's 40 breakages, and the list will most likely grow.

And if this change wasn't for nothing, I'd even understand this. Unfortunately... well, read along.

Shortest-stem makes Make more declarative!

* WARNING: Backward-incompatibility!
  The pattern-specific variables and pattern rules are now applied in the
  shortest stem first order instead of the definition order ...

You can define a "generic" rule, and then declare "amendments" to it by specifying, for example, prefixes to the filename, or the folders your target should be in. Now you can do it in arbitrary order, which makes the language more declarative. A good improvement, but, unfortunately, it really introduces nothing what couldn't be done before.

Linking change behavior

* WARNING: Backward-incompatibility!
  The library search behavior has changed to be compatible with the standard
  linker behavior. ...

That's the kind of change that contributes to "standard library" behavior: more flexibility with basically the same computational model. I could cheer upon this...if I used it at least once.

You can use something else instead of TABs

* New special variable: .RECIPEPREFIX allows you to reset the recipe
  introduction character from the default (TAB) to something else. ...

First, too late: Make is already notorious for its stupid tabs. Second, what other single character but tab could be used for this purpose? That's not even a regexp!

Of course, you can specify plain space character as this prefix. Then if your lines start with several spaces, the first would be recognized as prefix, and the rest would be ignored by the shell Make invokes. But then TABs won't work... So I doubt that this option would be used at all.

One shell for the whole rule

* New special target: .ONESHELL instructs make to invoke a single instance
  of the shell and provide it with the entire recipe, regardless of how many
  lines it contains. ...

First I thought, at last! I'm tired of writing multiline rules with these \ characters at the ends of the lines (that's especially painful if you have loops or conditionals inside the shell commands). However, I noticed that this is a global setting. So you can't just turn it on for certain rules; if you gonna turn it on, you have to write the whole makefile with these &&-s shell operators to mimic old behavior.

The rest

The rest is syntactical sugar and attempts to comply with the standards that evolve faster—and I actually couldn't imagine anything what evolved slower than standards—until this release, of course.

Conclusion

So the new release of Make is just a fucking disappointment. Yes, we certainly need something new, everyone says that (here's a link to deleted Tuomov's rant about Make, for example; thanks to QDiesel for that). More precisely, we have been needing it for several years. And I hope that in my search for the declarative language, I'll find something that could supersede this obsolete Make tool.

Read on | Comments (0) | Make a comment >>


Searching for the declarative language

I'm really tired of telling these dumb machines every instruction they should perform! I know that they only pretend to be dumb, and they can act clever if they're taught to.

After I implemented yet another graph traversing algorithm, after I wrote "if it doesn't exists, create, and only then proceed" for the million times, after I had troubles with parallelization of totally independent tasks just because I didn't write the program to be parallel, I felt that I'm done. Most programs I wrote had a similar scheme: loosely connected computations at different nodes with some dependencies between the results of them.

But over and over again I had to implement the exact sequence, while it really either doesn't matter, or is easily computable. I started looking for a language that takes this burden off me, for the language that allows me to declare my intent and build the sequence of actions on its own.

I discovered that there's a programming paradigm devoted explicitly to this: declare intent rather than steps. And, not least, I should be able to use it at work, to write production code with it! However, the languages I knew back then didn't support this paradigm. So I started looking, both for new languages and inside the things I already knew to spot declarativeness in them.

I was disappointed. Nearly nothing out there was really what I want. Here's what I've discovered so far.

Markov algorithms

What is Markov algorithm?

Markov algorithm is one of the basic forms of algorithm notations. It expresses a computations as a series of string rewrites. Given an input—a string—execution of Markov algorithm is a prioritized substitution of one substring with another, the substitutions being prespecified and immutable; the list of substitutions forms an algorithm. Here's a program that converts binary to unary:

|0 → 0||
1  → 0|
0  → 

You can see why this works on the Wikipedia page.

You can simulate a Turing machine with Markov algorithm. Just draw a "caret" on the string and express the algorithm as the caret movements. Inside the caret you may store and modify the "state" of a Turing machine. Here's how a Turing machine's rule "if a caret reads A and is in the fourth state, write B, change state to 10th and move right" would look like:

|4|A → B|10|

That is, Markov algorithms form a Turing-complete language.

This is a declarative language, about which almost everyone who studied computer science knows. (Well, at least, in Russia; due to national origin of its inventor it may be more widespread there). However, nearly no one could name it when asked about declarative languages. Sad that people don't establish connections between what they learn at the university and what they call "real life"...

Back to the topic, Markov algorithm is an ordered set of statements A → B, each of which really means "there can't be sequence A in the resultant string! But if there is, it should be replaced with B"

It's nevertheless questionable if Markov algorithm is declarative. Sure, it doesn't tell exactly how the computation should occur. Of course, the exact calculations can be easily inferred, but it is true for any declarative language.

The problem with it is that Markov algorithm is a mathematical abstraction. Although, following the notation of Markov algorithms, some useful languages were designed (one of them is the Refal language), they still resemble mathematical abstractions, and I can't see how I could use them in production.

Make

What I could—and do—use in production is Make, the language of Linux makefiles. It doesn't have a specific name, actually, so I'll call it "Make" here.

Basically, a structure of a Make program is a list of "rules". Each rule specifies a target (or a target pattern, like %.cpp), and a list of prerequisites. A rule means that in order to accomplish the target, you must

  1. get done with prerequisites
  2. execute commands specified within the rule.

The commands usually take prerequisites as input, but they don't have to. This creates an oriented graph of dependencies, which is walked until you reach the target. Sounds like a cool concept, which should have been backed up with some kind of mathematical theory and notation!..

In reality, it's not. Make evolved from (and, perhaps, still remains) a build system. And here go some gory details. The targets and prerequisites denote files in the filesystem, and the timestamp of these files (i.e. the time they were last modified) is used as a measure of whether you're "done" with prerequisites.

The rules are implemented in one of the shell languages you choose. Technically, they could be written in any language, but shell scripts are chosen because they're tied to work with files as input—and files are the primary objects Make works with. This fits building application perfectly, but you can go over this domain if you treat file system as just a key-value database.

However, the biggest limitation of Make is that it can't modify the dependency graph after the execution is started. There are techniques to overcome this restriction, but they're not generic, and are too tricky and fragile to be used in production.

Another thing is that Make is just...too old. Programming has changed a little since 1978, you know... Make is not flexible, has problems with debugging, and doesn't evolve much: its computational model is already exhausted.

Whenever

An esoteric language I was introduced to in a programmers.SE question is Whenever. Basically, a Whenever program consists of an unordered list of clauses. A clause can contain an operator. An operator is executed if the clause that contains it is in the to-do list, and if some conditions apply. Operators may add and/or remove clauses from the to-do list, or print text (and we know that printing text is no different from any useful work, actually).

Memory cells are implemented as possibility to have a clause several times on the to-do list and as expression that returns this number. The conditions mentioned above can refer to this expression.

Fairness is a property of a nondeterministic system, which can repeatedly and infinitely face a multiple choice, to not allow behavior when a certain item is never chosen past any moment of time. I wrote a big article about fairness, check it out for more details.

The language is called "Whenever" because its main feature is that it's not clear when a certain clause could be executed! Each clause in to-do list is considered for execution with uniform probability. (I think the equality of probability could be safely replaced with enforcing fairness). Sadly, the author was so excited with the concept of such "absence of urgency", that he overlooked the declarative nature of the language.

Here's a sample program, which is just a "while" loop. It should print A ten times:

1 defer (2) 3;
2 print ("A");
3 3;
4 defer (1 || N(3)<=10) -3#N(3),-5;
5 again (5) defer (1 || N(3)>10) 2,1;

Play with the language yourself! You can read more about the available commands, and download Java (sic!) interpreter at the Whenever language homepage. It seems that David Morgan-Mar is its author.

Whenever language's key difference from Make is that it has control over its own dependency graph. It can impose and drop dependencies as the clauses are executed. However, the weakness of this language is its esoteric nature coupled with absence of a developed mathematical model (it resembles several concepts at once, but not any particular one).

Still not what I want

So, my results of searching a declarative language, which could fit into production, are still not great. The languages I observed here are either too conceptual (Markov algorithms and Whenever), or too limited (Refal, Make). I believe this gap will be getting closer over time. Or, perhaps, I'm just searching in the wrong place? Maybe such languages as Haskell and Prolog can already do what I want, but I'm just not aware of it? And do I really know what I want?..

Well, this only means that there'll be another blog post when I get to it!

Read on | Comments (2) | Make a comment >>


The most awful limitation of Make

I was trying to invent the way for GNU Make to depend on contents of files, rather than on timestamps.

If what was implied is the content change between consequent make invocations (i.e. target should not be remade, if timestamp has changed, but the content did not), it's easy. Prototyped in my own Stackoverflow answer, the solution was simple:

content=.md5-$1-$(shell md5sum <$1 | sed 's/[[:space:]].*//')
define track
$(call content,$1):
        rm .md5-$1-*
        touch $$@
endef

$(eval $(call track,foo))
$(eval $(call track,bar))

all: $(call content,foo) $(call content,bar)
        ...

However, then I tried to solve more common use-case. The file gets updated, but if after update its content didn't change, then treat it as if it wasn't updated. I think it's impossible.

An only way to achieve it is to make makefile rebuild itself after such update. But it means that makefile should depend on this file, and this way the file will be rebuilt at each make invocation, regardless if it's really needed in this particular case.

In other words, once make established its plan of action, chose what targets will be rebuilt, nothing can change this plan. The targets will be executed in an order picked, and nothing that happens during the run will make the order or the set of them change.

And that's virtually the most important and the most restrictive make limitation. When I write my own build system, I'll make it powerful enough to solve this.

And for now, let's keep tracking timestamps, or search for better tools.

Read on | Comments (0) | Make a comment >>


Make: a Filesystem Transformation Prover

One of my favorite tools that belongs to Linux world is make. I'm a huge fan of it and, being a Linux programmer, use it in my everyday projects. But no, not as a build system I mostly use it. make is much more than just a casual build system.

Then, what is make? Having thought on that question, I think, I eventually devised a sound definition of what it actually does--or, at least, what its main activity is. Capability to build programs is just its tag feature that, unfortunately, hides more profound design principles.

What follows the red mark is just an utter garbage. While it may contain somewhat interesting observation, which are partially true, it is just wrong. I'll show it later.

make is a filesystem transformation prover. Its purpose is to prove that file system can be transformed from its current state to the specified, according to the rules in the makefile. The result is a proof, given (a) initial filesystem state, (b) set of inference rules and (c) target files that should be brought to up-to date state. make should provide the sequence of rules, in which every rule a: b c d ... is only triggered for an up-to-date b, c etc and not up-to-date a. Such makefile rules can be described as Assuming up-to-date-ness of b, c and d, infer up-to-date-ness of a after applying the commands listed in rule. And the sequence of rules built should have every target file (specified in make's command line) "inferred" this way.

Sounds familiar, right? Isn't it the same as proving logical formulas? It totally is. Up-to-date files from a given filesystem state are premises. Rules in makefile are the logical arguments (pattern rules resemble them even better). And targets are the formulas we're to prove.

Impressed? Now create a Makefile
aa%: a%
        touch $@
and try to make aaaaa in the folder where file named "a" exists. You'll fail to achieve this, and that's due to a huge limitation of make, which brings me back from mathematical dreams to reality: make is a build system. Flexible, but a build system.

But

However, make, upon successful construction one of the possible "proofs" for all targets, actually executes the commands along the sequence of rules. And these commands can be arbitrary, while soundness of the whole transformation requires each rule to update its target and nothing else. This shatters the clean concept described above a bit.

Much of functionality of make unmentioned here may be treated just as tricky preprocessing of a makefile to shorten the number of characters needed to describe a set of explicit rules.

At the same time, this doesn't always suffice all users. The thing is, the rule format in make is quite limited and they can reason only about "up-to-dateness" of files in a given state. While users need more complex inference rules, for example "Target is up-to-date if its argument didn't change since the previous make invocation". While such rules can be implemented in terms of existing capabilities of make (like this), it would still be nice if make provided them out of the box.

Let alone many other caveats in GNU Make. Make is surely not ideal.

However

But this is a positive post rather than negative. The declarative nature of make and its availability (all modern major Linux distributions provide it) make it a an invaluable tool if you need such a "filesystem transformation prover" as a part of your project. And it's all "Linux way": make can act as a manager that runs many small scripts (invoked via commands within a makefile or written inline) with specific narrow purposes.

I used it with various aims: to process header files, to run regression tests and to update a repository mirror; and I always found it a very flexible and robust tool. So, do not hesitate to use it. Make is more than just a build system. Though, just a little more.

Read on | Comments (0) | Make a comment >>


More posts about make >>