What can C++ Libraries Learn from Lua?

The Convenience of Text Pattern Matching

const char *text = "Some people, when confronted with a problem, think \"I know, I'll use regular expressions.\". Now they have two problems.";

(Example for std::regex from http://http://en.cppreference.com/w/cpp/regex)

Parsing text with Regular Expressions is a powerful technique traditionally available with so-called 'scripting' languages, from the grand-daddy Awk through Perl onwards. Regular expressions are first-class citizens of these languages and their users get adept at using them. Of course, there can be too much of a good thing, and sometimes those users forget that they have perfectly good if-statements for making complex matches easier to write and read. This is why I like Lua string patterns precisely because they are more limited in scope. Strictly speaking, although similar, they aren't regular expressions since they lack the 'alternation' operator |. But they are a good balance between simplicity and power, which is the global virtue of Lua, together with smallness. (Lua in total weighs less than the PCRE library.)

C++ and its older brother C are not very good at text manipulation, by the standards set by these dynamic languages. So this investigation is about how we can use regular expressions for text wrangling, first in C and then in C++, and how the standard ways can be improved. My method will simply be to use the Lua string functions as a design template, whenever appropriate.

The C Way

For some obvious reasons, regular expressions are not so easy to use in C, although part of the POSIX standard and thus always available on compliant systems. There is no equivalent to regexp literals (/..../) so you have to compile the expression explicitly before you use it. You must also create a buffer to receive the positions where the match and its submatches occured in the string.

 regex_t rx;
 regmatch_t matches[1];  // there's always one match
 regcomp(&rx,"[[:alpha:]]+",REG_EXTENDED);

 if (regexec(&rx,"*aba!",1,matches,0) == 0) {
     printf("regexec %d %d\n",matches[0].rm_so,matches[0].rm_eo);
 } else {
     printf("regexec no match\n");
 }
 ...
 // explicitly dispose of the compiled regex
 regfree(&rx);

Doing something more realistic that just matching is a more involved. Say we wish to go over all the word matches with length greater than six in the above useful quotation:

 const char *P = text;
 char buffer[256];  // words are usually smaller than this ;)
 while (regexec(&rx,P,1,matches,0) == 0) {
     int start = matches[0].rm_so, end = matches[0].rm_eo;
     int len = end-start;
     if (len > 6) {
         strncpy(buffer,P+start, len);
         buffer[len] = 0;
         printf("got %d '%s'\n",len,buffer);
     }
     P += end; // find the next match
 }

Contrast with the Lua solution - note how '%' is used instead of '\' in regular expressions, with the character class '%a' meaning any char in the alphabet:

 for word in text:gmatch('%a+') do
     if #word > 6 then
         print ("got",#word,word)
     end
 end

So clearly there is room for improvement!

A Higher-level C Wrapper

I am not alone in finding classical regexp notation painful, especially in C strings where the backslash is the escape character. E.g. if we wanted to use special characters literally: "\$\(([[:alpha]]+\))". The Lua notation for this expression would simply be "%$%(%a+)%)" which is easier on the eyes and the fingers. So I've provided an option to rx_new to convert Lua notation into classical notation. It is a very simple-minded translation: '%' becomes '\', '%%' becomes '%', and the Lua character classes 's','d','x','a','w' become the POSIX character classes 'space', 'digit', 'xdigit','alpha' and 'alnum' within brackets like '[[:space:]]'. The semantics are not at all changed - these regexps only look like Lua string patterns, although mostly equivalent.

An optional POSIX-only part of llib (see rx.cin the lllib-p directory) provides a higher-level wrapper. rx_find is very much like Lua's string.find although we don't have the convenience of multiple return values.

 Regex *r = rx_new("[[:alpha:]]+",0);
 int i1=0, i2=0;
 bool res = rx_find(r,"*aba!",&i1,&i2);
 printf("%d from %d to %d\n",res,i1,i2);

 // Now find all the words!
 int start = 0,end;
 while (rx_find(r,text,&start,&end)) {
     int len = end - start;
     if (len > 6) {
         strncpy(buffer,text+start,len);
         buffer[len] = 0;
         printf("[%s]\n",buffer);
     }
     start = end;  // find the next match
 }
 ...
 // generic disposal for any llib object
 unref(r);

The need for an extra 'matches' array has disappeared and is now managed transparently by the Regex type. It's pretty much how a Lua programmer would loop over matches, if string.gmatch wasn't appropriate, except for the C string fiddlng - which is essential when you specially don't want to allocate a lot of little strings dynamically.

Here is a cleaner solution, with some extra cost.

 int j1=0,j2;
 while (rx_find(r,text,&j1,&j2)) {
     str_t word = rx_group(r,0);
     if (array_len(word) > 6) {
         printf("%s\n",word);
     }
     unref(word);
     j1 = j2;
 }

rx_group will return the indicated match as a llib-allocated C string; could have used our friend strlen but array_len is going to be faster, since the size is in the hidden header; I've put in an unref to indicate that we are allocating these strings dynamically and they won't go away by themselves.

So, some code for extracting 'NAME=VALUE' pairs in a string separated by newlines (the ever-flexible C for-loop helps with not having to do j1=j2 at the end of the loop body, where these things tend to get forgotten, leading to irritatingly endless loops.)

 Regex *pairs = rx_new("(%a+)%s*=%s*([^\n]+)",RX_LUA);
 str_t test_text = "bob=billy\njoe = Mr Bloggs";
 for (int j1=0,j2; rx_find(r,test_text,&j1,&j2), j1=j2) {
     str_t name = rx_group(r,1);
     str_t value = rx_group(r,2);
     printf("%s: '%s'\n",name,value);
     dispose(name,value);
 }

This is easier to write and read, I believe. Since the loop counters are not used in the body of the loop, and since this is C and not C++, you can write a macro:

 #define FOR_MATCH(R,text) (int i1_=0,i2_; rx_find(R,text,&i1_,&i2_; _i1=i2_)

There are also functions to do substitutions, but I'll leave that to the documentation. llib is linked statically, so using part of it incurs little cost - I'd estimate about 25Kb extra in this case. You will probably need to make normal copies of these strings - a general function to copy strings and other llib arrays to a malloc'd block is here:

 void *copy_llib_array (const void *P) {
     int n = array_len(P) + 1;   // llib always over-allocates
     int nelem = obj_elem_size(P);
     void *res = malloc(n*nelem);
     memcpy(res,P,n*nelem);
     return P;
 }

News from the C++ Standards Committee

Some may think I have anti-C++ tendencies, but a professional never hates anything useful, especially if it pays the bills. So I was interested to see that regular expression support has arrived in C++.

 #include <regex>
 using namespace std; // so sue me
 ...
 // 'text' is the Useful Quote above...
 regex self("REGULAR EXPRESSIONS",
         regex_constants::ECMAScript | regex_constants::icase);
 if (regex_search(text, self)) {
     cout << "Text contains the phrase 'regular expressions'\n";
 }

That's not too bad - the Standard supports a number of regexp dialects, and in fact ECMAScript is the default. How about looking for words longer than six characters?

 regex word_regex("(\\S+)");
 string s = text;
 auto words_begin = sregex_iterator(s.begin(), s.end(), word_regex);
 sregex_iterator words_end;

 const int N = 6;
 cout << "Words longer than " << N << " characters:\n";
 for (sregex_iterator i = words_begin; i != words_end; ++i) {
     smatch match = *i;
     string match_str = match.str();
     if (match_str.size() > N) {
         cout << "  " << match_str << '\n';
     }
 }

Perhaps not the clearest API ever approved by the Standards Committee! We can make such an iteration easier with a helper class:

class regex_matches {

 const regex& rx;
 const string& s;

public:

 regex_matches(const regex& rx, const string& s): rx(rx),s(s) {
 }
 sregex_iterator begin() { return sregex_iterator(s.begin(),s.end(),rx); }
 sregex_iterator end() { return sregex_iterator(); }

}; ....

 regex_matches m(word_regex,s);
 for (auto match : m) {
     string match_str = match.str();
     if (match_str.size() > N) {
         cout << "  " << match_str << '\n';
     }
 }

We're finally getting to the point where a straightforward intent can be expressed concisely and clearly - this isn't so far from the Lua example. And it is portable.

Substituting text according to a pattern is a powerful thing that is used all the time in languages that support it, and std::regex_replace does a classic global substitution where the replacement is a string with group references.

Alas, there are some downsides. First, this does not work with the GCC 4.8 installed on my Ubuntu machines, but does work with the GCC 4.9 I have on Windows. Second, it took seven seconds to compile simple examples on my I3 laptop, which is an order of magntitude more than I expect from programs of this size. So, in this case, the future has not arrived.

A C++ Wrapper for POSIX Regular Expressions.

Portability is currently not one of my preoccupations, so I set out to do a modern C++ wrapping of the POSIX API, in a style similar to llib-p's Regex type. (Fortunately, the GnuWin32 project has made binaries for the GNU regex implementation available - although they are only 32-bit. The straight zip downloads are what you want, otherwise you will probably have unwelcome visistors on your machine.)

When testing this library on large-ish data, I received a shock. My favourite corpus is The Adventures of Sherlock Holmes from the Gutenberg project; just over 100,000 words, and this regexp library (from glibc) performs miserably to do something that is practically instantaneous in Lua. (std::regex is much better in this department.) So I've taken the trouble to extract the actual Lua pattern machinery and make it available directly from C++.

Let me jump immediately to the words-larger-than-six example. It is deliberately desiged to look like the Lua example:

 Rxp words("%a+",Rx::lua);  // simple Lua-style regexps
 for (auto M:  words.gmatch(text)) {
     if (M[0].size() > 6)
         cout << M[0] << "\n";
 }

When modern C++ cooks, it cooks. auto is a short word that can alias complicated types - (here just Rx::match), but the range-based for-loop is the best thing since std::string. And I've got my order-of-magnitude smaller compile time back, which is not an insignificant consideration.

(if you want to use the Lua engine, then replace the first declaration with simply Rxl words("%a+").)

I resisted the temptation to provide a split method; heaven knows the idea is useful but doesn't have to be implemented that way. It would return some concrete type like vector<string> and it would introduce a standard library dependency other than std::string into the library. Rather, Rx::match has a template method append_to which will take the results of the above iteration and use push_back on the passed container:

 vector<string> strings;
 words.gmatch(text).append_to(strings);

If you wanted a list<string> instead, then it trivially happens. You can append to a container and so forth without awkward splicing.

What would it mean to populate a map with matches? I don't think there's one answer, since there is no clear mapping, but here is one way of interpreting it; the expresson must have at least two submatches,and the first will be the key, and the second will be the value:

 Rx word_pairs("(%a+)=([^;]+)",Rx::lua);
 map<string,string> config;
 string test = "dog=juno;owner=angela"
 word_pairs.gmatch(test).fill_map(config);

Again, this will work with any smart array, not just std::map, as long as it follows the standard and defines mapped_type. The user of the class only pays for this method if it is used. These are considerations dear to the C++ mindset.

I'm an admirer of Lua's string.gsub, where the replacement can be three things:

• a string like '%1=%2', where the digits refer to the 'captured' group (0 is the whole match)
• a lookup table for the match
• a function that receives all the matches

The first case is the most useful. Here we have a match, where the submatch is the text we want to extract (called 'captures' in Lua.)

 Rx angles("<(%a+)>",Rx::lua);
 auto S = "hah <hello> you, hello <dolly> yes!";
 cout << angles.gsub(S,"[%1]") << endl;
 // --> hah [hello] you, hello [dolly] yes!

With the second case, I did not want to hardwire std::map but defined gsub as a template method applicable to any associative array.

 map<string,string> lookup = {
    {"bonzo","BONZO"},
    {"dog","DOG"}
 };
 Rx dollar("%$(%a+)",Rx::lua);
 string res = dollar.gsub("$bonzo is here, $dog! Look sharp!",lookup);
 // --> BONZO is here, DOG! Look sharp!

Forcing the third 'callable' case to overload correctly doesn't seem possible, so it has a distinct name:

 // we need a 'safe' getenv - use a lambda!
 res = dollar.gsub_fun("$HOME and $PATH",[](const Rx::match& m) {
     auto res = getenv(m[1].c_str());
     return res ? res : "";
 });
 // --> /home/user and /home/user/bin:/....

(One of the ways that GCC 4.9 makes life better is that generic lambdas can be written, and the argument type here just becomes auto&.)

In this way, string manipulation with regular expressions can be just as expressive in C++ as in Lua, or any other language with string pattern support.

The files (plus some 'tests') are available here, but remember this is very fresh stuff - currently in the 'working prototype' phase.

Transferable Design

The larger point here is that dynamically-typed languages can provide design lessons for statically-typed languages. C++ and C (particularly) are poor at string manipulation compared to these languages, but there is nothing special here about dynamic vs static: it is a question of designing libraries around known good practices in the 'scripting' universe and using them to make programming more fun and productive in C and C++.

There is a limit to how far one can go in C, but it's clear we can do better, even if llib itself might not feel like the solution. C is not good at 'internal' iterators since function pointers are awkward to use, and so 'external' iterators - using a loop - is better for custom subtitutions, etc.

C++ has already got a standard for regular expressions, but it will take a while for the world to catch up: we are being hammered here by heavy use of templates, all in headers. This is of course a classic example of too much of a good thing, because generic code is how C++ escapes the tyranny of type hierarchies, by compile-time static duck-typing. (The fancy word here is 'structural typing'.) For instance, Rxp::gsub can use anything that looks like a standard map.

Even there, I wanted to present an alternative design, not necessarily because I want you to use it, but because looking at alternatives is good mental exercise. The Standard is not Holy Scripture, and some parts of it aren't really ready, in a pragmatically useful way. In the comfort of our own company repository I can choose a solution that does not damage our productivity.

shmake - A Shell-based Build Tool

Beyond Make

Make is a marvelous tool in the Unix tradition: it does its one thing well. But modern software leans on build tools much more heavily, for conditional configuration and deployment, as well as for only rebuilding files when the files they depend on change. If Make was sufficient, then it would not be constantly being reinvented.

Make has always assumed that all the other Unix tools would be available, which makes cross-platform hard. So other build systems have appeared, either native build file generators (like CMake generating makefiles and MSBuild files) or directly tracking dependencies and running the compiler, like SCons or my own Lake.

shmake does not try to be cross-platform; its expressive power comes from the Unix shell. In fact, shmakefiles are shell scripts, except they are not executed standalone - rather they are run from within shmake itself. What it does is use shell as its DSL for expressing builds, rather than extending Make with shell-like constructs (like GNU Make) or inventing its own language (like CMake).

Assume a program consists of two source files and a header; here is a shmakefile for building it:

 #!/bin/sh
 . /tmp/shmake.sh

 COMPILE='gcc -c @(INPUT) -o @(TARGET)'
 LINK='gcc @(DEPS) -o @(TARGET)'

 T hello.o hello.c common.h "$COMPILE"
 T common.o common.c "$COMPILE"
 T hello hello.o common.o "$LINK"
 all hello

shmake thinks very much like Make; there are targets, but they're explicitly indicated by T: the target hello.o is the result of applying a compile command to hello.c, with common.h as an extra dependency. The object files are linked to give a target hello, and finally there's a special 'all' target which depends on hello. The commands use @() for variable expansion; for the first target, TARGET is hello.o, INPUT is hello.c and DEPS would be 'hello.c common.h'. Once the shmakefile is run within shmake, the target dependencies are checked and the tools invoked when a target is older than any of its inputs.

By itself, this isn't an improvement over Make. In particular, manually tracking dependencies (that hello.o depends also on common.h) is tedious and hard to get right. The point of this "hello build' is to emphasize that shmake is not limited to building programs, and is good for general dependency-based programming.

Convenient Compilation

For this simple program, the following shmakefile will do all the above, and more:

 simple$ cat shmakefile
 #!/bin/sh
 . /tmp/shmake.sh

 C hello hello.c common.c

 simple$ shmake
 compiling common.o
 compiling hello.o
 linking hello
 simple$ touch common.h
 simple$ shmake
 compiling hello.o
 linking hello

Note that shmake has detected the dependency of hello.c on common.h!

By default, shmake is not a chatty program (and the -q flag will make it even less chatty). To see the executed commands, use -v for verbose:

 simple$ shmake clean
 simple$ shmake -v
 gcc -c -Wall -MMD -O2 common.c -o common.o
 gcc -c -Wall -MMD -O2 hello.c -o hello.o
 gcc common.o hello.o -Wl,-s -o hello
 simple$ shmake clean
 simple$ shmake -v -g
 gcc -c -Wall -MMD -g common.c -o common.o
 gcc -c -Wall -MMD -g hello.c -o hello.o
 gcc common.o hello.o  -o hello

These are fairly generic flags (never compile C without -Wall; C's warnings are what most other languages would consider errors!) except for -MMD. This asks gcc to write a .d file, which shmake reads to build up target dependencies.

 simple$ cat hello.d
 hello.o: hello.c common.h

So 'C hello hello.c common.c' is a useful shortcut - under the hood, it expands to exactly the first explicit version. Note that since shmake is tracking all created targets, it knows how to construct a default 'clean' target, and the '-g' flag causes a non-optimized debug build to be made. If you pass a target name with extension .a, then C will build a static language; if you pass .so, it will build a shared library.

The implicit FAQ at this point might be:

• OK, that's convenient. But can these shortcuts be composed?
• This is a trivial program. How to make it link against libraries?
• What if I have my own special flags?

First, they are composable because they create targets, and targets can be made to depend on them, just as in Make.

 C first first/*.c
 C second second/*.c
 all first second

Second, the 'C' command supports the usual -I, -L -l flags which are passed to the compile or the link stage. But the 'S' command is more flexible and easier to read. 'S cflags' and 'S lflags' can be used to set custom compiler and link flags, which answers the third point. 'shmake CC=clang' works as expected - name-value pairs become environment variables passed to the script.

Here is a complete shmake file for building Lua - compare to the original makefile.

 #!/bin/sh
 . /tmp/shmake.sh

 LUA_LIB=liblua52.a
 S defines LUA_COMPAT_ALL
 S exports true
 S libs readline
 S libs m

 echo "Building Lua for $PLAT platform"

 case $PLAT in
 freebsd) 
     S defines LUA_USE_LINUX
     ;;
 Linux)
     S defines LUA_USE_LINUX
     S libs dl
     ;;
 ansi)
     S defines LUA_ANSI
     ;;
 posix)
     S defines LUA_USE_POSIX
     ;;
 solaris)
     S defines LUA_USE_POSIX LUA_USE_DLOPEN
     S libs dl
     ;;
 darwin)
     S defines LUA_USE_MACOSX
     ;;
 *)
     echo "unsupported platform $PLAT. One of freebsd,Linux,Darwin,posix,ansi,solaris"
     exit 1
     ;;
 esac

 C $LUA_LIB *.c -x 'lua.c luac.c'
 C lua lua.c $LUA_LIB
 C luac luac.c $LUA_LIB

 all lua luac

This is definitely easier to read than the original. One reason is the 'C' commands support wildcards, and the very useful -x,--exclude flag. Another is how the 'S' command makes conditional use of libraries and includes easier. The original makefile had to shell out make for each case; here we actually have a case statement since we have a real scripting language.

It is straightforward to get a debug build, just clean and use -g. It is also easier to modify. If the first line is replaced with:

 LIB_NAME=liblua52
 if [ -n "$BUILD_SO" ]; then
     LUA_LIB=$LIB_NAME.so
 else
     LUA_LIB=$LIB_NAME.a
     S exports true
 fi

then 'shmake BUILD_SO=1' will make you a shared library. (It will fail to build luac, which needs to be statically linked, but everything else works.)

Scripting with C

Our computers have been steadily getting more memory and faster I/O, even if individual core speeds have stalled. However, C has not been getting bigger. So on modern machines, C programs compile & link very quickly, even without tcc. Of course tcc is convenient because it can skip the irritating compile/link cycle. There is still that irritating boilerplate before you can write code. When testing llib, I found this shmakefile useful:

 c-script$ cat shmakefile
 #!/bin/sh
 . /tmp/shmake.sh

 # simple C script!
 S quiet true

 cfile=$P.c
 sfile=$P.s

 T  $cfile $sfile "
 cat >@(TARGET) <<block
 #include <stdio.h>
 #include <stdlib.h>
 #include <llib/all.h>
 #include <llib/template.h>

 int main(int argc, char **argv)
 {
 #line 1 \"@(INPUT)\"
 block
 cat @(INPUT) >> @(TARGET)
 cat >>@(TARGET) <<block
     return 0;
 }
 block
 "

 # llib needs C99 support
 C99 -g $P $cfile -n llib

 T all $P "./$P $A"

 c-script$ cat hello.s
 FOR (i,argc) printf("hello, world %s\n",argv[i]);
 c-script$ shmake P=hello A='one two three'
 hello, world ./hello
 hello, world one
 hello, world two
 hello, world three

Again, the shell script is doing all the work - 'hello.c' is a target which is generated by inserting 'hello.s' into some boilerplate. The compile step then creates the target 'hello' from 'hello.c', and finally the default target 'all' causes the program to be compiled and run. On this modest laptop, which most of you would scorn as a development platform, this compiles and executes practically instantly.

What is this '-n llib'? It means that the program needs llib. This is a useful concept borrowed from Lake. shmake resolves the need 'llib' by checking in turn if 'llib.need' or '~/.shmake/llib.need' exist. A need file is a simple config file that allows for expansions in the style of pkg-config's .pc files:

 c-script$ cat llib.need
 prefix=../..
 cflags=-I${prefix}
 libs=-L${prefix}/llib -lllib

If shmake cannot find llib.need, it invokes pkg-config.

To make this accessible everywhere, you could have a wrapper called 'crun' that looks like 'shmake -f ~/mystuff/cscript/shmakefile P=$1' and ensure that llib.need sits in ~/.shmake, with actual prefix of installed llib.

Needs are a useful solution to finding libraries, in the same style as pkg-config, but not requiring it to be present. (pkg-config is not universal, and many packages forget to use it even if it's around.) Even within the Linux ghetto, packagers cannot agree where to put things. For instance, on Debian the tcl headers are in /usr/include/tcl8.6, the Lua headers are in /usr/include/lua5.2; other distros may simply drop them in /usr/include or elsewhere.

Short and Sweet

shmake came out of a need to scratch an itch ('why is building software still so irritating?') and the need to test llib. To make things easier, llib is bundled with the tarball. There is no special install required - just copy shmake onto your path. It was written using the time-honoured Barbarian-in-a-Hurry (BIAH) methodology so there isn't much care about memory leaks. It may occaisionally try to eat your homework.

If nothing else, it is a concrete example of using shell as an embedded scripting language. Which is the the point I want to emphasize; these are shell scripts. There are many good resources on how to learn shell, and shell has many uses outside the niche of building, unlike gmake. If you have limited time to master things (and this is usually true these days) then it is better to master shell than gmake, precisely because it is more universally useful, and shmake makes that choice easier.

Pragmatic Modern C++ Survival Guide

Compiling: Brute Force Helps

C++ has a reputation for being the slowest mainstream language to compile, leading to light sabre office duels. A certain creative boredom can set in, when people start chasing the dream of the Suceessor to C++: "The creation myth of Go is something like this: Rob Pike, Ken Thompson, and Robert Griesemer were waiting for a particularly long C++ build to take place when they decided to theorize a new language" Zack Hubert. It was also a reason why I blew several years of my life on an interactive C++ interpreter.

The fact is, that C++ compilers are not slow. It is the incredible amount of inlined code that the compiler must digest before it can give you the proverbial Hello World application. The six or so lines of code pull in more than 17,000 lines of include files. The culprit here is actually C++ hanging onto an outdated compile/build cycle that was already quaint in 1973.

So the best way to improve build times is to get the best system you can afford, with particular attention to fast disk I/O, like keeping the compiler with its headers and libraries on a SSD. And remember that you will pay for heavily optimized builds; better to test unoptimized with debug information, since then your crash traces can make sense in a debugger. But mainly, the less included code a file needs to bring in, the better.

Embrace the Future

As the quote goes, the future is already here, it's just badly distributed. The modern standard (2011) is only now being widely available on all platforms. On Windows, Visual Studio 2013 is pretty much there and the Community Edition is available free with fairly liberal licensing; GCC 4.9 is available from the excellent TDM-GCC project. On Linux, Ubuntu 14.04 has GCC 4.8, and Clang 3.4 is easily available from the repositories; on OS X Clang is now standard.

I mention these choices because it's good to get a second opinion if you're struggling with weird errors; in particular, Clang gives better diagnostics than GCC, even if you would still be using GCC as your 'production' compiler. C++ is a standardized language with a standard library, and you can use that fact.

The new language is much more fun. The standard way to iterate over a container was:

 for (vector<int>::iterator it = vi.begin(); it != vi.end(); ++it) {
     int ival = *it;
     ....
 }

Now there is auto, much relief.

 for (auto it = vi.begin(); it != vi.end(); ++it) {
     int ival = *it;
     ....
 }

And finally, range-based for-loops.

 for (auto ival: vi) {
     ....
 }

You can even do this, thanks to std::initializer_list:

 for (auto i : {10,20,30,40}) {
   cout << i << endl;
 }

And the cost? Totally nada - the short form is completely equivalent to the first verbose form, and will work with any type that defines begin and end.

This new form of for loop is interesting, because it's the first piece of basic C++ syntax to be based on the underlying standard library. A big step, because the language does not provide batteries, just the means to create batteries. std::string is not hard-wired into the language, neither is std::map. The Go language decided to implement them in as primitives, just as Java allowed System.String to participate in operator overloading, which is otherwise not allowed. Both of these languages are reactions to C++, which achieve simplicity at the cost of generality. The unusually awkward error messages generated by C++ stem mainly from this not-baked-in philosophy, rampant function overloading, plus a possibily excessive reliance on generic programming.

auto represents a very cool idea, local type inference. It is an interesting idea that the new langauges of this century lean on this heavily, so that (for instance) idiomatic Go code has relatively few explicit type annotations. Meanwhile, the dynamic language people are adding type annotations, not necessarily for performance but rather for maintenance and better tooling. auto comes with a big gotcha ; since C++ is a value-based language, auto v = expression makes v the value type and does a copy; if you want a reference to that value, say auto& v = expression. (This burned me more than once, so I feel I ought to pass on the warning).

Move semantics are a powerful new idea. The concrete result of this is that you no longer need to worry about a function returning a std::vector, since the vector constructed in the function will be moved to the returned vector; basically its data (a pointer to some values) is moved to the new vector and zeroed out in the old vector. But the old advice to pass const &T if you just want to access a large object still stands, since C++ still copies across by value.

Parts of a system may want to keep objects from another part. This can be tricky, because we don't have a garbage collector cleaning up when the last 'owner' of the object is gone. Say a GUI object needs to keep a a persistent store object as part of its state; what must it do when it dies? It can ignore it, and assume that the persistent store people will free it one day. But how do the PSP know that their object isn't still being used? More machinery needs to be added, and so forth. (No wonder big C++ systems are like Heath Robinson/Rube Goldberg cartoons). Better to use a well-proven idea that has a standard implementation - shared ownership through smart pointers.

 class B {
     shared_ptr<A> a;

 public:
     B(shared_ptr<A> a) : a(a) { }

     int method() {
         // this is the underlying pointer
         cout << a.get() << endl;

         // otherwise behaves just like a regular pointer
        return a->method();
     }
 };

 void test() {
     shared_ptr<A> pa (new A());
     B b1(pa);   //b1 shares pa
     B b2(pa);   //b2 shared pa

     // exactly the same operation, since B's share an A
     b1.method();
     b2.method();

 } // b1 and b2 are now dead. No one left sharing pa, so it is deleted.

This is easier to show than explain - it is a way for objects to share an object, in such a way that when the last sharer goes, the shared object may be safely deleted. It works because the default destructor for B works by destructing its members, releasing the shared pointers. Smart pointers can be put into containers; when the container is destroyed, the shared pointers are again released. A note on Names: readability can be helped by typedefs:

 typedef std::vector<std::shared_ptr<A>> SharedVectorA;

Here's the new generalized typedef in action:

 template <class T>
 using SharedVector = std::vector<std::shared_ptr<T>>;

which can be used like so: SharedVector<A> my_a_list.

Dealing with Errors from the Compiler and Its Community

I must emphasize that C++ compilers, however stubborn and irritating they can be, do not embody active malice. They merely wish to share their understanding of the problem, in a verbose and pedantic way, like a person with no social skills. So the thing to do is concentrate on the first error and ignore the spew.

For example, a naive but hopeful person might think that iostreams can dump out vectors directly. Here are the first lines of the error (clang 3.4)

 for1.cpp:11:9: error: invalid operands to binary expression
 ('ostream' (aka 'basic_ostream<char>') and 'vector<int>')
 cout << vi << endl;
 ~~~~ ^  ~~

That's not too bad! Notice that the compiler uses the terminology of compilers, not humans. You have to know what a 'binary expression' is and that it has 'operands'. These terms are not part of C++, but part of how people talk about C++.

But the compiler will now go on for another ninety lines, telling you how it could not match vector<int> against any of the many overloads of ostream& << TYPE. These can be safely ignored, once you know that you can't dump out a vector directly.

GCC 4.8 makes a complete hash of it, however.

 for1.cpp:11:12: error: cannot bind 'std::ostream {aka std::basic_ostream<char>}' lvalue to 'std::basic_ostream<char>&&'
     cout << vi << endl;

Visual C++ 2010 is relatively clear, but thereafter degenerates into irrevalent confusion:

 for1.cpp(11) : error C2679: binary '<<' : no operator found which takes a right-
 hand operand of type 'std::vector<_Ty>' (or there is no acceptable conversion)

And so an important point: there is more than one compiler in the world, and some of them are more sensible than others.

We move on to the attitude of the community to errors. For instance, unlike many introductory texts, I don't bother to prefix standard types with std:: because the result is easier to read and type. You ought in any case know the commonly-used contents of the std namespace; if there is an ambiguity, then the compiler will tell you about it. The one thing you must not do is say using namespace std in a header file, because you will truly be Polluting the Global Namespace for everyone else. Anyway, this is one of the little things that can generate a lot of hot air on the Internet - it is a red flag to a certain kind of bull who believes that stylistic quirks represent major heresies. I'm reminded of Henry Higgins in My Fair Lady when he sings "An Englishman's way of speaking/Absolutely classifies him/The moment he talks/He makes some other Englishmen despise him/One common language I'm afraid we'll never get".

Like with compiler messages beyond the statement of the original error, it is a good idea to ignore random online opinion. Much better to read the classics - Dr Stroustrup himself is refreshingly pragmatic. The term 'Modern C++' has been redefined several times in the last twenty years, so remember that much earnest wisdom and advice has expired. It may be an unfashionable position, but reading blog posts and looking at Stackoverflow answers is not the way to learn a programming language like C++ properly. Go read a good book, basically. You're no longer forced to use paper.

Use a Good IDE

I suspect this will upset some bulls, because they themselves are super productive in the editors originally created by the Unix gods. Well, great for them, but I can't help feeling that the anecdotal experiences of a few very talented and focussed people does not represent universal good advice, and certainly isn't data. Personally I find the Eclipse CDT makes me more productive, and in my Windows days found Visual Studio equally effective. But I will qualify this: as a professional you should not be dependent on an environment that you cannot code outside - that's when you need a good general-purpose code editor in addition. For instance, when a program is first taking shape, a plain editor is less distracting; I don't write these articles in Word because it it is far too busy, and fails to undertand the creative writing process: first write, then edit; then format.

There is a learning curve with big IDEs, and you will have to read the manual. For instance, it is straightforward to bring a makefile-based project into Eclipse, but then you have to tell it what your include paths and defines are - it cannot deduce this from your makefile, (which is a task which many people find difficult anyway). Once it knows where to find the includes, the errors start disappearing, and the program becomes a live document, in the sense that any symbol becomes a link to its definition using ctrl-click, and so forth. If you type nonsense, the environment will start emitting yellow ink and finally red. This takes some getting used to, since it will often complain before you've finished an edit. Again, it's about filtering out irrelevant criticism. The benefits go beyond 'hyper-linking' to code completion, where ctrl-enter will complete functions and methods for you, and safe global renaming, which is useful for us who can never get names exactly right the first time. Having constant feedback about errors means that often your builds will be correct, first time.

So right tool for the job at hand, and knowing when a tool is appropriate. Investing some learning time in your tools always pays off. For instance, learn the keyboard shortcuts of your editor/IDE and you will not waste too much time with pointing your mouse around like a tourist trying to buy things in a foreign market. Learn your debugger well, because exploring the live state of a program is the best way to track down most bugs. C++ can be frustrating to debug, but GDB can be taught to display standard library objects in a clear way.

There seems to be some contradiction in what I'm saying: first I say that rich environments are too busy and distracting, and then say that they are enormously helpful. The trick is again to know what tool to use for what phase of a project.

For instance, I encourage you to write little test programs to get a feeling for language features - for this an IDE is irritating because setting up a new project is tedious. Better then to have a code editor which can run the compiler and indicate the errors. But for larger programs, you need to navigate the code base efficiently and perform incremental changes.

Good Design and Modularity

Of course, C++ does not have modules. They're likely to arrive in the coming years for the 2017 standard, but currently all we have are 'compilation units', aka 'files'. A class consists of a header file and an implementation file? Not necessarily. I think people are encouraged to keep classes separate to avoid source files getting too large. If a group of classes hunt in a pack then there's no reason why they can't live in the same file, and if nobody outside a file refers to a class, then it doesn't need to be defined in a header. Finally, if a class is just a collection of static functions, then that class is acting like a namespace and should be expressed as such - this is not Java.

Code that needs to see a class obviously needs to see its definition. A weakness of the current C++ compilation model is that such client code also needs to see non-public parts of the class definition, mostly because to create an object, whether directly on the stack or using new, it needs know how big that class is.

I am not a paranoid person so I don't worry about 'others' seeing the 'insides' of my class definition; but it does irritate me that client code depends on private members, and all classes that these members refer to. So any change in the implementaton of a class requires a rebuild of all code that refers to it, and all that code needs to pull in the private dependencies. One way around this irritation is the so-called 'Pointer to Implementation' pattern (PIMPL).

 // pimpl.h
 class Test {
 private:
    struct Data;
    Data *self;

 public:
    Test(double x, double y);
    ~Test();
    double add();
 };

Notice the private struct that isn't fully defined! So the first thing the implementation does is define that struct. Thereafter, the methods all refer to the hidden implementation pointer.

 // pimpl.cpp
 #include "pimpl.h"

 struct Test::Data {
    double x;
    double y;
 };

 Test::Test(double x, double y) {
    self = new Data();
    self->x = x;
    self->y = y;
 }

 Test::~Test() {
    delete self;
 }

 double Test::add() {
    return self->x + self->y;
 }

The cost is some extra allocation and redirection when accessing object state. But now imagine that you wish to expose a simple interface to a complicated object, then PIMPL will simplify the public interface and any pure implementation changes will not require a rebuild.

Another method which uses redirection is to define an interface:

 // Encoder.h
 #include <inttypes.h>

 class Encoder {
 public:
     virtual int encoder(uint8_t *out, const uint8_t *in, size_t insize) = 0;
     virtual int decode(uint8_t *out, const uint8_t *in, size_t insize) = 0;
 };

The derived subclasses do all the work:

 // simple-encoder.cpp
 #include "encoder.h"

 class SimpleEncoder: public Encoder {
     int encoder(uint8_t *out, const uint8_t *in, size_t insize) {
         ...
     }

     int decode(uint8_t *out, const uint8_t *in, size_t insize) {
         ...
     }
 };

 Encoder *SimpleEncoder_new() {
     return SimpleEncoder();
 }

Then the client code can get the proper subclass with a XXX_new function. This is not the most elegant way of creating subclasses, but it works and it's easy to understand; you add a new Encoder and add a reference to the builder in the file that's responsible for making new encoders. There is a little cost involved in the virtual method dispatch, but usually this will be dwarfed by the actual payload. (Again, premature optimization...).

The beauty of this decoupling is that the actual details of a particular encoder are kept within a single file, if it fits.

Good design is dividing larger systems into subsystems that are responsible for a single aspect of the problem. Networking, databases, GUI, etc are separate concerns and you make your later life easier by separating them out. And this makes for better builds on average, since modifying a subsystem implementation requires only compiling the affected files.