CrunchBang Linux

olegabrielz

#! Junkie

From: Norway

Registered: 2014-06-14

Posts: 442

The make command and multithreading

Everytime I see a howto regarding installation from source the recipe is basically:

./configure
make
sudo make install

I've noticed that the make command by default only run in one thread. And therefore with larger programs it takes forever to finish. Now that we have CPUs with multiple cores, and also cores with multi threading technology, the efficiency of the make command can be boosted up to use all available threads (or a number of threads that suits you).
This option gives make access to all available threads:

make -j

Or you can specify the number of threads you want to use by adding a suffix to the option:

make -j 6

I will assume that many people here already know of the -j (--jobs) option. But I didn't know about this until recently. And for me it was a great discovery

Last edited by olegabrielz (2014-12-27 00:47:36)

iMBeCil

WAAAT?

From: Edrychwch o'ch cwmpas

Registered: 2012-03-22

Posts: 1,026

Website

Re: The make command and multithreading

^+1

And, even better is:

$ export MAKE="make -j6"
$ make

and then you are sure that the 'sub-make'-s will be ran concurrently (multi-threaded), too.

What I find extremely useful is:

$ make > somefile 2>&1

It will store ALL messages from 'make' (so called 'stdout' and 'stderr') to 'somefile'. I find this indispensable for catching compilation errors ...

---

Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

olegabrielz

#! Junkie

From: Norway

Registered: 2014-06-14

Posts: 442

Re: The make command and multithreading

Thanks iMBeCil

I will try that out for sure!!

Last edited by olegabrielz (2014-12-26 14:46:23)

twoion

Emerald Caffeine

From: 星界

Registered: 2012-05-11

Posts: 1,648

Re: The make command and multithreading

Thanks iMBeCil

I will try that out for sure!!

Beware of caveats.

---

Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

olegabrielz

#! Junkie

From: Norway

Registered: 2014-06-14

Posts: 442

Re: The make command and multithreading

Thanks for the warning twoion

To be honest with you: That info was a bit cryptic to me. I have to do some reading before I can grasp what that warning actually means  :8

iMBeCil

WAAAT?

From: Edrychwch o'ch cwmpas

Registered: 2012-03-22

Posts: 1,026

Website

Re: The make command and multithreading

Thanks iMBeCil

I will try that out for sure!!

You're welcome, olegabrielz.

Beware of caveats.

If I understood correctly, sometimes certain 'make' arguments (defined in MAKE) are not really passed to sub-makes. But I'm not an expert in make-files and/or compilation (or even programming). Did you encounter such cases IRL?

However, I must say that I compile some stuff approx once a month (a GUI project including BLAS, FLTK ...) and I have never had problems with MAKE="make -j n" during compilation ...

Last edited by iMBeCil (2014-12-26 15:54:45)

---

Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

twoion

Emerald Caffeine

From: 星界

Registered: 2012-05-11

Posts: 1,648

Re: The make command and multithreading

For example, consider a project:

primary_project: helper_projects
  do_stuff

helper_projects: # are located in subdirs, maybe from a git-submodule
  make -C helper_project_1 # looks good right! they have their own makefile.
  make -C helper_project_2 # again!

Trying

$ make -Bj6

Will tell you to use '+'. -C is more intuitive (it's listed in make(1)!) to use than 'cd $subdir && $(MAKE)' because it makes sense to let make handle all the stuff and you have less to worry about, right? Haha, nope.

---

Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

iMBeCil

WAAAT?

From: Edrychwch o'ch cwmpas

Registered: 2012-03-22

Posts: 1,026

Website

Re: The make command and multithreading

For example, consider a project:

primary_project: helper_projects
  do_stuff

helper_projects: # are located in subdirs, maybe from a git-submodule
  make -C helper_project_1 # looks good right! they have their own makefile.
  make -C helper_project_2 # again!

Trying

$ make -Bj6

Will tell you to use '+'. -C is more intuitive (it's listed in make(1)!) to use than 'cd $subdir && $(MAKE)' because it makes sense to let make handle all the stuff and you have less to worry about, right? Haha, nope.

Thanks a lot, twoion, for simple and nice example.

I think now I understand ... that's what I expected more-less. Then, it is up to primary_project's maintainer to write MAKE-aware make files. Well, the worst thing which can happen is that compilation will not work, or it won't be as fast as possible.

One warning to OP, though: I did encounter cases where parallel compilation didn't work, while single thread one (like 'make -j 1') did. It was the 'race problem': due to 'make -j 4', one sub_project started to compile prematurely. It needed another sub_project to be compiled, which didn't compiled yet ... and there were all sort of errors. It was hard to catch, since sometimes compilation would work OK, and sometimes not .... because shell (or OS) on its on did decide how to arrange compilation processes ...

Moral of the story: if it doesn't compile with 'make -j 6', try 'make -j 1'-only first.

---

Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

olegabrielz

#! Junkie

From: Norway

Registered: 2014-06-14

Posts: 442

Re: The make command and multithreading

Thank you both iMBeCil and twoion

I just realized that I have to wade in a bit deeper to be able wrap my head around this make command  8)

mynis01

#! Die Hard

From: 127.0.0.1

Registered: 2010-07-02

Posts: 2,005

Re: The make command and multithreading

You guys might also find GNU Parallel to be of interest. It apparently works even with stuff other than compilers, but I haven't tested it myself.