SEARCH

Enter your search query in the box above ^, or use the forum search tool.

You are not logged in.

#1 2014-12-26 14:14:50

olegabrielz
#! Junkie
From: Norway
Registered: 2014-06-14
Posts: 442

The make command and multithreading

Everytime I see a howto regarding installation from source the recipe is basically:

./configure
make
sudo make install

I've noticed that the make command by default only run in one thread. And therefore with larger programs it takes forever to finish. Now that we have CPUs with multiple cores, and also cores with multi threading technology, the efficiency of the make command can be boosted up to use all available threads (or a number of threads that suits you).
This option gives make access to all available threads:

make -j

Or you can specify the number of threads you want to use by adding a suffix to the option:

make -j 6

I will assume that many people here already know of the -j (--jobs) option. But I didn't know about this until recently. And for me it was a great discovery smile

Last edited by olegabrielz (2014-12-27 00:47:36)

Offline

Be excellent to each other!

#2 2014-12-26 14:33:09

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2012-03-22
Posts: 1,026
Website

Re: The make command and multithreading

^+1

And, even better is:

$ export MAKE="make -j6"
$ make

and then you are sure that the 'sub-make'-s will be ran concurrently (multi-threaded), too.

What I find extremely useful is:

$ make > somefile 2>&1

It will store ALL messages from 'make' (so called 'stdout' and 'stderr') to 'somefile'. I find this indispensable for catching compilation errors ...


Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

Offline

#3 2014-12-26 14:45:28

olegabrielz
#! Junkie
From: Norway
Registered: 2014-06-14
Posts: 442

Re: The make command and multithreading

Thanks iMBeCil big_smile

I will try that out for sure!!

Last edited by olegabrielz (2014-12-26 14:46:23)

Offline

#4 2014-12-26 15:12:56

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: The make command and multithreading

olegabrielz wrote:

Thanks iMBeCil big_smile

I will try that out for sure!!

Beware of caveats.


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#5 2014-12-26 15:52:07

olegabrielz
#! Junkie
From: Norway
Registered: 2014-06-14
Posts: 442

Re: The make command and multithreading

Thanks for the warning twoion smile

To be honest with you: That info was a bit cryptic to me. I have to do some reading before I can grasp what that warning actually means  :8

Offline

#6 2014-12-26 15:53:49

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2012-03-22
Posts: 1,026
Website

Re: The make command and multithreading

olegabrielz wrote:

Thanks iMBeCil big_smile

I will try that out for sure!!

You're welcome, olegabrielz.

twoion wrote:

Beware of caveats.

If I understood correctly, sometimes certain 'make' arguments (defined in MAKE) are not really passed to sub-makes. But I'm not an expert in make-files and/or compilation (or even programming). Did you encounter such cases IRL?

However, I must say that I compile some stuff approx once a month (a GUI project including BLAS, FLTK ...) and I have never had problems with MAKE="make -j n" during compilation ...

Last edited by iMBeCil (2014-12-26 15:54:45)


Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

Offline

#7 2014-12-26 16:15:07

twoion
Emerald Caffeine
From: 星界
Registered: 2012-05-11
Posts: 1,648

Re: The make command and multithreading

For example, consider a project:

primary_project: helper_projects
  do_stuff

helper_projects: # are located in subdirs, maybe from a git-submodule
  make -C helper_project_1 # looks good right! they have their own makefile.
  make -C helper_project_2 # again!

Trying

$ make -Bj6

Will tell you to use '+'. -C is more intuitive (it's listed in make(1)!) to use than 'cd $subdir && $(MAKE)' because it makes sense to let make handle all the stuff and you have less to worry about, right? Haha, nope.


Tannhäuser ~ {www,pkg,ddl}.bunsenlabs.org/{gitlog,repoidx}

Offline

#8 2014-12-26 18:18:42

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2012-03-22
Posts: 1,026
Website

Re: The make command and multithreading

twoion wrote:

For example, consider a project:

primary_project: helper_projects
  do_stuff

helper_projects: # are located in subdirs, maybe from a git-submodule
  make -C helper_project_1 # looks good right! they have their own makefile.
  make -C helper_project_2 # again!

Trying

$ make -Bj6

Will tell you to use '+'. -C is more intuitive (it's listed in make(1)!) to use than 'cd $subdir && $(MAKE)' because it makes sense to let make handle all the stuff and you have less to worry about, right? Haha, nope.

Thanks a lot, twoion, for simple and nice example.

I think now I understand ... that's what I expected more-less. Then, it is up to primary_project's maintainer to write MAKE-aware make files. Well, the worst thing which can happen is that compilation will not work, or it won't be as fast as possible.

One warning to OP, though: I did encounter cases where parallel compilation didn't work, while single thread one (like 'make -j 1') did. It was the 'race problem': due to 'make -j 4', one sub_project started to compile prematurely. It needed another sub_project to be compiled, which didn't compiled yet ... and there were all sort of errors. It was hard to catch, since sometimes compilation would work OK, and sometimes not .... because shell (or OS) on its on did decide how to arrange compilation processes ...

Moral of the story: if it doesn't compile with 'make -j 6', try 'make -j 1'-only first. wink


Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

Offline

#9 2014-12-26 18:29:09

olegabrielz
#! Junkie
From: Norway
Registered: 2014-06-14
Posts: 442

Re: The make command and multithreading

Thank you both iMBeCil and twoion smile

I just realized that I have to wade in a bit deeper to be able wrap my head around this make command  8)

Offline

#10 2014-12-27 08:47:44

mynis01
#! Die Hard
From: 127.0.0.1
Registered: 2010-07-02
Posts: 2,005

Re: The make command and multithreading

You guys might also find GNU Parallel to be of interest. It apparently works even with stuff other than compilers, but I haven't tested it myself.

Offline

Board footer

Powered by FluxBB

Copyright © 2012 CrunchBang Linux.
Proudly powered by Debian. Hosted by Linode.
Debian is a registered trademark of Software in the Public Interest, Inc.
Server: acrobat

Debian Logo