GNU Make rule(s) from two lists

[Satoshi Yoda]

Good day! :3
I am learning Make for a few months, and got a problem that intuitively should have a simple solution, but I couldn't find it in documentation, and it is difficult to google...

Let's say I have 2 lists with the same length N in Makefile:

CPP_FILES = 1.cpp 2.cpp 3.cpp
O_FILES = one.o two.o three.o

Main point is that the element of the first list can not be simply calculated from the corresponding element from the second list.

And what I want is simple rules, like that or something:

one.o : 1.cpp
    g++ -c $^ -o $@

two.o : 2.cpp
    g++ -c $^ -o $@

three.o : 3.cpp
    g++ -c $^ -o $@

But without handwriting all N rules. Is there a way to do it?

[mbozzi]

Is there a way to do it?

Perhaps something like this?

.SUFFIXES:
TARGETS:=one.o two.o three.o four.o
PREREQS:=1.cpp 2.cpp 3.cpp 4.cpp

define GEN_RECIPE
$(1)
            g++ -c $$^ -o $$@
endef

$(foreach context,$(join $(addsuffix :,$(TARGETS)),$(PREREQS)),\
        $(eval $(call GEN_RECIPE,$(context))))

[kbw]

GNU make has a whole bunch of rules built in. For example, GNU make knows how to build a C and C++ program.

To demonstrate this:
* create a C hello world program called hello.c
* create a C++ hello world program called hello2.cpp

On the command line type:

make hello
make hello2

For your example you just need to define the add the names.

SRC1 = one.cpp one_options.cpp
SRC2 = two.cpp
SRC3 = three.cpp

all: one two three

one: $(SRC1:.cpp=.o)
    $(LINK.cpp) -o $@ $^ $(LFDLAGS)

two: $(SRC2:.cpp=.o)
    $(LINK.cpp) -o $@ $^ $(LFDLAGS)

three: $(SRC3:.cpp=.o)
    $(LINK.cpp) -o $@ $^ $(LFDLAGS)

all is a "phony" target. i.e. make will always try to create it, knowing that it doesn't actually exist.

So make tries to make all (being the first target). This is made up from one, two and three.

one is made up from one.o and one_options.o. And make knows how to do the rest. But I've gone the extra mile and supported projects with more than one source file by explicitly telling make how to put those .o files together. For a static or shared library, the rule will be different.

To see what rules make has, run:

make -np

-n - do nothing
-p - show definitions

Once you see the rules for compiling/linking, you'll see how you can pass compiler arguments in.

[Satoshi Yoda]

@mbozzi Thank you! You got the point, and your solution works for me =) But it looks a little like dark magic. I guess I will use it, and meanwhile read about some of these functions, so it will become less dark. Appreciate it a lot!

@kbw Thank you for the explanation of the basics and for example! Yet my main confusion was about building a relatively long list (not 3 but actually 33 or 1033 items) of .cpp files to .o files with completely different names (maybe I didn’t quite accurately put it, I’m sorry...) And I didn't want to add a new line to the Makefile each time I add another one .cpp into a project.

@kbw Maybe you can clarify a little, what does this part do?
$(SRC1:.cpp=.o)
Just for learning purposes ^_^

[mbozzi]

@mbozzi [It] looks a little like dark magic.

Glad it helped, but this is overkill. We could execute some shell commands in an if statement to the same effect.

GNU Make is metaprogrammable. Here's how it expands:

$(foreach context,$(join $(addsuffix :,$(TARGETS)),$(PREREQS)),\
        $(eval $(call GEN_RECIPE,$(context)))) ≡
$(foreach context,$(join $(addsuffix :,one.o two.o three.o four.o),1.cpp 2.cpp 3.cpp 4.cpp),\
        $(eval $(call GEN_RECIPE,$(context)))) ≡
$(foreach context,$(join one.o: two.o: three.o: four.o:,1.cpp 2.cpp 3.cpp 4.cpp),\
        $(eval $(call GEN_RECIPE,$(context)))) ≡
$(foreach context, one.o:1.cpp two.o:2.cpp three.o:3.cpp four.o:4.cpp,\
        $(eval $(call GEN_RECIPE,$(context)))) ≡
$(eval $(call GEN_RECIPE,one.o:1.cpp))
$(eval $(call GEN_RECIPE,two.o:2.cpp))
$(eval $(call GEN_RECIPE,three.o:3.cpp))
$(eval $(call GEN_RECIPE,four.o:4.cpp)) ≡
$(eval
one.o:1.cpp
            g++ -c $^ -o $@)
$(eval
two.o:2.cpp
            g++ -c $^ -o $@)
$(eval
three.o:3.cpp
            g++ -c $^ -o $@)
$(eval
four.o:4.cpp
            g++ -c $^ -o $@)

Function eval interprets its argument string as if it was Make source code. The effects are as if we wrote the required rules by hand.

[kbw]

Apologies.

$(SRC1:.cpp=.o)

means, replace all the .cpp file names in SCR1 with .o

So, just add your file names to SCR1 to get them built.

There’s a trick I use to get header file dependencies auto generated, but get it working with your .cpp files first.

The key to working with makefiles is to keep them as simple as possible. Stick to one directory, and if your project is partitioned into multiple directories, use a simple makefile for each, and let the makefile in the parent directory call them.

[doug4]

Main point is that the element of the first list can not be simply calculated from the corresponding element from the second list.

Can you go into the reason for this? The fact that your object file names don't correspond to their initial source file names sounds a bit odd to me. I'm not saying it's wrong, but it raises the question "Why?"

Is the reason you are compiling abc.cpp -> xyz.o compelling enough to justify abandoning the "normal" abc.cpp -> abc.o?

[Satoshi Yoda]

@doug4 About "Why?" Yes, there is some personal preferences that push me to abandon simple abc.cpp -> abc.o rules.
But this is a long story, and it is really subjective. Yes, definitely possible there is a better solution for my problems, but I need to start with something, and at the moment @mbozzi's foreach solution works just fine =)

So, the long story:
- My code contains several folders, and I expect it to possibly contain a lot more.
- I want my "build system" to be easily transferable from one project to another, because I have a lot of learning examples, simple projects, more complex ones, etc. So, just one Makefile sounds perfect.
- Firstly I started with simple general recommended strategy when each .cpp file compiled to .o file with the same name in the same directory. It works fine, and I used it for some time. But now I decided to try to move all .o files into separate folder (like temp/), because they create a little bit of a mess in the working folder (like code/). That maybe is not reasonable by itself, but I have 3 more connected reasons to build something not to the same folder:
1) I use sublime text for all the work, and .o files appears in the tree view thus eats valuable screen space that could be used for .h .cpp and other "important" files. And I don't want to exclude them with visible filter or something.
2) I read that www.gnu.org/software/make/manual/make.html#Automatic-Prerequisites (4.14 Generating Prerequisites Automatically) and I want to implement it in my build file at the moment. There it appears to be also a generated .d file for every .cpp file along with .o file, and I definitely do not want it to go to the main code directories, instead I want it to go to temp also.
3) I am learning Vulkan at the moment, and Vulkan projects contains a lot of GLSL shader files like .frag .vert .comp that compiles to .spv with something like that glslc abc.frag -o abc-frag.spv These text source files belongs to the same code directories as .cpp files, because they can be highly "intertwined" and work like a "complex solution". I definitely do not want to have separate shaders/ directory with all of them =) And there is also another thing: resulting binary .spv file must just go along with the executable, they are not "linked" to single file or something, so as the build result I want 1 main executable + 1 folder called shaders, that contains all .spv files with "nice" names, but does not contains all my code subfolders structure.

These 3 points lead me to that unobvious solution:
Compile all subfolders with working code into a single temp folder without subfolders, to do that I can replace / with - or something, like that:
code/scene/Scene.cpp -> temp/scene-Scene.o
in case of shaders like that:
code/batch/shader.vert -> build/shaders/batch-vert.spv
I can do this with just one line of code in Makefile, like that:
SPV_FILES = $(addprefix build/shaders/,$(addsuffix .spv,$(subst .,-,$(subst -shader,,$(subst code-,,$(subst /,-,${GLSL_FILES}))))))
That is a mess, yes, but it is just one line in one place. And it contains some hardcoded names, because it is work-in-progress, so I change it frequently.
But when I tried to create build target for these files it appeared that I need to do inverse string transform to specify the prerequisite:
build/shaders/batch-vert.spv -> code/batch/shader.vert
And I don't want to do it and maintain it, because it will be the second mess-code-line, and it should work exactly opposite to the first one, which leads to errors.

@kbw thanks for explanation :3 About several Makefiles in each directory, hmm... I did not think about it, maybe I can go to some existing solutions to clarify why is it needed and why is it used and when is it used...

@mbozzi
Thank you for expanding these functions, now I understand it =) Also I was confused with define ... endef construct, but I read about it, and it happens to be just multi-line declaration :3

[keskiverto]

- I want my "build system" to be easily transferable from one project to another, because I have a lot of learning examples, simple projects, more complex ones, etc. So, just one Makefile sounds perfect.
- Firstly I started with simple general recommended strategy when each .cpp file compiled to .o file with the same name in the same directory. It works fine, and I used it for some time. But now I decided to try to move all .o files into separate folder

Out-of-source build is a good practice.

You could look at CMake. It is a "build system". It writes Makefile(s) for you.
https://cmake.org/cmake/help/latest/guide/tutorial/index.html
Even in its tutorial we see out-of-source build:

mkdir Step1_build
cd Step1_build
cmake ../Step1
cmake --build .

Where the Step1_build is a temporary directory that will contain object files, etc.
The build directory could be anywhere, totally outside of the source tree.