Building libraries before using them

[adam2016]

Hi guys,

I've compiled a few libraries in the past such a SDL and FLTK

but why do you need to compile these libraries before you use them? it's much different to when compiling lets say a small project in an IDE, how come?

because from what I have noticed is that in these libraries such as FLTK there are many files included excluding the source and headers such folders like cmake, lib(static libraries), so why is there so many folders and files rather than not just including the actual cpp source and header files??

if anybody has any good resources/links I could read on this topic or even try to explain why this needs to be done I would be more than grateful, thanks :)


also is this compiling or building the libraries and what is the difference?

[poteto]

Mingw? Because mingw has 10 differen't flavors, 1000's of versions, and in general why use mingw when you can use msvc and have binaries built with VS 2015(14.0) are forward compatible with vs2017(14.1), vs2019(14.2), and (important to me) the only way to get the richest debug info on windows is from msvc (debatable, but C++ exception stack traces from SEH, core dumps, and compiling with debug flags can find C level mistakes like buffer corruption and memory leaks, is cool). But I use mingw because.... I don't know.

Also there is absolutely no reason to compile SDL since it is a C library, you either download the mingw version (.a files to link with), or msvc version (.lib files), C is just very very very easily binary compatible compared to C++ since it doesn't change much (even when things say things should be binary compatible there is always an exception if you look hard enough, but if it says it works, it should work, for the defaults atleast).

And for fltk, from my skimming of it, this library that isn't a DLL, its statically linked, msvc has this thing called incremental linking (I don't know much about it), but it is the default setting for a new project, and it prevents old 2015 static libraries from linking to newer versions (even if it's in the same major version). There is an option to make FLTK a DLL, so if you want to make your own github page for msvc binaries for fltk, you can try.

[Duthomhas]

Hey, woah there. There is absolutely nothing wrong with MinGW as a build environment -- plenty of people use it regularly to good effect.

And incremental linking has absolutely nothing to do with the ABI compatibility issues you get from mixing library versions (which you shouldn't do, because it causes problems).

For Windows, you generally can find precompiled libraries, but even on Windows you then have a maintenance nightmare for compiler + OS + architecture versions.

On Linux it is worse, making proper binary library distributions for every system impractical. Whenever you do find binary package repositories for something, it is because someone has an interest in both the OS and the library.

It makes life a whole lot simpler just to provide the source code and a build script.

"Compile" and "build" have overlapping meanings, but when considered distinct "compile" refers to compiling and linking a single component of a system whereas "build" refers to the entire system itself.

Hope this helps.

[poteto]

I assume when you say "mixing library versions which you shouldn't do, because it causes problems" isn't arguing against using library binaries (.lib) from vs2015 in vs2019, other than that, I would imagine that the format of the .obj were incompatible these would cause havoc to your code (this happens alot if you tinker with compiler flags but forget to recompile).

Other than that the incremental linking thing I said is wrong, it is probably a very specific problem for msvc where people version bump and forget to recompile everything and I misunderstood (I was just skimming through the net). Most likely .dll or statically linked libraries are treated similarly and they don't have any incremental linking on the boundary, and I just found out that msvc static libaries have the naming scheme of libXXXXX.lib, I now wonder where this naming scheme comes from.

[Duthomhas]

There is no One True Naming Scheme.

Stuff that starts with "lib" is a Linux convention, not Windows, so whenever you see it it is because of a build system that adheres to something Linux-y.

[adam2016]

but why the bloat?

I mean why couldn't libraries like fltk ( and many many others )just supply the cpp and header files, this way you could compile the code with your compiler( compile to a static lib ) and link it with your program,

seems like there is so many folders that includes more than the standard cpp and header files.

[Niccolo]

To great extent, even a complex library like Boost does offer, in a way, a method to compile to a static library for linking with code.

However, Boost in particular is many libraries in a collection. Each one is separated for selective use, and each one either builds it's own static library (or dynamic library), or is header only.

A library like FLTK, or wxWidgets, contains several components that may not be required when building an application. To make the library components selectable there may be some separation of those units into multiple library outputs.

Some have options to build separate component libraries or one large library.

Some GUI libraries include resources. These are data, some are images others are definition files of some kind, and they may be platform specific. In order to organize platform specific portions of a library, these are naturally divided by platform into various directories, and optionally included in a build depending on the target platform.

On some occasions cross compilation may be supported. This is where, for an example, one might build a Windows target on a Linux computer, or an Android target on a Windows computer.

In Boost, for example, a number of libraries abstract what are operating system concepts, like filesystem. Although we should opt for the standard library in modern C++, the std::filesystem came to the standard through Boost. For a long while, as a result, one would find that Boost organized that code so there was a layer of interface code, which presented the C++ abstractions we use to write applications, and another set of layers for each operating system supported which adapted that interface layer to the target operating system. This implies, as a result, a matrix of platform specific libraries to be built.

"I mean why couldn't libraries like fltk ( and many many others )just supply the cpp and header files..."

If this were all there was, how would one select those portions specific to a target platform? How would one know which portions are required to build a particular component of the library?

What is usually included is some kind of definition of the build, often using CMake or similar tool, which can be used either to build the library, or can create a solution file for some particular IDE to open (and subsequently build from there).

[adam2016]

Thanks Niccolo :)

for example in the fltk zip folder, when unzipped there are many different folders

obviously yes I understand why cmake is there, this is to actually build the static library in this case.

but why is there so many folders such as FL,fluid,GL,html,ide,jpeg,lib,src,zlib

in the folder IDE there are several sub folders such as different versions of microsoft visual studio and xcode.

lets take visualC2010 sub folder there are many .vcxproj files and also subfolders containing .obj and .pdb files, what are these used for and why are they needed?

thanks

[Niccolo]

Each of the folders you list have a different component.

The html folder, for example, probably contains code for a specialized control which can render html content, presumably from a website but possibly loaded from a file, to show formatted text inside a control. That feature is not frequently used for some types of applications, but is otherwise rather popular for other types of applications. Making it a separate component gives the consumers of this library a choice, which leads to faster builds when not used.

Similarly, GL probably refers to an OpenGL control, which allows you to write an application that can display OpenGL 3D models in a control. Many applications have no use for this, and some operating systems might not even support it. As a result, it is made an optional component.

The jpeg directory probably includes one of the handful of typical and widely used C sources for decoding jpeg images. This library is not written by the team producing FLTK, but in order to provide the ability to load and display jpeg images, they include this separate library. This is known as a dependency. Such a dependency is a 3rd party library that is require by the FLTK library in order to support jpeg image loading and saving.

The zlib directory probably includes yet another dependency, one of the handful of open source zip compression and decompression libraries. This allows FLTK applications to open and decompress zip files, but the zlib library is not written by the authors of FLTK. It is included as a dependency.

The lib is probably a more general directory, perhaps the output of building the library, or perhaps some additional dependency of small code.

The other libraries, for which I'm not sure of their purpose, likely follow the same basic concepts of dependencies or separation of components.

For any large project it becomes important to keep control of the various components which make up the whole target. For the most part, only simple applications use a single directory for the entire collection of source files. This eventually leads to clutter and confusion. Separating components into various directories isolates related topics like chapters in a book.

Each of the vcxproj files are containers of definitions for one of the Visual Studio IDE's. A single sln file will refer to each vcxproj, to create a final product comprised of several components. They are to the IDE what makefiles are to the old make build system. They list what source files are required to build a product, what compiler flags and options to use, etc.

The obj files are the intermediate object files, which is the result of compilation. They are used by the linker to assemble the final executable or library. The pdb files are Microsoft's extension for the "program database", a file of information providing debug information for the libraries and executables that are built. They are used by debuggers to give human readable names to variables and functions, and to map the executable binary code for the debugger's features.

[adam2016]

thanks Niccolo,

great insight and great breakdown, it's amazing how much technical details and intricacies go into libraries like this,

but the bigger question is where do you learn this type of stuff?

most C++ books and online sources will teach you how to code, algorithms and data structures different libraries etc etc but none of them ( or at least that I know ) don't teach you any of this stuff, how bigger libraries are built, how the build system works, how platforms cause libraries problems and the works around these using different make files etc etc

where do you learn these type of things? is there any good books? is there any good online resources that teaches you these types of things?

thanks :)

[Duthomhas]

Time and experience. Get a job. Volunteer for an Open Source project. Read everything you can get your hands on. Write your own mid-sized program (5000-10,000 LoC).

A good start is also to simply download and compile/modify an oss project, even if you will never contribute anything useful to the community....

[Niccolo]

@Duthomhas has the goods on this one. If there is a book on these points, I've never seen it.

These organizational attributes are just part of the natural evolution of using these systems.

When I started out, the primary tool was make under UNIX, or, on DOS, one of the Borland IDE's, primitive as they were.

I assume the notion of chapters in a book made sense, because that is likely the origin of that style of project organization.

As to what those dependency libraries involve, that is, as @Duthomhas pointed out, something most of us just observed when the earliest libraries came to our attention that used them as prerequisites.

Many of the file extension types also come to us through usage. The Microsoft IDE has used several different ones over the decades, all in the same basic plan. The "obj" file is observed simply by command line compilation and linking.

In a way, this is kind of like learning to swim by diving in and emerging without drowning. Some just don't make it, but fortunately they can move on to other pursuits when it is software development.

So, I would append @Duthomhas point with this:

Assume others have climbed the high mountains, and what you see is the result of many such minds having figured out good practices and follow along so as to absorb by emulation. Keep your wits about you and forge ahead with determination that if they can do this, you can.

It is so in many industries and arts. What is taught in universities are they theories and facts. The reason physicians go through a few years as interns before they're really trusted on their own is exactly this kind of thing. Mechanics freshly minted from technical schools are still not familiar with the practical techniques used by professionals, until they watch over the shoulders of the elders.

Here, though, you have a much easier opportunity, because through the Internet you can access some of the most sophisticated work products of great minds (and a lot of mediocrity as well), which affords all of us the opportunity to observe over the shoulders of thousands we would never have time to actually meet.

[Duthomhas]

+1

Keep in mind too that programmers are not a hive mind. Every now and then someone takes a look at things and says "I could do that better", then write code, and other people like the code and use it. We get things like CMake, which is in many ways a significant improvement over Make and Autotools. It can still be a pain to use, but...

The point is that there are a gazillion ways to do things, some of which are very popular and/or well-known, and a few of which are simply garbage, and a wide swath of stuff in-between.

Learn the really popular common tools, find one or two you like regardless, and ignore the rest until you have to learn it.

[mbozzi]

See, e.g., Colby Pike's "pitchfork layout" for a reasonable project organization:
https://api.csswg.org/bikeshed/?force=1&url=https://raw.githubusercontent.com/vector-of-bool/pitchfork/develop/data/spec.bs