Multi-component CMake-based project with Git X-Modules

How to build a multi-component project without installing all libraries.

At the time of writing this, there's no default solution for dependency management in the C world. There're several competing tools like Basel, Meson, conan, etc but none of them is recognized as standard de facto like Maven in Java world.

In this post, we will describe a CMake-based setup of a project consisting of several modules, which:

• can be built and installed independently;
• can be build together, so all the app and library sources can be edited, debugged, and committed simultaneously.

This setup resolves the pain of "fix and build the library" -> "install the library" -> "build the app with the library" -> "test the fix from the app" cycle.

As an example, we will create an app that computes SHA-1 checksum of its argument. To compute SHA-1 checksum we will use an external library by Steve Reid.

Project structure. Currently, the SHA-1 library in question has only a Makefile to build the project. We want to add a CMake-related build file (CMakeLists.txt) to this library. As we don't have write access to this repository we will fork this project to our Git server. For that, we will use Atlassian Bitbucket Server/Data Center as one of the most popular self-hosting Git solutions. It's also convenient because there's a special Git X-Modules App for it. If you are on any other Git Server, you may still use Git X-Modules as a command-line tool.

The SHA-1 library component will be called 'sha-1'. Then we will create a 'checksum' component that uses 'sha-1' library and produces an executable file. One should be able to build this component independently assuming 'sha-1' library is installed (into /usr/lib and /usr/include or any other OS-specific directory for libraries).

Finally, we will create a 'checksum-project' that includes both 'sha-1' and 'checksum' components and can be built without having to install 'sha-1' library to the system. This will allow us to open, edit, and build all components in an IDE.

From the Git perspective, one can use Git submodules for that. But this solution causes a lot of problems. For example, the Projectile package of Emacs doesn't currently work well with submodules. We will use a better solution: Git X-Modules.

Step 1. Create 'sha-1' repository with Atlassian Bitbucket Server/Data Center interface.

We want now to fork https://github.com/clibs/sha1 project there. To do that, clone the sha1 project from GitHub and push it to our self-hosted Bitbucket Server repository (we assume it's running on example.org domain):

git clone https://github.com/clibs/sha1.git sha-1
cd sha-1
git remote set-url origin http://example.org/scm/checksum/sha-1.git
git push origin master

Step 2. Add our custom CMakeLists.txt to 'sha-1' project with the following content:

cmake_minimum_required(VERSION 3.8 FATAL_ERROR)
project(sha1)

add_library(sha1 sha1.c)
target_include_directories(sha1 PUBLIC .)
set_target_properties(sha1 PROPERTIES PUBLIC_HEADER "sha1.h")
install(TARGETS sha1 LIBRARY DESTINATION lib ARCHIVE DESTINATION lib PUBLIC_HEADER DESTINATION include)

The install() command tells CMake to install the library to /usr/local/lib and the header to /usr/local/include, so it can be easily included by the app.

The target_include_directories() call is not necessary to build this project but will be useful later for the multi-module project structure.

Commit and push the change:

git add CMakeLists.txt
git commit -m "CMakeLists.txt added."
git push origin master

Make sure CMake can build the project. We can use these old-fashioned commands:

mkdir build
cd build
cmake ..
make

To install the library into the system one can use "sudo make install" command but later we will describe how to create a multi-component structure that doesn't need installation. So we do not install the library.

Step 3. Create a Git repository for our app.

Create a 'checksum' repository using Atlassian Bitbucket Server/Data Center interface.

Clone and "cd" into the repository:

git clone http://example.org/scm/checksum/checksum.git checksum
cd checksum

Create the main.c file with the following content:

#include <sha1.h>
#include <stdio.h>
#include <string.h>

#define SHA1_SIZE_IN_BYTES 20

void print_usage(char* command) {
  fprintf(stderr, "Usage: %s STRING\n", command);
}

void print_hex(char* buffer, int buffer_size) {
  for(int i = 0; i < buffer_size; i++) {
    printf("%02x", buffer[i] & 0xff);
  }
}

int main(int argc, char** argv) {
  if (argc != 2) {
    print_usage(argv[0]);
    return 1;
  }
  char sha1[SHA1_SIZE_IN_BYTES + 1];
  char* str = argv[1];

  SHA1(sha1, str, strlen(str));
  sha1[SHA1_SIZE_IN_BYTES] = '\0';

  print_hex(sha1, SHA1_SIZE_IN_BYTES);
  printf("\n");

  return 0;
}

Then create CMakeLists.txt file with this content:

cmake_minimum_required(VERSION 3.8 FATAL_ERROR)
project(checksum)

add_executable(checksum main.c)

target_include_directories(checksum PRIVATE sha1)
target_link_libraries(checksum PRIVATE sha1)

The main.c is simple: it just calls SHA1() function from 'sha-1' library and CMakeLists.txt adds a dependency on 'sha-1' library. Commit and push the change:

git add main.c CMakeLists.txt
git commit -m "Initial commit."
git push origin master

If 'sha-1' is installed on the system with "sudo make install" command of "Step 2", our checksum app can be build using CMake. But if 'sha-1' is not installed, CMake build command will fail because it can't find the header file for the library:

mkdir build
cd build
cmake ..
make

Scanning dependencies of target checksum
[ 50%] Building C object CMakeFiles/checksum.dir/main.c.o
/home/dmit10/work/blog/02-cmake-post/checksum/main.c:1:10: fatal error: sha1.h: No such file or directory
 #include <sha1.h>
          ^~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/checksum.dir/build.make:63: CMakeFiles/checksum.dir/main.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:73: CMakeFiles/checksum.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

Step 4. Multi-module structure. Now we want to create a multi-module repository 'checksum-project' with the following structure:

CMakeLists.txt
checksum/    <-- 'checksum' module insertion point
libs/sha-1/  <-- 'sha-1' library insertion point

If you don't have the Git X-Modules app for Bitbucket installed, install it now. To do so, go to Administration | Find new apps | Search the Marketplace and type "X-Modules".

Now create a new Git repository and name it 'checksum-project'.

Once the Git X-Modules app is installed, there will be an X-Modules button on the Git repository page. Click it.

As the repository is empty, click "Create Default Branch".

Then click "Add Module" to add the 'sha-1' library.

Choose the 'sha-1' repository.

Choose the 'master' branch in the 'sha-1' repository.

Make sure the 'sha-1' module is inserted into 'libs/sha-1' so "This Repository Path" should be 'libs/sha-1'.

Click "Add Module". Do not "Apply Changes" yet.

Now add another module to add 'checksum' app to our multi-module structure. To do that click 'Add Module' again:

Then choose the 'checksum' repository.

Choose the 'master' branch in it.

Make sure the 'checksum' repository is inserted into the 'checksum' directory, so "This Repository Path" should be 'checksum'. The directory name is arbitrary (as well as "libs/sha-1"), we will specify it later in add_subdirectory() call.

Click 'Add Module'

Apply the changes.

Now 'checksum-project' repository contains 2 directories: 'checksum' with 'checksum' module and 'libs/sha-1' with 'sha-1' library. They are inserted into 'checksum-project' as normal directories in Git.

Step 5. Create a multi-module CMake configuration in 'checksum-project'.

Clone and 'cd' into the 'checksum-project':

git clone http://example.org/scm/checksum/checksum-project.git checksum-project/
cd checksum-project/

Create CMakeLists.txt with the following content:

cmake_minimum_required(VERSION 3.8 FATAL_ERROR)
project(checksum-project)

add_subdirectory(libs/sha-1)
add_subdirectory(checksum)

Commit and push the file:

git add CMakeLists.txt
git commit -m "CMakeLists.txt configuration added."
git push origin master

That's all!

The project can now be built without installation:

mkdir build
cd build
cmake ..
make

[ 25%] Building C object libs/sha-1/CMakeFiles/sha1.dir/sha1.c.o
[ 50%] Linking C static library libsha1.a
[ 50%] Built target sha1
[ 75%] Building C object checksum/CMakeFiles/checksum.dir/main.c.o
[100%] Linking C executable checksum
[100%] Built target checksum

Now let's try our app:

checksum/checksum foobar
8843d7f92416211de9ebb963ff4ce28125932878

and compare it with standard 'sha1sum' utility:

echo -n foobar | sha1sum
8843d7f92416211de9ebb963ff4ce28125932878  -

All sources are in the same repository as normal Git directories. Now one can modify 'libs/sha-1' and 'checksum/' inside the 'checksum-project' directly and these changes will be synchronized with corresponding 'sha-1' and 'checksum' repositories. And vice versa. Have fun!