Skip Top Navigation Bar

Jextract - The Native Library Binding Extraction Tool

 

Introduction

When developing Java applications there can be use cases when those require access to libraries and third-party memory written in other programming languages. Project Panama was designed to meet the demand for better developer support in accessing native libraries, particularly for those developed with C/C++. Interaction between the JVM and the "foreign" (non-Java) APIs has been made simpler with Foreign Function and Memory API (FFM API). The FFM API became a final feature in JDK 22 and adds support for foreign memory access, as well as for foreign function calls.

The jextract tool parses header files (.h) of native libraries, and generates Java code, named bindings, which internally use the Foreign Function and Memory API. Thanks to the output of jextract you can directly use a Java model of the native libraries you are interested in.

This tutorial will walk you through how to obtain and run the jextract tool, but also on how to use the Java code that it generates.

 

Get jextract

Pre-built binaries for jextract are periodically released here. These binaries are built from the master branch of the jextract repository, and target the foreign memory access and function API in the latest JDK.

Download the binary suitable to your operating system and you can start using it. Alternatively, you can build jextract from the latest sources by following the instructions inside its repository.

⚠️ If you are using macOS Catalina or later you may need to remove the quarantine attribute from the bits before you can use the jextract binaries. Run the following command to achieve this: 

sudo xattr -r -d com.apple.quarantine path/to/jextract/folder/

 

Command Line Options

-D or --define-macro <macro>=<value>

Defines <macro> to <value> (or 1 if <value> omitted).

--header-class-name <name>

Designates the name of the generated header class. If this option is not specified, then header class name is derived from the header file name. For example, class "foo_h" for header "foo.h". If multiple headers are specified, then this option is mandatory.

-t, --target-package <package>

Defines the target package name for the generated classes. If this option is not specified, then unnamed package is used.

-I, --include-dir <dir>

Appends a directory to the include search paths. Include search paths are searched in order. For example, if -I foo -I bar is specified, header files will be searched in "foo" first, then (if nothing is found) in "bar".

-l, --library <name \| path>

Specifies a shared library that should be loaded by the generated header class. If starts with :, then what follows is interpreted as a library path. Otherwise, <libspec> denotes a library name. Examples:

  • -l GL,
  • -l :libGL.so.1
  • -l :/usr/lib/libGL.so.1.

--use-system-load-library

Libraries specified using -l are loaded in the loader symbol lookup (using either System::loadLibrary, or System::load). This option is useful if the libraries must be loaded from one of the paths in java.library.path.

--output <path>

Defines where to place generated files.

--dump-includes <file>

Dumps included symbols into specified file.

To filter on specific elements, jextract can generate a dump of all the symbols encountered in a header file. This dump can be manipulated, and then used as an argument file (with the @argfile syntax) to generate bindings only for a subset of symbols seen by jextract.

--include-[function,constant,struct,union,typedef,var]<String>

Includes a symbol of the given name and kind in the generated bindings. When one of these options is specified, any symbol that is not matched by any specified filters is omitted from the generated bindings.

--version

Prints tool version information, the JDK for which it was built, clang version and exits.

 

How to Run jextract

Let's take the following example: render a teapot using functions from freeglut (for Windows use freeglut MSVC package). freeglut is an open-source alternative to the OpenGL Utility Toolkit (GLUT) library.

⚠️ If you are using macOS Catalina or later you should install also mesa-glu, an open-source library that provides additional utility functions to complement the core OpenGL specification:

brew install mesa-glu

Typically, a native library has an include directory which contains all the header files that define the interface of the library, with one main header file. The freeglut library located at /path/to/freeglut/ has a directory path/to/freeglut/version/include where the header files are stored.

Let's open a terminal window in the root directory of the Java project you're working on, which has a src source directory corresponding to the root package, and run jextract to transform the main freeglut header into Java code:

# macOS and Linux compatible command
jextract --output src \
    -l :/opt/homebrew/Cellar/freeglut/3.6.0/lib/libglut.3.dylib \
    -I /opt/homebrew/Cellar/freeglut \
    -I /opt/homebrew/Cellar/mesa \
    -I /opt/homebrew/Cellar/mesa-glu \
    -t org.freeglut \
    /opt/homebrew/Cellar/freeglut/3.6.0/include/GL/freeglut.h

This command will transform the header file /opt/homebrew/Cellar/freeglut/3.6.0/include/GL/freeglut.h into corresponding Java classes by taking into account the following options:

  • --output src to store the output in src root directory;
  • -l :/opt/homebrew/Cellar/freeglut/3.6.0/lib/libglut.3.dylib instructs jextract that the generated bindings should load from the library path /opt/homebrew/Cellar/freeglut/3.6.0/lib/libglut.3.dylib;
  • -I /opt/homebrew/Cellar/freeglut, -I /opt/homebrew/Cellar/mesa, -I /opt/homebrew/Cellar/mesa-glu specify the header file search directories. These locations are used to find header files included through #include in the main header file;
  • -t org.freeglut specifies the target package to which the generated classes and interfaces will belong. (jextract will automatically create the package structure under the src directory specified through --output);
  • /opt/homebrew/Cellar/freeglut/3.6.0/include/GL/freeglut.h is the main header file of the native library you want to generate bindings for.

The equivalent command on Windows is similar:

# Windows PowerShell command
jextract --output src `
  -I "\path\to\freeglut\include" `
  --use-system-load-library `
  "-l" opengl32 `
  "-l" glu32 `
  "-l" freeglut `
  "-t" "org.freeglut" `
  "\path\to\freeglut\include\GL\glut.h"

Filtering

Some libraries are incredibly large (such as Windows.h), and you might not be need all the jextract generated code. For such situations, you can use jextract's --include-XYZ command line options to only generate classes only for the elements you specify.

To know which symbols you can filter, jextract can generate a dump of all the symbols encountered in a header file:

# macOS and Linux compatible command
jextract --output src \
    -l :/opt/homebrew/Cellar/freeglut/3.6.0/lib/libglut.3.dylib \
    -I /opt/homebrew/Cellar/freeglut \
    -I /opt/homebrew/Cellar/mesa \
    -I /opt/homebrew/Cellar/mesa-glu \
    -t org.freeglut \
    --dump-includes glut.symbols \
    /opt/homebrew/Cellar/freeglut/3.6.0/include/GL/freeglut.h

We obtain a glut.symbols file containing almost 5000 lines like the following:

## Extracted from: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/OpenGL.framework/Headers/gl.h

--include-function glAccum                                 # header: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/OpenGL.framework/Headers/gl.h
--include-function glActiveTexture                         # header: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/OpenGL.framework/Headers/gl.h
--include-function glAlphaFunc                             # header: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/OpenGL.framework/Headers/gl.h
--include-function glAreTexturesResident                   # header: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/OpenGL.framework/Headers/gl.h
--include-function glArrayElement                          # header: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/OpenGL.framework/Headers/gl.h
--include-function glAttachShader                          # header: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/OpenGL.framework/Headers/gl.h
...

You can edit it to contain only the symbols you need and then use it as an argument file (using the @argfile syntax) to generate bindings only for a subset of symbols seen by jextract:

# macOS and Linux compatible command
jextract --output src \
    -l :/opt/homebrew/Cellar/freeglut/3.6.0/lib/libglut.3.dylib \
    -I /opt/homebrew/Cellar/freeglut \
    -I /opt/homebrew/Cellar/mesa \
    -I /opt/homebrew/Cellar/mesa-glu \
    -t org.freeglut @glut.symbols \
    /opt/homebrew/Cellar/freeglut/3.6.0/include/GL/freeglut.h

⚠️ If you remove a declaration that is needed by another included structure, jextract will report the missing dependency and terminate without generating any bindings:

$ jextract --include-var aVar test.h
 
ERROR: aVar depends on A which has been excluded

 

Integrate Code Generated by jextract

Most of the methods that jextract generates are static, and are designed to be imported using import static. To access the code that jextract generates for the header file freeglut.h, only the following two wildcard imports are needed:

import org.freeglut.*;
import static org.freeglut.freeglut_h.*;

The import static org.freeglut.freeglut_h.*; statement will import all the static functions and fields from the class that jextract generates for the main header file of the library. This includes methods to access functions, global variables, macros, enums, primitive typedefs, and layouts for builtin C types. The import org.freeglut.*; statement imports all the other classes generated by jextract, which can include:

  • classes representing structs or unions,
  • function types,
  • and struct or union typedefs.

Now let's write the Teapot.java code. To speed up things, will take some inspiration from the Teapot.c native variant of the code:

#include <GL/glut.h>

void display(void)
{
  //Clear color and depth buffers
  glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
  glPushMatrix();
  // Assign a color to the teapot
  glColor3f(0.0, 1.0, 0.0);
  // Rotation
  glRotatef(10, 0.0, 0.0, 1.0);
  glRotatef(10, 0.0, 1.0, 0.0);

  //Draw
  glutWireTeapot(1);
  glPopMatrix();
  //Must swap the buffer in double buffer mode
  glutSwapBuffers();
}

void init(void)
{
  glClearColor(0.0, 0.0, 0.0, 0.0);
  //Model(Object coordinates), View (Camera coordinates), Projection (Screen coordinates)
  glMatrixMode(GL_PROJECTION);
  gluPerspective(40.0, 1.0, 1.0, 10.0);
  glMatrixMode(GL_MODELVIEW);
  gluLookAt(0.0, 0.0, 5.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.);
}

int main(int argc, char **argv)
{
  glutInit(&argc, argv);
  glutInitDisplayMode(GLUT_DOUBLE | GLUT_RGB | GLUT_DEPTH);
  glutInitWindowSize(500, 500);
  glutCreateWindow("Hello Panama!");
  init();
  glutDisplayFunc(display);
  glutMainLoop();
  return 0;
}

We can keep the display method in Java and change init to a constructor: 

import java.lang.foreign.Arena;
import java.lang.foreign.SegmentAllocator;

import org.freeglut.*;
import static org.freeglut.freeglut_h.*;

public class Teapot {

    Teapot(SegmentAllocator allocator) {
        // Reset Background
        glClearColor(0f, 0f, 0f, 0f);
        //Model(Object coordinates), View (Camera coordinates), Projection (Screen coordinates)
        glMatrixMode(GL_PROJECTION());
        gluPerspective(40.0, 1.0, 1.0, 10.0);
        glMatrixMode(GL_MODELVIEW());
        gluLookAt(0.0, 0.0, 5.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.);
    }

    void display() {
        glClear(GL_COLOR_BUFFER_BIT() | GL_DEPTH_BUFFER_BIT());
        glPushMatrix();
        glColor3f(0.0f, 1.0f, 0.0f);
        glRotatef(10, 0.0f, 0.0f, 1.0f);
        glRotatef(10, 0.0f, 1.0f, 0.0f);
        glutWireTeapot(1);
        glPopMatrix();
        glutSwapBuffers();
    }

    public static void main(String[] args) {
        try (var arena = Arena.ofConfined()) {
            var argc = arena.allocateFrom(C_INT, 0);
            glutInit(argc, argc);
            glutInitDisplayMode(GLUT_DOUBLE() | GLUT_RGB() | GLUT_DEPTH());
            glutInitWindowSize(500, 500);
            glutCreateWindow(arena.allocateFrom("Hello Panama!"));
            var teapot = new Teapot(arena);
            var displayStub = glutDisplayFunc$callback.allocate(teapot::display, arena);
            glutDisplayFunc(displayStub);
            glutMainLoop();
        }
    }
}

This example uses Arena.ofConfined(), a confined Arena because the application has a determined lifetime. The scope of a confined arena is alive from when it is created to when it is closed. A confined arena has an owner thread and that is the thread that created it. Just the owner thread can access the memory segments allocated in a confined arena. If you try to close a confined arena with a thread other than the owner thread, you will get an exception.

The memory segment argc is allocated with the arena by invoking Arena.allocateFrom(OfInt,int) and is initialized on this line: var argc = arena.allocateFrom(C_INT, 0);. C_INT is a constant generated by jextract, that has the value ValueLayout.JAVA_INT, and 0 is the value used to initialize the memory segment.

GLUT library initialization inside the main method is kept as similar as possible to its native variant:

    public static void main(String[] args) {
        try (var arena = Arena.ofConfined()) {
            var argc = arena.allocateFrom(C_INT, 0);
            glutInit(argc, argc);
            glutInitDisplayMode(GLUT_DOUBLE() | GLUT_RGB() | GLUT_DEPTH());
            glutInitWindowSize(500, 500);
            glutCreateWindow(arena.allocateFrom("Hello Panama!"));
            var teapot = new Teapot(arena);
            var displayStub = glutDisplayFunc$callback.allocate(teapot::display, arena);
            glutDisplayFunc(displayStub);
            glutMainLoop();
        }
    }

As the generated code for glutCreateWindow requires a MemorySegment, you allocate it with the previously initialized arena and store the window title in the off-heap memory associated with the memory segment:

glutCreateWindow(arena.allocateFrom("Hello Panama!"));

Finally, you can create a teapot by calling the Teapot.java constructor with the arena and then invoke display callback for the current window.

var teapot = new Teapot(arena);
var displayStub = glutDisplayFunc$callback.allocate(teapot::display, arena);
glutDisplayFunc(displayStub);
glutMainLoop();

To run the Teapot.java example you should first compile the freeglut generated code: 

# macOS and Linux compatible command
javac -d . src/org/freeglut/*.java

# For windows there are too many sources for command line. Put them into separate file
ls -r src/*.java | %{ $_.FullName } | Out-File sources.txt
javac -d classes '@sources.txt'

Next, let's launch the Teapot.java program:

# macOS and Linux compatible command
java -XstartOnFirstThread --enable-native-access=ALL-UNNAMED src/Teapot.java

# Windows PowerShell command
java -cp classes `
  --enable-native-access=ALL-UNNAMED `
  -D"java.library.path=C:\Windows\System32`;\path\to\freeglut\bin\x64" `
  src\Teapot.java

You should see a green teapot:

teapot

Tracing

When debugging an application is useful to inspect the parameters passed to a native call. Code generated by jextract supports tracing of native calls, meaning parameters passed to native calls can be printed on the standard output.

To enable the tracing support, just pass the -Djextract.trace.downcalls=true flag as a VM argument when launching your application:

# macOS and Linux compatible command
java -XstartOnFirstThread -Djextract.trace.downcalls=true --enable-native-access=ALL-UNNAMED src/Teapot.java

# Windows PowerShell command
java -cp classes --enable-native-access=ALL-UNNAMED `
  -D"jextract.trace.downcalls=true" `
  -D"java.library.path=C:\Windows\System32`;\path\to\freeglut\bin\x64" `
  src\Teapot.java

Bellow you can observe an excerpt of the previous command's output:

glutInit(MemorySegment{ address: 0x600001d9c080, byteSize: 4 }, MemorySegment{ address: 0x600001d9c080, byteSize: 4 })
glutInitDisplayMode(18)
glutInitWindowSize(500, 500)
glutCreateWindow(MemorySegment{ address: 0x600001d99070, byteSize: 14 })
glClearColor(0.0, 0.0, 0.0, 0.0)
glShadeModel(7425)
glLightfv(16384, 4611, MemorySegment{ address: 0x600001da4710, byteSize: 16 })
glLightfv(16384, 4608, MemorySegment{ address: 0x600001da4830, byteSize: 16 })
glLightfv(16384, 4609, MemorySegment{ address: 0x600001da4830, byteSize: 16 })
glLightfv(16384, 4610, MemorySegment{ address: 0x600001da4830, byteSize: 16 })
glMaterialfv(1028, 5633, MemorySegment{ address: 0x13789af10, byteSize: 452 })
glEnable(2896)
glEnable(16384)
glEnable(2929)
glutDisplayFunc(MemorySegment{ address: 0x11456c0c0, byteSize: 0 })
glutIdleFunc(MemorySegment{ address: 0x1145b6ac0, byteSize: 0 })
glutMainLoop()

 

Programming Language Support

Foreign Function and Memory API and jextract support C header files, yet other languages have C interoperability. You can still use jextract to integrate with libraries written in those language through an intermediate C layer. Checkout the table below to understand which other languages can work with jextract and how to do that:

Language Method of access
C++ C++ allows declaring C methods using extern "C" syntax, and many C++ libraries have a C interface alongside. Jextract can consume such a C interface, which can then be used to access the library in question.
Rust The Rust ecosystem has a tool called cbindgen which can be used to generate a C interface for a Rust library. Such generated C interface can then be consumed by jextract, and be used to access the library in question.

 

More Learning


Last update: December 17, 2024