docs: update README to include info about external grammars

itsaky · itsaky · commit 6a50a04d426b · 2023-06-18T20:19:40.000+05:30
diff --git a/README.md b/README.md
@@ -59,95 +59,143 @@ git clone --recurse-submodules https://github.com/AndroidIDEOfficial/android-tre
 
 A normal Gradle build (`./gradlew build`) can be executed in order to build everything for Android _and_ the host OS. To build `android-tree-sitter` and the grammars _only_ for the host OS, you can execute `buildForHost` task on appropriate subprojects.
 
-## Examples
+## Adding grammars
 
-First, load the shared libraries somewhere in your application:
+The Gradle modules for the grammars are almost identical, with only minor differences in the CMakeLists file and the Java binding class.
 
-```java
-    public class App {
-      static {
-        // load the tree sitter library
-        TreeSitter.loadLibrary();
-        
-        // native libraries for languages are automatically loaded
-        // no need to load them manually
-      }
-    }
-```
+These Gradle modules are automatically generated by the [`DynamicModulePlugin`](https://github.com/AndroidIDEOfficial/android-tree-sitter/tree/dev/build-logic/ats/src/main/java/com/itsaky/androidide/treesitter/DynamicModulePlugin.kt). The generation process relies on the [`grammars.json`](https://github.com/AndroidIDEOfficial/android-tree-sitter/tree/dev/grammars/grammars.json) file. More information about the structure of this JSON file can be found in the [README](https://github.com/AndroidIDEOfficial/android-tree-sitter/blob/dev/grammars/README.md) under the `grammars` directory.
+
+Apart from the `DynamicModulePlugin`, there are [other Gradle plugins](https://github.com/AndroidIDEOfficial/android-tree-sitter/tree/dev/build-logic/ats/src/main/java/com/itsaky/androidide/treesitter) which are used to configure and build the grammars effectively.
 
-Then, you can create a `TSParser`, set the language, and start parsing:
-
-```java 
-    try (TSParser parser = new TSParser()) {
-      parser.setLanguage(TSLanguagePython.newInstance());
-      try (TSTree tree = parser.parseString("def foo(bar, baz):\n  print(bar)\n  print(baz)", TSInputEncoding.TSInputEncodingUTF8 /*specify encoding, default is UTF-8*/)) {
-        TSNode root = tree.getRootNode();
-        assertEquals(1, root.getChildCount());
-        assertEquals("module", root.getType());
-        assertEquals(0, root.getStartByte());
-        assertEquals(44, root.getEndByte());
-
-        TSNode function = root.getChild(0);
-        assertEquals("function_definition", function.getType());
-        assertEquals(5, function.getChildCount());
-      }
-    }
+The common configuration for the grammars can be found in the [`build.gradle.kts`](https://github.com/AndroidIDEOfficial/android-tree-sitter/blob/52cc0400feee5079cac25b27d1e7b673ee53f436/build.gradle.kts#L136) file. This is where you can make changes or adjustments to the module configuration that applies to all grammars.
+
+The generated modules are located in the `rootDir/build/grammar-modules` directory. This is where you can find the output of the module generation process.
+
+To add a new grammar to the project, follow these steps:
+
+1. Begin by adding the grammar source code to the `grammars` directory. To accomplish this, you can add a submodule using the following command:
+```bash
+git submodule add <remote_url> grammars/<language_name>
 ```
+2. The `language_name` should be the simple name of the language, without the `tree-sitter-` prefix. This name is used to generate both the shared library and the Gradle module. For example, if the `language_name` is `abc`:
+    - The module `tree-sitter-abc` will be automatically generated.
+    - The name of the resulting shared library will be `libtree-sitter-abc.so`.
+3. After adding the grammar source, update the `grammars.json` file to include the newly added grammar in the project.
+4. Finally, sync the project to trigger the generation of the module for the newly added grammar.
+
+## Loading external grammars
+
+You have two ways to load grammars that are not published along with this project :
 
-For debugging, it can be helpful to see string representation of the tree:
+- Package the grammar with your application.
+- Load the grammar at runtime using `TSLanguage.loadLanguage`.
 
+`TSLanguage.loadLanguage` uses `dlopen` to load the library and must be CAREFULLY used. Also, grammars that are loaded using this method must be closed when they are not used.
+
+> **_Prefer using the first method whenever possible._**
+
+### Package the grammar with your application
+
+You can package the grammar in your Android application as you would package any other shared library :
+
+- Include the `libtree-sitter-myLang.so` file in the `jniLibs` directory of your project.
+- Create a native method in a Java class which will return the pointer to the language :
 ```java
-    try (TSParser parser = new TSParser()) {
-      parser.setLanguage(TSLanguagePython.newInstance());
-      try (TSTree tree = parser.parseString("print(\"hi\")")) {
-        assertEquals(
-          "(module (expression_statement (call function: (identifier) arguments: (argument_list (string)))))",
-          tree.getRootNode().getNodeString()
-        );
-      }
-    }
+package com.my.app;
+
+public class MyClass {
+
+  static {
+    System.loadLibrary("tree-sitter-myLang");
+  }
+
+  public static native long myLang();
+}
 ```
 
-If you're going to be traversing a tree, then you can use the `walk` method, which is much more efficient than the above getters:
+- Write the C/C++ implementation for the method :
+```c++
+extern "C" TSLanguage *tree_sitter_myLang();
+
+extern "C"
+JNIEXPORT jlong JNICALL
+Java_com_my_app_MyClass_myLang(JNIEnv *env, jclass clazz) {
+  // simply cast the language pointer to jlong
+  return (jlong) tree_sitter_myLang();
+}
+```
+
+- Create and use the `TSLanguage` instance :
 
 ```java
-    try (TSParser parser = new TSParser()) {
-      parser.setLanguage(TSLanguagePython.newInstance());
-      try (TSTree tree = parser.parseString("def foo(bar, baz):\n  print(bar)\n  print(baz)")) {
-        try (TSTreeCursor cursor = tree.getRootNode().walk()) {
-          assertEquals("module", cursor.getCurrentTreeCursorNode().getType());
-          cursor.gotoFirstChild();
-          assertEquals("function_definition", cursor.getCurrentTreeCursorNode().getType());
-          cursor.gotoFirstChild();
-          assertEquals("def", cursor.getCurrentTreeCursorNode().getType());
-          cursor.gotoNextSibling();
-          cursor.gotoParent();
-        }
-      }
+final TSLanguage myLang = new TSLanguage("myLang", MyClass.myLang());
+
+// use it with TSParser
+try (final var parser = new TSParser()) {
+  parser.setLanguage(myLang);
+  ...
+}
 ```
 
-For more examples, see the tests in `android-tree-sitter/src/test/java/com/itsaky/androidide/treesitter`.
+### Load grammars at runtime
 
-## Adding more grammars
+`TSLanguage` provides `loadLanguage(String, String)` method which can be used to load the grammars at runtime. This method uses `dlopen` to load the shared library, get the language instance and return its pointer. Use this method CAREFULLY.
 
-The Gradle modules for the grammars are almost identical, with only minor differences in the CMakeLists file and the Java binding class.
+The language instances created using this method **MUST** be closed using `TSLanguage.close()`. Calling the `close` method ensures that the underlying `dlopen`'ed library handle is closed using `dlclose`.
 
-These Gradle modules are automatically generated by the [`DynamicModulePlugin`](https://github.com/AndroidIDEOfficial/android-tree-sitter/tree/dev/build-logic/ats/src/main/java/com/itsaky/androidide/treesitter/DynamicModulePlugin.kt). The generation process relies on the [`grammars.json`](https://github.com/AndroidIDEOfficial/android-tree-sitter/tree/dev/grammars/grammars.json) file. More information about the structure of this JSON file can be found in the [README](https://github.com/AndroidIDEOfficial/android-tree-sitter/blob/dev/grammars/README.md) under the `grammars` directory.
+Usage :
+```java
+// provide the path to the shared library and the name of the language
+// the name is used to cache the language instance
+// further invocations of this method with the same lang name returns the
+// cached language instance
+final TSLanguage myLang = TSLanguage.loadLanguage("/path/to/libtree-sitter-myLang.so", "myLang");
+
+if (myLang != null) {
+  // loaded successfully
+} else {
+  // failed to load the language
+  // see logcat for details
+}
+```
 
-Apart from the `DynamicModulePlugin`, there are [other Gradle plugins](https://github.com/AndroidIDEOfficial/android-tree-sitter/tree/dev/build-logic/ats/src/main/java/com/itsaky/androidide/treesitter) which are used to configure and build the grammars effectively.
+Use this language :
+```java
+try (final var parser = new TSParser()) {
+  parser.setLanguage(myLang);
+  ...
+}
+```
 
-The common configuration for the grammars can be found in the [`build.gradle.kts`](https://github.com/AndroidIDEOfficial/android-tree-sitter/blob/52cc0400feee5079cac25b27d1e7b673ee53f436/build.gradle.kts#L136) file. This is where you can make changes or adjustments to the module configuration that applies to all grammars.
+You don't have to keep a reference to `myLang`. Once loaded, the language can be accessed using `TSLanguageCache` :
+```java
+// returns the 'myLang' instance i.e. both are same
+final TSLanguage cachedMyLang = TSLanguageCache.get("myLang");
+```
 
-The generated modules are located in the `rootDir/build/grammar-modules` directory. This is where you can find the output of the module generation process.
+**DO NOT FORGET** to close the language :
+```java
+// this closes the underlying library handle
+myLang.close();
+```
 
-To add a new grammar to the project, follow these steps:
+## Examples
+
+For examples, see the [tests](https://github.com/AndroidIDEOfficial/android-tree-sitter/tree/dev/android-tree-sitter/src/test/java/com/itsaky/androidide/treesitter).
+
+## License
 
-1. Begin by adding the grammar source code to the `grammars` directory. To accomplish this, you can add a submodule using the following command:
-```bash
-git submodule add <remote_url> grammars/<language_name>
 ```
-2. The `language_name` should be the simple name of the language, without the `tree-sitter-` prefix. This name is used to generate both the shared library and the Gradle module. For example, if the `language_name` is `abc`:
-    - The module `tree-sitter-abc` will be automatically generated.
-    - The name of the resulting shared library will be `libtree-sitter-abc.so`.
-3. After adding the grammar source, update the `grammars.json` file to include the newly added grammar in the project.
-4. Finally, sync the project to trigger the generation of the module for the newly added grammar.
+android-tree-sitter library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Lesser General Public
+License as published by the Free Software Foundation; either
+version 2.1 of the License, or (at your option) any later version.
+
+android-tree-sitter library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Lesser General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with android-tree-sitter.  If not, see <https://www.gnu.org/licenses/>.
+```
