Skip to content

Commit 69ac47f

Browse files
meltsufinsuztomo
andauthored
feat(java): migrate Java container from sdk-platform-java (#2670)
This PR moves Java Librarian container implementation WIP from `github.com/googleapis/sdk-platform-java/internal/librariangen` to `github.com/googleapis/librarian/internal/container/java` while preserving the git history. Here are the steps that were followed: ``` # 1. Clone the source repository git clone [email protected]:googleapis/sdk-platform-java.git cd sdk-platform-java # 2. Filter the repository to keep only the 'internal/librariangen' subdirectory git filter-repo --subdirectory-filter internal/librariangen/ --force # 3. Go back to the parent directory cd .. # 4. Clone the destination repository git clone [email protected]:googleapis/librarian.git cd librarian # 5. Create a new branch for the migration git checkout -b feat/migrate-librariangen-java main # 6. Add the filtered repository as a remote git remote add sdk-platform-java-librariangen ../sdk-platform-java git fetch sdk-platform-java-librariangen # 7. Merge the histories. This command creates a merge commit but keeps the content of the 'librarian' repository. git merge --allow-unrelated-histories --no-ff sdk-platform-java-librariangen/main -s ours --no-commit # 8. Read the tree from the filtered repository into the desired subdirectory git read-tree --prefix=internal/container/java/ -u sdk-platform-java-librariangen/main # 9. Commit the changes git commit -m "feat(internal/container/java): migrate librariangen from sdk-platform-java" ``` Since we are still in early stages of implementing the Java container, the plan is to squash-and-merge this PR. original commits: ``` commit 0a1bbea44d6a1b024c569db89fef8d08ad7bd2d2 Author: Tomo Suzuki <[email protected]> Date: Thu Oct 23 23:58:54 2025 -0400 chore(librariangen): Generate to use languagecontainer.Run (#3968) commit 452d703b703ab3222fd1a7060ed5e1ac6363322b Author: Mike Eltsufin <[email protected]> Date: Thu Oct 23 17:08:42 2025 -0400 feat(librariangen): generate grpc stubs and resource helpers (#3967) * Introduces the generation of gRPC stubs and resource helpers. * Modifies the `Generate` function to create output directories for GAPIC, gRPC, and proto files. * Updates the `invokeProtoc` function to pass an `OutputConfig` struct. * Updates the `restructureOutput` function to handle gRPC stubs and resource names. * Adds a `copyAndMerge` function to merge resource name files into the proto destination. * Updates the `cleanupIntermediateFiles` function to remove the GAPIC, gRPC, and proto directories. * Updates tests to reflect these changes. * Improvements in error handling. commit a26a6d937e0523b9a6c193504645f96649e48573 Author: Tomo Suzuki <[email protected]> Date: Thu Oct 23 10:15:24 2025 -0400 chore(librariangen): languagecontainer package to parse release-init request (#3965) Introducing languagecontainer package. * **New languagecontainer package**: A new `languagecontainer` package has been introduced to encapsulate language-specific container operations and command execution logic, promoting modularity. This package parses the request JSON file and calls the corresponding implementation method in each language container. The languagecontainer package itself should not have language-specific implementation. * **release-init command parsing**: The `languagecontainer.Run` function now includes robust logic to parse `release-init` requests from JSON files, handle command-line flags, and invoke the `ReleaseInit` function. * Why isn't this providing an interface? It's because if `LanguageContainer` is an interface, then there would be package name conflict of `languagecontainer/release` package (language agostic) and `release` package (language-specific. In this case Java-specific). [Here is a piece of code](#2516 (comment)). * **Main command dispatch refactor**: The `librariangen` `main.go` has been refactored to delegate non-`generate` commands to the new `languagecontainer.Run` function, centralizing command execution and wiring up `release.Init` for the `release-init` command. * **message.Change struct update**: The `SourceCommitHash` field in the `message.Change` struct has been renamed to `CommitHash` for improved clarity and consistency. This is due to the recent renaming of the field and [google-cloud-go/internal/librariangen/request.Change](https://github.com/googleapis/google-cloud-go/blob/7a85df39319e3a4870d4ad413f714ae5edd78ac8/internal/librariangen/request/request.go#L60) already has the field. The user (Java's language container in this case) doesn't have to implement the parsing logic and the tests. I moved the argument parsing tests to languagecontainer/languagecontainer_test.go from main_test.go. commit f22935d55d811acfa6903f9778d02b94aed34d13 Author: Tomo Suzuki <[email protected]> Date: Mon Oct 20 21:57:59 2025 -0400 chore(librariangen): introduce message package and add ReleaseInitRequest (#3958) commit 2f6c75da3021d030c7a192f1fbb4b30908ef9dad Author: Mike Eltsufin <[email protected]> Date: Fri Oct 17 11:29:31 2025 -0400 feat(librariangen): add generate package (#3952) Based on https://github.com/googleapis/google-cloud-go/tree/main/internal/librariangen/generate with adaptation for Java. Currently it's just the scaffolding and more work is needed to generate a usable GAPIC library. The `generate` package contains the core logic for the generation process, including: - Reading and parsing the `generate-request.json` from librarian. - Parsing `BUILD.bazel` files in the googleapis repository to extract GAPIC configuration. - Building and executing `protoc` with the `gapic-generator-java` plugin. - Unzipping and restructuring the generated files into the final library layout. A `run-generate-library.sh` script is included for local development and end-to-end testing of the generation process. Additionally, a `go.work` file has been added to the root of the repository to support the multi-module workspace structure. commit feabef32c4c45be0fb1db3615568365b902ece24 Author: Mike Eltsufin <[email protected]> Date: Thu Oct 16 10:25:53 2025 -0400 feat(librariangen): add bazel package (#3940) Based on https://github.com/googleapis/google-cloud-go/tree/main/internal/librariangen/bazel with adaptation for Java. commit 598de0697957ea3b05b1347c0898108648c3d7d1 Author: Mike Eltsufin <[email protected]> Date: Mon Oct 13 13:37:41 2025 -0400 feat(librariangen): add protoc package (#3935) This will be used for constructing the protoc command for generating Java GAPICs. Based on https://github.com/googleapis/google-cloud-go/tree/main/internal/librariangen/protoc with the following important changes: * The `Build` function in `protoc.go` is updated to construct the `protoc` command with the correct arguments for Java GAPIC generation. * The tests in `protoc_test.go` are updated to reflect the changes in `protoc.go`. * The `gapicImportPath` is removed from the test configuration, as it is not relevant for Java.' * The testdata is included with modifications for Java. commit f32325e1d04a804e881b08091704a81c29461eb7 Author: Mike Eltsufin <[email protected]> Date: Mon Oct 13 10:46:41 2025 -0400 test(librariangen): add codecov and improve main.go coverage (#3936) Refactors the main function to be more testable by extracting the logic into a `runCLI` function. The logging setup is also extracted into a `parseLogLevel` function to allow for easier testing. This change increases the test coverage of main.go from 65.5% to 96.9%. commit fe44aede2aba9627db4aa0397d2ac4de353cea03 Author: Mike Eltsufin <[email protected]> Date: Fri Oct 10 20:01:26 2025 -0400 feat(librariangen): add request package (#3933) This will be used for parsing Librarian CLI requests. Copied from https://github.com/googleapis/google-cloud-go/tree/main/internal/librariangen/request. commit f6b0b47a14eddd9c76540362b5922a5acc56b5d4 Author: Mike Eltsufin <[email protected]> Date: Fri Oct 10 20:01:07 2025 -0400 feat(librariangen): add execv package (#3932) This will be used for executing commands like protoc. Copied from https://github.com/googleapis/google-cloud-go/tree/main/internal/librariangen/execv. commit 4c5a1c417a7033deb82039239728309b35e3e70b Author: Tomo Suzuki <[email protected]> Date: Thu Oct 9 16:05:42 2025 -0400 build: AR exit gate YAML to declare dependencies.gitSource (#3929) This should suppress the warning in b/450542318. commit 8f7ef85481fd9a926abcabb6de6451cd5f5faeee Author: Tomo Suzuki <[email protected]> Date: Wed Oct 8 17:18:00 2025 -0400 build: prepare Cloud Build YAML file for librarian-java (#3928) commit 29d188d20e530570f46774b37d88703822774a33 Author: Mike Eltsufin <[email protected]> Date: Wed Oct 8 13:32:54 2025 -0400 feat(librariangen): scaffold Java language container for Librarian (#3926) This change introduces the initial scaffolding for `librariangen`, a containerized Go application that will serve as the Java-specific code generator within the Librarian pipeline. The primary goal of this change is to establish the foundational components for the Java language container, including: * A new `librariangen` CLI application: Written in Go, this will be the entry point for Java code generation tasks invoked by Librarian. It currently includes basic command dispatching and support for `--version`. * A multi-stage `Dockerfile`: This creates a minimal container image for librariangen. * Unit tests and CI: A new GitHub Actions workflow has been added to build and test the librariangen executable. This scaffolding is the first step in implementing the full code generation logic within the Librarian system, eventually replacing the Java hermetic code generation system. The code is largely adapted from the Go language container at https://github.com/googleapis/google-cloud-go/tree/main/internal/librariangen. Fixes: #2500 ``` --------- Co-authored-by: Tomo Suzuki <[email protected]>
1 parent 50046f5 commit 69ac47f

36 files changed

+3295
-2
lines changed

all_test.go

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ import (
3333

3434
var noHeaderRequiredFiles = []string{
3535
".gcloudignore",
36-
".github/CODEOWNERS",
36+
"CODEOWNERS",
3737
".gitignore",
3838
"Dockerfile",
3939
"LICENSE",
@@ -51,6 +51,7 @@ var ignoredExts = map[string]bool{
5151
".yaml": true,
5252
".txt": true,
5353
".webp": true,
54+
".sh": true,
5455
}
5556

5657
var ignoredDirs = []string{
@@ -84,7 +85,7 @@ func TestHeaders(t *testing.T) {
8485
}
8586
return nil
8687
}
87-
if slices.Contains(noHeaderRequiredFiles, path) || ignoredExts[filepath.Ext(path)] {
88+
if slices.Contains(noHeaderRequiredFiles, filepath.Base(path)) || ignoredExts[filepath.Ext(path)] {
8889
return nil
8990
}
9091

internal/container/java/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
workspace/

internal/container/java/Dockerfile

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Copyright 2025 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# This Dockerfile creates a MOSS-compliant image for the Go librariangen,
16+
# which is invoked by the Librarian tool. It uses a multi-stage build to
17+
# create a minimal final image.
18+
19+
# --- Builder Stage ---
20+
# This stage builds the librariangen binary using the MOSS-compliant base image.
21+
FROM marketplace.gcr.io/google/debian12:latest AS builder
22+
23+
# Set environment variables for tool versions for easy updates.
24+
ENV GO_VERSION=1.24.0
25+
26+
# Install build dependencies.
27+
RUN apt-get update && \
28+
apt-get install -y \
29+
build-essential \
30+
ca-certificates \
31+
curl \
32+
wget && \
33+
rm -rf /var/lib/apt/lists/*
34+
35+
# Install the specific Go version required for compatibility.
36+
RUN wget https://go.dev/dl/go${GO_VERSION}.linux-amd64.tar.gz -O go.tar.gz && \
37+
tar -C /usr/local -xzf go.tar.gz && \
38+
rm go.tar.gz
39+
ENV PATH=/usr/local/go/bin:$PATH
40+
RUN go version
41+
42+
WORKDIR /src
43+
44+
# Copy go.mod and go.sum to download dependencies before copying all the source.
45+
# This allows Docker to cache the dependencies layer.
46+
# COPY go.mod go.sum ./
47+
# RUN go mod download
48+
49+
# Copy all source code.
50+
COPY . .
51+
52+
# Build the librariangen binary.
53+
RUN CGO_ENABLED=0 GOOS=linux go build -v -o /librariangen .
54+
55+
# --- Final Stage ---
56+
# This stage creates the final, minimal image with the compiled binary and
57+
# all required runtime dependencies pinned to specific versions for compatibility.
58+
FROM marketplace.gcr.io/google/debian12:latest
59+
60+
# Copy the compiled librariangen binary from the builder stage.
61+
COPY --from=builder /librariangen /usr/local/bin/librariangen
62+
63+
# Set the entrypoint for the container to run the compiled librariangen.
64+
# The Librarian will provide commands like 'generate' as arguments.
65+
ENTRYPOINT [ "/usr/local/bin/librariangen" ]

internal/container/java/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Java GAPIC Generator for Librarian (librariangen)
2+
3+
This directory contains the source code for `librariangen`, a containerized Go application that serves as the Java-specific code generator within the Librarian pipeline. Its responsibility is to generate release-ready Java GAPIC client libraries from `googleapis` API definitions, replacing the legacy Java hermetic code generation toolchain.
Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
// Copyright 2025 Google LLC
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package bazel
16+
17+
import (
18+
"errors"
19+
"fmt"
20+
"log/slog"
21+
"os"
22+
"path/filepath"
23+
"regexp"
24+
"strconv"
25+
"strings"
26+
"sync"
27+
)
28+
29+
// Config holds configuration extracted from a googleapis BUILD.bazel file.
30+
type Config struct {
31+
gapicYAML string
32+
grpcServiceConfig string
33+
restNumericEnums bool
34+
serviceYAML string
35+
transport string
36+
hasGAPIC bool
37+
}
38+
39+
// HasGAPIC indicates whether the GAPIC generator should be run.
40+
func (c *Config) HasGAPIC() bool { return c.hasGAPIC }
41+
42+
// GapicYAML is the GAPIC config file in the API version directory in googleapis.
43+
func (c *Config) GapicYAML() string { return c.gapicYAML }
44+
45+
// ServiceYAML is the client config file in the API version directory in googleapis.
46+
func (c *Config) ServiceYAML() string { return c.serviceYAML }
47+
48+
// GRPCServiceConfig is name of the gRPC service config JSON file.
49+
func (c *Config) GRPCServiceConfig() string { return c.grpcServiceConfig }
50+
51+
// Transport is typically one of "grpc", "rest" or "grpc+rest".
52+
func (c *Config) Transport() string { return c.transport }
53+
54+
// HasRESTNumericEnums indicates whether the generated client should support numeric enums.
55+
func (c *Config) HasRESTNumericEnums() bool { return c.restNumericEnums }
56+
57+
// Validate ensures that the configuration is valid.
58+
func (c *Config) Validate() error {
59+
if c.hasGAPIC {
60+
if c.serviceYAML == "" {
61+
return errors.New("librariangen: serviceYAML is not set")
62+
}
63+
}
64+
return nil
65+
}
66+
67+
var javaGapicLibraryRE = regexp.MustCompile(`java_gapic_library\((?s:.)*?\)`)
68+
69+
// Parse reads a BUILD.bazel file from the given directory and extracts the
70+
// relevant configuration from the java_gapic_library rule.
71+
func Parse(dir string) (*Config, error) {
72+
c := &Config{}
73+
fp := filepath.Join(dir, "BUILD.bazel")
74+
data, err := os.ReadFile(fp)
75+
if err != nil {
76+
return nil, fmt.Errorf("librariangen: failed to read BUILD.bazel file %s: %w", fp, err)
77+
}
78+
content := string(data)
79+
80+
gapicLibraryBlock := javaGapicLibraryRE.FindString(content)
81+
if gapicLibraryBlock != "" {
82+
c.hasGAPIC = true
83+
c.grpcServiceConfig = findString(gapicLibraryBlock, "grpc_service_config")
84+
c.serviceYAML = strings.TrimPrefix(findString(gapicLibraryBlock, "service_yaml"), ":")
85+
c.transport = findString(gapicLibraryBlock, "transport")
86+
if c.restNumericEnums, err = findBool(gapicLibraryBlock, "rest_numeric_enums"); err != nil {
87+
return nil, fmt.Errorf("librariangen: failed to parse BUILD.bazel file %s: %w", fp, err)
88+
}
89+
c.gapicYAML = strings.TrimPrefix(findString(gapicLibraryBlock, "gapic_yaml"), ":")
90+
}
91+
if err := c.Validate(); err != nil {
92+
return nil, fmt.Errorf("librariangen: invalid bazel config in %s: %w", dir, err)
93+
}
94+
slog.Debug("librariangen: bazel config loaded", "conf", c)
95+
return c, nil
96+
}
97+
98+
var reCache = &sync.Map{}
99+
100+
func getRegexp(key, pattern string) *regexp.Regexp {
101+
val, ok := reCache.Load(key)
102+
if !ok {
103+
val = regexp.MustCompile(pattern)
104+
reCache.Store(key, val)
105+
}
106+
return val.(*regexp.Regexp)
107+
}
108+
109+
func findString(content, name string) string {
110+
re := getRegexp("findString_"+name, fmt.Sprintf(`%s\s*=\s*(?:"([^"]+)"|'([^']+)'){1}`, name))
111+
match := re.FindStringSubmatch(content)
112+
if len(match) > 2 {
113+
if match[1] != "" {
114+
return match[1] // Double-quoted
115+
}
116+
return match[2] // Single-quoted
117+
}
118+
slog.Debug("librariangen: failed to find string attr in BUILD.bazel", "name", name)
119+
return ""
120+
}
121+
122+
func findBool(content, name string) (bool, error) {
123+
re := getRegexp("findBool_"+name, fmt.Sprintf(`%s\s*=\s*(\w+)`, name))
124+
if match := re.FindStringSubmatch(content); len(match) > 1 {
125+
if b, err := strconv.ParseBool(match[1]); err == nil {
126+
return b, nil
127+
}
128+
return false, fmt.Errorf("librariangen: failed to parse bool attr in BUILD.bazel: %q, got: %q", name, match[1])
129+
}
130+
slog.Debug("librariangen: failed to find bool attr in BUILD.bazel", "name", name)
131+
return false, nil
132+
}

0 commit comments

Comments
 (0)