Skip to content

Commit bc3ed4a

Browse files
authored
Licenses_cpp: Parses NOTICES and DEPS (flutter#172044)
changes: 1) Now NOTICES files can be parsed 1) Now DEP files are parsed to make sure we are scanning cipd dependencies. Previously we just checked git repos. 1) Added a `secondary` directory to `data` that allows us to inject our own licenses. In a perfect world we wouldn't have this. When the dust settles it may not be necessary anymore. After this we are about 60 copyrights shy of having everything. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent fdcb8f1 commit bc3ed4a

File tree

14 files changed

+14922
-9233
lines changed

14 files changed

+14922
-9233
lines changed

engine/src/flutter/ci/licenses_cpp.sh

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,5 +83,10 @@ fi
8383
--licenses_path "$LICENSES_OUTPUT_PATH" \
8484
--v $VERBOSITY
8585

86-
git diff --no-index --ignore-cr-at-eol "$LICENSES_PATH" "$LICENSES_OUTPUT_PATH"
86+
git diff \
87+
--no-index \
88+
--ignore-cr-at-eol \
89+
--ignore-matching-lines="^You may obtain a copy of this library's Source Code Form from:.*" \
90+
"$LICENSES_PATH" \
91+
"$LICENSES_OUTPUT_PATH"
8792
rm "$LICENSES_OUTPUT_PATH"

engine/src/flutter/sky/packages/sky_engine/LICENSE_CPP

Lines changed: 14279 additions & 9196 deletions
Large diffs are not rendered by default.

engine/src/flutter/tools/licenses_cpp/BUILD.gn

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ source_set("licenses") {
88
"src/comments_util.h",
99
"src/data.cc",
1010
"src/data.h",
11+
"src/deps_parser.cc",
12+
"src/deps_parser.h",
1113
"src/filter.cc",
1214
"src/filter.h",
1315
"src/license_checker.cc",
@@ -52,6 +54,7 @@ executable("licenses_cpp_testrunner") {
5254
sources = [
5355
"src/catalog_unittests.cc",
5456
"src/comments_unittests.cc",
57+
"src/deps_parser_unittests.cc",
5558
"src/filter_unittests.cc",
5659
"src/license_checker_unittests.cc",
5760
]

engine/src/flutter/tools/licenses_cpp/data/README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ validate the engine repository. There are 3 main divisions of this data:
66
- include.txt -- A list of all the files that will be checked.
77
- exclude.txt -- A list of all the files that will be excluded.
88
- data/ -- A catalog of all the accepted and known licenses.
9+
- secondary/ -- Secondary licenses to be included.
910

1011
All regex are in the [re2 format](https://github.com/google/re2/wiki/syntax).
1112

@@ -34,3 +35,8 @@ show up in source code or in its own LICENSE file. The format is the following:
3435
regexes.
3536
3) Remaining lines - Matcher regex that will be used to extract the full text
3637
of the accepted license.
38+
39+
## secondary/
40+
41+
This directory structure needs to match the one found in the working directory.
42+
License files here will be added verbatim to the output.

engine/src/flutter/tools/licenses_cpp/data/exclude.txt

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,13 @@
88
.*/tests/.*
99
.*third_party/googletest/.*
1010
^build/.*
11+
^buildtools/.*
1112
^impeller/fixtures/.*
13+
^prebuilts/.*
1214
^shell/platform/fuchsia/dart-pkg/zircon_ffi/lib/zircon_ffi.dart
1315
^sky/packages/sky_engine/LICENSE
1416
^testing/.*
17+
^third_party/android_tools/.*
1518
^third_party/angle/third_party/.*
1619
^third_party/angle/tools/.*
1720
^third_party/angle/util/.*
@@ -20,10 +23,14 @@
2023
^third_party/dart/third_party/binary_size/.*
2124
^third_party/dart/third_party/binaryen/.*
2225
^third_party/dart/third_party/d3/.*
26+
^third_party/dart/third_party/devtools/web/.*\.frag$
27+
^third_party/dart/third_party/devtools/web/.*\.js$
2328
^third_party/dart/third_party/pkg/.*
2429
^third_party/dart/third_party/requirejs/.*
2530
^third_party/depot_tools/.*
2631
^third_party/freetype2/docs/.*
32+
^third_party/google_fonts_for_unit_tests/.*
33+
^third_party/gradle/.*
2734
^third_party/harfbuzz/.*\.py
2835
^third_party/harfbuzz/perf/.*
2936
^third_party/harfbuzz/src/harfbuzz.cc
@@ -32,6 +39,7 @@
3239
^third_party/imgui/.*
3340
^third_party/inja/third_party/amalgamate/.*
3441
^third_party/inja/third_party/include/doctest/.*
42+
^third_party/java/.*
3543
^third_party/libpng/contrib/.*
3644
^third_party/libwebp/.*\.py
3745
^third_party/libwebp/examples/.*

engine/src/flutter/tools/licenses_cpp/data/include.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,5 @@
2525
.*\.txt$
2626
.*\.ucm$
2727
.*\.vert$
28+
^third_party/dart/tools/sdks/dart-sdk/bin/resources/devtools/assets/NOTICES$
29+
^third_party/dart/third_party/devtools/web/assets/NOTICES$
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
Copyright (c) 2009 Florian Loitsch.
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy
4+
of this software and associated documentation files (the "Software"), to deal
5+
in the Software without restriction, including without limitation the rights
6+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
copies of the Software, and to permit persons to whom the Software is
8+
furnished to do so, subject to the following conditions:
9+
10+
The above copyright notice and this permission notice shall be included in all
11+
copies or substantial portions of the Software.
12+
13+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
19+
SOFTWARE.

engine/src/flutter/tools/licenses_cpp/src/data.cc

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,8 @@ const char* kExcludeFilename = "exclude.txt";
1212
namespace fs = std::filesystem;
1313

1414
absl::StatusOr<Data> Data::Open(std::string_view data_dir) {
15-
fs::path include_path = fs::path(data_dir) / kIncludeFilename;
15+
fs::path data_path = fs::path(data_dir);
16+
fs::path include_path = data_path / kIncludeFilename;
1617
absl::StatusOr<Filter> include_filter = Filter::Open(include_path.string());
1718
if (!include_filter.ok()) {
1819
return absl::InvalidArgumentError("Can't open include.txt at " +
@@ -33,7 +34,11 @@ absl::StatusOr<Data> Data::Open(std::string_view data_dir) {
3334
catalog.status().ToString());
3435
}
3536

36-
return Data{.include_filter = std::move(*include_filter),
37-
.exclude_filter = std::move(*exclude_filter),
38-
.catalog = std::move(*catalog)};
37+
fs::path secondary_dir = data_path / "secondary";
38+
39+
return Data{
40+
.include_filter = std::move(*include_filter),
41+
.exclude_filter = std::move(*exclude_filter),
42+
.catalog = std::move(*catalog),
43+
.secondary_dir = fs::exists(secondary_dir) ? secondary_dir : fs::path()};
3944
}

engine/src/flutter/tools/licenses_cpp/src/data.h

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
#ifndef FLUTTER_TOOLS_LICENSES_CPP_SRC_DATA_H_
66
#define FLUTTER_TOOLS_LICENSES_CPP_SRC_DATA_H_
77

8+
#include <filesystem>
9+
810
#include "flutter/third_party/abseil-cpp/absl/status/statusor.h"
911
#include "flutter/tools/licenses_cpp/src/catalog.h"
1012
#include "flutter/tools/licenses_cpp/src/filter.h"
@@ -17,6 +19,7 @@ struct Data {
1719
Filter include_filter;
1820
Filter exclude_filter;
1921
Catalog catalog;
22+
std::filesystem::path secondary_dir;
2023
};
2124

2225
#endif // FLUTTER_TOOLS_LICENSES_CPP_SRC_DATA_H_
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
// Copyright 2013 The Flutter Authors. All rights reserved.
2+
// Use of this source code is governed by a BSD-style license that can be
3+
// found in the LICENSE file.
4+
5+
#include "flutter/tools/licenses_cpp/src/deps_parser.h"
6+
7+
#include <re2/re2.h>
8+
9+
#include <string>
10+
#include <string_view>
11+
#include <vector>
12+
13+
#include "flutter/third_party/re2/re2/re2.h"
14+
15+
DepsParser::DepsParser() = default;
16+
17+
DepsParser::~DepsParser() = default;
18+
19+
// TODO(gaaclarke): Convert this to flex/bison, this is getting a bit
20+
// complicated. It technically doesn't handle commas in trailing comments.
21+
std::vector<std::string> DepsParser::Parse(std::string_view input) {
22+
std::vector<std::string> result;
23+
re2::StringPiece mutable_input(input);
24+
25+
// 1. Find the start of the deps block
26+
if (!re2::RE2::FindAndConsume(&mutable_input, R"((?:^|\n)deps\s*=\s*\{)")) {
27+
return result;
28+
}
29+
30+
// 2. Find the matching closing brace for the entire deps block
31+
size_t brace_close_pos = std::string_view::npos;
32+
int brace_count = 1;
33+
for (size_t i = 0; i < mutable_input.length(); ++i) {
34+
if (mutable_input[i] == '{') {
35+
brace_count++;
36+
} else if (mutable_input[i] == '}') {
37+
brace_count--;
38+
if (brace_count == 0) {
39+
brace_close_pos = i;
40+
break;
41+
}
42+
}
43+
}
44+
45+
if (brace_close_pos == std::string_view::npos) {
46+
return result;
47+
}
48+
49+
// 3. Extract the content of the deps block
50+
std::string_view deps_content = mutable_input.substr(0, brace_close_pos);
51+
52+
// 4. Filter out comments and build a new string
53+
std::string cleaned_content;
54+
size_t current_pos_clean = 0;
55+
while (current_pos_clean < deps_content.length()) {
56+
size_t next_newline = deps_content.find('\n', current_pos_clean);
57+
if (next_newline == std::string_view::npos) {
58+
next_newline = deps_content.length();
59+
}
60+
61+
std::string_view line = deps_content.substr(
62+
current_pos_clean, next_newline - current_pos_clean);
63+
64+
size_t first_char = line.find_first_not_of(" \t");
65+
if (first_char != std::string_view::npos) {
66+
line = line.substr(first_char);
67+
} else {
68+
line = "";
69+
}
70+
71+
if (line.empty() || line[0] != '#') {
72+
cleaned_content.append(line);
73+
cleaned_content.append("\n");
74+
}
75+
76+
current_pos_clean = next_newline + 1;
77+
}
78+
79+
// 5. Parse the cleaned content
80+
size_t current_pos = 0;
81+
while (current_pos < cleaned_content.length()) {
82+
// Find the start of the key
83+
size_t key_start = cleaned_content.find('\'', current_pos);
84+
if (key_start == std::string_view::npos) {
85+
break;
86+
}
87+
key_start++;
88+
89+
// Find the end of the key
90+
size_t key_end = cleaned_content.find('\'', key_start);
91+
if (key_end == std::string_view::npos) {
92+
break;
93+
}
94+
95+
std::string_view key =
96+
std::string_view(&cleaned_content[key_start], key_end - key_start);
97+
98+
// Find the colon after the key
99+
size_t colon_pos = cleaned_content.find(':', key_end);
100+
if (colon_pos == std::string_view::npos) {
101+
break;
102+
}
103+
104+
// Find the start of the value
105+
size_t value_start =
106+
cleaned_content.find_first_not_of(" \t\n", colon_pos + 1);
107+
if (value_start == std::string_view::npos) {
108+
break;
109+
}
110+
111+
// Find the end of the value
112+
size_t value_end = std::string_view::npos;
113+
int brace_level = 0;
114+
int bracket_level = 0;
115+
for (size_t i = value_start; i < cleaned_content.length(); ++i) {
116+
char c = cleaned_content[i];
117+
if (c == '{') {
118+
brace_level++;
119+
} else if (c == '}') {
120+
brace_level--;
121+
} else if (c == '[') {
122+
bracket_level++;
123+
} else if (c == ']') {
124+
bracket_level--;
125+
} else if (c == ',' && brace_level == 0 && bracket_level == 0) {
126+
value_end = i;
127+
break;
128+
}
129+
}
130+
131+
if (value_end == std::string_view::npos) {
132+
value_end = cleaned_content.length();
133+
}
134+
135+
std::string_view value = std::string_view(&cleaned_content[value_start],
136+
value_end - value_start);
137+
138+
// Check for 'dep_type': 'cipd'
139+
if (value.find("'dep_type': 'cipd'") != std::string_view::npos) {
140+
result.emplace_back(key);
141+
}
142+
143+
current_pos = value_end + 1;
144+
}
145+
146+
return result;
147+
}

0 commit comments

Comments
 (0)