Skip to content

Commit b664703

Browse files
shhdgitshhdgit
authored
tweak: set default plan base on the TOC (#637)
* feat: implement TOC-based file filtering for documentation generation - Added functions to extract file paths from TOC navigation and filter documentation files based on TOC content. - Integrated TOC filtering into the createDocs function to optimize the build process by only including relevant files. - Enhanced logging to provide insights on the number of files processed and filtered during the build. * feat: implement TiDB Cloud TOC file handling and default plan determination - Added new functions to retrieve and process TOC files specific to TiDB Cloud, categorizing them into dedicated, starter, and essential types. - Integrated TOC file handling into the documentation generation process, optimizing the build by filtering relevant files. - Enhanced the useCloudPlan and DocTemplate components to support default plan determination based on TOC presence, improving user experience for TiDB Cloud articles. - Introduced logging for better visibility into the number of files processed and their categorization. * feat: enhance TOC processing and path generation for TiDB Cloud documentation - Updated TOC handling to include prefix support for file paths, allowing for better organization of documentation based on TOC structure. - Refactored path generation logic to accommodate optional prefixes, improving URL construction for various documentation types. - Enhanced file extraction functions to utilize extended folder information, ensuring accurate file paths are generated during the build process. - Improved logging for skipped nodes during filtering, providing better insights into the documentation generation process. * refactor: clean up unused interface in create-pages and enhance cloud mode session handling - Removed the unused TocQueryData interface from create-pages.ts to streamline the codebase. - Added logic in useCloudPlanNavigate to store cloud mode in session storage when it is set, improving state management for user navigation. * chore: test * chore: update production workflow and enhance TOC processing - Changed the GitHub Actions workflow reference from "tweak-plan-select" to "master" for production deployment. - Modified the `mdxAstToToc` function to include an optional parameter for filtering whitelist content, improving TOC processing. - Renamed `WHITE_LIST` to `WHITELIST` for consistency in naming conventions. - Added logic to filter out nodes based on a specific heading in the TOC, enhancing content management during documentation generation. * fix: rename SKIP_MODE_HEADING constant for consistency - Updated the SKIP_MODE_HEADING constant from "_BUILD_WHITELIST" to "_BUILD_ALLOWLIST" to align with current naming conventions and improve clarity in the codebase. * feat: improve TOC filtering logic for file inclusion - Enhanced the filtering mechanism in the `filterNodesByToc` function to first check for exact matches and then allow for prefix matches, improving the accuracy of file inclusion based on the TOC structure. - This change optimizes the documentation generation process by ensuring relevant files are included even when only a prefix match is available. * refactor: enhance TOC filtering logic to support anchor-less file matching - Updated the `filterNodesByToc` function to improve the inclusion logic by checking for direct file references in the TOC and allowing matches without anchor parts. This change enhances the accuracy of file filtering during documentation generation. * fix: enhance TOC filtering to skip additional headings - Updated the `filterWhitelistContent` function to allow skipping headings that start with an underscore, improving the filtering logic for whitelist content in the TOC.
1 parent 25d188c commit b664703

File tree

9 files changed

+494
-74
lines changed

9 files changed

+494
-74
lines changed

‎gatsby/cloud-plan.ts‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
import { mdxAstToToc, TocQueryData } from "./toc";
2+
import { generateConfig } from "./path";
3+
import { extractFilesFromToc } from "./toc-filter";
4+
5+
/**
6+
* Get files from different TOC types for tidbcloud
7+
* Returns a Map where key is "locale/repo/version" and value is object with dedicated, starter, essential file sets
8+
*/
9+
export async function getTidbCloudFilesFromTocs(
10+
graphql: any
11+
): Promise<
12+
Map<
13+
string,
14+
{ dedicated: Set<string>; starter: Set<string>; essential: Set<string> }
15+
>
16+
> {
17+
const tocQuery = await graphql(`
18+
{
19+
allMdx(
20+
filter: { fileAbsolutePath: { regex: "/tidbcloud/.*TOC.*md$/" } }
21+
) {
22+
nodes {
23+
id
24+
slug
25+
mdxAST
26+
parent {
27+
... on File {
28+
relativePath
29+
}
30+
}
31+
}
32+
}
33+
}
34+
`);
35+
36+
if (tocQuery.errors) {
37+
console.error(tocQuery.errors);
38+
}
39+
40+
const tocNodes = tocQuery.data!.allMdx.nodes;
41+
const tidbCloudTocFilesMap = new Map<
42+
string,
43+
{ dedicated: Set<string>; starter: Set<string>; essential: Set<string> }
44+
>();
45+
46+
tocNodes.forEach((node: TocQueryData["allMdx"]["nodes"][0]) => {
47+
const { config } = generateConfig(node.slug);
48+
const toc = mdxAstToToc(node.mdxAST.children, config);
49+
const files = extractFilesFromToc(toc);
50+
51+
// Create a key for this specific locale/repo/version combination
52+
const key = `${config.locale}/${config.repo}/${
53+
config.version || config.branch
54+
}`;
55+
56+
// Determine TOC type based on filename
57+
const relativePath = node.parent.relativePath;
58+
let tocType: "dedicated" | "starter" | "essential" = "dedicated";
59+
60+
if (relativePath.includes("TOC-tidb-cloud-starter")) {
61+
tocType = "starter";
62+
} else if (relativePath.includes("TOC-tidb-cloud-essential")) {
63+
tocType = "essential";
64+
}
65+
66+
// Initialize the entry if it doesn't exist
67+
if (!tidbCloudTocFilesMap.has(key)) {
68+
tidbCloudTocFilesMap.set(key, {
69+
dedicated: new Set(),
70+
starter: new Set(),
71+
essential: new Set(),
72+
});
73+
}
74+
75+
// Add files to the appropriate TOC type
76+
const entry = tidbCloudTocFilesMap.get(key)!;
77+
entry[tocType] = new Set(files);
78+
79+
console.info(`TOC ${key} (${tocType}): found ${files.length} files`);
80+
});
81+
82+
return tidbCloudTocFilesMap;
83+
}
84+
85+
/**
86+
* Determine the inDefaultPlan value for a tidbcloud article based on TOC presence
87+
*/
88+
export function determineInDefaultPlan(
89+
fileName: string,
90+
pathConfig: any,
91+
tidbCloudTocFilesMap: Map<
92+
string,
93+
{ dedicated: Set<string>; starter: Set<string>; essential: Set<string> }
94+
>
95+
): string | null {
96+
// Only apply this logic for tidbcloud articles
97+
if (pathConfig.repo !== "tidbcloud") {
98+
return null;
99+
}
100+
101+
const key = `${pathConfig.locale}/${pathConfig.repo}/${
102+
pathConfig.version || pathConfig.branch
103+
}`;
104+
105+
const tocData = tidbCloudTocFilesMap.get(key);
106+
if (!tocData) {
107+
return null;
108+
}
109+
110+
const { dedicated, starter, essential } = tocData;
111+
112+
// Check if article is in TOC.md (dedicated)
113+
if (dedicated.has(fileName)) {
114+
return "dedicated";
115+
}
116+
117+
// Check if article is in TOC-tidb-cloud-starter.md but not in TOC.md
118+
if (starter.has(fileName) && !dedicated.has(fileName)) {
119+
return "starter";
120+
}
121+
122+
// Check if article is only in TOC-tidb-cloud-essential.md
123+
if (
124+
essential.has(fileName) &&
125+
!dedicated.has(fileName) &&
126+
!starter.has(fileName)
127+
) {
128+
return "essential";
129+
}
130+
131+
return null;
132+
}

‎gatsby/create-pages.ts‎

Lines changed: 39 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@ import {
1414
} from "./path";
1515
import { docs as DOCS_CONFIG } from "../docs/docs.json";
1616
import { cpMarkdown } from "./cp-markdown";
17+
import {
18+
getTidbCloudFilesFromTocs,
19+
determineInDefaultPlan,
20+
} from "./cloud-plan";
21+
import { getFilesFromTocs, filterNodesByToc } from "./toc-filter";
1722

1823
interface PageQueryData {
1924
allMdx: {
@@ -35,6 +40,12 @@ export const createDocs = async (createPagesArgs: CreatePagesArgs) => {
3540
// const template = resolve(__dirname, '../src/doc/index.tsx')
3641
const template = resolve(__dirname, "../src/templates/DocTemplate.tsx");
3742

43+
// First, get the list of files that should be built based on TOC content
44+
const tocFilesMap = await getFilesFromTocs(graphql);
45+
46+
// Get tidbcloud specific TOC files for plan determination
47+
const tidbCloudTocFilesMap = await getTidbCloudFilesFromTocs(graphql);
48+
3849
const docs = await graphql<PageQueryData>(`
3950
{
4051
allMdx(
@@ -66,10 +77,25 @@ export const createDocs = async (createPagesArgs: CreatePagesArgs) => {
6677
sig.error(docs.errors);
6778
}
6879

69-
const nodes = docs.data!.allMdx.nodes.map((node) => {
70-
const { config, name, filePath } = generateConfig(node.slug);
71-
return { ...node, pathConfig: config, name, filePath };
72-
});
80+
const nodes = filterNodesByToc(
81+
docs.data!.allMdx.nodes.map((node) => {
82+
const { config, name, filePath } = generateConfig(node.slug);
83+
return { ...node, pathConfig: config, name, filePath };
84+
}),
85+
tocFilesMap
86+
);
87+
88+
sig.info(
89+
`Building ${nodes.length} files after TOC filtering (from ${
90+
docs.data!.allMdx.nodes.length
91+
} total files)`
92+
);
93+
94+
// Log some statistics about the filtering
95+
const filteredByToc = docs.data!.allMdx.nodes.length - nodes.length;
96+
if (filteredByToc > 0) {
97+
sig.info(`Filtered out ${filteredByToc} files not referenced in TOCs`);
98+
}
7399

74100
const versionRecord = nodes.reduce(
75101
(acc, { pathConfig, name }) => {
@@ -112,6 +138,13 @@ export const createDocs = async (createPagesArgs: CreatePagesArgs) => {
112138
)
113139
.filter(Boolean);
114140

141+
// Determine inDefaultPlan for tidbcloud articles
142+
const inDefaultPlan = determineInDefaultPlan(
143+
name,
144+
pathConfig,
145+
tidbCloudTocFilesMap
146+
);
147+
115148
cpMarkdown(`${node.slug}.md`, path, name);
116149
createPage({
117150
path,
@@ -136,12 +169,13 @@ export const createDocs = async (createPagesArgs: CreatePagesArgs) => {
136169
banner: true,
137170
feedback: true,
138171
},
172+
inDefaultPlan,
139173
},
140174
});
141175

142176
// create redirects
143177
if (node.frontmatter.aliases) {
144-
node.frontmatter.aliases.forEach((fromPath) => {
178+
node.frontmatter.aliases.forEach((fromPath: string) => {
145179
createRedirect({
146180
fromPath,
147181
toPath: path,

‎gatsby/create-types.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ export const createExtraType = ({ actions }: CreatePagesArgs) => {
5858
if (!slug.endsWith("TOC"))
5959
throw new Error(`unsupported query in ${slug}`);
6060
const { config } = generateConfig(slug);
61-
const res = mdxAstToToc(mdxAST.children, config);
61+
const res = mdxAstToToc(mdxAST.children, config, undefined, true);
6262
mdxNode.nav = res;
6363
return res;
6464
},
@@ -94,7 +94,7 @@ export const createExtraType = ({ actions }: CreatePagesArgs) => {
9494
if (!slug.endsWith("TOC-tidb-cloud-starter"))
9595
throw new Error(`unsupported query in ${slug}`);
9696
const { config } = generateConfig(slug);
97-
const res = mdxAstToToc(mdxAST.children, config);
97+
const res = mdxAstToToc(mdxAST.children, config, undefined, true);
9898
mdxNode.starterNav = res;
9999
return res;
100100
},
@@ -130,7 +130,7 @@ export const createExtraType = ({ actions }: CreatePagesArgs) => {
130130
if (!slug.endsWith("TOC-tidb-cloud-essential"))
131131
throw new Error(`unsupported query in ${slug}`);
132132
const { config } = generateConfig(slug);
133-
const res = mdxAstToToc(mdxAST.children, config);
133+
const res = mdxAstToToc(mdxAST.children, config, undefined, true);
134134
mdxNode.essentialNav = res;
135135
return res;
136136
},

‎gatsby/path.ts‎

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,14 @@ export function generateUrl(filename: string, config: PathConfig) {
55
const lang = config.locale === Locale.en ? "" : `/${config.locale}`;
66

77
if (filename === "") {
8-
return `${lang}/${config.repo}/${config.version ? config.version : ""}`;
8+
return `${lang}/${config.repo}${
9+
config.version ? `/${config.version}` : ""
10+
}${config.prefix ? `/${config.prefix}` : ""}`;
911
}
1012

11-
return `${lang}/${config.repo}/${
12-
config.version ? config.version + "/" : ""
13-
}${filename}`;
13+
return `${lang}/${config.repo}${config.version ? `/${config.version}` : ""}${
14+
config.prefix ? `/${config.prefix}` : ""
15+
}/${filename}`;
1416
}
1517

1618
export const generateDocHomeUrl = (filename: string, config: PathConfig) => {
@@ -54,21 +56,28 @@ export function generateConfig(slug: string): {
5456

5557
let name = rest[rest.length - 1];
5658
name = name === "_index" ? "" : name;
59+
let prefix = undefined;
5760

5861
if (repo === Repo.tidbcloud) {
59-
if (filePath.includes("starter/")) {
60-
name = `starter/${name}`;
61-
} else if (filePath.includes("essential/")) {
62-
name = `essential/${name}`;
63-
} else if (filePath.includes("dedicated/")) {
62+
if (slug.includes("starter/")) {
63+
prefix = "starter";
64+
} else if (slug.includes("essential/")) {
65+
prefix = "essential";
66+
} else if (slug.includes("dedicated/")) {
6467
if (!!name) {
65-
name = `dedicated/${name}`;
68+
prefix = "dedicated";
6669
}
6770
}
6871
}
6972

7073
return {
71-
config: { locale, repo, branch, version: branchToVersion(repo, branch) },
74+
config: {
75+
locale,
76+
repo,
77+
branch,
78+
version: branchToVersion(repo, branch),
79+
prefix,
80+
},
7281
filePath,
7382
name,
7483
};

0 commit comments

Comments
 (0)