Skip to content

Commit 3f26583

Browse files
Mpdreamzreakaleekleemthompo
authored
feature/publish links index lamdba (#668)
* stage * Simplify function * ensure binary is named bootstrap * add lambda docker build * update lambda build * Add `free-disk-space` input (#614) * Add `free-disk-space` input * Use exact commit * Simply if check on Free Disk Space (#616) * Default CurrentUrlPath to empty string while we investigate further (#618) * Fix `free-disk-space` input attempt 2 (#617) * Just compare with repository name for now (#619) * Add update-reference-index action (#622) * Update links-index.json whenever a PR get's closed (#623) * Update links-index.json whenever we push a links.json (#624) * Continue-on-error temporarily for validate-inbound-links (#627) * docs-preview: Use `since_last_remote_commit: true` for changed-files action (#628) * On ci skip if /docs folder has no docset.yml silently (#630) * On ci skip if /docs folder has no docset.yml silently * skip false not in finally * Add log message * Revert "docs-preview: Use `since_last_remote_commit: true` for changed-files …" (#629) This reverts commit ed33817. * Feature: allow the docs-build action to hint to subsequent steps to skip (#631) * Set correct deployment status (#632) * Optimize preview build to avoid checkout on pull_request* events (#633) * Add `PrimaryNav` feature (#604) * Add latest header design and adjust pages navigation * Fix unintended merge changes * fix * Pass markdownparser to directive html renderer * Change feature flag from UpperCamelCase to kebab-case * Update docs/_docset.yml * Refactor feature flags --------- Co-authored-by: Martijn Laarman <[email protected]> * Revert "Add `PrimaryNav` feature (#604)" (#635) This reverts commit 8edd22a. * Add smoke test (#637) * Add smoke test * fix * Run free-disk-space action in security-docs (#639) * Add `primary-nav` feature (#636) * Add latest header design and adjust pages navigation * Fix unintended merge changes * fix * Pass markdownparser to directive html renderer * Change feature flag from UpperCamelCase to kebab-case * Update docs/_docset.yml * Refactor feature flags * Fix case where targetUrl is null or empty * Revert Commands.cs * Don't fail-fast on matrix * Handle the case where there is no root index.md file * Fix * test * ok * fix * Fix --------- Co-authored-by: Martijn Laarman <[email protected]> * Inbound link checking, fully validate if found in links-index (#643) * Add `landing-page-path` output and use it in preview workflow (#642) * Add first-page-path output and use it in preview build workflow * Refactor * Add to test * Fix smoke test * Refactor * Fix * Fix smoke test * Fix output name in usage and add output to action.yml * Refactor smoke-tests * Fix preview-build.yml (#644) * Remove unused workflow (#646) * Add validate command for single published repository (#648) Update command to be rooted to `docs-assembler inbound-links <command>` * Refactor link index command into dedicated LinkRegistry module (#649) Moved the "create-index" command from InboundLinkCommands to a new LinkRegistryCommands class for better modularity and maintainability. Updated program registration to include the new LinkRegistryCommands module and renamed the command to "update" for clarity. * Better error messages for cross_link errors (From `docs-builder` and `docs-assembler`) (#651) * Ensure inbound links validation skips docset.yml declared repositories validation * Add warning that link links to repository not in yet in the registry * Add better errors to docs-builder to (link to links.json file) * Add logging to git remote resolving (#652) * Update cross-repo links syntax info (#654) * If GITHUB_REPOSITORY is set use it verbatim (#655) * Share inbound-links commands with `docs-builder` (#657) * Fix `environment_url` in preview deployment (#658) * Adjust links and wording in primary and secondary navigation (#659) * Add mobile navigations (#662) * Add mobile navigations * Remove landing-page feature toggle * Add extension hook points to all special usecases to inject themselves into docs build process (#666) * Move detection-rules support behind an extension system * Move EnabledExtensions to Configuration * add untracked files * Add documentation for extensions * Add more extensions points to isolate the DetectionRule extension further * Isolate file scanning in DocumentationSet * Do a ful parse * Rename binary to follow kebab-case convention * Move lambda.DockerFile * Update lambda.DockerFile * Update README.md for publish-links-index-lambda with build instructions * Update src/docs-lambda-index-publisher/lambda.DockerFile Co-authored-by: Jan Calanog <[email protected]> --------- Co-authored-by: Jan Calanog <[email protected]> Co-authored-by: Liam Thompson <[email protected]>
1 parent 21738a8 commit 3f26583

File tree

8 files changed

+194
-2
lines changed

8 files changed

+194
-2
lines changed

‎Directory.Build.props

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,4 +31,4 @@
3131
<ItemGroup>
3232
<PackageReference Include="MinVer" Version="6.0.0" PrivateAssets="all" />
3333
</ItemGroup>
34-
</Project>
34+
</Project>

‎docs-builder.sln

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "update-reference-index", "u
7171
actions\update-reference-index\action.yml = actions\update-reference-index\action.yml
7272
EndProjectSection
7373
EndProject
74+
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "docs-lambda-index-publisher", "src\docs-lambda-index-publisher\docs-lambda-index-publisher.csproj", "{C559D52D-100B-4B2B-BE87-2344D835761D}"
75+
EndProject
7476
Global
7577
GlobalSection(SolutionConfigurationPlatforms) = preSolution
7678
Debug|Any CPU = Debug|Any CPU
@@ -117,6 +119,10 @@ Global
117119
{4CCE599A-B9FE-4DF2-8763-34CF0A99D4AA}.Debug|Any CPU.Build.0 = Debug|Any CPU
118120
{4CCE599A-B9FE-4DF2-8763-34CF0A99D4AA}.Release|Any CPU.ActiveCfg = Release|Any CPU
119121
{4CCE599A-B9FE-4DF2-8763-34CF0A99D4AA}.Release|Any CPU.Build.0 = Release|Any CPU
122+
{C559D52D-100B-4B2B-BE87-2344D835761D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
123+
{C559D52D-100B-4B2B-BE87-2344D835761D}.Debug|Any CPU.Build.0 = Debug|Any CPU
124+
{C559D52D-100B-4B2B-BE87-2344D835761D}.Release|Any CPU.ActiveCfg = Release|Any CPU
125+
{C559D52D-100B-4B2B-BE87-2344D835761D}.Release|Any CPU.Build.0 = Release|Any CPU
120126
EndGlobalSection
121127
GlobalSection(NestedProjects) = preSolution
122128
{4D198E25-C211-41DC-9E84-B15E89BD7048} = {BE6011CC-1200-4957-B01F-FCCA10C5CF5A}
@@ -132,5 +138,6 @@ Global
132138
{6E2ED6CC-AFC1-4E58-965D-6AEC500EBB46} = {245023D2-D3CA-47B9-831D-DAB91A2FFDC7}
133139
{6554F917-73CE-4B3D-9101-F28EAA762C6B} = {245023D2-D3CA-47B9-831D-DAB91A2FFDC7}
134140
{9FEC15F6-13F8-40B1-A66A-EB054E49E680} = {245023D2-D3CA-47B9-831D-DAB91A2FFDC7}
141+
{C559D52D-100B-4B2B-BE87-2344D835761D} = {BE6011CC-1200-4957-B01F-FCCA10C5CF5A}
135142
EndGlobalSection
136143
EndGlobal

‎src/Elastic.Markdown/SourceGenerationContext.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,4 +17,4 @@ namespace Elastic.Markdown;
1717
[JsonSerializable(typeof(GitCheckoutInformation))]
1818
[JsonSerializable(typeof(LinkIndex))]
1919
[JsonSerializable(typeof(LinkIndexEntry))]
20-
internal sealed partial class SourceGenerationContext : JsonSerializerContext;
20+
public sealed partial class SourceGenerationContext : JsonSerializerContext;

‎src/docs-lambda-index-publisher/Program.cs

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
// Licensed to Elasticsearch B.V under one or more agreements.
2+
// Elasticsearch B.V licenses this file to you under the Apache 2.0 License.
3+
// See the LICENSE file in the project root for more information
4+
5+
using System.Diagnostics;
6+
using System.Text;
7+
using Amazon.Lambda.Core;
8+
using Amazon.Lambda.RuntimeSupport;
9+
using Amazon.S3;
10+
using Amazon.S3.Model;
11+
using Elastic.Markdown.CrossLinks;
12+
13+
await LambdaBootstrapBuilder.Create(Handler)
14+
.Build()
15+
.RunAsync();
16+
17+
static async Task<string> Handler(ILambdaContext context)
18+
{
19+
var sw = Stopwatch.StartNew();
20+
IAmazonS3 client = new AmazonS3Client();
21+
var bucketName = "elastic-docs-link-index";
22+
var request = new ListObjectsV2Request
23+
{
24+
BucketName = bucketName,
25+
MaxKeys = 5
26+
};
27+
28+
var linkIndex = new LinkIndex
29+
{
30+
Repositories = []
31+
};
32+
try
33+
{
34+
ListObjectsV2Response response;
35+
do
36+
{
37+
response = await client.ListObjectsV2Async(request, CancellationToken.None);
38+
foreach (var obj in response.S3Objects)
39+
{
40+
if (!obj.Key.StartsWith("elastic/", StringComparison.OrdinalIgnoreCase))
41+
continue;
42+
43+
var tokens = obj.Key.Split('/');
44+
if (tokens.Length < 3)
45+
continue;
46+
47+
var repository = tokens[1];
48+
var branch = tokens[2];
49+
50+
var entry = new LinkIndexEntry
51+
{
52+
Repository = repository,
53+
Branch = branch,
54+
ETag = obj.ETag.Trim('"'),
55+
Path = obj.Key
56+
};
57+
if (linkIndex.Repositories.TryGetValue(repository, out var existingEntry))
58+
existingEntry[branch] = entry;
59+
else
60+
linkIndex.Repositories.Add(repository, new Dictionary<string, LinkIndexEntry>
61+
{
62+
{ branch, entry }
63+
});
64+
Console.WriteLine(entry);
65+
}
66+
67+
// If the response is truncated, set the request ContinuationToken
68+
// from the NextContinuationToken property of the response.
69+
request.ContinuationToken = response.NextContinuationToken;
70+
} while (response.IsTruncated);
71+
}
72+
catch (AmazonS3Exception ex)
73+
{
74+
return $"Error encountered on server. Message:'{ex.Message}' getting list of objects.";
75+
}
76+
77+
var json = LinkIndex.Serialize(linkIndex);
78+
79+
using var stream = new MemoryStream(Encoding.UTF8.GetBytes(json));
80+
await client.UploadObjectFromStreamAsync(bucketName, "link-index.json", stream, new Dictionary<string, object>(), CancellationToken.None);
81+
return $"Finished in {sw}";
82+
}

‎src/docs-lambda-index-publisher/README.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Index Registry Update Lambda Function
2+
3+
From a linux `x86_64` machine you can use the followint to build a AOT binary that will run
4+
5+
on a vanilla `Amazon Linux 2023` without any dependencies.
6+
7+
```bash
8+
docker build . -t publish-links-index:latest -f src/docs-lambda-index-publisher/lambda.DockerFile
9+
```
10+
11+
Then you can copy the published artifacts from the image using:
12+
13+
```bash
14+
docker cp (docker create --name tc publish-links-index:latest):/app/.artifacts/publish ./.artifacts && docker rm tc
15+
```
16+
17+
The `bootstrap` binary should now be available under:
18+
19+
```
20+
.artifacts/publish/docs-lambda-index-publisher/release_linux-x64/bootstrap
21+
```
22+

‎src/docs-lambda-index-publisher/aws-lambda-tools-defaults.json

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"Information": [
3+
"This file provides default values for the deployment wizard inside Visual Studio and the AWS Lambda commands added to the .NET Core CLI.",
4+
"To learn more about the Lambda commands with the .NET Core CLI execute the following command at the command line in the project root directory.",
5+
"dotnet lambda help",
6+
"All the command line options for the Lambda command can be specified in this file."
7+
],
8+
"profile": "",
9+
"region": "",
10+
"configuration": "Release",
11+
"function-runtime": "provided.al2",
12+
"function-memory-size": 512,
13+
"function-timeout": 180,
14+
"function-handler": "Elastic.Documentation.Lambda.LinkIndexUploader"
15+
}

‎src/docs-lambda-index-publisher/docs-lambda-index-publisher.csproj

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
<Project Sdk="Microsoft.NET.Sdk">
2+
<PropertyGroup>
3+
<OutputType>Exe</OutputType>
4+
<TargetFramework>net9.0</TargetFramework>
5+
<ImplicitUsings>enable</ImplicitUsings>
6+
<Nullable>enable</Nullable>
7+
<InvariantGlobalization>true</InvariantGlobalization>
8+
9+
<AssemblyName>bootstrap</AssemblyName>
10+
<AWSProjectType>Lambda</AWSProjectType>
11+
12+
<IsPublishable>true</IsPublishable>
13+
<PublishAot>true</PublishAot>
14+
<PublishTrimmed>true</PublishTrimmed>
15+
<EnableSdkContainerSupport>true</EnableSdkContainerSupport>
16+
<TrimmerSingleWarn>false</TrimmerSingleWarn>
17+
<DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
18+
19+
<RootNamespace>Elastic.Documentation.Lambda.LinkIndexUploader</RootNamespace>
20+
</PropertyGroup>
21+
<ItemGroup>
22+
<PackageReference Include="Amazon.Lambda.RuntimeSupport" Version="1.12.3"/>
23+
<PackageReference Include="Amazon.Lambda.Core" Version="2.5.0"/>
24+
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.4.4"/>
25+
<PackageReference Include="AWSSDK.S3" Version="3.7.414.5"/>
26+
</ItemGroup>
27+
<ItemGroup>
28+
<ProjectReference Include="..\Elastic.Markdown\Elastic.Markdown.csproj"/>
29+
</ItemGroup>
30+
</Project>
31+

‎src/docs-lambda-index-publisher/lambda.DockerFile

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
FROM public.ecr.aws/amazonlinux/amazonlinux:2023 AS base
2+
3+
ARG TARGETARCH
4+
ARG TARGETOS
5+
6+
WORKDIR /app
7+
8+
RUN rpm --import https://packages.microsoft.com/keys/microsoft.asc
9+
10+
RUN curl -o /etc/yum.repos.d/microsoft-prod.repo https://packages.microsoft.com/config/fedora/39/prod.repo
11+
12+
RUN dnf update -y
13+
RUN dnf install -y dotnet-sdk-9.0
14+
RUN dnf install -y npm
15+
RUN dnf install -y git
16+
RUN dnf install -y clang
17+
18+
#COPY ./src/ ./src/
19+
#COPY ./tests/ ./tests/
20+
#COPY ./docs/*.csproj ./docs/
21+
#COPY ./.github/*.csproj ./.github/
22+
#COPY ./build/*.fsproj ./build/
23+
#COPY ./*.sln ./
24+
25+
COPY . .
26+
27+
ENV DOTNET_NOLOGO=true \
28+
DOTNET_CLI_TELEMETRY_OPTOUT=true
29+
30+
RUN arch=$TARGETARCH \
31+
&& if [ "$arch" = "amd64" ]; then arch="x64"; fi \
32+
&& echo $TARGETOS-$arch > /tmp/rid
33+
34+
RUN dotnet publish src/docs-lambda-index-publisher -r linux-x64 -c Release
35+

0 commit comments

Comments
 (0)