CCGenerateLOD – New Version Produces Encrypted 0_n.json File, Upload to Reality Data Management Fails

[VietBuiHa]

Hi Bentley team,

We're currently using the CCGenerateLOD tool to convert 3SM and 3MX files into web-ready format. We recently tried a newer version of the tool (downloaded from https://kb.orbitgt.com/external/ccgeneratelod), and while the conversion completes, the resulting 0_n.json file appears to be encrypted or obfuscated.

This encrypted output fails to upload to Reality Data Management service.

Interestingly, when using a previous version of CCGenerateLOD, the 0_n.json file is not encrypted and the upload to Reality Data Management works without issue.

Could you please clarify:

Is the encryption of 0_n.json in the new version intentional?

Is there a configuration option to generate an unencrypted version?

Are there any known compatibility changes with the Reality Data Management service?

We’d really appreciate any guidance or clarification on how to proceed with the newer version.

Best regards,
Viet Bui

[jeroenvermeulenbentley]

ServiceNow
https://dev.azure.com/bentleycs/ProductAdvancement/_workitems/edit/1670277

DevOpsTask
https://dev.azure.com/bentleycs/OrbitGT/_workitems/edit/1738444

[jeroenvermeulenbentley]

The updated CCLODPlugin generates compressed .json files that require to be uploaded with content encoding flag to iTwin Reality Management for consuming web apps to be de-compressed by the browser.
Compressed tiles reduce storage and improves streaming performance.
iTwin Capture upload client app is now setting the content encoding flag properly.

[jeroenvermeulenbentley]

@VietBuiHa
Can you review and share your feedback?

[VietBuiHa]

Hi, thanks for your response, we can't use iTwin Capture client app to upload our files to Reality Data Management. We're creating an automation service to convert the 3sm, 3mx automatically and then upload it to Reality Data Management through Reality Data API, Everything happens in the cloud, that's why we need to use CCGenerateLOD.

[VietBuiHa]

Following is the code we use to upload the reality data into Reality Data Management through the API, could you please guide us how to upload the compressed tiles, in this new CCGenerateLOD verions?

public async Task<string> CreateRealityData(string filename)
{
    using (var httpClient = new HttpClient())
    {
        using (var request = new HttpRequestMessage(HttpMethod.Post, "https://api.bentley.com/reality-management/reality-data/"))
        {
            request.Headers.TryAddWithoutValidation("Accpect", "application/vnd.bentley.itwin-platform.v1+json");
            request.Headers.TryAddWithoutValidation("Authorization", $"Bearer {(await _bentleyService.Get2LeggedToken())}");

            dynamic realityData = new ExpandoObject();
            realityData.iTwinId = _iTwinId;
            realityData.displayName = Path.GetFileNameWithoutExtension(filename);
            realityData.classification = "Model";

            switch (Path.GetExtension(filename))
            {
                case ".3sm":
                    realityData.type = "RealityMesh3DTiles";
                    realityData.rootDocument = "n_0.json";
                    break;
                case ".las":
                    realityData.type = "OPC";
                    realityData.rootDocument = $"{Path.GetFileNameWithoutExtension(filename)}.opc";
                    break;
                case ".e57":
                    realityData.type = "OPC";
                    realityData.rootDocument = $"{Path.GetFileNameWithoutExtension(filename)}.opc";
                    break;
                case ".zip":
                    var targetFile = Directory.EnumerateFiles(Utils.GetRootPath(filename), "*.*", SearchOption.AllDirectories)
                        .Where(file => file.ToLower().EndsWith("obj") || file.ToLower().EndsWith("3mx"))
                        .FirstOrDefault();
                    switch (Path.GetExtension(targetFile))
                    {
                        case ".obj":
                            realityData.type = "RealityMesh3DTiles";
                            realityData.rootDocument = "n_0.json";
                            break;
                        case ".3mx":
                            realityData.type = "RealityMesh3DTiles";
                            realityData.rootDocument = "n_0.json";
                            break;
                        default:
                            break;
                    }
                    break;
                default:
                    break;
            }

            request.Content = new StringContent(JsonConvert.SerializeObject(realityData));
            request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");

            var response = await httpClient.SendAsync(request);

            if (response.IsSuccessStatusCode)
            {
                dynamic content = JsonConvert.DeserializeObject<dynamic>(await response.Content.ReadAsStringAsync());
                return content.realityData.id;
            }
            else
            {
                var message = await response.Content.ReadAsStringAsync();
                Console.WriteLine(message);
                _logger.LogError(message);
                throw new Exception("Create Reality Data Failed!");
            }
        }
    }
}

[VietBuiHa]

@jeroenvermeulenbentley, this hasn't been resolved yet, but I see you closed the topic?

[jeroenvermeulenbentley]

My mistake.
Reviewing your comment and will get back to you tomorrow.

[jeroenvermeulenbentley]

The CCGenerateLOD doesn't upload data. Reality Management provides a Write SAS URL for a Reality Data Azure Container.
The client writing data into the Azure Container is required to set the content encoding flag for files uploaded into the Azure Container.
Which Bentley Application or Service hosts the CCGenereateLOD plugin?

[VietBuiHa]

We do the upload by ourselves, using the Reality Data Management API. As you can see in my code, we create the new reality data using the API, I'm wondering what the required content encoding flag is for the compressed RealityMesh3DTiles?

We download the CCgenerateLOD and host it on the cloud (Azure Kubernetes Service). The previous version works like a charm, we only have a problem with the newest version when n_0.json is obfuscated or compressed

[VietBuiHa]

You can see we generate the SAS token from the RealityData and upload the tiles to the Azure Container

public async Task<string> GetSASToken(string id)
{
    using (var httpClient = new HttpClient())
    {
        using (var request = new HttpRequestMessage(HttpMethod.Get, @$"https://api.bentley.com/reality-management/reality-data/{id}/writeaccess?iTwinId={_iTwinId}"))
        {
            request.Headers.TryAddWithoutValidation("Accpect", "application/vnd.bentley.itwin-platform.v1+json");
            request.Headers.TryAddWithoutValidation("Authorization", $"Bearer {(await _bentleyService.Get2LeggedToken())}");
            var response = await httpClient.SendAsync(request);
            if (response.IsSuccessStatusCode)
            {
                dynamic content = JsonConvert.DeserializeObject<dynamic>(await response.Content.ReadAsStringAsync());
                return content._links.containerUrl.href;
            }
            else
            {
                _logger.LogError(await response.Content.ReadAsStringAsync());
                throw new Exception("Get Reality Data SAS Failed!");
            }
        }
    }
}

[GlenBentleyUS]

@cnovel @LotteauL,
Hi Cyril, Liv, Is there any way we can resolve this outside of using Manage and Extract?

[cnovel]

Not really my area of expertise, but I remember that for Web3SM, user have to upload with the correct "ContentEncoding" header set to "deflate". In C#, it would look like this:

await blobClient.UploadAsync(output, new BlobUploadOptions
{
    HttpHeaders = new BlobHttpHeaders
    {
        ContentType = "text/plain",
        ContentEncoding = "deflate"
    }
});

with output being a stream of what they need to upload. Adding @BeChaRem who might be able to confirm.