A little confused about the process

[Andrewp2]

Hi, I'm trying to use Slang. Here's a brief description of what I'm doing: I want to use slang rather than wgsl for writing graphics code in a Bevy-based game. There's some slang-rs bindings, but they don't include the reflection API and I want to use that. What I do instead is my build process compiles a short cpp script that uses slang to compile all the slang shaders to glsl, which wgpu can use, and a reflection.json which captures the automatically generated layout. I can then read the reflection.json within the rust code and use that to match up the actual buffers+textures to the layout that slang has generated.

To compile a slang script, what I'm currently doing is:

1. Create a global session.
2. Create a session w/ target description, search paths etc.
3. Compile each .slang file to a module.
4. For each module, find each entry point. For each entry point, create a composed component type that includes all modules and the specific entry point.
5. Link that composed component type together to get a linked program.
6. Write the glsl file out for this entry point.
7. Write out the reflection data to reflection.json based on the linked program.

So, something seems obviously off about step 4 to me. If I have 100 different shaders, some subset of them will contain global shader parameters, but I don't want each entry point to contain all of the global shader parameters that are used across all the shaders. In fact, I don't think step 4 as written even actually works.

Instead, I want the generated glsl for a specific entry point to only contain the global shader parameters that it actually uses. However, I don't want to have to manually declare for each entry point in all of my 100 shaders what specific global parameters that entry point ends up using, and have to manually create the composed component type that only includes the specific modules that entry point uses.

---

Let me give you an example (written in WGSL) so it's a little clearer. I'm trying to write a compute-based raytracer. Maybe my first module is a raytracing prelude that includes buffers for a BVH:

@group(0) @binding(0)
var<storage, read> lbvh: LBVH;
// other global shader parameters needed for tracing a ray go here...
fn trace_ray(ray: Ray) -> TraceResults {
  // tracing ray using BVH code here...
}
// other helper functions go here...

And then I have a main pass, which has a buffer for lights and other parameters not listed.

@group(0) @binding(4)
var<storage, read> lights: Lights;
// more parameters...
// MIS functions and other stuff that I didn't need in the basic raytracing module
// has an entry point

And finally I have a raytracing-based prepass, which doesn't need the lights buffer (or the other parameters) because it's just writing out a depth texture and a normals texture.

@group(0) @binding(4)
var normals: texture_storage_2d<rgba8unorm, write>;
@group(0) @binding(5)
var depth: texture_storage_2d<rgba32float, write>;
// entry point below

Now both the prepass and the main pass require the raytracing-prelude module in order to function, because they both need a BVH to trace a ray. If I were to rewrite each of these files as slang shaders, each one would be its own module, and prepass and mainpass would both import raytracingPrelude;. But when I want to compile these shaders (lets take prepass as an example) I don't want to include mainpass's global shader parameters, because they would be unnecessary. I also don't want to have to special-case each entry point inside the compilation code, instead the fact that I'm not importing the mainpass module inside the prepass should be enough information to know to not include the mainpasses global shader parameters (the Lights buffer).

[Andrewp2]

Here's the CPP script for reference:

#include <slang.h>
#include <slang-com-ptr.h>
#include <iostream>
#include <filesystem>
#include <fstream>
#include <cstdlib>
#include <vector>
#include <string>
#include <nlohmann/json.hpp>

using Slang::ComPtr;

#define RETURN_ON_FAIL(x) \
  {                       \
    auto _res = x;        \
    if (_res != 0)        \
    {                     \
      return -1;          \
    }                     \
  }

namespace fs = std::filesystem;

template <typename... TArgs>
inline void reportError(const char *format, TArgs... args)
{
  printf(format, args...);
#ifdef _WIN32
  char buffer[4096];
  sprintf_s(buffer, format, args...);
  _Win32OutputDebugString(buffer);
#endif
}

inline void diagnoseIfNeeded(slang::IBlob *diagnosticsBlob)
{
  if (diagnosticsBlob != nullptr)
  {
    reportError("%s", (const char *)diagnosticsBlob->getBufferPointer());
  }
}

void reflectShaderProgram(nlohmann::json &shader_reflection, slang::ProgramLayout *program_layout)
{
  unsigned parameterCount = program_layout->getParameterCount();
  nlohmann::json parameters = nlohmann::json::array();
  for (unsigned pp = 0; pp < parameterCount; pp++)
  {
    slang::VariableLayoutReflection *parameter = program_layout->getParameterByIndex(pp);
    const char *parameterName = parameter->getName();
    nlohmann::json param_info;
    param_info["name"] = parameterName ? parameterName : "";
    slang::TypeLayoutReflection *typeLayout = parameter->getTypeLayout();
    if (typeLayout)
    {
      param_info["type"] = typeLayout->getType()->getName();
    }
    parameters.push_back(param_info);
  }
  shader_reflection["parameters"] = parameters;

  unsigned entryPointCount = program_layout->getEntryPointCount();
  nlohmann::json entry_points = nlohmann::json::array();
  for (unsigned ee = 0; ee < entryPointCount; ee++)
  {
    slang::EntryPointLayout *entryPointLayout = program_layout->getEntryPointByIndex(ee);
    if (entryPointLayout)
    {
      nlohmann::json entry_info;
      entry_info["name"] = entryPointLayout->getName();
      entry_info["stage"] = entryPointLayout->getStage();
      entry_points.push_back(entry_info);
    }
  }
  shader_reflection["entry_points"] = entry_points;
}

int main(int argc, char *argv[])
{
  (void)argc; // Suppress unused parameter warnings
  (void)argv;

  fs::path out_dir = "compiled_shaders";
  fs::create_directories(out_dir);
  ComPtr<slang::IGlobalSession> slangGlobalSession;
  RETURN_ON_FAIL(slang::createGlobalSession(slangGlobalSession.writeRef()));
  slang::TargetDesc targetDesc = {};
  targetDesc.format = SLANG_GLSL;
  targetDesc.profile = slangGlobalSession->findProfile("glsl_460");
  slang::SessionDesc sessionDesc = {};
  sessionDesc.targets = &targetDesc;
  sessionDesc.targetCount = 1;
  const char *searchPaths[] = {"assets/shaders"};
  sessionDesc.searchPaths = searchPaths;
  sessionDesc.searchPathCount = 1;
  ComPtr<slang::ISession> session;
  RETURN_ON_FAIL(slangGlobalSession->createSession(sessionDesc, session.writeRef()));
  std::vector<ComPtr<slang::IModule>> modules;
  for (const auto &entry : fs::recursive_directory_iterator("assets/shaders"))
  {
    if (entry.is_regular_file() && entry.path().extension() == ".slang")
    {
      fs::path shader_path = entry.path();
      std::string shader_name = shader_path.stem().string();

      ComPtr<slang::IBlob> diagnosticBlob;
      ComPtr<slang::IModule> module;
      module = session->loadModule(shader_name.c_str(), diagnosticBlob.writeRef());
      diagnoseIfNeeded(diagnosticBlob);
      if (!module)
      {
        return -1;
      }
      modules.push_back(module);
    }
  }
  nlohmann::json reflection_data = nlohmann::json::array();
  for (auto &slangModule : modules)
  {
    std::string shader_name = slangModule->getName();
    std::cout << "Compiling shader: " << shader_name << std::endl;
    const int entry_point_count = slangModule->getDefinedEntryPointCount();
    std::cout << entry_point_count << " entry points found" << std::endl;
    if (entry_point_count == 0)
    {
      continue;
    }
    for (int i = 0; i < entry_point_count; i++)
    {
      ComPtr<slang::IEntryPoint> entryPoint;
      SlangResult result = slangModule->getDefinedEntryPoint(i, entryPoint.writeRef());
      RETURN_ON_FAIL(result);
      // slang::IComponentType *componentTypes[] = {slangModule.get(), entryPoint.get()};
      std::vector<slang::IComponentType *> componentTypes;
      for (auto &mod : modules)
      {
        componentTypes.push_back(mod.get());
      }
      componentTypes.push_back(entryPoint.get());
      ComPtr<slang::IComponentType> composedProgram;
      {
        ComPtr<slang::IBlob> diagnosticsBlob;
        result = session->createCompositeComponentType(
            componentTypes.data(), (SlangInt)componentTypes.size(), composedProgram.writeRef(), diagnosticsBlob.writeRef());
        std::cout << "create composite component type..." << std::endl;
        diagnoseIfNeeded(diagnosticsBlob);
        RETURN_ON_FAIL(result);
      }
      ComPtr<slang::IComponentType> linkedProgram;
      {
        ComPtr<slang::IBlob> diagnosticsBlob;
        result = composedProgram->link(linkedProgram.writeRef(), diagnosticsBlob.writeRef());
        std::cout << "create linked program..." << std::endl;
        diagnoseIfNeeded(diagnosticsBlob);
        RETURN_ON_FAIL(result);
      }
      ComPtr<slang::IBlob> glslBlob;
      {
        ComPtr<slang::IBlob> diagnosticsBlob;
        result = linkedProgram->getEntryPointCode(
            i, 0, glslBlob.writeRef(), diagnosticsBlob.writeRef());
        std::cout << "create glsl blob..." << std::endl;
        diagnoseIfNeeded(diagnosticsBlob);
        RETURN_ON_FAIL(result);
      }

      fs::path output_shader_path = out_dir / (shader_name + ".glsl");
      std::ofstream shader_file(output_shader_path);
      shader_file.write(
          (const char *)glslBlob->getBufferPointer(), glslBlob->getBufferSize());
      shader_file.close();
      std::cout << "wrote glsl blob..." << std::endl;

      nlohmann::json shader_reflection_json;
      shader_reflection_json["shader_name"] = shader_name;

      slang::ProgramLayout *program_layout = linkedProgram->getLayout();
      std::cout << "got program layout..." << std::endl;
      if (program_layout)
      {
        reflectShaderProgram(shader_reflection_json, program_layout);
      }
    }
    fs::path reflection_path = out_dir / "reflection.json";
    std::ofstream reflection_file(reflection_path);
    reflection_file << reflection_data.dump(4);
    reflection_file.close();

    return 0;
  }

[csyonghe]

@Andrewp2 There is a way to query whether a parameter is used in the final shader. The API is called IMetadata::isParameterLocationUsed.

You can get this IMetadata from IComponentType::getEntryPointMetadata.

To integrate this into your current reflection workflow, simply call isParameterLocationUsed for each resource binding, and if it is not used, don't include it in your output json.

You can also take a look at slang-reflection-test-main.cpp, which implements a reflection->json printer for our internal testing purposes.

[laurooyen]

@Andrewp2 In case you're still interested, I just added support for the reflection API to my slang-rs bindings. A handfull of reflection methods and the IMetadata interface are still missing, but support for them is planned short-term.

[Andrewp2]

For anyone in this in the future:

Slang now has a toJson API directly, which makes this easier.

Steps:

1. Create global session
2. Create session
3. For each slang file, create a module if and only if it has at least one entry point (getDefinedEntryPointCount() > 0)
4. For each module, for each entry point index, get that defined entry point, create a composite component type with just that entry point, link it, optionally produce GLSL or other languages, and then just use toJson on the linked program to get the full reflection file.

I didn't use slang-rs because I think at the exact moment I was solving this it was missing something, but I think it has everything I was missing now.