Analyze API surface to simplify its usage

[spoenemann]

As discussed in today's dev meeting, we want to look at Langium's API surface and then decide whether to restructure it in order to simplify its usage. I made a simple program to list the top-level symbols exposed by the langium package. Here is the result:

Total: 238

# OBJECT (21)
AllSemanticTokenModifiers
AllSemanticTokenTypes
DefaultNameRegexp
DefaultSemanticTokenOptions
Disposable
DocumentState
DocumentValidator
DONE_RESULT
EMPTY_SCOPE
EMPTY_STREAM
EmptyFileSystem
Formatting
GrammarAST
LangiumGrammarModule
NL
NLEmpty
RangeComparison
Reduction
SemanticTokensDecoder
UriUtils
ValidationCategory

# CLASS (76)
AbstractAstReflection
AbstractCallHierarchyProvider
AbstractCstNode
AbstractExecuteCommandHandler
AbstractFormatter
AbstractGoToImplementationProvider
AbstractInlayHintProvider
AbstractLangiumParser
AbstractParserErrorMessageProvider
AbstractSemanticTokenProvider
AbstractSignatureHelpProvider
AbstractTypeDefinitionProvider
AstNodeHoverProvider
CompositeCstNodeImpl
CompositeGeneratorNode
ContextCache
CstNodeBuilder
DefaultAstNodeDescriptionProvider
DefaultAstNodeLocator
DefaultCommentProvider
DefaultCompletionProvider
DefaultConfigurationProvider
DefaultDefinitionProvider
DefaultDocumentBuilder
DefaultDocumentHighlightProvider
DefaultDocumentSymbolProvider
DefaultDocumentValidator
DefaultFoldingRangeProvider
DefaultFuzzyMatcher
DefaultIndexManager
DefaultJsonSerializer
DefaultLangiumDocumentFactory
DefaultLangiumDocuments
DefaultLanguageServer
DefaultLexer
DefaultLinker
DefaultNameProvider
DefaultNodeFormatter
DefaultNodeKindProvider
DefaultReferenceDescriptionProvider
DefaultReferences
DefaultReferencesProvider
DefaultRenameProvider
DefaultScopeComputation
DefaultScopeProvider
DefaultServiceRegistry
DefaultTokenBuilder
DefaultValueConverter
DefaultWorkspaceManager
DefaultWorkspaceSymbolProvider
Deferred
DisposableCache
DocumentCache
EmptyFileSystemProvider
ErrorWithLocation
FormattingRegion
IndentNode
JSDocDocumentationProvider
LangiumCompletionParser
LangiumParser
LangiumParserErrorMessageProvider
LeafCstNodeImpl
MapScope
MultilineCommentHoverProvider
MultiMap
MutexLock
NewLineNode
RootCstNodeImpl
SemanticTokensBuilder
SimpleCache
StreamImpl
StreamScope
TreeStreamImpl
URI
ValidationRegistry
WorkspaceCache

# FUNCTION (138)
addCallHierarchyHandler
addCodeActionHandler
addCodeLensHandler
addCompletionHandler
addConfigurationChangeHandler
addDiagnosticsHandler
addDocumentHighlightsHandler
addDocumentLinkHandler
addDocumentsHandler
addDocumentSymbolHandler
addExecuteCommandHandler
addFindReferencesHandler
addFoldingRangeHandler
addFormattingHandler
addGoToDeclarationHandler
addGotoDefinitionHandler
addGoToImplementationHandler
addGoToTypeDefinitionHandler
addHoverHandler
addInlayHintHandler
addRenameHandler
addSemanticTokenHandler
addSignatureHelpHandler
addWorkspaceSymbolHandler
assertUnreachable
assignMandatoryAstProperties
compareRange
convertBigint
convertBoolean
convertDate
convertID
convertInt
convertNumber
convertString
copyAstNode
createCallHierarchyRequestHandler
createCompletionParser
createDefaultModule
createDefaultSharedModule
createGrammarConfig
createLangiumGrammarServices
createLangiumParser
createRequestHandler
createServerRequestHandler
createServicesForGrammar
delayNextTick
diagnosticData
eagerLoad
equalURI
escapeRegExp
expandToNode
expandToString
expandToStringWithNL
expandTracedToNode
expandTracedToNodeIf
findAssignment
findCommentNode
findDeclarationNodeAtOffset
findFirstFeatures
findLeafNodeAtOffset
findLocalReferences
findNameAssignment
findNextFeatures
findNodeForKeyword
findNodeForProperty
findNodesForKeyword
findNodesForKeywordInternal
findNodesForProperty
findRootNode
flattenCst
getAllReachableRules
getCaseInsensitivePattern
getContainerOfType
getCrossReferenceTerminal
getDiagnosticRange
getDocument
getEntryRule
getHiddenRules
getInteriorNodes
getNextNode
getPreviousNode
getStartlineNode
getTerminalParts
hasContainerOfType
inject
inRange
interpretAstReflection
interruptAndCheck
isAstNode
isAstNodeDescription
isAstNodeWithComment
isCommentNode
isCommentTerminal
isCompositeCstNode
isCstChildNode
isGeneratorNode
isIMultiModeLexerDefinition
isJSDoc
isLeafCstNode
isLinkingError
isMultilineComment
isNamed
isNewLineNode
isOperationCancelled
isReference
isRootCstNode
isTokenTypeArray
isTokenTypeDictionary
isWhitespaceRegExp
joinToNode
joinTracedToNode
joinTracedToNodeIf
linkContentToContainer
loadGrammarFromJson
mergeCompletionProviderOptions
mergeSignatureHelpOptions
normalizeEOL
parseJSDoc
partialMatches
partialRegex
prepareLangiumParser
relativeURI
setInterruptionPeriod
startCancelableOperation
startLanguageServer
stream
streamAllContents
streamAst
streamContents
streamCst
streamReferences
toDiagnosticSeverity
toDocumentSegment
tokenToRange
toString
toStringAndTrace
traceToNode
traceToNodeIf

# STRING (1)
EOL

# SYMBOL (2)
DatatypeSymbol
OperationCancelled

[spoenemann]

My observations:

• The module keyword of TypeScript is deprecated, the equivalent namespace should be used instead. The concept of "internal modules" was from the early days of the language, when using JavaScript's global scope was still common.
• The TypeScript documentation recommends to use ES modules (files with imports / exports) instead of namespaces. But that recommendation is only valid for dependencies within the code base.
• We use index files to gather everything that should be exposed to consumers of the langium package. There's a main index file (import from 'langium') and additional entry points for code that is kept separate (e.g. 'langium/test').
• 238 exported symbols (not counting the types) seems like a lot. What's positive about it: the editor can auto-complete a symbol and add it to the list of imports, so discoverability is good. What's negative: some symbols can't be understood without context or reading documentation, e.g. NL.
• The concept of ES modules doesn't help with these package exports, so namespaces are the way to go to put symbols into context.
• A subset of the symbols is going to be moved to a separate package export (Export LSP services via dedicated ESM export #1166). These don't need a namespace because the import path already gives some context.
• The VS Code API makes heavy use of namespaces (I counted 14). We don't have to go that far, but it's a good example for splitting the API surface into different topics.

My thoughts and recommendations:

• I wouldn't move everything into namespaces to keep the good discoverability through auto-completion and to avoid excessive API breakage.
• I would move most of the utility functions, constants and data classes into namespaces. Example: grammar-utils (Split grammar and main export #1171 (comment)).
• I would merge Split grammar and main export #1171 and then create a PR with a proposal for exported namespaces. We can discuss which symbols to move within that PR.
• I propose to declare the main branch as preparation for v3.0.0 and create a new maintenance-2.x branch for v2.2.0 etc.

[sailingKieler]

Wrt.

The concept of ES modules doesn't help with these package exports, so namespaces are the way to go to put symbols into context.

How do grouped exports like in https://github.com/eclipse-langium/langium/blob/main/packages/langium/src/index.ts#L24-L26 relate to that?

[spoenemann]

In my understanding, import * as xyz wraps the exports of the imported module into a namespace local to that file. Effectively, we create an object xyz that contains all those symbols.

This is the easiest way to create a namespace to be consumed only from the outside of the package, while keeping direct access to the individual symbols within the package code.