feat: python language features inside proc python code block

[jingjiajie]

New libs are installed, please run npm install

Summary
"Pytight" language server is integrated, which is the Microsoft's official python language server.

Thanks to the Pyright, the following features are supported:
Code completion
Hover hint
Outline for python code
and functions shown in the picture:

Due to reasons, the following features are not yet included:
Python code formatting: It's not included in the Pyright language server. It has to be a feature we separately implement in the future.
Folding range for Python code: Because we set to use the Language Server side Folding range parsing rather than using the configuration file, it only recognizes the Folding ranges we returned from language server. However, the Python folding range is naturally implemented with configuration file, so it's ignored. We have to treat it as a separate feature and manually implement it with our language server in the future.

a simple test code:

proc python;
submit;
        class Jonas:
            def name(self) -> str:
                return "Jonas"

        def salute() -> None:
            jonas = Jonas()
            print("This man is %s!" % jonas.name())
endsubmit;
run;

proc python;
    submit;
salute()

[jingjiajie]

@scnwwu In main branch, formatting the above test code, the code disappearing problem occurs again just like before. Would you mind to have a check again? :)

[Zhirong2022]

@scnwwu In main branch, formatting the above test code, the code disappearing problem occurs again just like before. Would you mind to have a check again? :)

@jingjiajie, I have raised the issue #992 to track the changes. Thanks.

[scnwwu]

@jingjiajie, I failed to build your PR code on my PC.
ENOENT: no such file or directory
vscode-sas-extension\server\node_modules\pyright-internal-lsp\dist\packages\pyright-internal\typeshed-fallback
I did npm install but still no such directory

[jingjiajie]

@jingjiajie, I failed to build your PR code on my PC. ENOENT: no such file or directory vscode-sas-extension\server\node_modules\pyright-internal-lsp\dist\packages\pyright-internal\typeshed-fallback I did npm install but still no such directory

Fixed

[scnwwu]

There's a UX problem with this PR. The context menu item "Go to Definition", "Go to Declaration" etc. now always display. For most SAS code except PROC PYTHON, they have no functionality. It may be confusing for some user who rarely use Python.

[scnwwu]

Got unexpected "p" is not defined error with below code in the editor

proc python terminate;
submit;
#Reference to variable defined in previous PROC PYTHON call
print("x = " + x)
def my_function():
    print("Inside the proc step")
endsubmit;
run;

Got unexpected String literal is unterminated error with below code in the editor

proc python;
  interactive;
  x = '''
    endinteractive
  '''
  print('hello')
endinteractive;
run;

[scnwwu]

It does NOT work in browser build.

[jingjiajie]

There's a UX problem with this PR. The context menu item "Go to Definition", "Go to Declaration" etc. now always display. For most SAS code except PROC PYTHON, they have no functionality. It may be confusing for some user who rarely use Python.

@scnwwu It's controlled by the InitializeResult returned by onInitialize() at server side. I'm not sure if there's a way to dynamically control this. Could have a future investigation. Please let me know if you have more ideas about this design.

It does NOT work in browser build.

I found Microsoft maintained two language servers for Python, one is for desktop(Pyright https://github.com/microsoft/pyright), another is for browser(https://github.com/microsoft/vscode-python). In this PR we integrated the Pyright, so the browser version will become another work after this.
Currently I haven't had a future look at the differecies between them, but it seems that the file system is the biggest difference. Desktop version reads local python executables, libs, etc. while browser version should have totally different mechanisms. Multi-threading is another big difference. There are also many node features required by desktop version for specific features, which browser version implements differently. So that's why there have to be two version.

[scnwwu]

There's a UX problem with this PR. The context menu item "Go to Definition", "Go to Declaration" etc. now always display. For most SAS code except PROC PYTHON, they have no functionality. It may be confusing for some user who rarely use Python.

@scnwwu It's controlled by the InitializeResult returned by onInitialize() at server side. I'm not sure if there's a way to dynamically control this. Could have a future investigation. Please let me know if you have more ideas about this design.

Would you take a look at https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#client_registerCapability
So only register textDocument/definition when found proc python. I don't know if it will work.

[jingjiajie]

There's a UX problem with this PR. The context menu item "Go to Definition", "Go to Declaration" etc. now always display. For most SAS code except PROC PYTHON, they have no functionality. It may be confusing for some user who rarely use Python.

@scnwwu It's controlled by the InitializeResult returned by onInitialize() at server side. I'm not sure if there's a way to dynamically control this. Could have a future investigation. Please let me know if you have more ideas about this design.

Would you take a look at https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#client_registerCapability So only register textDocument/definition when found proc python. I don't know if it will work.

Good. It seems to work!

[jingjiajie]

Got unexpected "p" is not defined error with below code in the editor

proc python terminate;
submit;
#Reference to variable defined in previous PROC PYTHON call
print("x = " + x)
def my_function():
    print("Inside the proc step")
endsubmit;
run;

Got unexpected String literal is unterminated error with below code in the editor

proc python;
  interactive;
  x = '''
    endinteractive
  '''
  print('hello')
endinteractive;
run;

@scnwwu I found they are caused by Lexer.startFrom(). The CodeZoneManager reads token by calling the Lexer.startFrom() in the middle of embedded code. Without having read "proc python" in previous lines, the Lexer can't regard the token as a part of embedded code, instead it regards it as a regular proc stmt, then gives wrong result.

For example, the correct token should be 'print("x = " + x)' because it's in a proc python block, but if use Lexer.startFrom() from the start position of this line, the token returned is "print", that's wrong.

Is it possible to remove all the occurance of Lexer.startFrom() function, instead use SyntaxProvider.getSyntax()? It returns correct tokens. However it seems to be a big complexity, I'll need your confirmation.

[scnwwu]

Got unexpected "p" is not defined error with below code in the editor

proc python terminate;
submit;
#Reference to variable defined in previous PROC PYTHON call
print("x = " + x)
def my_function():
    print("Inside the proc step")
endsubmit;
run;

Got unexpected String literal is unterminated error with below code in the editor

proc python;
  interactive;
  x = '''
    endinteractive
  '''
  print('hello')
endinteractive;
run;

@scnwwu I found they are caused by Lexer.startFrom(). The CodeZoneManager reads token by calling the Lexer.startFrom() in the middle of embedded code. Without having read "proc python" in previous lines, the Lexer can't regard the token as a part of embedded code, instead it regards it as a regular proc stmt, then gives wrong result.

For example, the correct token should be 'print("x = " + x)' because it's in a proc python block, but if use Lexer.startFrom() from the start position of this line, the token returned is "print", that's wrong.

Is it possible to remove all the occurance of Lexer.startFrom() function, instead use SyntaxProvider.getSyntax()? It returns correct tokens. However it seems to be a big complexity, I'll need your confirmation.

@jingjiajie, please reset context in Lexer.startFrom()
Add this.context.embeddedLangState = EmbeddedLangState.NONE; under
https://github.com/sassoftware/vscode-sas-extension/blob/main/server/src/sas/Lexer.ts#L1027
Then the code zone looks correct. And the 1st snippet looks good.
I don't know what's the problem for the 2nd snippet. Would you provide more details?

[jingjiajie]

All reported issues so far have been fixed.

[scnwwu]

@jingjiajie, the build is hang. Would you please address it?

[Zhirong2022]

Put mouse indicator in the python code, try to 'Run Region' from the context menu, it will run proc python code block and others together.

proc print data=sashelp.cars(obs=1);
run;

proc python;
submit;
import random

if __name__ == '__main__':
    # print
    print('Hello World!')

    # 4 random number
    i = 0
    rand_list = []
    while i < 4:
        rand_num = random.randint(0, 499)
        if rand_num not in rand_list:
            rand_list.append(rand_num)
            i += 1
    # ouput
    print(rand_list)
endsubmit;
run;

[Zhirong2022]

Python language features are disabled if the 'proc python' code block locates in a customized code block.

*region;
proc print data=sashelp.cars;
run;

proc python;
submit;
my_function()
print("y = " + y)
endsubmit;
run;
*endregion;

[Zhirong2022]

Error on 'run;' unexpectedly if only having comments in python code

proc python;
    interactive;
    #comment
    endinteractive;
run;

[jingjiajie]

Create a new .sas file and type code below. The code has syntax error as expected since it has no colon next to submit. Perform 'Format document' through ContextMenu.

1. It will keep processing and no further response.
2. Stop the process and correct the code, there is no syntax highlighting anymore if trying to type something new.

proc python;
    submit
    endsubmit;
    run;

@Zhirong2022 Fixed.

[Zhirong2022]

The syntax help and the keyword is not consistent, hover mouse over 'endinteractive'.

proc python;
interactive;

endinteractive;
run;

[jingjiajie]

The syntax help and the keyword is not consistent, hover mouse over 'endinteractive'.

proc python;
interactive;

endinteractive;
run;

@Zhirong2022 Fixed.

[Zhirong2022]

All raised issues have fixed.