Real time syntax checking of raku (perl6) script with language server in VIM

Time:2021-6-3

======
Update
2021.02.07: update syntax check code

Previously, I sorted out a raku script as the language server of raku in VIM / neovimIt can check the syntax of raku script after the file is saved

Because the previous language server script usedraku -c foobar.rakuStatement to check the syntax of the script. The above syntax check requires file data, so the contents of the editor buffer must be written to the file before syntax check can be carried out. Based on the above method, if real-time syntax check is needed, the possible way is to create a temporary file, When a content change event occurs, the contents of the buffer are written to a temporary fileraku -c temporary_file.rakuTo check the syntax. The method of creating temporary files is also at presentAtom raku plug inandVs Code raku plug inThe strategy used

I’m lookingRaku parserCode, find the originalNqp provides the function of parsing raku script and syntax checkingThrough the function provided by nqp, you can get rid of the temporary file, and use language server to do real-time syntax check for raku script edited in VIM / neovim. At the end of the article, I sorted out a new script with relevant configuration instructions. If you only want to know how to configure it, you can jump to the end directly
Real time syntax checking of raku (perl6) script with language server in VIM

Syntax analysis of nqp raku

The raku parsing function in nqp is very easy to call in raku scripts

use nqp;
my $*LINEPOSCACHE;

my $code = Q:to[_END_];
foor 1..10 {
}
_END_

my $compiler := nqp::getcomp('Raku') // nqp::getcomp('perl6');
my $g := nqp::findmethod(
  $compiler,'parsegrammar'
)($compiler);

my $a := nqp::findmethod(
  $compiler,'parseactions'
)($compiler);

try {
  $g.parse( $code, :p( 0 ), :actions( $a ) );
}

$!.say;

say "Hello I am still alive";

Save the script above and run the script. You will get the following error message:
Real time syntax checking of raku (perl6) script with language server in VIM

Change line 22 of the above script to$!.perl.say;, you can see the data type and content structure of the error message, and find that the information in the original error message can be used directly without further text analysis. Therefore, you only need to extract the information in the error object
Real time syntax checking of raku (perl6) script with language server in VIM

After testing several rounds of different errors, we know that the data structure returned by different types of error information is slightly different. Use the following code to parse different error information, and get the line of error information, error information and error severity

#!/usr/bin/env raku

use nqp;
my $*LINEPOSCACHE;
my $code = Q:to[_END_];
foor 1..10 {
}
_END_

my $compiler := nqp::getcomp('Raku') // nqp::getcomp('perl6');
my $g := nqp::findmethod(
  $compiler,'parsegrammar'
)($compiler);

my $a := nqp::findmethod(
  $compiler,'parseactions'
)($compiler);

try {
  $g.parse( $code, :p( 0 ), :actions( $a ));
}

if ($!) {
  my $line-number;
  my $severity = 1;
  my $message = "";
  # https://docs.raku.org/type-exception.html
  # https://github.com/rakudo/rakudo/blob/ca7bc91e71afe9373b57cd629215f843e8026df1/src/core.c/Exception.pm6
  given $!.WHO {
    when "X::Syntax::Malformed" {
      $line-number = $!.line;
      $message = $!.message;
    }
    when "X::Undeclared::Symbols" {
      if $!.unk_routines {
        $line-number = $!.unk_routines.values.min[0];
      } else {
        $line-number = $!.unk_types.values.min[0];
      }
      $message = $!.message;
    } 
    when "X::Comp::Group" {
      if $!.panic.WHO eq "X::Comp::AdHoc" {
        $line-number = $!.panic.line;
      } else {
        $line-number = $!.panic.unk_routines.values.min[0];
      }
      $message = $!.message;
    } 
    when "X::AdHoc" {
      $line-number = 0;
      $message = $!.payload ~ "\n" ~ $!.backtrace.Str;
    }
    default {
      $line-number = $!.line - 1;
      $message = $!.message;
    }
  }
  say "line-number is:" ~ $line-number;
  say "message is :" ~ $message;
}

In this way, we can get structured error information
Real time syntax checking of raku (perl6) script with language server in VIM

Language server

Basically, the problem has been solved, but in order to realize real-time syntax checking, we need to modify the previous language server script slightly

The editor returns the full content of the buffer text

We hope that the language server can check the syntax of scripts without relying on temporary files. Then the information returned by the editor to the language server needs to include the script content. This is controlled by the return value passed to the editor by the language server during initialization

By settingtextDocumentSync => 1To make the editor always return the full script after the file changes

sub initialize(%params) {
  %(
    capabilities => {
      # TextDocumentSyncKind.Full
      # Documents are synced by always sending the full content of the document.
      textDocumentSync => 1,

      # Provide outline view support (not)
      documentSymbolProvider => False,

      # Provide hover support (not)
      hoverProvider => False
    }
  )
}

Handling text content change events

Now, after each change of text, the editor will pass the text content in the buffer to the language server. The type information of the editor text change event will be marked as textdocument / didchange in the editor passing to the language server. Add the corresponding conditional execution statement block (lines 15-17) in the language server, and callcheck-syntaxCheck the grammar

sub process_request(%request) {
  # TODO throw an exception if a method is called before $initialized = True
  # debug-log(%request);
  given %request<method> {
    when 'initialize' {
      my $result = initialize(%request<params>);
      send-json-response(%request<id>, $result);
    }
    when 'textDocument/didOpen' {
      check-syntax(%request, "open");
    }
    when 'textDocument/didSave' {
      check-syntax(%request, "save");
    }
    when 'textDocument/didChange' {
      check-syntax(%request, "change");
    }
    when 'shutdown' {
      # Client requested to shutdown...
      send-json-response(%request<id>, Any);
    }
    when 'exit' {
      exit 0;
    }
  }
}

Syntax checking function

Modify the syntax check function to get the script content (line 6) from the editor for syntax check when the text changes. In other cases, read the file content for syntax check

sub check-syntax(%params, $type) {

  my $uri = %params<params><textDocument><uri>;
  my $code;
  if ($type eq "change") {
    $code = %params<textDocument><text> || %params<params><contentChanges>[0]<text>;
  } else {
    my $file;
    if $uri ~~ /file\:\/\/(.+)/ {
      $file = $/[0].Str;
    }
    $code = $file.IO.slurp;
  }

  my @problems = parse-error($code) || [];

  my %parameters = %(
    uri         => $uri,
    diagnostics => @problems
  );
  send-json-request('textDocument/publishDiagnostics', %parameters);

  return;
}

Perform grammar checking

The function that performs syntax checking in the above script isparse-errorIt checks the syntax of the script in string format, and then outputs the formatted error message. The content of this function is modified by the previous nqp syntax check demo script

sub extract-info($error) {
    my $line-number;
    my $severity = 1;
    my $message = "";

    given $error.WHO {
      when "X::Syntax::Malformed" {
        $line-number = $error.line;
        $message = $error.message;
      }
      when "X::Undeclared::Symbols" {
        if $error.unk_routines {
          say "ok";
          $line-number = $error.unk_routines.values.min[0];
        } else {
          $line-number = $error.unk_types.values.min[0];
        }
        $message = $error.message;
      } 
      when "X::Comp::Group" {
        if $error.panic.line < Inf {
          $line-number = $error.panic.line;
        } else {
          $line-number = $error.panic.unk_routines.values.min[0];
        }
        $message = $error.message;
      } 
      when "X::AdHoc" {
        $line-number = 0;
        $message = $error.payload ~ "\n" ~ $error.backtrace.Str;
      }
      default {
        $line-number = $error.line;
        $message = $error.message;
      }
    }

    return { line-number => $line-number, severity => $severity, message => $message };
}

sub parse-error($code) is export {

  my $*LINEPOSCACHE;
  my $problems;

  my $compiler := nqp::getcomp('Raku') // nqp::getcomp('perl6');
  my $g := nqp::findmethod(
    $compiler,'parsegrammar'
  )($compiler);

  #$g.HOW.trace-on($g);

  my $a := nqp::findmethod(
    $compiler,'parseactions'
  )($compiler);

  try {
    $g.parse( $code, :p( 0 ), :actions( $a ));
  }

  if ($!) {
    my %info = extract-info($!);
    # https://docs.raku.org/type-exception.html
    # https://github.com/rakudo/rakudo/blob/ca7bc91e71afe9373b57cd629215f843e8026df1/src/core.c/Exception.pm6

    $problems = ({
      range => {
        start => {
          line      => %info<line-number> - 1,
          character => 0
        },
        end => {
          line      => %info<line-number> - 1,
          character => 99
        },
      },
      severity => %info<severity>,
      source   => 'Raku',
      message  => %info<message>
    });
  }
  # say debug-log(@problems);
  return $problems;
}

use

Finally, we get an updated raku language server script to realize real-time syntax checking

#!/usr/bin/env raku
use JSON::Fast;
use nqp;

# No standard input/output buffering to prevent unwanted hangs/failures/waits
$*OUT.out-buffer = False;
$*ERR.out-buffer = False;

debug-log("?: Starting raku-langserver... Reading/writing stdin/stdout.");

start-listen();

sub start-listen() is export {
  my %request;

  loop {
    my $content-length = get_content_length();

    if $content-length == 0 {
      next;
    }

    # debug-log("length is: " ~ $content-length);

    %request = read_request($content-length);

    unless %request {
      next;
    }

    # debug-log(%request);
    process_request(%request);

  }
}


sub get_content_length {

  my $content-length = 0;
  for $*IN.lines -> $line {

    # we're done here
    last if $line eq '';

    # Parse HTTP-style header
    my ($name, $value) = $line.split(': ');
    if $name eq 'Content-Length' {
      $content-length += $value;
    }
  }

  # If no Content-Length in the header
  return $content-length;
}

sub read_request($content-length) {
  my $json    = $*IN.read($content-length).decode;
  my %request = from-json($json);

  return %request;
}

sub process_request(%request) {
  # TODO throw an exception if a method is called before $initialized = True
  # debug-log(%request);
  given %request<method> {
    when 'initialize' {
      my $result = initialize(%request<params>);
      send-json-response(%request<id>, $result);
    }
    when 'textDocument/didOpen' {
      check-syntax(%request, "open");
    }
    when 'textDocument/didSave' {
      check-syntax(%request, "save");
    }
    when 'textDocument/didChange' {
      check-syntax(%request, "change");
    }
    when 'shutdown' {
      # Client requested to shutdown...
      send-json-response(%request<id>, Any);
    }
    when 'exit' {
      exit 0;
    }
  }
}

sub debug-log($text) is export {
  $*ERR.say($text);
}

sub send-json-response($id, $result) {
  my %response = %(
    jsonrpc => "2.0",
    id       => $id,
    result   => $result,
  );
  my $json-response = to-json(%response, :!pretty);
  my $content-length = $json-response.chars;
  my $response = "Content-Length: $content-length\r\n\r\n" ~ $json-response;
  print($response);
}


sub send-json-request($method, %params) {
  my %request = %(
    jsonrpc  => "2.0",
    'method' => $method,
    params   => %params,
  );
  my $json-request = to-json(%request);
  my $content-length = $json-request.chars;
  my $request = "Content-Length: $content-length\r\n\r\n" ~ $json-request;
  # debug-log($request);
  print($request);
}

sub initialize(%params) {
  %(
    capabilities => {
      # TextDocumentSyncKind.Full
      # Documents are synced by always sending the full content of the document.
      textDocumentSync => 1,

      # Provide outline view support (not)
      documentSymbolProvider => False,

      # Provide hover support (not)
      hoverProvider => False
    }
  )
}

sub check-syntax(%params, $type) {

  my $uri = %params<params><textDocument><uri>;
  my $code;
  if ($type eq "change") {
    $code = %params<textDocument><text> || %params<params><contentChanges>[0]<text>;
  } else {
    my $file;
    if $uri ~~ /file\:\/\/(.+)/ {
      $file = $/[0].Str;
    }
    return unless $file.IO.e;
    $code = $file.IO.slurp;
  }

  my @problems = parse-error($code) || [];


  my %parameters = %(
    uri         => $uri,
    diagnostics => @problems
  );
 
  send-json-request('textDocument/publishDiagnostics', %parameters);


  return;
}

grammar ErrorMessage {
  token TOP { <Error>+ }
  token Error { <Warning> || <Missing-libs> || <Undeclared-name> || <Missing-generics> }

  rule Undeclared-name { <ErrorInit> Undeclared <Undeclared-type>s?\:\r?\n\s+<Name> used at lines? <Linenum>\.? <Message> .* }
  rule Missing-generics{ <ErrorInit> <Error-type> <-[\:]>+\:<Linenum> \s* "------>" <Message>? }
  rule Missing-libs { <ErrorInit> Could not find <Name> in\:<-[\:]>+\:<Linenum> }
  token Warning { "Potential difficulties:" \n <Error-type> <-[\:]>+\:<Linenum> \s* "------>" <Message>? }

  token Error-type { \N* }
  token Undeclared-type { routine || name }
  token Name { <-[\'\s]>+ }
  token Linenum { \d+ }
  token Message { .* }
  token ErrorInit { '[31m===[0mSORRY![31m===[0m' \N+ }
}

class ErrorMessage-actions {
  method TOP ($/) {
    make $<Error>.map( -> $e {
      my $line-number;
      my $message;
      my $severity = 1;

      given $e {
        when $e<Missing-libs> {
          $line-number = $e<Missing-libs><Linenum>.Int;
          $message = qq[Could not find Library $e<Missing-libs><Name>];
        }
        when $e<Missing-generics> {
          $line-number = $e<Missing-generics><Linenum>.Int;
          $message = qq[$e<Missing-generics><Error-type>\n{$e<Missing-generics><Message>.trim.subst(/\x1b\[\d+m/, '', :g)}];
        }
        when $e<Undeclared-name> {
          $line-number = $e<Undeclared-name><Linenum>.Int;
          $message = qq[Undelcared $e<Undeclared-name><Undeclared-type> $e<Undeclared-name><Name>. {$e<Undeclared-name><Message>.trim.subst(/\x1b\[\d+m/, '', :g)}];
        }
        when $e<Warning> {
          $line-number = $e<Warning><Linenum>.Int;
          $message = qq[{$e<Warning><Error-type>.trim}\n{$e<Warning><Message>.trim.subst(/\x1b\[\d+m/, '', :g)}];
          $severity = 3;
        }
      }

      my Bool $vim = True;
      $line-number-- if $vim;

      ({
        range => {
          start => {
            line      => $line-number,
            character => 0
          },
          end => {
            line      => $line-number + 1,
            character => 0
          },
        },
        severity => $severity,
        source   => 'raku -c',
        message  => $message
      })
    })
  }
}

sub extract-info($error) {
    my $line-number;
    my $severity = 1;
    my $message = "";

    given $error.WHO {
      when "X::Syntax::Malformed" {
        $line-number = $error.line;
        $message = $error.message;
      }
      when "X::Undeclared::Symbols" {
        if $error.unk_routines {
          $line-number = $error.unk_routines.values.min[0];
        } else {
          $line-number = $error.unk_types.values.min[0];
        }
        $message = $error.message;
      } 
      when "X::Comp::Group" {
        if $error.panic.line < Inf {
          $line-number = $error.panic.line;
        } else {
          $line-number = $error.panic.unk_routines.values.min[0];
        }
        $message = $error.message;
      } 
      when "X::AdHoc" {
        $line-number = 0;
        $message = $error.payload ~ "\n" ~ $error.backtrace.Str;
      }
      default {
        $line-number = $error.line;
        $message = $error.message;
      }
    }

    return { line-number => $line-number, severity => $severity, message => $message };
}

sub parse-error($code) is export {

  my $*LINEPOSCACHE;
  my $problems;

  my $compiler := nqp::getcomp('Raku') // nqp::getcomp('perl6');
  my $g := nqp::findmethod(
    $compiler,'parsegrammar'
  )($compiler);

  #$g.HOW.trace-on($g);

  my $a := nqp::findmethod(
    $compiler,'parseactions'
  )($compiler);

  try {
    $g.parse( $code, :p( 0 ), :actions( $a ));
  }

  if ($!) {
    my %info = extract-info($!);
    # https://docs.raku.org/type-exception.html
    # https://github.com/rakudo/rakudo/blob/ca7bc91e71afe9373b57cd629215f843e8026df1/src/core.c/Exception.pm6

    $problems = ({
      range => {
        start => {
          line      => %info<line-number> - 1,
          character => 0
        },
        end => {
          line      => %info<line-number> - 1,
          character => 99
        },
      },
      severity => %info<severity>,
      source   => 'Raku',
      message  => %info<message>
    });
  }
  # say debug-log(@problems);
  return $problems;
}

Use it in the same way as beforeCOC plug in)

Save the above script in a file/foo/bar/raku-lsp.rakuAnd add execution permission to the filechmod +x /foo/bar/raku-lsp.raku.

Then configure the COC plug-in in VIM or neovim

  1. Open profile:CocConfig
  2. Add the language server script above:
{
    "languageserver" : {
        "raku": {
            "command": "/foo/bar/raku-lsp.raku",
            "args": ["--vim"],
            "filetypes": ["raku", "rakumod", "pl6", "p6", "pm6"]
        }
    }
}

Restart the VIM or neovim editor and open a raku file. Just write a few lines to see the effect