Doesn't work direct from Ruby CLI #31

Fryguy · 2020-12-10T22:41:53Z

As a test, I tried the following

foo.rb:

class Foo
  3.times |x|
    puts x
  end
end

And none of the following work

$ ruby -r dead_end foo.rb
foo.rb:6: syntax error, unexpected end, expecting end-of-input
$ ruby -r dead_end -c foo.rb
foo.rb:6: syntax error, unexpected end, expecting end-of-input

I assume this is probably a Ruby ordering thing in that it parses your script before applying the requires, but it's very surprising.

If instead, I do this, it works fine.

bar.rb:

require "foo"

Not sure if there's an actual fix that can be done if this is a Ruby core issue, but wanted to bring to your attention, particularly since one of your goals is eventually getting this into ruby core in a similar way to did_you_mean.

schneems · 2020-12-11T03:37:52Z

Thanks for trying it out. It does work if you just require the file:

$ ruby -r dead_end -r ./foo.rb -e puts 'hi'
Run `$ dead_end /private/tmp/foo.rb` for more options

DeadEnd: Unmatched `end` detected

This code has an unmatched `end`. Ensure that all `end` lines
in your code have a matching syntax keyword  (`def`,  `do`, etc.)
and that you don't have any extra `end` lines.

file: /private/tmp/foo.rb
simplified:

      1  class Foo
    ❯ 2    3.times |x|
    ❯ 4    end
      5  end

The reason has to do with how I'm hooking into Ruby to know when a syntax error has occurred. I'm monkeypatching require, load, and require_relative https://github.com/zombocom/dead_end/blob/a598e7617e06b6016277b90944c5d2bd51ba70d1/lib/dead_end/auto.rb#L16-L20

It looks like running Ruby code directly from $ ruby doesn't use one of these. There is a CLI that ships with this gem though:

$ dead_end foo.rb

DeadEnd: Unmatched `end` detected

This code has an unmatched `end`. Ensure that all `end` lines
in your code have a matching syntax keyword  (`def`,  `do`, etc.)
and that you don't have any extra `end` lines.

file: /private/tmp/foo.rb
simplified:

      1  class Foo
    ❯ 2    3.times |x|
    ❯ 4    end
      5  end

Not sure if there's an actual fix that can be done if this is a Ruby core issue, but wanted to bring to your attention, particularly since one of your goals is eventually getting this into ruby core in a similar way to did_you_mean

100% thanks! Every little bit of behavior that's unexpected is good to know about.

This commit adds the `dead_end` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. ## What is dead_end? When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem. ## Demo Generate a file with a syntax error: ``` $ echo "require_relative 'syntax_error'" > test.rb $ cat > syntax_error.rb <<EOF class Dog < Animal def sit super end def bark puts "woof" def happy tail.wag end end EOF ``` Build and run ruby: ``` $ make $ make runruby generating vm_call_iseq_optimized.inc vm_call_iseq_optimized.inc unchanged RUBY_ON_BUG='gdb -x ./.gdbinit -p' ./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems ./test.rb Run `$ dead_end /Users/rschneeman/Documents/projects/ruby/syntax_error.rb` for more options DeadEnd: Missing `end` detected This code has a missing `end`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/ruby/syntax_error.rb simplified: 1 class Dog < Animal ❯ 6 def bark ❯ 9 def happy ❯ 11 end 12 end /Users/rschneeman/Documents/projects/ruby/lib/dead_end/auto.rb:30:in `require': /Users/rschneeman/Documents/projects/ruby/syntax_error.rb:12: syntax error, unexpected end-of-input, expecting `end' (SyntaxError) ``` ## Status This commit is a proof of concept spike (https://en.wikipedia.org/wiki/Spike_(software_development)). If core likes this approach, then I can refine and improve it. ## Known issues - This implementation did not wire up any code that updates the vendored gem version from the source. It would be not difficult to add https://github.com/ruby/ruby/blob/41867532ac07515543e4d7e75094edeff09de743/tool/sync_default_gems.rb - This implementation does not add `dead_end` as a feature to be enabled yet: https://github.com/ruby/ruby/blob/cb4e2cb55a59833fc4f1f6db2f3082d1ffcafc80/ruby.c#L97. - DeadEnd's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations ruby/syntax_suggest#31. - We could extend or modify the Ruby API to hook into a better place with guidance and approval.

Adds the `dead_end` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. Reference ticket: https://bugs.ruby-lang.org/issues/18159 close ruby#4845 ## What is dead_end? When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem. ## Demo Generate a file with a syntax error: ``` $ echo "require_relative 'syntax_error'" > test.rb $ cat > syntax_error.rb <<EOF class Dog < Animal def sit super end def bark puts "woof" def happy tail.wag end end EOF ``` ## Known issues - DeadEnd's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations ruby/syntax_suggest#31. - We could extend or modify the Ruby API to hook into a better place with guidance and approval.

Adds the `dead_end` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. close ruby#4845 ## What is dead_end? When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem. ## Demo Generate a file with a syntax error: ``` $ echo "require_relative 'syntax_error'" > test.rb $ cat > syntax_error.rb <<EOF class Dog < Animal def sit super end def bark puts "woof" def happy tail.wag end end EOF ``` Build and run ruby: ``` $ make $ make runruby generating vm_call_iseq_optimized.inc vm_call_iseq_optimized.inc unchanged RUBY_ON_BUG='gdb -x ./.gdbinit -p' ./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems ./test.rb Run `$ dead_end /Users/rschneeman/Documents/projects/ruby/syntax_error.rb` for more options DeadEnd: Missing `end` detected This code has a missing `end`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/ruby/syntax_error.rb simplified: 1 class Dog < Animal ❯ 6 def bark ❯ 9 def happy ❯ 11 end 12 end /Users/rschneeman/Documents/projects/ruby/lib/dead_end/auto.rb:30:in `require': /Users/rschneeman/Documents/projects/ruby/syntax_error.rb:12: syntax error, unexpected end-of-input, expecting `end' (SyntaxError) ``` ## Known issues - DeadEnd's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations ruby/syntax_suggest#31. - We could extend or modify the Ruby API to hook into a better place with guidance and approval.

As pointed out in #31 the current version of dead_end only works if the developer requires dead_end and then invokes `require`. This PR takes advantage of this change in Ruby 3.2 ruby/ruby#5516 to modify the `SyntaxError` directly. This behavior will only be fixed for Ruby 3.2+ Additionally we are still not able to handle the case where a program is streamed to ruby and does not exist on disk: ``` $ echo "def foo" | ruby ``` As the SyntaxError does not provide us with the contents of the script. ``` $ echo "def foo" | ruby -:1: syntax error, unexpected end-of-input def foo ```

Currently dead_end works by monkey patching require. This causes confusion and problems as other tools are not expecting this. For example zombocom/derailed_benchmarks#204 and #124. This PR utilizes the new SyntaxError#detailed_message as introduced in ruby/ruby#5516 that will be released in Ruby 3.2. That means that developers using dead_end with Ruby 3.2+ will experience more consistent behavior. ## Limitations As pointed out in #31 the current version of dead_end only works if the developer requires dead_end and then invokes `require`. This behavior is still not fixed for Ruby 3.2+ ``` $ ruby -v ruby 3.2.0preview1 (2022-04-03 master f801386f0c) [x86_64-darwin20] $ cat monkeypatch.rb SyntaxError.prepend Module.new { def detailed_message(highlight: nil, **) message = super message += "Monkeypatch worked\n" message end } # require_relative "bad.rb" # Note that i am commenting # out the require, but leaving # in the monkeypatch ⛄️ 3.2.0 🚀 /tmp $ cat bad.rb def lol_i-am-a-synt^xerror ⛄️ 3.2.0 🚀 /tmp $ ruby -r./monkeypatch.rb bad.rb bad.rb:1: syntax error, unexpected '-', expecting ';' or '\n' def lol_i-am-a-synt^xerror ``` Additionally we are still not able to handle the case where a program is streamed to ruby and does not exist on disk: ``` $ echo "def foo" | ruby ``` As the SyntaxError does not provide us with the contents of the script. ``` $ echo "def foo" | ruby -:1: syntax error, unexpected end-of-input def foo ```

Adds the `dead_end` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. close ruby#4845 ## What is dead_end? When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem. ## Demo Generate a file with a syntax error: ``` $ echo "require_relative 'syntax_error'" > test.rb $ cat > syntax_error.rb <<EOF class Dog < Animal def sit super end def bark puts "woof" def happy tail.wag end end EOF ``` Build and run ruby: ``` $ make $ make runruby generating vm_call_iseq_optimized.inc vm_call_iseq_optimized.inc unchanged RUBY_ON_BUG='gdb -x ./.gdbinit -p' ./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems ./test.rb Run `$ dead_end /Users/rschneeman/Documents/projects/ruby/syntax_error.rb` for more options DeadEnd: Missing `end` detected This code has a missing `end`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/ruby/syntax_error.rb simplified: 1 class Dog < Animal ❯ 6 def bark ❯ 9 def happy ❯ 11 end 12 end /Users/rschneeman/Documents/projects/ruby/lib/dead_end/auto.rb:30:in `require': /Users/rschneeman/Documents/projects/ruby/syntax_error.rb:12: syntax error, unexpected end-of-input, expecting `end' (SyntaxError) ``` ## Known issues - DeadEnd's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations ruby/syntax_suggest#31. - We could extend or modify the Ruby API to hook into a better place with guidance and approval.

When there is a syntax error inside of an eval, but the parent code is valid a syntax error is raised inside of the eval instead of inside the parent code. When this happens Ruby thinks the filename is `(eval)` but this does not exist on disk so dead end cannot read it. Here's the current behavior: ``` $ cat /tmp/scratch.rb $stderr = STDOUT eval("\"\xf0".force_encoding("utf-8")) $ ruby -I./lib -rdead_end /tmp/scratch.rb DeadEnd: Could not find filename from "(eval):1: invalid multibyte char (UTF-8) (SyntaxError)\n" /tmp/scratch.rb:2:in `eval': (eval):1: invalid multibyte char (UTF-8) (SyntaxError) (eval):1: unterminated string meets end of file from /tmp/scratch.rb:2:in `<main>' ``` Since we will never be able to read `(eval):1` we can special case this "file name" and skip emitting the warning. Same problem with code that is streamed to ruby: ``` $ echo "def foo" | ruby -rdead_end ``` However we're not currently capturing this error due to #31. You can still get a warning by using `DEBUG=1` environment variable for these cases in case there is a bug in this behavior and we need to debug it.

Adds the `dead_end` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. close ruby#4845 ## What is dead_end? When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem. ## Demo Generate a file with a syntax error: ``` $ echo "require_relative 'syntax_error'" > test.rb $ cat > syntax_error.rb <<EOF class Dog < Animal def sit super end def bark puts "woof" def happy tail.wag end end EOF ``` Build and run ruby: ``` $ make $ make runruby generating vm_call_iseq_optimized.inc vm_call_iseq_optimized.inc unchanged RUBY_ON_BUG='gdb -x ./.gdbinit -p' ./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems ./test.rb Run `$ dead_end /Users/rschneeman/Documents/projects/ruby/syntax_error.rb` for more options DeadEnd: Missing `end` detected This code has a missing `end`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/ruby/syntax_error.rb simplified: 1 class Dog < Animal ❯ 6 def bark ❯ 9 def happy ❯ 11 end 12 end /Users/rschneeman/Documents/projects/ruby/lib/dead_end/auto.rb:30:in `require': /Users/rschneeman/Documents/projects/ruby/syntax_error.rb:12: syntax error, unexpected end-of-input, expecting `end' (SyntaxError) ``` ## Known issues - DeadEnd's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations ruby/syntax_suggest#31. - We could extend or modify the Ruby API to hook into a better place with guidance and approval.

Adds the `syntax_suggest` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. Reference ticket: https://bugs.ruby-lang.org/issues/18159 close ruby#4845 ## What is syntax_suggest? When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem. > Note: Previously this tool was named `dead_end`. ## Known issues - SyntaxSearch's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations ruby/syntax_suggest#31. - This causes failure in the test suite (test_expected_backtrace_location_when_inheriting_from_basic_object_and_including_kernel) and confusion when inspecting backtraces if there's a different error when trying to require a file such as measuring memory (ruby/syntax_suggest#124 (comment)). - Discussed fix. We previously talked about opening up `SyntaxError` to be monkeypatched in the same way that other gems hook into `NoMethodError`. This is currently not possible and requires development work. When we last talked about it at RubyKaigi Nobu expressed an ability to make such a change.

Adds the `syntax_suggest` syntax error display tool to Ruby through the same mechanism as `error_highlight` and `did_you_mean`. Reference ticket: https://bugs.ruby-lang.org/issues/18159 close #4845 ## What is syntax_suggest? When a syntax error is raised by requiring a file, dead_end will use a combination of indentation and lexing to identify the problem. > Note: Previously this tool was named `dead_end`. ## Known issues - SyntaxSearch's approach of showing syntax errors only works through integration with `require`, `load`, `autoload`, and `require_relative` (since it monkeypatches them to detect syntax errors). It does not work with direct Ruby file invocations ruby/syntax_suggest#31. - This causes failure in the test suite (test_expected_backtrace_location_when_inheriting_from_basic_object_and_including_kernel) and confusion when inspecting backtraces if there's a different error when trying to require a file such as measuring memory (ruby/syntax_suggest#124 (comment)). - Discussed fix. We previously talked about opening up `SyntaxError` to be monkeypatched in the same way that other gems hook into `NoMethodError`. This is currently not possible and requires development work. When we last talked about it at RubyKaigi Nobu expressed an ability to make such a change.

schneems · 2022-12-30T03:07:42Z

Works with Ruby 3.2 now. Closing

schneems mentioned this issue Sep 15, 2021

Spike adding dead_end to Ruby ruby/ruby#4845

Closed

schneems mentioned this issue Apr 28, 2022

Add syntax_suggest as default gem ruby/ruby#5859

Merged

schneems mentioned this issue May 18, 2022

Annotate syntax error without require #139

Merged

schneems mentioned this issue May 24, 2022

Stop warning when on eval & streaming code #143

Merged

schneems closed this as completed Dec 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Doesn't work direct from Ruby CLI #31

Doesn't work direct from Ruby CLI #31

Fryguy commented Dec 10, 2020

schneems commented Dec 11, 2020 •

edited

Loading

schneems commented Dec 30, 2022

Doesn't work direct from Ruby CLI #31

Doesn't work direct from Ruby CLI #31

Comments

Fryguy commented Dec 10, 2020

schneems commented Dec 11, 2020 • edited Loading

schneems commented Dec 30, 2022

schneems commented Dec 11, 2020 •

edited

Loading