[GR-37533] Merge master into 24.0 release branch for feature freeze

PullRequest: truffleruby/4144

Author: eregon

3rd_party_licenses.txt

@@ -1308,7 +1308,7 @@ of
 */
 ================================================================================
 
-ffi 1.14.2
+ffi 1.15.5
 
 Copyright (c) 2008-2016, Ruby FFI project contributors
 All rights reserved.

CHANGELOG.md

@@ -3,6 +3,8 @@
 New features:
 
 * C/C++ extensions are now compiled using the system toolchain and executed natively instead of using GraalVM LLVM (Sulong). This leads to faster startup, no warmup, better compatibility, smaller distribution and faster installation for C/C++ extensions (#3118, @eregon).
+* Full suport for the Ruby 3.2 and Ruby 3.3 syntax by adopting the [Prism](https://github.com/ruby/prism) parser, which is about twice as fast as the old parser (#3117, #3038, #3039, @andrykonchin, @eregon).
+* Pattern matching is now fully supported, with the exception of Find pattern (`in [*, a, *]`) (#3332, #2683, @eregon, @razetime).
 
 Bug fixes:
 
@@ -12,6 +14,13 @@ Bug fixes:
 * Fix `Proc#parameters` and return all the numbered parameters lower than the used explicitly ones (@andrykonchin).
 * Fix some C API functions which were failing when called with Ruby values represented as Java primitives (#3352, @eregon).
 * Fix `IO.select([io], nil, [io])` on macOS, it was hanging due to a bug in macOS `poll(2)` (#3346, @eregon, @andrykonchin).
+* Run context cleanup such as showing the output of tools when `SignalException` and `Interrupt` escape (@eregon).
+* Handle a new variable inside the `case` target expression correctly (#3377, @eregon).
+* The arguments of `Thread.new(*args, &block)` need to be marked as shared between multiple threads (#3179, @eregon).
+* Fix `Range#bsearch` and raise `TypeError` when range boundaries are non-numeric and block not passed (@andrykonchin).
+* Fix using the `--cpusampler` profiler when there are custom unblock functions for `rb_thread_call_without_gvl()` (#3013, @eregon).
+* Fix recursive raising `FrozenError` exception when redefined `#inspect` modifies an object (#3388, @andrykonchin).
+* Fix `Integer#div` returning the wrong object type when the divisor is a `Rational` (@simonlevasseur, @nirvdrum).
 
 Compatibility:
 
@@ -21,6 +30,21 @@ Compatibility:
 * Promote `File#path` and `File#to_path` to `IO#path` and `IO#to_path` and make IO#new accept an optional `path:` keyword argument (#3039, @moste00)
 * Display "unhandled exception" as the message for `RuntimeError` instances with an empty message (#3255, @nirvdrum).
 * Set `RbConfig::CONFIG['configure_args']` for openssl and libyaml (#3170, #3303, @eregon).
+* Support `Socket.sockaddr_in(port, Socket::INADDR_ANY)` (#3361, @mtortonesi).
+* Implement the `Data` class from Ruby 3.2 (#3039, @moste00, @eregon).
+* Make `Coverage.start` and `Coverage.result` accept parameters (#3149, @mtortonesi, @andrykonchin).
+* Implement `rb_check_funcall()` (@eregon).
+* Implement `MatchData#{byteoffset,deconstruct,deconstruct_keys}` from Ruby 3.2 (#3039, @rwstauner).
+* Add `Integer#ceildiv` method (#3039, @simonlevasseur, @nirvdrum).
+* Implement `Class#attached_object` method (#3039, @andrykonchin).
+* Fix `ENV#{clone,dup}` and raise `TypeError` (#3039, @andrykonchin).
+* Fix `Coverage.supported?` and raise `TypeError` if argument is not Symbol (#3039, @andrykonchin).
+* Accept options argument to `Regexp.{new,compile}` of String and warn for unknown types (#3039, @rwstauner).
+* Implement `Time#deconstruct_keys` from Ruby 3.2 (#3039, @rwstauner).
+* Do not autosplat a proc that accepts a single positional argument and keywords (#3039, @andrykonchin).
+* Support passing anonymous * and ** parameters as method call arguments (#3039, @andrykonchin).
+* Handle either positional or keywords arguments by default in `Struct.new` (#3039, @rwstauner).
+* Promote `Set` class to core library (#3039, @andrykonchin).
 
 Performance:
 

LICENCE.md

@@ -1,6 +1,6 @@
 # TruffleRuby Licence
 
-TruffleRuby is copyright (c) 2013-2023 Oracle and/or its affiliates, and is made
+TruffleRuby is copyright (c) 2013-2024 Oracle and/or its affiliates, and is made
 available to you under the terms of any one of the following three licenses:
 
 * Eclipse Public License version 2.0, or

README.md

@@ -90,7 +90,7 @@ TruffleRuby is actively tested on the following systems:
 * Ubuntu 16.04, 18.04, 20.04, 22.04 (all LTS)
 * Fedora 37, 38
 * Debian 10, 11, 12
-* macOS 10.14 (Mojave) for amd64, 11 (Big Sur) for aarch64
+* macOS 11 (Big Sur)
 
 Architectures:
 
@@ -182,7 +182,7 @@ For known vulnerabilities in Ruby, please refer to the [known-cves](doc/user/kno
 
 ## Licence
 
-TruffleRuby is copyright (c) 2013-2023 Oracle and/or its affiliates, and is made
+TruffleRuby is copyright (c) 2013-2024 Oracle and/or its affiliates, and is made
 available to you under the terms of any one of the following three licenses:
 
 * Eclipse Public License version 2.0, or

ci.jsonnet

@@ -251,7 +251,7 @@ local part_definitions = {
       platform_name:: "DarwinAMD64",
       "$.cap":: {
         # GR-45839, GR-46279: exclude macmini_late_2014_8gb, they are too slow, have too little RAM and cause various timeouts
-        normal_machine: ["darwin_mojave", "!macmini_late_2014_8gb"],
+        normal_machine: ["darwin_bigsur", "!macmini_late_2014_8gb"],
       },
       environment+: {
         LANG: "en_US.UTF-8",
@@ -521,14 +521,14 @@ local composition_environment = utils.add_inclusion_tracking(part_definitions, "
       "ruby-test-specs-linux-amd64-stable":    $.platform.linux  + $.jdk.stable + $.env.jvm + gate_no_build + $.use.build + $.run.test_unit_tck + native_config + $.run.test_specs + { timelimit: "01:20:00" },
       "ruby-test-specs-linux-amd64-latest":    $.platform.linux  + $.jdk.latest + $.env.jvm + gate_no_build + $.use.build + $.run.test_unit_tck + native_config + $.run.test_specs + { timelimit: "01:20:00" },
       "ruby-test-specs-darwin-amd64-stable":   $.platform.darwin_amd64 + $.jdk.stable + $.env.jvm + gate_no_build + $.use.build + $.run.test_unit_tck + native_config + $.run.test_specs + { timelimit: "01:40:00" },
-      # "ruby-test-specs-darwin-amd64-latest":   $.platform.darwin_amd64 + $.jdk.latest + $.env.jvm + gate_no_build + $.use.build + $.run.test_unit_tck + native_config + $.run.test_specs + { timelimit: "01:40:00" }, # GR-49652
+      "ruby-test-specs-darwin-amd64-latest":   $.platform.darwin_amd64 + $.jdk.latest + $.env.jvm + gate_no_build + $.use.build + $.run.test_unit_tck + native_config + $.run.test_specs + { timelimit: "01:40:00" },
       "ruby-test-specs-darwin-aarch64-stable": $.platform.darwin_aarch64 + $.jdk.stable + $.env.jvm + gate_no_build + $.use.build + $.run.test_unit_tck + native_config + $.run.test_specs + { timelimit: "01:40:00" },
       "ruby-test-specs-darwin-aarch64-latest": $.platform.darwin_aarch64 + $.jdk.latest + $.env.jvm + gate_no_build + $.use.build + $.run.test_unit_tck + native_config + $.run.test_specs + { timelimit: "01:40:00" },
       "ruby-test-fast-linux-aarch64":   $.platform.linux_aarch64 + $.jdk.stable + $.env.jvm + gate + $.run.test_fast + native_config + { timelimit: "45:00" },
       "ruby-test-fast-linux-amd64":     $.platform.linux  + $.jdk.stable + $.env.jvm + gate + $.run.test_fast + { timelimit: "45:00" },  # To catch missing slow tags
-      "ruby-test-mri-asserts":          $.platform.linux  + $.jdk.stable + $.env.jvm + gate + $.run.test_mri_fast + { timelimit: "01:30:00" },
+      "ruby-test-mri-asserts":          $.platform.linux  + $.jdk.stable + $.env.jvm + gate + $.run.test_mri_fast + { timelimit: "01:15:00" },
       "ruby-test-mri-linux-amd64":      $.platform.linux  + $.jdk.stable + $.env.native + gate + $.run.test_mri + { timelimit: "01:20:00" },
-      "ruby-test-mri-linux-aarch64":    $.platform.linux_aarch64 + $.jdk.stable + $.env.native + gate + $.run.test_mri + { timelimit: "01:20:00" },
+      # "ruby-test-mri-linux-aarch64":  $.platform.linux_aarch64 + $.jdk.stable + $.env.native + gate + $.run.test_mri + { timelimit: "01:30:00" }, # GR-51361
       "ruby-test-mri-darwin-amd64":     $.platform.darwin_amd64 + $.jdk.stable + $.env.native + gate + $.run.test_mri + { timelimit: "01:30:00" },
       "ruby-test-mri-darwin-aarch64":   $.platform.darwin_aarch64 + $.jdk.stable + $.env.native + gate + $.run.test_mri + { timelimit: "01:30:00" },
       "ruby-test-integration-linux-amd64": $.platform.linux  + $.jdk.stable + $.env.jvm + gate + $.run.test_integration,
@@ -551,7 +551,7 @@ local composition_environment = utils.add_inclusion_tracking(part_definitions, "
       "ruby-test-svm-ce-linux-amd64-stable":    $.platform.linux          + $.jdk.stable + $.env.native    + $.env.gdb_svm + gate + native_tests + $.env.host_inlining_log,
       "ruby-test-svm-ce-linux-amd64-latest":    $.platform.linux          + $.jdk.latest + $.env.native    + $.env.gdb_svm + gate + native_tests,
       "ruby-test-svm-ce-darwin-amd64-stable":   $.platform.darwin_amd64   + $.jdk.stable + $.env.native    + $.env.gdb_svm + gate + native_tests,
-      # "ruby-test-svm-ce-darwin-amd64-latest":   $.platform.darwin_amd64   + $.jdk.latest + $.env.native    + $.env.gdb_svm + gate + native_tests, # GR-49652
+      "ruby-test-svm-ce-darwin-amd64-latest":   $.platform.darwin_amd64   + $.jdk.latest + $.env.native    + $.env.gdb_svm + gate + native_tests,
       "ruby-test-svm-ce-darwin-aarch64-stable": $.platform.darwin_aarch64 + $.jdk.stable + $.env.native    +                 gate + native_tests,
       "ruby-test-svm-ce-darwin-aarch64-latest": $.platform.darwin_aarch64 + $.jdk.latest + $.env.native    +                 gate + native_tests,
       "ruby-test-svm-ee-linux-amd64":    $.platform.linux          + $.jdk.stable + $.env.native_ee + $.env.gdb_svm + gate + native_tests + $.env.host_inlining_log + { timelimit: "01:30:00" },

ci/common.jsonnet

@@ -344,7 +344,8 @@ local common_json = import "../common.json";
     },
 
     local linux   = { os:: "linux",   capabilities+: [self.os] },
-    local darwin  = { os:: "darwin",  capabilities+: [self.os] },
+    # Run darwin jobs on Big Sur or later by excluding all older versions
+    local darwin  = { os:: "darwin",  capabilities+: [self.os, "!darwin_sierra", "!darwin_mojave", "!darwin_catalina"] },
     local windows = { os:: "windows", capabilities+: [self.os] },
 
     local amd64   = { arch:: "amd64",   capabilities+: [self.arch] },

common.json

@@ -44,13 +44,13 @@
     "labsjdk-ee-21Debug": {"name": "labsjdk",   "version": "ee-21.0.1+11-jvmci-23.1-b26-debug", "platformspecific": true },
     "labsjdk-ee-21-llvm": {"name": "labsjdk",   "version": "ee-21.0.1+11-jvmci-23.1-b26-sulong", "platformspecific": true },
 
-    "oraclejdk-latest":       {"name": "jpg-jdk",   "version": "22",      "build_id": "27", "release": true, "platformspecific": true, "extrabundles": ["static-libs"]},
-    "labsjdk-ce-latest":      {"name": "labsjdk",   "version": "ce-22+27-jvmci-b01", "platformspecific": true },
-    "labsjdk-ce-latestDebug": {"name": "labsjdk",   "version": "ce-22+27-jvmci-b01-debug", "platformspecific": true },
-    "labsjdk-ce-latest-llvm": {"name": "labsjdk",   "version": "ce-22+27-jvmci-b01-sulong", "platformspecific": true },
-    "labsjdk-ee-latest":      {"name": "labsjdk",   "version": "ee-22+27-jvmci-b01", "platformspecific": true },
-    "labsjdk-ee-latestDebug": {"name": "labsjdk",   "version": "ee-22+27-jvmci-b01-debug", "platformspecific": true },
-    "labsjdk-ee-latest-llvm": {"name": "labsjdk",   "version": "ee-22+27-jvmci-b01-sulong", "platformspecific": true }
+    "oraclejdk-latest":       {"name": "jpg-jdk",   "version": "22",      "build_id": "33", "release": true, "platformspecific": true, "extrabundles": ["static-libs"]},
+    "labsjdk-ce-latest":      {"name": "labsjdk",   "version": "ce-22+33-jvmci-b01", "platformspecific": true },
+    "labsjdk-ce-latestDebug": {"name": "labsjdk",   "version": "ce-22+33-jvmci-b01-debug", "platformspecific": true },
+    "labsjdk-ce-latest-llvm": {"name": "labsjdk",   "version": "ce-22+33-jvmci-b01-sulong", "platformspecific": true },
+    "labsjdk-ee-latest":      {"name": "labsjdk",   "version": "ee-22+33-jvmci-b01", "platformspecific": true },
+    "labsjdk-ee-latestDebug": {"name": "labsjdk",   "version": "ee-22+33-jvmci-b01-debug", "platformspecific": true },
+    "labsjdk-ee-latest-llvm": {"name": "labsjdk",   "version": "ee-22+33-jvmci-b01-sulong", "platformspecific": true }
   },
 
   "eclipse": {

doc/contributor/how-to-guide.md

@@ -1139,24 +1139,6 @@ jt -q ruby -e 'puts Truffle::Debug.java_class_of([])'
 RubyArray
 ```
 
-### Truffle::Debug.yarp_serialize
-
-```
-jt -q ruby -e 'puts Truffle::Debug.yarp_serialize("1").dump'
-"YARP\x00\x04\x00@7\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00L\f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00S\x19\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00&\b\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00"
-```
-
-### Truffle::Debug.yarp_parse
-
-```
-jt -q ruby -e 'puts Truffle::Debug.yarp_parse("[].to_ary")'
-ProgramNode
-  Scope
-  StatementsNode
-    CallNode
-      ArrayNode
-```
-
 ### Truffle::Debug.ast
 
 ```

doc/contributor/interop_details.md

@@ -78,6 +78,28 @@
 
 # Behavior of interop messages for Ruby objects
 
+## toDisplayString()
+
+When interop message `toDisplayString` is sent
+- to 
+  it returns a String representing the object.
+- otherwise
+  it returns the same as `subject.inspect`.
+
+## MetaObject related messages
+
+When interop message `hasMetaObject` is sent
+- to **`nil`**, **`:symbol`**, **a `String`**, **a `BigDecimal`**, **an `Object`**, **a frozen `Object`**, **a `StructWithValue`**, **a `Class`**, **a `Hash`**, **an `Array`**, **an `Exception`**, **an `Exception` with a cause**, **`proc {...}`**, **`lambda {...}`**, **a `Method`**, **a `Truffle::FFI::Pointer`**, **polyglot pointer**, **polyglot members**, **polyglot array** or **polyglot hash**
+  it returns true.
+- otherwise
+  it returns false.
+
+When interop message `getMetaObject` is sent
+- to 
+  it returns the Ruby exception's class.
+- otherwise
+  it returns the same Ruby class as `subject.class`.
+
 ## `null` related messages
 
 When interop message `isNull` is sent
@@ -329,7 +351,7 @@ When interop message `writeMember` is sent
   it writes the given value under the given name.
 - to **a `StructWithValue`**
   it writes the value to the given struct member.
-- to **a `BigDecimal`**, **`nil`**, **`:symbol`** or **a frozen `Object`**
+- to **`nil`**, **`false`**, **`true`**, **`:symbol`**, **an `Integer`**, **a `Float`**, **a `BigDecimal`** or **a frozen `Object`**
   it fails with `UnsupportedMessageError` when the receiver is frozen.
 - otherwise
   it fails with `UnsupportedMessageError`.

doc/contributor/parser.md

@@ -8,14 +8,74 @@
 
 ## Print Detailed Prism AST
 
+### From the Prism repository
+
 ```bash
-cd prism
-bundle exec rake
-bin/parse -e '1&.itself'
+$ cd prism
+$ bundle exec rake compile
+$ bin/parse -e '1&.itself'
+@ ProgramNode (location: (1,0)-(1,9))
+├── locals: []
+└── statements:
+    @ StatementsNode (location: (1,0)-(1,9))
+    └── body: (length: 1)
+        └── @ CallNode (location: (1,0)-(1,9))
+            ├── flags: safe_navigation
+            ├── receiver:
+            │   @ IntegerNode (location: (1,0)-(1,1))
+            │   └── flags: decimal
+            ├── call_operator_loc: (1,1)-(1,3) = "&."
+            ├── name: :itself
+            ├── message_loc: (1,3)-(1,9) = "itself"
+            ├── opening_loc: ∅
+            ├── arguments: ∅
+            ├── closing_loc: ∅
+            └── block: ∅
 ```
 
+### From TruffleRuby
+
 We can also see what the AST as Java nodes and without extra location fields looks like on TruffleRuby with:
 ```bash
-cd truffleruby
-jt -q ruby -e 'puts Truffle::Debug.yarp_parse(ARGV[0])' -- '1&.itself'
+$ cd truffleruby
+$ jt -q ruby tool/parse_ast.rb '1&.itself'
+Source:
+1&.itself
+
+AST:
+ProgramNode
+  locals:
+  statements: StatementsNode
+    body:
+      CallNode[Li]
+        flags: 1
+        receiver: IntegerNode
+          flags: 2
+        name: "itself"
+        arguments: null
+        block: null
+```
+
+```bash
+$ jt -q ruby tool/parse_ast.rb some_file.rb
+```
+
+You can also compare to JRuby's AST with:
+```bash
+$ jruby tool/parse_ast.rb '1&.itself'
+Source:
+1&.itself
+
+AST:
+RootNode line: 0
+  CallNode*[lazy]:itself line: 0
+    FixnumNode line: 0, long: 1
+, null, null
+```
+
+## Print the Truffle AST
+
+```bash
+$ cd truffleruby
+$ jt -q ruby tool/truffle_ast.rb '1&.itself'
 ```

doc/contributor/prism.md

@@ -321,9 +321,3 @@ file to fix it.
 
 You can also recompute the tags automatically for an entire test file with
 `jt retag test/mri/tests/file.rb`.
-
-## Building the parser
-
-TruffleRuby uses the Jay parser generator. A copy of this is located in
-`tool/jay`. The command `jt build parser` will build Jay, if needed, and then
-regenerate the parser. We check the generated parser into the source repository.

doc/contributor/updating-jruby-parser.md

@@ -5,7 +5,7 @@ which are the only supported ways to use TruffleRuby.
 
 ## TruffleRuby
 
-TruffleRuby is copyright (c) 2013-2023 Oracle and/or its
+TruffleRuby is copyright (c) 2013-2024 Oracle and/or its
 affiliates, and is made available to you under the terms of any one of the
 following three licenses:
 
@@ -165,9 +165,14 @@ files.
 
 #### FFI
 
-TruffleRuby includes parts of the FFI gem 1.14.2. The FFI gem is copyright
-2008-2016, Ruby FFI project contributors, and covered by the three-clause BSD
-licence (see `ffi.txt`).
+TruffleRuby includes parts of the FFI gem (version as described in [lib/truffle/ffi/version.rb](../../lib/truffle/ffi/version.rb)).
+The FFI gem is copyright  2008-2016, Ruby FFI project contributors, and covered by the three-clause BSD licence (see `ffi.txt`).
+
+#### Prism
+
+TruffleRuby uses the [Prism](https://github.com/ruby/prism) Ruby parser
+(version as described in [src/main/c/yarp/include/prism/version.h](../../src/main/c/yarp/include/prism/version.h)),
+copyright Shopify Inc. and is available under an MIT licence (see `src/main/c/yarp/LICENSE.md`).
 
 # Java dependencies
 
@@ -203,14 +208,6 @@ Engine Yard and are released under an MIT licence (see `mit.txt`).
 We do not distribute the FFI Specs, but they are copyright 2008-2014
 Ruby-FFI contributors and are released under an MIT licence (see `mit.txt`).
 
-## Jay
-
-TruffleRuby uses the Jay parser generator, modified from
-https://github.com/jruby/jay revision `9ffc59a`. Jay is copyright 1989 The
-Regents of the University of California, ported by Axel T. Schreiner, and is
-covered by the three-clause BSD licence (see `jay.txt`). We only distribute Jay
-in the source repository - it isn't part of releases.
-
 ## Written offer for source code
 
 For any software that you receive from Oracle in binary form which is licensed