Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Trim non-ASCII characters #44

Merged
merged 1 commit into from
Jul 18, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions assets/sig/generated/actionpack.rbs
Original file line number Diff line number Diff line change
Expand Up @@ -4228,7 +4228,7 @@ module ActionController
# The primary options are:
# * <tt>:partial</tt> - See <tt>ActionView::PartialRenderer</tt> for details.
# * <tt>:file</tt> - Renders an explicit template file. Add <tt>:locals</tt> to pass in, if so desired.
# It shouldnt be used directly with unsanitized user input due to lack of validation.
# It shouldn(trim non-ascii characters)t be used directly with unsanitized user input due to lack of validation.
# * <tt>:inline</tt> - Renders an ERB template string.
# * <tt>:plain</tt> - Renders provided text and sets the content type as <tt>text/plain</tt>.
# * <tt>:html</tt> - Renders the provided HTML safe string, otherwise
Expand Down Expand Up @@ -10166,7 +10166,7 @@ module ActionDispatch
#
# You can specify unicode character routes in your router:
#
# get "こんにちは", to: "welcome#index"
# get "(trim non-ascii characters)", to: "welcome#index"
#
# == Routing to Rack Applications
#
Expand Down
6 changes: 3 additions & 3 deletions assets/sig/generated/actionview.rbs
Original file line number Diff line number Diff line change
Expand Up @@ -6108,7 +6108,7 @@ module ActionView
# number_to_currency(1234567890.50) # => $1,234,567,890.50
# number_to_currency(1234567890.506) # => $1,234,567,890.51
# number_to_currency(1234567890.506, precision: 3) # => $1,234,567,890.506
# number_to_currency(1234567890.506, locale: :fr) # => 1 234 567 890,51
# number_to_currency(1234567890.506, locale: :fr) # => 1 234 567 890,51 (trim non-ascii characters)
# number_to_currency("123a456") # => $123a456
#
# number_to_currency("123a456", raise: true) # => InvalidNumberError
Expand Down Expand Up @@ -9563,10 +9563,10 @@ module ActionView
#
# The view class must have the following methods:
#
# * <tt>View.new(lookup_context, assigns, controller)</tt> Create a new
# * <tt>View.new(lookup_context, assigns, controller)</tt> (trim non-ascii characters) Create a new
# ActionView instance for a controller and we can also pass the arguments.
#
# * <tt>View#render(option)</tt> Returns String with the rendered template.
# * <tt>View#render(option)</tt> (trim non-ascii characters) Returns String with the rendered template.
#
# Override this method in a module to change the default behavior.
def view_context: () -> untyped
Expand Down
12 changes: 6 additions & 6 deletions assets/sig/generated/activerecord.rbs
Original file line number Diff line number Diff line change
Expand Up @@ -19061,14 +19061,14 @@ module ActiveRecord
# Finds the first record with the given attributes, or creates a record
# with the attributes if one is not found:
#
# # Find the first user named "Penélope" or create a new one.
# User.find_or_create_by(first_name: 'Penélope')
# # => #<User id: 1, first_name: "Penélope", last_name: nil>
# # Find the first user named "Pen(trim non-ascii characters)lope" or create a new one.
# User.find_or_create_by(first_name: 'Pen(trim non-ascii characters)lope')
# # => #<User id: 1, first_name: "Pen(trim non-ascii characters)lope", last_name: nil>
#
# # Find the first user named "Penélope" or create a new one.
# # Find the first user named "Pen(trim non-ascii characters)lope" or create a new one.
# # We already have one so the existing record will be returned.
# User.find_or_create_by(first_name: 'Penélope')
# # => #<User id: 1, first_name: "Penélope", last_name: nil>
# User.find_or_create_by(first_name: 'Pen(trim non-ascii characters)lope')
# # => #<User id: 1, first_name: "Pen(trim non-ascii characters)lope", last_name: nil>
#
# # Find the first user named "Scarlett" or create a new one with
# # a particular last name.
Expand Down
94 changes: 47 additions & 47 deletions assets/sig/generated/activesupport.rbs
Original file line number Diff line number Diff line change
Expand Up @@ -917,8 +917,8 @@ module ActiveSupport
# Option Class Result
# :redis Proc -> options[:redis].call
# :redis Object -> options[:redis]
# :url String -> Redis.new(url: )
# :url Array -> Redis::Distributed.new([{ url: }, { url: }, ])
# :url String -> Redis.new(url: (trim non-ascii characters))
# :url Array -> Redis::Distributed.new([{ url: (trim non-ascii characters) }, { url: (trim non-ascii characters) }, (trim non-ascii characters)])
#
# No namespace is set by default. Provide one if the Redis cache
# server is shared with other apps: <tt>namespace: 'myapp-cache'</tt>.
Expand Down Expand Up @@ -4217,7 +4217,7 @@ module ActiveSupport
# 1234567890.50.to_s(:currency) # => "$1,234,567,890.50"
# 1234567890.506.to_s(:currency) # => "$1,234,567,890.51"
# 1234567890.506.to_s(:currency, precision: 3) # => "$1,234,567,890.506"
# 1234567890.506.to_s(:currency, locale: :fr) # => "1 234 567 890,51 "
# 1234567890.506.to_s(:currency, locale: :fr) # => "1 234 567 890,51 (trim non-ascii characters)"
# -1234567890.50.to_s(:currency, negative_format: '(%u%n)')
# # => "($1,234,567,890.50)"
# 1234567890.50.to_s(:currency, unit: '&pound;', separator: ',', delimiter: '')
Expand Down Expand Up @@ -5167,15 +5167,15 @@ class String
# grapheme clusters ("perceptual characters") by truncating at combining
# characters.
#
# >> "🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪".size
# >> "(trim non-ascii characters)".size
# => 20
# >> "🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪".bytesize
# >> "(trim non-ascii characters)".bytesize
# => 80
# >> "🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪🔪".truncate_bytes(20)
# => "🔪🔪🔪🔪…"
# >> "(trim non-ascii characters)".truncate_bytes(20)
# => "(trim non-ascii characters)"
#
# The truncated text ends with the <tt>:omission</tt> string, defaulting
# to "", for a total length not exceeding <tt>bytesize</tt>.
# to "(trim non-ascii characters)", for a total length not exceeding <tt>bytesize</tt>.
def truncate_bytes: (untyped truncate_at, ?omission: ::String omission) -> untyped

# Truncates a given +text+ after a given number of words (<tt>words_count</tt>):
Expand Down Expand Up @@ -5461,13 +5461,13 @@ class String
# encapsulates the original string. A Unicode safe version of all the String methods are defined on this proxy
# class. If the proxy class doesn't respond to a certain method, it's forwarded to the encapsulated string.
#
# >> "lj".mb_chars.upcase.to_s
# => "LJ"
# >> "(trim non-ascii characters)".mb_chars.upcase.to_s
# => "(trim non-ascii characters)"
#
# NOTE: Ruby 2.4 and later support native Unicode case mappings:
#
# >> "lj".upcase
# => "LJ"
# >> "(trim non-ascii characters)".upcase
# => "(trim non-ascii characters)"
#
# == Method chaining
#
Expand Down Expand Up @@ -6529,7 +6529,7 @@ module ActiveSupport
#
# PLANETS.map { |planet| planet.capitalize }
# # => DEPRECATION WARNING: PLANETS is deprecated! Use PLANETS_POST_2006 instead.
# (Backtrace information)
# (Backtrace information(trim non-ascii characters))
# ["Mercury", "Venus", "Earth", "Mars", "Jupiter", "Saturn", "Uranus", "Neptune"]
module DeprecatedConstantAccessor
def self.included: (untyped base) -> untyped
Expand Down Expand Up @@ -6670,7 +6670,7 @@ module ActiveSupport
# example.old_request.to_s
# # => DEPRECATION WARNING: @request is deprecated! Call request.to_s instead of
# @request.to_s
# (Backtrace information)
# (Backtrace information(trim non-ascii characters))
# "special_request"
#
# example.request.to_s
Expand Down Expand Up @@ -6698,7 +6698,7 @@ module ActiveSupport
#
# PLANETS.map { |planet| planet.capitalize }
# # => DEPRECATION WARNING: PLANETS is deprecated! Use PLANETS_POST_2006 instead.
# (Backtrace information)
# (Backtrace information(trim non-ascii characters))
# ["Mercury", "Venus", "Earth", "Mars", "Jupiter", "Saturn", "Uranus", "Neptune"]
class DeprecatedConstantProxy < Module
def self.new: (*untyped args, **untyped kwargs) { () -> untyped } -> untyped
Expand Down Expand Up @@ -8167,15 +8167,15 @@ module ActiveSupport
# Replaces non-ASCII characters with an ASCII approximation, or if none
# exists, a replacement character which defaults to "?".
#
# transliterate('Ærøskøbing')
# transliterate('(trim non-ascii characters)r(trim non-ascii characters)sk(trim non-ascii characters)bing')
# # => "AEroskobing"
#
# Default approximations are provided for Western/Latin characters,
# e.g, "ø", "ñ", "é", "ß", etc.
# e.g, "(trim non-ascii characters)", "(trim non-ascii characters)", "(trim non-ascii characters)", "(trim non-ascii characters)", etc.
#
# This method is I18n aware, so you can set up custom approximations for a
# locale. This can be useful, for example, to transliterate German's "ü"
# and "ö" to "ue" and "oe", or to add support for transliterating Russian
# locale. This can be useful, for example, to transliterate German's "(trim non-ascii characters)"
# and "(trim non-ascii characters)" to "ue" and "oe", or to add support for transliterating Russian
# to ASCII.
#
# In order to make your custom transliterations available, you must set
Expand All @@ -8185,15 +8185,15 @@ module ActiveSupport
# i18n:
# transliterate:
# rule:
# ü: "ue"
# ö: "oe"
# (trim non-ascii characters): "ue"
# (trim non-ascii characters): "oe"
#
# # Or set them using Ruby
# I18n.backend.store_translations(:de, i18n: {
# transliterate: {
# rule: {
# 'ü' => 'ue',
# 'ö' => 'oe'
# '(trim non-ascii characters)' => 'ue',
# '(trim non-ascii characters)' => 'oe'
# }
# }
# })
Expand All @@ -8210,10 +8210,10 @@ module ActiveSupport
#
# Now you can have different transliterations for each locale:
#
# transliterate('Jürgen', locale: :en)
# transliterate('J(trim non-ascii characters)rgen', locale: :en)
# # => "Jurgen"
#
# transliterate('Jürgen', locale: :de)
# transliterate('J(trim non-ascii characters)rgen', locale: :de)
# # => "Juergen"
#
# Transliteration is restricted to UTF-8, US-ASCII and GB18030 strings
Expand All @@ -8224,23 +8224,23 @@ module ActiveSupport
# a 'pretty' URL.
#
# parameterize("Donald E. Knuth") # => "donald-e-knuth"
# parameterize("^très|Jolie-- ") # => "tres-jolie"
# parameterize("^tr(trim non-ascii characters)s|Jolie-- ") # => "tres-jolie"
#
# To use a custom separator, override the +separator+ argument.
#
# parameterize("Donald E. Knuth", separator: '_') # => "donald_e_knuth"
# parameterize("^très|Jolie__ ", separator: '_') # => "tres_jolie"
# parameterize("^tr(trim non-ascii characters)s|Jolie__ ", separator: '_') # => "tres_jolie"
#
# To preserve the case of the characters in a string, use the +preserve_case+ argument.
#
# parameterize("Donald E. Knuth", preserve_case: true) # => "Donald-E-Knuth"
# parameterize("^très|Jolie-- ", preserve_case: true) # => "tres-Jolie"
# parameterize("^tr(trim non-ascii characters)s|Jolie-- ", preserve_case: true) # => "tres-Jolie"
#
# It preserves dashes and underscores unless they are used as separators:
#
# parameterize("^très|Jolie__ ") # => "tres-jolie__"
# parameterize("^très|Jolie-- ", separator: "_") # => "tres_jolie--"
# parameterize("^très_Jolie-- ", separator: ".") # => "tres_jolie--"
# parameterize("^tr(trim non-ascii characters)s|Jolie__ ") # => "tres-jolie__"
# parameterize("^tr(trim non-ascii characters)s|Jolie-- ", separator: "_") # => "tres_jolie--"
# parameterize("^tr(trim non-ascii characters)s_Jolie-- ", separator: ".") # => "tres_jolie--"
#
# If the optional parameter +locale+ is specified,
# the word will be parameterized as a word of that language.
Expand Down Expand Up @@ -9083,7 +9083,7 @@ module ActiveSupport
# in the resulting list are Chars instances instead of String. This makes
# chaining methods easier.
#
# 'Café périferôl'.mb_chars.split(/é/).map { |part| part.upcase.to_s } # => ["CAF", " P", "RIFERÔL"]
# 'Caf(trim non-ascii characters) p(trim non-ascii characters)rifer(trim non-ascii characters)l'.mb_chars.split(/(trim non-ascii characters)/).map { |part| part.upcase.to_s } # => ["CAF", " P", "RIFER(trim non-ascii characters)L"]
def split: (*untyped args) -> untyped

# Works like <tt>String#slice!</tt>, but returns an instance of
Expand All @@ -9099,20 +9099,20 @@ module ActiveSupport

# Reverses all characters in the string.
#
# 'Café'.mb_chars.reverse.to_s # => 'éfaC'
# 'Caf(trim non-ascii characters)'.mb_chars.reverse.to_s # => '(trim non-ascii characters)faC'
def reverse: () -> untyped

# Limits the byte size of the string to a number of bytes without breaking
# characters. Usable when the storage for a string is limited for some
# reason.
#
# 'こんにちは'.mb_chars.limit(7).to_s # => "こん"
# '(trim non-ascii characters)'.mb_chars.limit(7).to_s # => "(trim non-ascii characters)"
def limit: (untyped limit) -> untyped

# Capitalizes the first letter of every word, when possible.
#
# "ÉL QUE SE ENTERÓ".mb_chars.titleize.to_s # => "Él Que Se Enteró"
# "日本語".mb_chars.titleize.to_s # => "日本語"
# "(trim non-ascii characters)L QUE SE ENTER(trim non-ascii characters)".mb_chars.titleize.to_s # => "(trim non-ascii characters)l Que Se Enter(trim non-ascii characters)"
# "(trim non-ascii characters)".mb_chars.titleize.to_s # => "(trim non-ascii characters)"
def titleize: () -> untyped

# Returns the KC normalization of the string by default. NFKC is
Expand All @@ -9126,20 +9126,20 @@ module ActiveSupport

# Performs canonical decomposition on all the characters.
#
# 'é'.length # => 2
# 'é'.mb_chars.decompose.to_s.length # => 3
# '(trim non-ascii characters)'.length # => 2
# '(trim non-ascii characters)'.mb_chars.decompose.to_s.length # => 3
def decompose: () -> untyped

# Performs composition on all the characters.
#
# 'é'.length # => 3
# 'é'.mb_chars.compose.to_s.length # => 2
# '(trim non-ascii characters)'.length # => 3
# '(trim non-ascii characters)'.mb_chars.compose.to_s.length # => 2
def compose: () -> untyped

# Returns the number of grapheme clusters in the string.
#
# 'क्षि'.mb_chars.length # => 4
# 'क्षि'.mb_chars.grapheme_length # => 3
# '(trim non-ascii characters)'.mb_chars.length # => 4
# '(trim non-ascii characters)'.mb_chars.grapheme_length # => 3
def grapheme_length: () -> untyped

# Replaces all ISO-8859-1 or CP1252 characters by their UTF-8 equivalent
Expand Down Expand Up @@ -9195,13 +9195,13 @@ module ActiveSupport
# Unpack the string at grapheme boundaries. Returns a list of character
# lists.
#
# Unicode.unpack_graphemes('क्षि') # => [[2325, 2381], [2359], [2367]]
# Unicode.unpack_graphemes('Café') # => [[67], [97], [102], [233]]
# Unicode.unpack_graphemes('(trim non-ascii characters)') # => [[2325, 2381], [2359], [2367]]
# Unicode.unpack_graphemes('Caf(trim non-ascii characters)') # => [[67], [97], [102], [233]]
def unpack_graphemes: (untyped string) -> untyped

# Reverse operation of unpack_graphemes.
#
# Unicode.pack_graphemes(Unicode.unpack_graphemes('क्षि')) # => 'क्षि'
# Unicode.pack_graphemes(Unicode.unpack_graphemes('(trim non-ascii characters)')) # => '(trim non-ascii characters)'
def pack_graphemes: (untyped unpacked) -> untyped

# Decompose composed characters to the decomposed form.
Expand Down Expand Up @@ -9877,7 +9877,7 @@ module ActiveSupport
# number_to_currency(1234567890.50) # => "$1,234,567,890.50"
# number_to_currency(1234567890.506) # => "$1,234,567,890.51"
# number_to_currency(1234567890.506, precision: 3) # => "$1,234,567,890.506"
# number_to_currency(1234567890.506, locale: :fr) # => "1 234 567 890,51 "
# number_to_currency(1234567890.506, locale: :fr) # => "1 234 567 890,51 (trim non-ascii characters)"
# number_to_currency('123a456') # => "$123a456"
#
# number_to_currency("123a456", raise: true) # => InvalidNumberError
Expand Down Expand Up @@ -10511,7 +10511,7 @@ module ActiveSupport
# Be sure to re-raise unhandled exceptions if this is what you expect.
#
# begin
#
# (trim non-ascii characters)
# rescue => exception
# rescue_with_handler(exception) || raise
# end
Expand Down
2 changes: 1 addition & 1 deletion assets/sig/generated/railties.rbs
Original file line number Diff line number Diff line change
Expand Up @@ -4415,7 +4415,7 @@ end

module Rails
class Secrets
# Greatly inspired by Ara T. Howard's magnificent sekrets gem. 😘
# Greatly inspired by Ara T. Howard's magnificent sekrets gem. (trim non-ascii characters)
# :nodoc:
class MissingKeyError < RuntimeError
def initialize: () -> untyped
Expand Down
5 changes: 5 additions & 0 deletions bin/to-ascii.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!ruby

ARGV.each do |p|
File.write p, File.read(p).gsub(/[^[:ascii:]]+/, '(trim non-ascii characters)')
end