Case folding fixes (#133)
* Fixes allowing for “Full” folding and NFKC_CaseFold compliance. * Only include C (Common) and F (Full) foldings from CaseFolding.txt. Removed S (Simple) since F & S are specified to be exclusive. * Extend UTF8PROC_IGNORE to also ignore unassigned codepoints (such as \u2065) which are specified as being discarded by NFKC_CF. * Document the changes to UTF8PROC_IGNORE in header. * Add NFKC_CF helper function with documentation. * restore old IGNORE behavior, add UTF8PROC_STRIPNA, rename to utf8proc_NFKC_Casefold, add a test * success message * test that IGNORE does not strip NA * data update * NFKC_Casefold shouldn't strip NA
This commit is contained in:
committed by
GitHub
parent
48949bd3eb
commit
bdc8b9e4b2
@@ -104,7 +104,7 @@ $excl_version = $excl_version.chomp.split("\n").collect { |e| e.hex }
|
||||
$case_folding_string = File.open("CaseFolding.txt", :encoding => 'utf-8').read
|
||||
$case_folding = {}
|
||||
$case_folding_string.chomp.split("\n").each do |line|
|
||||
next unless line =~ /([0-9A-F]+); [CFS]; ([0-9A-F ]+);/i
|
||||
next unless line =~ /([0-9A-F]+); [CF]; ([0-9A-F ]+);/i
|
||||
$case_folding[$1.hex] = $2.split(" ").collect { |e| e.hex }
|
||||
end
|
||||
|
||||
|
||||
Reference in New Issue
Block a user