Commit Graph

24 Commits

Author SHA1 Message Date
Steven G. Johnson
d4a58cfec5
update data and algorithms for Unicode 11 (#140) 2018-07-24 13:18:48 -04:00
Steven G. Johnson
02f4e1890c
charwidth=1 for soft hyphen and unassigned codepoints (#135)
* use width=1 for soft hyphen and for unassigned/PUA codepoints

* don't count unassigned codepoints when comparing with system wcwidth

* more tests

* indentation fixes

* NEWS for 135

* remove special-casing for arabic control characters affecting a span of numbers, which are sometimes zero-width and sometimes not

* regenerate
2018-07-24 10:45:02 -04:00
Steven G. Johnson
d81308faba
uppercase mapping ß (U+00df) to ẞ (U+1E9E) (#134)
* uppercase(0x00df) = 0x1e9e

* tests for titlecase and u+00df uppercase

* NEWS, another test
2018-05-02 14:18:26 -04:00
Steven G. Johnson
bdc8b9e4b2
Case folding fixes (#133)
* Fixes allowing for “Full” folding and NFKC_CaseFold compliance.

* Only include C (Common) and F (Full) foldings from CaseFolding.txt. Removed S (Simple) since F & S are specified to be exclusive.
* Extend UTF8PROC_IGNORE to also ignore unassigned codepoints (such as \u2065) which are specified as being discarded by NFKC_CF.

* Document the changes to UTF8PROC_IGNORE in header.

* Add NFKC_CF helper function with documentation.

* restore old IGNORE behavior, add UTF8PROC_STRIPNA, rename to utf8proc_NFKC_Casefold, add a test

* success message

* test that IGNORE does not strip NA

* data update

* NFKC_Casefold shouldn't strip NA
2018-05-02 08:15:02 -04:00
Steven G. Johnson
ba042cf728 missing return code, success message in test/misc.c 2018-04-27 09:10:38 -04:00
Steven G. Johnson
d050c4636a make internal function static 2018-04-27 08:57:54 -04:00
Steven G. Johnson
53d7968055 added test for #128 2018-04-27 08:46:44 -04:00
Steven G. Johnson
b4621f43c3 new utf8proc_map_custom for hooking in user-defined custom mappings (#89)
* new utf8proc_map_custom for hooking in user-defined custom mappings

* whoops, add test program

* NEWS, version bump for 2.1

* change test functions to static so that gcc doesn't complain about missing prototypes
2016-11-30 10:40:26 -05:00
Benito van der Zander
eeebf70bcf Smaller tables (#68)
* convert sequences to utf-16 (saves 25kb)

* store sequence length in properties instead using -1 termination (saves 10kb)

* cache index for slightly faster data creation

* store lower/upper/title mapping in sequence array (saves 25kb). Add utf8proc_totitle, as title_mapping cannot be used to get the title codepoint anymore. Rename xxx_mapping to xxx_seqindex, so programs assuming a value with the old meaning fail at compile time

* change combination array data type to uint16 (saves 40kb)

* merge 1st and 2nd comb index (saves 50kb)

* kill empty prefix/suffix in combination array (saves 50kb)

* there was no need to have a separate combination start array, it can be merged in a single array

* some fixes

* mark the table as const again

* and regen
2016-07-12 11:51:50 -04:00
Michaël Meyer
1f17487aa9 Fix overrun 2016-02-04 04:06:28 +01:00
Peter Colberg
4b16193a25 Fix sscanf argument type for format %x 2015-10-30 15:27:18 -04:00
Peter Colberg
14b57791d8 Fix missing static declarations for internal functions 2015-10-30 15:24:34 -04:00
Peter Colberg
6acc41dfe9 Fix implicit function declarations 2015-10-30 15:22:09 -04:00
Peter Colberg
548497a398 Move common test functions to separate module
This resolves warnings for missing function prototypes.
2015-10-30 15:13:48 -04:00
Peter Colberg
09360de186 Do not export internal unsafe_encode_char() 2015-10-29 00:45:39 -04:00
Steven G. Johnson
6a7f92da64 fix #46 (make sure symbol-like codepoints have nonzero width even if they aren't in Unifont) 2015-06-24 14:07:15 -04:00
Steven G. Johnson
a8fb4b1772 add toupper/tolower functions (for JuliaLang/julia#11471) 2015-05-29 22:00:30 -04:00
ScottPJones
6a229a6776 Add tests for valid codepoints and iterate function 2015-05-29 20:11:10 +02:00
Scott Paul Jones
6249e6b8b1 Fix #34 handle 66 Unicode non-characters, also improve performance and surrogate handling 2015-05-29 19:50:03 +02:00
Tony Kelman
0a818c7003 Prefix other C99 typedefs with utf8proc_ 2015-04-06 22:36:33 -07:00
Tony Kelman
ad27722923 Use a new typedef utf8proc_ssize_t to avoid define collisions
with MSVC
2015-04-05 20:06:13 -07:00
Steven G. Johnson
c851c67888 put the API version as #defines in the header file (as discussed in #30) 2015-03-27 12:35:41 -04:00
Steven G. Johnson
a4c84d2063 fix #2: add charwidth function 2015-03-12 12:10:19 -04:00
Steven G. Johnson
90721f2d39 directory cleanup: move tests and data into subdirectories 2015-03-06 17:36:08 -05:00