Timerix/utf8proc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
C library for processing UTF-8 and UTF-32 data
39 Commits 1 Branch 0 Tags 1.5 MiB
C 98.2%
Ruby 0.8%
Makefile 0.5%
Julia 0.3%
CMake 0.2%
0f9e9796e6
Go to file
Steven G. Johnson 0f9e9796e6 fix #15: redefine UTF8PROC_CATEGORY_CN to 0 consistent with what we actually return
2014-12-12 18:25:07 -05:00
bench added GNU libunistring benchmark 2014-07-19 14:55:25 -04:00
.gitignore added normalization and encoding test for #13 2014-08-07 16:52:16 -04:00
.travis.yml Add travis file for testing 2014-08-07 17:04:03 -04:00
data_generator.rb Replace all explicitly marked regions with Ruby file read and regex section matches 2014-07-18 10:46:11 -04:00
LICENSE.md README updates 2014-12-07 21:29:34 -05:00
lump.txt import of utf8proc-v1.1.6 2014-07-15 15:29:52 -04:00
Makefile s/LIBRARY_EXPORTS/MOJIBAKE_EXPORTS/ 2014-09-23 11:08:52 -07:00
mojibake.h fix #15: redefine UTF8PROC_CATEGORY_CN to 0 consistent with what we actually return 2014-12-12 18:25:07 -05:00
NEWS.md a few typofixes 2014-08-12 21:30:59 +01:00
normtest.c added normalization and encoding test for #13 2014-08-07 16:52:16 -04:00
README.md README updates 2014-12-07 21:29:34 -05:00
utf8proc_data.c Update utf8proc_data.c (generated by data_generator.rb) 2014-07-18 10:46:11 -04:00
utf8proc.c utf8proc_version should return a different version string than utf8proc 2014-12-12 14:20:53 -05:00

README.md

libmojibake

Build Status

libmojibake is a development fork of the utf8proc library from Jan Behrens and the rest of the Public Software Group, who deserve nearly all of the credit for this package: a small, clean C library that provides Unicode normalization, case-folding, and other operations for data in the UTF-8 encoding. The main difference from utf8proc is that the Unicode support in libmojibake is more up-to-date (Unicode 7 vs. Unicode 5).

The reason for this fork is that utf8proc is used for basic Unicode support in the Julia language and the Julia developers wanted Unicode 7 support and other features, but the Public Software Group is currently occupied with other projects. As we implement and test new features in libmojibake, we are contributing patches back to utf8proc with the hope that they can be merged upstream.

(The original utf8proc package also includes Ruby and PostgreSQL plug-ins. We removed those from libmojibake in order to focus exclusively on the C library for the time being. We will strive to keep API changes to a minimum, so libmojibake should still be usable with the old plug-in code.)

Like utf8proc, the libmojibake package is licensed under the free/open-source MIT "expat" license (plus certain Unicode data governed by the similarly permissive Unicode data license); please see the included LICENSE.md file for more detailed information.

Quick Start

For compilation of the C library run make.

General Information

The C library is found in this directory after successful compilation and is named libmojibake.a (for the static library) and libmojibake.so (for the dynamic library).

The Unicode version being supported is 7.0.0. (Grapheme segmentation is currently based on version 4.1.0 of Unicode Standard Annex #29, but we hope to update this soon.)

For Unicode normalizations, the following options are used:

  • Normalization Form C: STABLE, COMPOSE
  • Normalization Form D: STABLE, DECOMPOSE
  • Normalization Form KC: STABLE, COMPOSE, COMPAT
  • Normalization Form KD: STABLE, DECOMPOSE, COMPAT

C Library

The documentation for the C library is found in the utf8proc.h header file. utf8proc_map is function you will most likely be using for mapping UTF-8 strings, unless you want to allocate memory yourself.

To Do

See the Github issues list.

Contact

Bug reports, feature requests, and other queries can be filed at the libmojibake issues page on Github.