Commit Graph
49 Commits
Author SHA1 Message Date
José Valim 1b3efa9625 Update to Unicode 9.0.0
The update instructions have also been outlined
on top of the unicode.ex file.

Closes #4864.
2016-11-20 00:13:12 +01:00
Miles Starkenburg 52ecfb0754 Fix nfd normalization bug (#4606)
* Fix nfd normalization bug

* Change String.normalize return type to binary
2016-05-13 09:28:50 +02:00
eksperimental bc8c2a55e9 Standardize unicode and friends (#4569)
* Unicode is a proper noun, so it should be capitalized

* Replace 'char data' with 'chardata'

* Correct use of a/an articles
2016-05-05 09:21:05 +02:00
Aleksei Magusev 6c96174a0b Merge pull request #4535 from lexmag/string-trim-pad
Introduce String.pad_{leading,trailing}/3 and String.trim{,_leading,_trailing}/2
2016-04-25 20:02:19 +02:00
Aleksei Magusev 0ea04744bc Introduce String.trim{,_leading,_trailing}/2 2016-04-25 19:59:24 +02:00
eksperimental 685e87bdf0 Formmating: Add white space around vertical bar (#4507) 2016-04-25 00:55:49 +02:00
José Valim 7da99878cb Fold decomposition recursions at compile time
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 16:55:29 +02:00
José Valim 3693966251 Do not reorder starting classes
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:38:14 +02:00
José Valim d1b6a8d553 Handle compositios with non-zero combining class
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:38:10 +02:00
José Valim 0de30c7af5 More optimizations and improvements to normalization
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:38:07 +02:00
José Valim 196d225489 Make decomposition recursive and consider exclusion list
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:38:00 +02:00
José Valim a9e246ecc6 Use combining_classes from UnicodeData
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:37:56 +02:00
José Valim 60d7771f41 Rely only on UnicodeData for composition/decomposition 2016-03-29 00:54:20 +02:00
José Valim 0bf5293aab Clean up unicode range parsing 2016-03-18 16:59:13 +01:00
José Valim 1d5ca3beed Update Unicode to 8.0.0 (equivalent pending) 2016-03-15 15:58:01 +01:00
José Valim d3ff4691b8 Avoid regular expressions 2016-03-15 15:43:32 +01:00
Abel Muiño c465f80a6d New definition of whitespace & breakable whitespace
All whitespace can be removed by `strip` but only breakable whitespace
can be used as a delimiter by `split`, with updated docs for String.split/1
2016-03-15 14:54:11 +01:00
José Valim 5b77dd9a14 Remove debug_info from unicode 2016-02-01 09:21:49 +01:00
José Valim 0c4a1f189f Do not include debug_info in metadata String modules 2016-01-08 10:11:07 +01:00
Aleksei Magusev 612fcc4760 Eliminate spaces in bitstring definitions 2015-11-28 23:48:48 +01:00
José Valim cfc4bc20d1 Remove uses of soon to be deprecated Dict 2015-10-23 15:04:24 +02:00
Bryan Endersstocker e12adea1d4 Add NFC support to String.normalize/2 2015-10-03 11:08:55 -04:00
Bryan Endersstocker 180f3184c6 Optimize Unicode normalization
Also extracted String.Normalizer module from String.Unicode.
2015-09-26 10:59:04 -04:00
Bryan Endersstocker a65b8c27da Add String.normalize/2 2015-09-26 00:12:44 -04:00
Bryan Enders 4fb6446145 Rename String.is_equivalent/2 to String.equivalent?/2 2015-09-25 15:01:29 -04:00
Bryan Enders f01ad6df36 Add String.is_equivalent/2 to test canonical Unicode equivalence 2015-09-25 13:35:16 -04:00
José Valim 5ed087ef50 Move away from HashDict and HashSet 2015-09-25 12:46:22 +02:00
eksperimental 8787e7019e Format raise message consistently
Kernel.raise messages should not start with uppercase, and should not
have trailing punctuation.

Mix.raise messages should start with uppercase, and should not have a trailing
puntuation.

    # starting with uppercase
    # exclude (Mix.raise, and any message that starts with all uppecase such as: I or IO
    ag '(?<!Mix\.)raise\s+\"+(?!([A-Z]+\b))[A-Z]'

    # Mix.raise should start with upper case
    # except when `mix` or `rebar` are mentioned
    ag 'Mix\.raise\s+\"(?!(mix|rebar))+[a-z]'

    # ending in period
    ag "raise\s+\"+.+\.\"\s*\n"
2015-09-08 00:44:17 +07:00
José Valim 34bbb3d83e Optimize many functions in String
Many functions relied on next_grapheme, which would traverse
the whole string generating a lot of graphemes as garbage along
the way.

This commit introduces next_grapheme_size, which returns the
size in bytes of the next grapheme, avoiding the generation
of the garbage and optimizing functions that need just the
byte_size.

Functions optimized are: String.at, String.split_at, String.slice
and String.length.
2015-08-02 13:49:50 +02:00
José Valim 5dafe3ab7b Speed up upcase and downcase for large strings 2015-06-05 13:12:25 +02:00
José Valim 4d8c6cbb57 Handle corner cases for small strings in rstrp 2015-06-02 10:13:45 +02:00
José Valim dbfbac4c86 Optimize rstrip
This new implementation is no longer linear without affecting
smaller samples. For a string that is 100 bytes long, it is
25x faster than the previous implementation.
2015-06-02 09:52:13 +02:00
eksperimental 8cc6ed995f fix typo in previous commit. bodyless defmacro added to show proper arg in b/1, t/1 2015-03-28 19:19:37 +07:00
eksperimental fdc0221517 rename function arg. when "other" was used 2015-03-27 02:37:36 +07:00
jw2013 3dd9df184e fix wrong CRLF grapheme 2014-11-08 12:45:41 -08:00
José Valim 64b68edd45 Do not consider subpatterns on Regex.split/3 2014-07-30 11:13:18 +02:00
José Valim 182cbfe541 Update to Unicode 7.0.0 2014-06-17 17:17:32 +02:00
José Valim 1038018e84 Migrate from size to byte_size and tuple_size 2014-06-06 13:33:02 +02:00
José Valim 540c763c85 Move binary and list conversions to String and List 2014-05-17 17:02:46 +02:00
José Valim 3bb8e17e9e Add iodata_to_binary and chardata_to_string to IO 2014-05-17 13:06:40 +02:00
José Valim 56f15d9f4e Do not add spaces after { and before }
This makes the source code consistent with the result
returned by inspect/2.
2014-04-21 19:06:35 +02:00
José Valim 024edf3d40 Rename iolist_* to iodata_* 2014-02-23 11:32:03 +01:00
José Valim 3512c1a860 Start migrating lc to for comprehensions 2014-02-17 14:45:04 +01:00
José Valim ba54afbf45 Move to new sigils syntax 2014-02-05 14:39:04 +01:00
José Valim 0802014779 No longer compile unicode in parallel
Compiling in parallel only bought us 2s (out of 24s) since
compiling graphemes is what takes most of the time but it
considerably increases memory usage, by about 30%, since both
compilation targets are loaded into memory at once.
2014-01-22 17:51:24 +01:00
José Valim 87fee37711 Change String.next_grapheme/1 and String.next_codepoint/1 to return nil on string end
This is more composable in the sense can use in in if/cond constructs
and also makes it easy to build streams. Closes #1721.
2014-01-14 16:15:54 +01:00
José Valim 0538e46023 Compile unicode modules in parallel 2014-01-09 20:12:20 +01:00
José Valim a829da1775 Reduce the size of String.Graphemes module
Based on a suggestion @krestenkrab and an initial implementation
proposed in #1902, we can reduce the total size of the String.Graphemes
beam file by ensuring the function clauses for next_grapheme are
similar.

We have done this by implementing next_grapheme in a tail recursive
fashion and using binary_part to select the first bits. The end result
is a consistently 33% faster graphemes operation for ascii strings
and only 10% slower operation for heavy unicode strings (which may
be further optimazable).
2013-11-29 12:48:01 +01:00
José Valim 94579848cc Move unicode from priv/ to unicode/ 2013-11-05 08:17:33 +01:00