Commit Graph
79 Commits
Author SHA1 Message Date
Xavier Noria b784c9315d Clarify String.next_codepoint/1 re invalid UTF-8 (#8489) 2018-12-08 16:41:57 +01:00
Sun Yaozhu 6760ba44c7 Fix ZWJ handling in Unicode grapheme clusters (#8361) 2018-11-04 19:56:36 +01:00
José Valim df26e1bcbd Print number of tests from graphemes tests 2018-10-02 12:25:50 +02:00
José Valim 99868597b1 Rely on Erlang/OTP normalization which is faster 2018-07-12 13:28:58 +02:00
José Valim c43136216b Update to Unicode 11 2018-06-06 11:22:36 +02:00
Michał Muskała 7cfb63c636 Use binary_part/3 instead of :binary.part/3
binary_part is a guard BIF and is slightly more efficient to use then
:binary.part which does a fully qualified external call.
2018-04-30 13:22:19 +02:00
José Valim 0db0b85f52 Consider case ignorable characters on Greek downcasing, closes #7149
Note this does not impact the runtime cost of other downcasing
operations. The final beam file grew only in 8kb.
2018-01-09 13:59:13 +01:00
José Valim 3f95b4a17d Improvements to greek handling of downcase, closes #7149 2017-12-26 23:11:17 +01:00
José Valim 8fd0141297 Optimize String.downcase/1
Thanks to @michalmuskala for feedback and review.
2017-12-20 12:58:17 +01:00
José Valim 2a6a545b8b Provide :greek and :ascii mappings in String upcase/downcase/capitalize
Closes #7105.
2017-12-20 11:54:06 +01:00
José Valim e43f5f53b2 Do not indent right-side of pipelines in the formatter (#7103) 2017-12-11 21:07:30 +01:00
José Valim 27aa867b2e Minimize generated beam size by using ranges
This approach is less performant when processing
sigma but compiles twice faster and takes two times
less space in disk.
2017-10-30 18:45:16 +01:00
Ben Olive 6f40fae504 Handle final sigma unicode exception
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2017-10-30 18:45:16 +01:00
José Valim dec9e6f0a8 Improvements to previously formatted files 2017-10-11 22:59:30 +02:00
José Valim 940d2f61c0 Make sure to not break guards on clauses with comments 2017-10-11 21:25:21 +02:00
José Valim c6233421e5 Format more files with little formatting requirement 2017-10-11 00:28:10 +02:00
Frank McGeough 24acb605ba Run formatter on lib/elixir/unicode/tokenizer.ex (#6813) 2017-10-10 18:10:35 +02:00
Guilherme Pasqualino 5c21265cc8 Run code formatter on lib/elixir/unicode/unicode.ex (#6795) 2017-10-10 12:24:15 +02:00
Michał Muskała dae60dbea4 Add is_binary guard to String.length (#6430)
This produces a bit more readable message if not a string is passed into `String.length`, then failing in the `next_grapheme_size` function.
2017-08-03 21:04:12 +02:00
Michał Muskała c71926f8db Optimise binary pattern matching in String.Break (#6340)
In downcase and upcase pattern match tail as ::bits instead of ::binary. This
saves us a check on each iteration if the tail is a proper binary. We only do
this check in the very beginning.

In trim_leading additionally make sure every head of the recursive function
pattern matches on a binary in first argument so the single binary match context
optimisation can be applied.
2017-07-16 14:29:45 +02:00
José Valim e8690e412a Update to Unicode 10 (#6293) 2017-07-04 09:57:07 +02:00
Andrea Leopardi 7d873750b1 Use :binary.split/3 in String.Break.split/1 (#6257) 2017-06-25 18:53:11 +02:00
José Valim 19b1668cfd Do not rely on on_load because it breaks releases 2017-05-29 15:50:06 +02:00
José Valim 2f663ce3a2 Add non-quoted Unicode atoms and variables (#6158)
It follows Unicode Annex #31.

See the Unicode Syntax document for a more in-depth reference.
2017-05-27 22:43:05 +02:00
José Valim 50d16830fd Properly slice titlecase_once binaries 2017-05-19 12:37:05 +02:00
Aleksei Magusev 6898fd98d7 Correct module name for grapheme tests 2017-02-23 23:58:20 +01:00
José Valim de1b731afb Add grapheme tests 2017-02-13 15:44:59 +01:00
José Valim b1c567b5ef Incorporate new grapheme rules in Unicode 9 2017-02-13 14:34:04 +01:00
Andrea Leopardi 7222c1bce3 Remove leftover trailing whitespace from the code base
[ci skip]
2017-01-13 15:01:47 +01:00
José Valim cae33236ad Speed up String.split/1 2016-11-21 19:03:16 +01:00
José Valim 1b3efa9625 Update to Unicode 9.0.0
The update instructions have also been outlined
on top of the unicode.ex file.

Closes #4864.
2016-11-20 00:13:12 +01:00
Miles Starkenburg 52ecfb0754 Fix nfd normalization bug (#4606)
* Fix nfd normalization bug

* Change String.normalize return type to binary
2016-05-13 09:28:50 +02:00
eksperimental bc8c2a55e9 Standardize unicode and friends (#4569)
* Unicode is a proper noun, so it should be capitalized

* Replace 'char data' with 'chardata'

* Correct use of a/an articles
2016-05-05 09:21:05 +02:00
Aleksei Magusev 6c96174a0b Merge pull request #4535 from lexmag/string-trim-pad
Introduce String.pad_{leading,trailing}/3 and String.trim{,_leading,_trailing}/2
2016-04-25 20:02:19 +02:00
Aleksei Magusev 0ea04744bc Introduce String.trim{,_leading,_trailing}/2 2016-04-25 19:59:24 +02:00
eksperimental 685e87bdf0 Formmating: Add white space around vertical bar (#4507) 2016-04-25 00:55:49 +02:00
José Valim 7da99878cb Fold decomposition recursions at compile time
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 16:55:29 +02:00
José Valim 3693966251 Do not reorder starting classes
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:38:14 +02:00
José Valim d1b6a8d553 Handle compositios with non-zero combining class
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:38:10 +02:00
José Valim 0de30c7af5 More optimizations and improvements to normalization
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:38:07 +02:00
José Valim 196d225489 Make decomposition recursive and consider exclusion list
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:38:00 +02:00
José Valim a9e246ecc6 Use combining_classes from UnicodeData
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
2016-03-29 13:37:56 +02:00
José Valim 60d7771f41 Rely only on UnicodeData for composition/decomposition 2016-03-29 00:54:20 +02:00
José Valim 0bf5293aab Clean up unicode range parsing 2016-03-18 16:59:13 +01:00
José Valim 1d5ca3beed Update Unicode to 8.0.0 (equivalent pending) 2016-03-15 15:58:01 +01:00
José Valim d3ff4691b8 Avoid regular expressions 2016-03-15 15:43:32 +01:00
Abel Muiño c465f80a6d New definition of whitespace & breakable whitespace
All whitespace can be removed by `strip` but only breakable whitespace
can be used as a delimiter by `split`, with updated docs for String.split/1
2016-03-15 14:54:11 +01:00
José Valim 5b77dd9a14 Remove debug_info from unicode 2016-02-01 09:21:49 +01:00
José Valim 0c4a1f189f Do not include debug_info in metadata String modules 2016-01-08 10:11:07 +01:00
Aleksei Magusev 612fcc4760 Eliminate spaces in bitstring definitions 2015-11-28 23:48:48 +01:00