José Valim
1b3efa9625
Update to Unicode 9.0.0
...
The update instructions have also been outlined
on top of the unicode.ex file.
Closes #4864 .
2016-11-20 00:13:12 +01:00
Miles Starkenburg
52ecfb0754
Fix nfd normalization bug ( #4606 )
...
* Fix nfd normalization bug
* Change String.normalize return type to binary
2016-05-13 09:28:50 +02:00
eksperimental
bc8c2a55e9
Standardize unicode and friends ( #4569 )
...
* Unicode is a proper noun, so it should be capitalized
* Replace 'char data' with 'chardata'
* Correct use of a/an articles
2016-05-05 09:21:05 +02:00
Aleksei Magusev
6c96174a0b
Merge pull request #4535 from lexmag/string-trim-pad
...
Introduce String.pad_{leading,trailing}/3 and String.trim{,_leading,_trailing}/2
2016-04-25 20:02:19 +02:00
Aleksei Magusev
0ea04744bc
Introduce String.trim{,_leading,_trailing}/2
2016-04-25 19:59:24 +02:00
eksperimental
685e87bdf0
Formmating: Add white space around vertical bar ( #4507 )
2016-04-25 00:55:49 +02:00
José Valim
7da99878cb
Fold decomposition recursions at compile time
...
Signed-off-by: José Valim <jose.valim@plataformatec.com.br >
2016-03-29 16:55:29 +02:00
José Valim
3693966251
Do not reorder starting classes
...
Signed-off-by: José Valim <jose.valim@plataformatec.com.br >
2016-03-29 13:38:14 +02:00
José Valim
d1b6a8d553
Handle compositios with non-zero combining class
...
Signed-off-by: José Valim <jose.valim@plataformatec.com.br >
2016-03-29 13:38:10 +02:00
José Valim
0de30c7af5
More optimizations and improvements to normalization
...
Signed-off-by: José Valim <jose.valim@plataformatec.com.br >
2016-03-29 13:38:07 +02:00
José Valim
196d225489
Make decomposition recursive and consider exclusion list
...
Signed-off-by: José Valim <jose.valim@plataformatec.com.br >
2016-03-29 13:38:00 +02:00
José Valim
a9e246ecc6
Use combining_classes from UnicodeData
...
Signed-off-by: José Valim <jose.valim@plataformatec.com.br >
2016-03-29 13:37:56 +02:00
José Valim
60d7771f41
Rely only on UnicodeData for composition/decomposition
2016-03-29 00:54:20 +02:00
José Valim
0bf5293aab
Clean up unicode range parsing
2016-03-18 16:59:13 +01:00
José Valim
1d5ca3beed
Update Unicode to 8.0.0 (equivalent pending)
2016-03-15 15:58:01 +01:00
José Valim
d3ff4691b8
Avoid regular expressions
2016-03-15 15:43:32 +01:00
Abel Muiño
c465f80a6d
New definition of whitespace & breakable whitespace
...
All whitespace can be removed by `strip` but only breakable whitespace
can be used as a delimiter by `split`, with updated docs for String.split/1
2016-03-15 14:54:11 +01:00
José Valim
5b77dd9a14
Remove debug_info from unicode
2016-02-01 09:21:49 +01:00
José Valim
0c4a1f189f
Do not include debug_info in metadata String modules
2016-01-08 10:11:07 +01:00
Aleksei Magusev
612fcc4760
Eliminate spaces in bitstring definitions
2015-11-28 23:48:48 +01:00
José Valim
cfc4bc20d1
Remove uses of soon to be deprecated Dict
2015-10-23 15:04:24 +02:00
Bryan Endersstocker
e12adea1d4
Add NFC support to String.normalize/2
2015-10-03 11:08:55 -04:00
Bryan Endersstocker
180f3184c6
Optimize Unicode normalization
...
Also extracted String.Normalizer module from String.Unicode.
2015-09-26 10:59:04 -04:00
Bryan Endersstocker
a65b8c27da
Add String.normalize/2
2015-09-26 00:12:44 -04:00
Bryan Enders
4fb6446145
Rename String.is_equivalent/2 to String.equivalent?/2
2015-09-25 15:01:29 -04:00
Bryan Enders
f01ad6df36
Add String.is_equivalent/2 to test canonical Unicode equivalence
2015-09-25 13:35:16 -04:00
José Valim
5ed087ef50
Move away from HashDict and HashSet
2015-09-25 12:46:22 +02:00
eksperimental
8787e7019e
Format raise message consistently
...
Kernel.raise messages should not start with uppercase, and should not
have trailing punctuation.
Mix.raise messages should start with uppercase, and should not have a trailing
puntuation.
# starting with uppercase
# exclude (Mix.raise, and any message that starts with all uppecase such as: I or IO
ag '(?<!Mix\.)raise\s+\"+(?!([A-Z]+\b))[A-Z]'
# Mix.raise should start with upper case
# except when `mix` or `rebar` are mentioned
ag 'Mix\.raise\s+\"(?!(mix|rebar))+[a-z]'
# ending in period
ag "raise\s+\"+.+\.\"\s*\n"
2015-09-08 00:44:17 +07:00
José Valim
34bbb3d83e
Optimize many functions in String
...
Many functions relied on next_grapheme, which would traverse
the whole string generating a lot of graphemes as garbage along
the way.
This commit introduces next_grapheme_size, which returns the
size in bytes of the next grapheme, avoiding the generation
of the garbage and optimizing functions that need just the
byte_size.
Functions optimized are: String.at, String.split_at, String.slice
and String.length.
2015-08-02 13:49:50 +02:00
José Valim
5dafe3ab7b
Speed up upcase and downcase for large strings
2015-06-05 13:12:25 +02:00
José Valim
4d8c6cbb57
Handle corner cases for small strings in rstrp
2015-06-02 10:13:45 +02:00
José Valim
dbfbac4c86
Optimize rstrip
...
This new implementation is no longer linear without affecting
smaller samples. For a string that is 100 bytes long, it is
25x faster than the previous implementation.
2015-06-02 09:52:13 +02:00
eksperimental
8cc6ed995f
fix typo in previous commit. bodyless defmacro added to show proper arg in b/1, t/1
2015-03-28 19:19:37 +07:00
eksperimental
fdc0221517
rename function arg. when "other" was used
2015-03-27 02:37:36 +07:00
jw2013
3dd9df184e
fix wrong CRLF grapheme
2014-11-08 12:45:41 -08:00
José Valim
64b68edd45
Do not consider subpatterns on Regex.split/3
2014-07-30 11:13:18 +02:00
José Valim
182cbfe541
Update to Unicode 7.0.0
2014-06-17 17:17:32 +02:00
José Valim
1038018e84
Migrate from size to byte_size and tuple_size
2014-06-06 13:33:02 +02:00
José Valim
540c763c85
Move binary and list conversions to String and List
2014-05-17 17:02:46 +02:00
José Valim
3bb8e17e9e
Add iodata_to_binary and chardata_to_string to IO
2014-05-17 13:06:40 +02:00
José Valim
56f15d9f4e
Do not add spaces after { and before }
...
This makes the source code consistent with the result
returned by inspect/2.
2014-04-21 19:06:35 +02:00
José Valim
024edf3d40
Rename iolist_* to iodata_*
2014-02-23 11:32:03 +01:00
José Valim
3512c1a860
Start migrating lc to for comprehensions
2014-02-17 14:45:04 +01:00
José Valim
ba54afbf45
Move to new sigils syntax
2014-02-05 14:39:04 +01:00
José Valim
0802014779
No longer compile unicode in parallel
...
Compiling in parallel only bought us 2s (out of 24s) since
compiling graphemes is what takes most of the time but it
considerably increases memory usage, by about 30%, since both
compilation targets are loaded into memory at once.
2014-01-22 17:51:24 +01:00
José Valim
87fee37711
Change String.next_grapheme/1 and String.next_codepoint/1 to return nil on string end
...
This is more composable in the sense can use in in if/cond constructs
and also makes it easy to build streams. Closes #1721 .
2014-01-14 16:15:54 +01:00
José Valim
0538e46023
Compile unicode modules in parallel
2014-01-09 20:12:20 +01:00
José Valim
a829da1775
Reduce the size of String.Graphemes module
...
Based on a suggestion @krestenkrab and an initial implementation
proposed in #1902 , we can reduce the total size of the String.Graphemes
beam file by ensuring the function clauses for next_grapheme are
similar.
We have done this by implementing next_grapheme in a tail recursive
fashion and using binary_part to select the first bits. The end result
is a consistently 33% faster graphemes operation for ascii strings
and only 10% slower operation for heavy unicode strings (which may
be further optimazable).
2013-11-29 12:48:01 +01:00
José Valim
94579848cc
Move unicode from priv/ to unicode/
2013-11-05 08:17:33 +01:00