lib/unicore/README.perl

# The goal is for perl to compile and reasonably run any version of Unicode.
# Working reasonably well doesn't mean that the test suite will run without
# showing errors.  A few of the very-Unicode specific test files have been
# modified to account for different versions, but most have not.  For example,
# some tests use characters that aren't encoded in all Unicode versions; others
# have hard-coded the General Categories for a code point that were correct at
# the time the test was written.  Perl itself will not compile under Unicode
# releases prior to 3.0 without a simple change to Unicode::Normalize.
# mktables contains instructions for this.

# The *.txt files were copied from

# 	ftp://www.unicode.org/Public/UNIDATA

# (which always points to the latest version) with subdirectories 'extracted' and
# 'auxiliary'.  Older versions are located under Public with an appropriate name.
# They are also available via http at www.unicode.org/versions/
#

# The Unihan files were not included due to space considerations.  Also NOT
# included were any *.html files.  It is possible to add the Unihan files and
# have some properties from them automatically compiled.  By editing mktables
# (see instructions near its beginning) you can add other Unihan properties.

# The file named 'version' should exist and be a single line with the Unicode
# version, like:
#
# 5.2.0
#
# (without the initial '# ')

# To be 8.3 filesystem friendly, the names of some of the input files have been
# changed from the values that are in the Unicode DB.  Not all of the Test
# files are currently used, so may not be present, so some of the mv's can
# fail.  The .html Test files are not touched.

mv PropertyValueAliases.txt PropValueAliases.txt
mv NamedSequencesProv.txt NamedSqProv.txt
mv NormalizationTest.txt NormTest.txt
mv DerivedAge.txt DAge.txt
mv DerivedCoreProperties.txt DCoreProperties.txt
mv DerivedNormalizationProps.txt DNormalizationProps.txt

# Some early releases don't have the extracted directory, and hence these files
# should be moved to it.
mkdir extracted 2>/dev/null
mv DerivedBidiClass.txt DerivedBinaryProperties.txt extracted 2>/dev/null
mv DerivedCombiningClass.txt DerivedDecompositionType.txt extracted 2>/dev/null
mv DerivedEastAsianWidth.txt DerivedGeneralCategory.txt extracted 2>/dev/null
mv DerivedJoiningGroup.txt DerivedJoiningType.txt extracted 2>/dev/null
mv DerivedLineBreak.txt DerivedNumericType.txt DerivedNumericValues.txt extracted 2>/dev/null

mv extracted/DerivedBidiClass.txt extracted/DBidiClass.txt
mv extracted/DerivedBinaryProperties.txt extracted/DBinaryProperties.txt
mv extracted/DerivedCombiningClass.txt extracted/DCombiningClass.txt
mv extracted/DerivedDecompositionType.txt extracted/DDecompositionType.txt
mv extracted/DerivedEastAsianWidth.txt extracted/DEastAsianWidth.txt
mv extracted/DerivedGeneralCategory.txt extracted/DGeneralCategory.txt
mv extracted/DerivedJoiningGroup.txt extracted/DJoinGroup.txt
mv extracted/DerivedJoiningType.txt extracted/DJoinType.txt
mv extracted/DerivedLineBreak.txt extracted/DLineBreak.txt
mv extracted/DerivedNumericType.txt extracted/DNumType.txt
mv extracted/DerivedNumericValues.txt extracted/DNumValues.txt
rmdir extracted 2>/dev/null     # Will fail if non-empty, but if it is empty
                                # was an early release that didn't have it.

mv auxiliary/GraphemeBreakTest.txt auxiliary/GCBTest.txt
mv auxiliary/LineBreakTest.txt auxiliary/LBTest.txt
mv auxiliary/SentenceBreakTest.txt auxiliary/SBTest.txt
mv auxiliary/WordBreakTest.txt auxiliary/WBTest.txt

# If you have the Unihan database (5.2 and above), you should also do the
# following:

mv Unihan_DictionaryIndices.txt UnihanIndicesDictionary.txt
mv Unihan_DictionaryLikeData.txt UnihanDataDictionaryLike.txt
mv Unihan_IRGSources.txt UnihanIRGSources.txt
mv Unihan_NumericValues.txt UnihanNumericValues.txt
mv Unihan_OtherMappings.txt UnihanOtherMappings.txt
mv Unihan_RadicalStrokeCounts.txt UnihanRadicalStrokeCounts.txt
mv Unihan_Readings.txt UnihanReadings.txt
mv Unihan_Variants.txt UnihanVariants.txt

# If you download everything, the names of files that are not used by mktables
# are not changed by the above, and hence may not work correctly as-is on 8.3
# filesystems.

# mktables is used to generate the tables used by the rest of Perl.  It will
# warn you about any *.txt and *.html files in the directory substructure that
# it doesn't know about.  You should remove any so-identified, or edit mktables
# to add them to its lists to process.  You can run
#
#    mktables -globlist
#
# to have it try to process these tables generically.

# COMPILING ON OLDER UNICODE VERSIONS
#
# To compile perl for use with an older Unicode release, delete everything in
# the lib/unicore directory except mktables and Makefile.  Then download the
# Unicode-supplied files for the desired version to that directory  (A url for
# these is given earlier in this file).  Then create the 'version' file with a
# single line, like '6.1.0'.  Do a 'make test' from the project level.  You
# will get some porting errors for needing to regen.  Regenerate what it tells
# you are needed, and make test again.  If you compile an old enough version,
# you will also have to download a few files from later Unicode versions,
# following the instructions that will be given if warranted.  It should
# compile in any release without warnings, except for some casing conflicts
# in Unicode 2.1.8, and some extraneous files will show up in very early
# releases of the form qr/diff.*\.txt/.  If you add Unihan.txt, one line is in error in
#
# Other glitches are noted in mktables under 'UNICODE VERSIONS NOTES'

# FOR PUMPKINS
#
# The files are inter-related.  If you take the latest UnicodeData.txt, for
# example, but leave the older versions of other files, there can be subtle
# problems.  So get everything available from Unicode, and delete those which
# aren't needed.
#
# When moving to a new version of Unicode, you need to update 'version' by hand
#
#	p4 edit version
# 	...
#
# You should look in the Unicode release notes (which are probably towards the
# bottom of http://www.unicode.org/reports/tr44/) to see if any properties have
# newly been moved to be Obsolete, Deprecated, or Stabilized.  The full names
# for these should be added to the respective lists near the beginning of
# mktables, using an 'if' to add them for just this Unicode version going
# forward, so that mktables can continue to be used for earlier Unicode
# versions.
#
# When putting out a new Perl release, think about if any of the Deprecated
# properties should be moved to Suppressed.
#
# perlrecharclass.pod has a list of all the characters that are white space,
# which needs to be updated if there are changes.  A quick way to check if
# there have been changes would be to see if the number of such characters
# listed in perluniprops.pod (generated by running mktables) for the property
# \p{White_Space} is no longer 25.  Further investigation would then be
# necessary to classify the new characters as horizontal and vertical.
#
# The code in regexec.c for the \X match construct is intimately tied to the
# regular expression in UAX #29 (http://www.unicode.org/reports/tr29/).  You
# should see if it has changed, and if so, regexec.c should be modified.  The
# current one is
# ( CRLF
# | Prepend* ( RI-sequence | Hangul-Syllable | !Control )
#   ( Grapheme_Extend | SpacingMark )*
# | . )
#
# mktables has many checks to warn you if there are unexpected or novel things
# that it doesn't know how to handle.
#
# Module::CoreList should be changed to include the new release
#
# Also, you should regen l1_char_class_tab.h, by
#
# perl regen/mk_L_charclass.pl
#
# and, regen charclass_invlists.h by
#
# perl regen/mk_invlists.pl
#
# Finally:
#
# 	p4 submit
#
# --
# jhi@iki.fi; updated by nick@ccl4.org, public@khwilliamson.com