[flang] Adjust prescanner fix for preprocessing (#122779)

Commas being optional in FORMAT statements, the tokenization of things
like 3I9HHOLLERITH is tricky. After tokenizing the initial '3', we d

[flang] Adjust prescanner fix for preprocessing (#122779)

Commas being optional in FORMAT statements, the tokenization of things
like 3I9HHOLLERITH is tricky. After tokenizing the initial '3', we don't
want to then take apparent identifier "I9HHOLLERITH" as the next token.
So the prescanner just consumes the letter ("I") as its own token in
this context.

A recent bug report complained that this can lead to incorrect results
when (in this case) the letter is a defined preprocessing macro. I
updated the prescanner to check that the letter is actually followed by
an instance of a problematic Hollerith literal.

And this broke two tests in the Fujitsu Fortran test suite that Linaro
runs, as it couldn't detect a following Hollerith literal that wasn't on
the same source line. We can't do look-ahead line continuation
processing in NextToken(), either.

So here's a second attempt at fixing the original problem: namely, the
letter that follows a decimal integer token is checked to see whether
it's the name of a defined macro.

show more ...

[flang] Refine tokenization trick that hid macro name (#121990)

In order to properly expose the Hollerith editing item in something like
FORMAT(3I9HHOLLERITH) as its own token, the tokenization rou

[flang] Refine tokenization trick that hid macro name (#121990)

In order to properly expose the Hollerith editing item in something like
FORMAT(3I9HHOLLERITH) as its own token, the tokenization routine in the
prescanner has special handling for digit strings followed by letters
("3I" above). This handler's effects are too broad, and prevent a macro
name from being recognized as such in a reported bug; make the test for
a hidden Hollerith more precise.

Fixes https://github.com/llvm/llvm-project/issues/121931.

show more ...

[FLANG][OPENMP] Fix handling of continuation lines in mixed OpenMP an… (#120714)

…d Fortran free-form
OpenMP feature was not enabled in the flang-new for the continuation
line, when we used the c

[FLANG][OPENMP] Fix handling of continuation lines in mixed OpenMP an… (#120714)

…d Fortran free-form
OpenMP feature was not enabled in the flang-new for the continuation
line, when we used the continuation line marker in combination of
free-form and OpenMP directive, it was throwing an error. PR is the fix
for that issue.

Added a fix for the following issue
https://github.com/llvm/llvm-project/issues/89559

show more ...

[flang] Fix continuation when line begins with empty macro expansion (#117407)

A free form source line that begins with a macro should still be
classified as a source line, and have its continuatio

[flang] Fix continuation when line begins with empty macro expansion (#117407)

A free form source line that begins with a macro should still be
classified as a source line, and have its continuation lines work, even
if the macro expands to an empty replacement.

Fixes https://github.com/llvm/llvm-project/issues/117297.

show more ...

[flang] Don't tokenize an exponent that isn't one (#117061)

The character 'e' or 'd' (either case) shouldn't be tokenized as part of
a real literal during preprocessing if it is not followed by an

[flang] Don't tokenize an exponent that isn't one (#117061)

The character 'e' or 'd' (either case) shouldn't be tokenized as part of
a real literal during preprocessing if it is not followed by an
optionally-signed digit string. Doing so prevents it from being
recognized as a macro name, or as the start of one.

Fixes https://github.com/llvm/llvm-project/issues/115676.

show more ...

[flang] Don't check fixed form label field too early (#117040)

When a fixed form source line begins with the name of a macro, don't
emit the usual warning message about a non-decimal character in t

[flang] Don't check fixed form label field too early (#117040)

When a fixed form source line begins with the name of a macro, don't
emit the usual warning message about a non-decimal character in the
label field. (The check for a macro was only being applied to free form
source lines, and the label field checking was unconditional).

Fixes https://github.com/llvm/llvm-project/issues/116914.

show more ...

[flang] Retain spaces when preprocessing fixed-form source (#112417)

When running fixed-form source through the compiler under -E, don't
aggressively remove space characters, since the parser won't

[flang] Retain spaces when preprocessing fixed-form source (#112417)

When running fixed-form source through the compiler under -E, don't
aggressively remove space characters, since the parser won't be parsing
the result and some tools might need to see the spaces in the -E
preprocessed output.

Fixes https://github.com/llvm/llvm-project/issues/112279.

show more ...

[flang] Tag warnings with LanguageFeature or UsageWarning (#110304)

(This is a big patch, but it's nearly an NFC. No test results have
changed and all Fortran tests in the LLVM test suites work as

[flang] Tag warnings with LanguageFeature or UsageWarning (#110304)

(This is a big patch, but it's nearly an NFC. No test results have
changed and all Fortran tests in the LLVM test suites work as expected.)

Allow a parser::Message for a warning to be marked with the
common::LanguageFeature or common::UsageWarning that controls it. This
will allow a later patch to add hooks whereby a driver will be able to
decorate warning messages with the names of its options that enable each
particular warning, and to add hooks whereby a driver can map those
enumerators by name to command-line options that enable/disable the
language feature and enable/disable the messages.

The default settings in the constructor for LanguageFeatureControl were
moved from its header file into its C++ source file.

Hooks for a driver to use to map the name of a feature or warning to its
enumerator were also added.

To simplify the tagging of warnings with their corresponding language
feature or usage warning, to ensure that they are properly controlled by
ShouldWarn(), and to ensure that warnings never issue at code sites in
module files, two new Warn() member function templates were added to
SemanticsContext and other contextual frameworks. Warn() can't be used
before source locations can be mapped to scopes, but the bulk of
existing code blocks testing ShouldWarn() and FindModuleFile() before
calling Say() were convertible into calls to Warn(). The ones that were
not convertible were extended with explicit calls to
Message::set_languageFeature() and set_usageWarning().

show more ...

[flang][preprocessor] Don't expand INCLUDE under -E by default (#110333)

Fortran INCLUDE lines have (until now) been treated like #include
directives. This isn't how things work with other Fortran

[flang][preprocessor] Don't expand INCLUDE under -E by default (#110333)

Fortran INCLUDE lines have (until now) been treated like #include
directives. This isn't how things work with other Fortran compilers when
running under the -E option for preprocessing only, so stop doing it by
default, and add -fpreprocess-include-lines to turn it back on when
desired.

show more ...

[flang] Tidy uses of raw_string_ostream (NFC)

As specified in the docs,
1) raw_string_ostream is always unbuffered and
2) the underlying buffer may be used directly

( 65b13610a5226b84889b923bae884b

[flang] Tidy uses of raw_string_ostream (NFC)

As specified in the docs,
1) raw_string_ostream is always unbuffered and
2) the underlying buffer may be used directly

( 65b13610a5226b84889b923bae884ba395ad084d for further reference )

Avoid unneeded calls to raw_string_ostream::str(), to avoid excess indirection.

show more ...

[flang][preprocessor] Change handling of macros in text from Fortran … (#108113)

…INCLUDE lines

The compiler current treats an INCLUDE line as essentially a synonym for
a preprocessing #include

[flang][preprocessor] Change handling of macros in text from Fortran … (#108113)

…INCLUDE lines

The compiler current treats an INCLUDE line as essentially a synonym for
a preprocessing #include directive. The causes macros that have been
defined at the point where the INCLUDE line is processed to be replaced
within the text of the included file.

This behavior is surprising to users who expect an INCLUDE line to be
expanded into its contents *after* preprocessing has been applied to the
original source file, with no further macro expansion.

Change INCLUDE line processing to use a fresh instance of Preprocessor
containing no macro definitions except _CUDA in CUDA Fortran
compilations and, if the original file was being preprocessed, the
standard definitions of __FILE__, __LINE__, and so forth.

show more ...

[flang][preprocessor] Extend handling of line continuation replacements (#107010)

Codes using traditional C preprocessors will sometimes put a keyword
macro name in a free form continuation line in

[flang][preprocessor] Extend handling of line continuation replacements (#107010)

Codes using traditional C preprocessors will sometimes put a keyword
macro name in a free form continuation line in order to get macro
replacement of part of an identifier, as in

call subr_&
&N&
&(1.)

where N is a keyword macro. f18 already handles this case, but not when
there is white space between the macro name and the following
continuation marker character '&'. Allow white space to appear.

Fixes https://github.com/llvm/llvm-project/issues/106931.

show more ...

[flang] Accept a non-breaking space character in source (#106611)

Accept non-breaking space characters (Latin-1 '\xa0', UTF-8 '\xc2'
'\xa0') in source code, converting them into regular spaces in t

[flang] Accept a non-breaking space character in source (#106611)

Accept non-breaking space characters (Latin-1 '\xa0', UTF-8 '\xc2'
'\xa0') in source code, converting them into regular spaces in the
cooked character stream when not in character literals.

show more ...

[flang] Handle pp-directives better in line continuation (#105572)

The code for detecting and processing some preprocessing directives
(conditional compilation and #line) while skipping comments be

[flang] Handle pp-directives better in line continuation (#105572)

The code for detecting and processing some preprocessing directives
(conditional compilation and #line) while skipping comments between one
source or compiler directive line and its continuations wasn't correctly
handling the case of such a directive following an explicit ampersand.

Fixes https://github.com/llvm/llvm-project/issues/100730 and
https://github.com/llvm/llvm-project/issues/100345.

show more ...

[flang][preprocessor] Fix handling of #line before free-form continua… (#100178)

…tion

See new test. A #line (or #) directive after a line ending with & and
before its continuation shouldn't eli

[flang][preprocessor] Fix handling of #line before free-form continua… (#100178)

…tion

See new test. A #line (or #) directive after a line ending with & and
before its continuation shouldn't elicit an error about mismatched
parentheses.

Fixes https://github.com/llvm/llvm-project/issues/100073.

show more ...

[flang][preprocessor] Handle initial "MACRO&" with no space (#98684)

The prescanner checks lines that begin with a keyword macro name to see
whether they should be treated as a comment or compiler

[flang][preprocessor] Handle initial "MACRO&" with no space (#98684)

The prescanner checks lines that begin with a keyword macro name to see
whether they should be treated as a comment or compiler directive
instead of a source line. This fails when the potential keyword macro
name is extended with identifier characters via Fortran line
continuation. Disable line continuation during this check.

show more ...

[flang] Accept a compiler directive sentinel after a semicolon (#96966)

Don't treat !DIR$ or an active !$ACC, !$OMP, &c. as a comment when they
appear after a semicolon, but instead treat them as a

[flang] Accept a compiler directive sentinel after a semicolon (#96966)

Don't treat !DIR$ or an active !$ACC, !$OMP, &c. as a comment when they
appear after a semicolon, but instead treat them as a compiler directive
sentinel.

show more ...

[flang][preprocessor] Expand some keyword macros in quoted character … (#96987)

…literals

To help port codes from compilers using pre-ANSI C preprocessors, which
didn't care much about context w

[flang][preprocessor] Expand some keyword macros in quoted character … (#96987)

…literals

To help port codes from compilers using pre-ANSI C preprocessors, which
didn't care much about context when replacing macros, support the
replacement of keyword macros in quoted character literals when (and
only when) the name of the keyword macro constitutes the entire
significant portion of a free form continuation line. See the new test
case for a motivating example.

Fixes https://github.com/llvm/llvm-project/issues/96781.

show more ...

[flang][preprocessing] Mix preprocessing directives with free form li… (#96244)

…ne continuation

Allow preprocessing directives to appear between a source line and its
continuation, including co

[flang][preprocessing] Mix preprocessing directives with free form li… (#96244)

…ne continuation

Allow preprocessing directives to appear between a source line and its
continuation, including conditional compilation directives (#if, #ifdef,
&c.).

Fixes https://github.com/llvm/llvm-project/issues/95476.

show more ...

[flang] Fix crash due to overly broad assertion (#95809)

Fix https://github.com/llvm/llvm-project/issues/95689 and add a
regression test.

[flang][preprocessor] Fixed-form continuation across preprocessing di… (#95332)

…rective

Implement fixed-form line continuation when the continuation line is the
result of text produced by an #i

[flang][preprocessor] Fixed-form continuation across preprocessing di… (#95332)

…rective

Implement fixed-form line continuation when the continuation line is the
result of text produced by an #include or other preprocessing directive.
This accommodates the somewhat common practice of putting dummy or
actual arguments into a header file and #including it into several code
sites.

Fixes https://github.com/llvm/llvm-project/issues/78928.

show more ...

[flang] Disable Fortran free form line continuation in non-source lin… (#94663)

…e produced by keyword macro replacement

When later initial keyword macro replacement will yield a line that is
no

[flang] Disable Fortran free form line continuation in non-source lin… (#94663)

…e produced by keyword macro replacement

When later initial keyword macro replacement will yield a line that is
not Fortran source, don't interpret "&" as a Fortran source line
continuation marker during tokenization of the line.

Fixes https://github.com/llvm/llvm-project/issues/82579.

show more ...

[flang][preprocessing] Handle #include after & line continuation (#93382)

Some applications like to use a CPP-style #include directive to pull in
a common list of arguments, dummy arguments, or COM

[flang][preprocessing] Handle #include after & line continuation (#93382)

Some applications like to use a CPP-style #include directive to pull in
a common list of arguments, dummy arguments, or COMMON block variables
after a free-form & line continuation marker. This works naturally with
compilers that run an actual cpp pass over the input before doing
anything specific to Fortran, but it's a case that I missed with this
integrated preprocessor.

show more ...

[flang] Ensure all warning/portability messages are guarded by Should… (#90518)

…Warn()

Many warning messages were being emitted unconditionally. Ensure that
all warnings are conditional on a tr

[flang] Ensure all warning/portability messages are guarded by Should… (#90518)

…Warn()

Many warning messages were being emitted unconditionally. Ensure that
all warnings are conditional on a true result from a call to
common::LanguageFeatureControl::ShouldWarn() so that it is easy for a
driver to disable them all, or, in the future, to provide per-warning
control over them.

show more ...

[flang][preprocessor] Support \ as line continuation (#89970)

When prescanning a Fortran source file with preprocessing enabled in
free source form, interpret a line-ending backslash as a source li

[flang][preprocessor] Support \ as line continuation (#89970)

When prescanning a Fortran source file with preprocessing enabled in
free source form, interpret a line-ending backslash as a source line
continuation marker as a C preprocessor would. This usage isn't
completely portable, but it is supported by GNU Fortran and appears in
the source for FPM package manager.

show more ...