[CrashReproducer] Add support for merging -ivfsoverlay

Merge all VFS mapped files inside -ivfsoverlay inputs into the vfs
overlay provided by the crash reproducer. This is the last missing piece
to

[CrashReproducer] Add support for merging -ivfsoverlay

Merge all VFS mapped files inside -ivfsoverlay inputs into the vfs
overlay provided by the crash reproducer. This is the last missing piece
to allow crash reproducers to fully work with user frameworks; when
combined with headermaps, it allows clang to find additional frameworks.

rdar://problem/27913709

llvm-svn: 290326

show more ...

[CrashReproducer] Setup a module collector callback for HeaderInclude

Collect missing include that cannot be fetched otherwise (e.g. when
using headermaps).

rdar://problem/27913709

llvm-svn: 289361

[VFS] Add 'ignore-non-existent-contents' field to YAML files

Add 'ignore-non-existent-contents' to tell the VFS whether an invalid path
obtained via 'external-contents' should cause iteration on the

[VFS] Add 'ignore-non-existent-contents' field to YAML files

Add 'ignore-non-existent-contents' to tell the VFS whether an invalid path
obtained via 'external-contents' should cause iteration on the VFS to stop.

If 'true', the VFS should ignore the entry and continue with the next. Allows
YAML files to be shared across multiple compiler invocations regardless of
prior existent paths in 'external-contents'. This global value is overridable
on a per-file basis.

This adds the parsing and write test part, but use by VFS comes next.

Differential Revision: https://reviews.llvm.org/D23422

rdar://problem/27531549

llvm-svn: 278456

show more ...

[NFC] Header cleanup

Summary: Removed unused headers, replaced some headers with forward class declarations

Patch by: Eugene <claprix@yandex.ru>

Differential Revision: https://reviews.llvm.org/D20

[NFC] Header cleanup

Summary: Removed unused headers, replaced some headers with forward class declarations

Patch by: Eugene <claprix@yandex.ru>

Differential Revision: https://reviews.llvm.org/D20100

llvm-svn: 275882

show more ...

[ModuleMap][CrashReproducer] Collect headers from inner frameworks

(1) Collect headers under inner frameworks (frameworks inside other
other frameworks).
(2) Make sure we also collect the right head

[ModuleMap][CrashReproducer] Collect headers from inner frameworks

(1) Collect headers under inner frameworks (frameworks inside other
other frameworks).
(2) Make sure we also collect the right header files inside them.

More info on (2):

Consider a dummy framework module B, with header Frameworks/B/B.h. Now
consider that another framework A, with header Frameworks/A/A.h, has a
layout with a inner framework Frameworks/A/Frameworks/B/B.h, where the
"B/B.h" part is a symlink for Frameworks/B/B.h. Also assume that
Frameworks/A/A.h includes <B/B.h>.

When parsing header Frameworks/A/A.h, framework module lookup is
performed in search for B, and it happens that
"Frameworks/A/Frameworks/B/B.h" path is registered in the module instead
of real "Frameworks/B/B.h". This occurs because
"Frameworks/A/Frameworks/B/B.h" is scanned first by the FileManager,
when looking for inner framework modules under Frameworks/A/Frameworks.
This makes Frameworks/A/Frameworks/B/B.h the default cached named inside
the FileManager for the B.h file UID.

This leads to modules being built without consistent paths to underlying
header files. This is usually not a problem in regular compilation flow,
but it's an issue when running the crash reproducer. The issue is that
clangs collect "Frameworks/A/Frameworks/B/B.h" but not
"Frameworks/B/B.h" into the VFS, leading to err_mmap_umbrella_clash. So
make sure we also collect the original header.

Differential Revision: http://reviews.llvm.org/D20194

rdar://problem/25880368

llvm-svn: 269502

show more ...

[CrashReproducer] Always use realpath for destination

When running reproducer scripts we need that original symlinks from the
source filesystem are reproduced in the VFS so that different virtual
pa

[CrashReproducer] Always use realpath for destination

When running reproducer scripts we need that original symlinks from the
source filesystem are reproduced in the VFS so that different virtual
paths can map to the same file, allowing the FileManager to share the
same UID between these virtual entries. This avoids all sorts of module
redefinition errors when using frameworks.

llvm-svn: 268825

show more ...

[CrashReproducer] Change module map callback signature. NFC

Use a StringRef instead of a FileEntry in the moduleMapAddHeader
callback to allow more flexibility on what to collect on further
patches.

[CrashReproducer] Change module map callback signature. NFC

Use a StringRef instead of a FileEntry in the moduleMapAddHeader
callback to allow more flexibility on what to collect on further
patches. This changes the interface I introduced in r264971.

llvm-svn: 268819

show more ...

[CrashReproducer] Setup 'use-external-names' in YAML files.

Hide the real paths when rebuilding from VFS by setting up the crash
reproducer to use 'use-external-names' = false. This way we avoid
mod

[CrashReproducer] Setup 'use-external-names' in YAML files.

Hide the real paths when rebuilding from VFS by setting up the crash
reproducer to use 'use-external-names' = false. This way we avoid
module redifinition errors and consistently use the same paths against
all modules.

With this change on Darwin we are able to simulate a crash for a simple
application using "Foundation/Foundation.h" (which relies on a bunch of
different frameworks and headers) and successfully rebuild all the
modules by relying solely at the VFS overlay.

llvm-svn: 266234

show more ...

Don't use PATH_MAX.

This is a SmallVector anyway, and so the exact size doesn't matter.

clang\lib\Frontend\ModuleDependencyCollector.cpp(83) : error C2065: 'PATH_MAX' : undeclared identifier
clang\

Don't use PATH_MAX.

This is a SmallVector anyway, and so the exact size doesn't matter.

clang\lib\Frontend\ModuleDependencyCollector.cpp(83) : error C2065: 'PATH_MAX' : undeclared identifier
clang\lib\Frontend\ModuleDependencyCollector.cpp(83) : error C2975: 'InternalLen' : invalid template argument for 'llvm::SmallString', expected compile-time constant expression
llvm\include\llvm/ADT/SmallString.h(24) : see declaration of 'InternalLen'

llvm-svn: 265634

show more ...

[CrashReproducer] Use toUppercase from include/clang/Basic/CharInfo.h

Use toUppercase instead of ::toupper()

llvm-svn: 265632

[CrashReproducer] Change std::toupper to ::toupper

Attempt to fix windows bots

llvm-svn: 265630

[CrashReproducer] Setup 'case-sensitive' in YAML files.

The crash reproducer was not setting up case sensitivity in the
VFS yaml files, which defaults to true. Make the crash reproducer
explicitly s

[CrashReproducer] Setup 'case-sensitive' in YAML files.

The crash reproducer was not setting up case sensitivity in the
VFS yaml files, which defaults to true. Make the crash reproducer
explicitly set that flag based on the case sensitivity of the .cache
path where vfs and modules are dumped.

llvm-svn: 265622

show more ...

[CrashReproducer] Move ModuleDependencyCollector method around. NFC

llvm-svn: 265621

[CrashReproducer] Add a module map callback for added headers

The current ModuleDependencyCollector has a AST listener to collect
header files present in loaded modules, but this isn't enough to col

[CrashReproducer] Add a module map callback for added headers

The current ModuleDependencyCollector has a AST listener to collect
header files present in loaded modules, but this isn't enough to collect
all headers needed in the crash reproducer. One of the reasons is that
the AST writer doesn't write symbolic link header paths in the pcm modules,
this makes the listeners on the reader only able to collect the real files.

Since the module maps could contain submodules that use headers which
are symbolic links, not collecting those forbid the reproducer scripts
to regen the modules.

For instance:

usr/include/module.map:
...
module pthread {
header "pthread.h"
export *

module impl {
header "pthread_impl.h"
export *
}
}
...

usr/include/pthread/pthread_impl.h
usr/include/pthread_impl.h -> pthread/pthread_impl.h

The AST dump for the module above:

<SUBMODULE_HEADER abbrevid=6/> blob data = 'pthread_impl.h'
<SUBMODULE_TOPHEADER abbrevid=7/> blob data = '/<path_to_sdk>/usr/include/pthread/pthread_impl.h'

Note that we don't have "usr/include/pthread_impl.h" which is requested
by the module.map in case we want to reconstruct the module in the
reproducer. The reason the original symbolic link path isn't used is
because the headers are kept by name and requested through the
FileManager, which unique files and returns the real path only.

To fix that, add a callback to be invoked everytime a header is added
while parsing module maps and hook that up to the module dependecy
collector. This callback is only registered when generating the
reproducer.

Differential Revision: http://reviews.llvm.org/D18585

rdar://problem/24499339

llvm-svn: 264971

show more ...

[CrashReproducer] Cleanup and move functionality around in ModuleDependencyCollector. NFC

- Make ModuleDependencyCollector use the DependencyCollector interface
- Move some methods from ModuleDepend

[CrashReproducer] Cleanup and move functionality around in ModuleDependencyCollector. NFC

- Make ModuleDependencyCollector use the DependencyCollector interface
- Move some methods from ModuleDependencyListener to ModuleDependencyCollector
in order to share common functionality with other future possible
callbacks.

llvm-svn: 264808

show more ...

Reapply [2] [VFS] Add 'overlay-relative' field to YAML files

This reapplies r261552 and r263748. Fixed testcase to reapply.

The VFS overlay mapping between virtual paths and real paths is done thro

Reapply [2] [VFS] Add 'overlay-relative' field to YAML files

This reapplies r261552 and r263748. Fixed testcase to reapply.

The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML files, which contains hardcoded paths
to the real files.

When a module compilation crashes, headers are dumped into <name>.cache/vfs
directory and are mapped via the <name>.cache/vfs/vfs.yaml. The script
generated for reproduction uses -ivfsoverlay pointing to file to gather the
mapping between virtual paths and files inside <name>.cache/vfs. Currently, we
are only capable of reproducing such crashes in the same machine as they
happen, because of the hardcoded paths in 'external-contents'.

To be able to reproduce a crash in another machine, this patch introduces a new
option in the VFS yaml file called 'overlay-relative'. When it's equal to
'true' it means that the provided path to the YAML file through the
-ivfsoverlay option should also be used to prefix the final path for every
'external-contents'.

Example, given the invocation snippet "... -ivfsoverlay
<name>.cache/vfs/vfs.yaml" and the following entry in the yaml file:

"overlay-relative": "true",
"roots": [
...
"type": "directory",
"name": "/usr/include",
"contents": [
{
"type": "file",
"name": "stdio.h",
"external-contents": "/usr/include/stdio.h"
},
...

Here, a file manager request for virtual "/usr/include/stdio.h", that will map
into real path "/<absolute_path_to>/<name>.cache/vfs/usr/include/stdio.h.

This is a useful feature for debugging module crashes in machines other than
the one where the error happened.

Differential Revision: http://reviews.llvm.org/D17457

rdar://problem/24499339

llvm-svn: 263893

show more ...

Revert "Reapply [VFS] Add 'overlay-relative' field to YAML files"

Tests failing on
http://bb.pgr.jp/builders/cmake-clang-x86_64-linux/builds/46102

This reverts commit a1683cd6c9e07359c09f86e98a4db6

Revert "Reapply [VFS] Add 'overlay-relative' field to YAML files"

Tests failing on
http://bb.pgr.jp/builders/cmake-clang-x86_64-linux/builds/46102

This reverts commit a1683cd6c9e07359c09f86e98a4db6b4e1bc51fc.

llvm-svn: 263750

show more ...

Reapply [VFS] Add 'overlay-relative' field to YAML files

This reapplies r261552.

The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML

Reapply [VFS] Add 'overlay-relative' field to YAML files

This reapplies r261552.

The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML files, which contains hardcoded paths
to the real files.

When a module compilation crashes, headers are dumped into <name>.cache/vfs
directory and are mapped via the <name>.cache/vfs/vfs.yaml. The script
generated for reproduction uses -ivfsoverlay pointing to file to gather the
mapping between virtual paths and files inside <name>.cache/vfs. Currently, we
are only capable of reproducing such crashes in the same machine as they
happen, because of the hardcoded paths in 'external-contents'.

To be able to reproduce a crash in another machine, this patch introduces a new
option in the VFS yaml file called 'overlay-relative'. When it's equal to
'true' it means that the provided path to the YAML file through the
-ivfsoverlay option should also be used to prefix the final path for every
'external-contents'.

Example, given the invocation snippet "... -ivfsoverlay
<name>.cache/vfs/vfs.yaml" and the following entry in the yaml file:

"overlay-relative": "true",
"roots": [
...
"type": "directory",
"name": "/usr/include",
"contents": [
{
"type": "file",
"name": "stdio.h",
"external-contents": "/usr/include/stdio.h"
},
...

Here, a file manager request for virtual "/usr/include/stdio.h", that will map
into real path "/<absolute_path_to>/<name>.cache/vfs/usr/include/stdio.h.

This is a useful feature for debugging module crashes in machines other than
the one where the error happened.

Differential Revision: http://reviews.llvm.org/D17457

rdar://problem/24499339

llvm-svn: 263748

show more ...

[VFS] Remove wrong header include

Follow up from r263686. Forgot to remove the wrong header file.

llvm-svn: 263691

Reapply [2]: [VFS] Add support for handling path traversals

This was applied twice r261551 and 263617 and later reverted because:

(1) Windows bot failing on unittests. Change the current behavior t

Reapply [2]: [VFS] Add support for handling path traversals

This was applied twice r261551 and 263617 and later reverted because:

(1) Windows bot failing on unittests. Change the current behavior to do
not handle path traversals on windows.

(2) Windows bot failed to include llvm/Config/config.h in order to use
HAVE_REALPATH. Use LLVM_ON_UNIX instead, as done in lib/Basic/FileManager.cpp.

Handle ".", ".." and "./" with trailing slashes while collecting files
to be dumped into the vfs overlay directory.

Include the support for symlinks into components. Given the path:

/install-dir/bin/../lib/clang/3.8.0/include/altivec.h, if "bin"
component is a symlink, it's not safe to use `path::remove_dots` here,
and `realpath` is used to get the right answer. Since `realpath`
is expensive, we only do it at collecting time (which only happens
during the crash reproducer) and cache the base directory for fast lookups.

Overall, this makes the input to the VFS YAML file to be canonicalized
to never contain traversal components.

Differential Revision: http://reviews.llvm.org/D17104

rdar://problem/24499339

llvm-svn: 263686

show more ...

Revert r263617, "Reapply: [VFS] Add support for handling path traversals"

It broke standalone clang build.

llvm-svn: 263636

Revert r263622, "clang/lib/Frontend/ModuleDependencyCollector.cpp: Use clang/Config/config.h instead of llvm's."

I'll commit better fix(es) later. "llvm/Config/config.h" shouldn't be available here.

Revert r263622, "clang/lib/Frontend/ModuleDependencyCollector.cpp: Use clang/Config/config.h instead of llvm's."

I'll commit better fix(es) later. "llvm/Config/config.h" shouldn't be available here.

llvm-svn: 263623

show more ...

clang/lib/Frontend/ModuleDependencyCollector.cpp: Use clang/Config/config.h instead of llvm's.

llvm-svn: 263622

Reapply: [VFS] Add support for handling path traversals

This is originally r261551, reverted because of windows bots failing on
unittests. Change the current behavior to do not handle path traversal

Reapply: [VFS] Add support for handling path traversals

This is originally r261551, reverted because of windows bots failing on
unittests. Change the current behavior to do not handle path traversals
on windows.

Handle ".", ".." and "./" with trailing slashes while collecting files
to be dumped into the vfs overlay directory.

Include the support for symlinks into components. Given the path:

/install-dir/bin/../lib/clang/3.8.0/include/altivec.h, if "bin"
component is a symlink, it's not safe to use `path::remove_dots` here,
and `realpath` is used to get the right answer. Since `realpath`
is expensive, we only do it at collecting time (which only happens
during the crash reproducer) and cache the base directory for fast lookups.

Overall, this makes the input to the VFS YAML file to be canonicalized
to never contain traversal components.

Differential Revision: http://reviews.llvm.org/D17104

rdar://problem/24499339

llvm-svn: 263617

show more ...

Revert "[VFS] Add support for handling path traversals"

This reverts commit r261551 due to failing tests in windows bots:

http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10054

Failin

Revert "[VFS] Add support for handling path traversals"

This reverts commit r261551 due to failing tests in windows bots:

http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10054

Failing Tests (4):
Clang-Unit :: Basic/BasicTests.exe/VFSFromYAMLTest.CaseInsensitive
Clang-Unit :: Basic/BasicTests.exe/VFSFromYAMLTest.DirectoryIteration
Clang-Unit :: Basic/BasicTests.exe/VFSFromYAMLTest.MappedFiles
Clang-Unit :: Basic/BasicTests.exe/VFSFromYAMLTest.UseExternalName

llvm-svn: 261654

show more ...