Revert BTF commit series.

The initial patch was not reviewed, and does not have any tests;
it should not have been merged.

This reverts 344395, 344390, 344387, 344385, 344381, 344376,
and 344366.

Revert BTF commit series.

The initial patch was not reviewed, and does not have any tests;
it should not have been merged.

This reverts 344395, 344390, 344387, 344385, 344381, 344376,
and 344366.

llvm-svn: 344405

show more ...

[BPF] Add BTF generation for BPF target

BTF is the debug format for BPF, a kernel virtual machine
and widely used for tracing, networking and security, etc ([1]).

Currently only instruction streams

[BPF] Add BTF generation for BPF target

BTF is the debug format for BPF, a kernel virtual machine
and widely used for tracing, networking and security, etc ([1]).

Currently only instruction streams are passed to kernel,
the kernel verifier verifies them before execution. In order to
provide better visibility of bpf programs to user space
tools, some debug information, e.g., function names and
debug line information are desirable for kernel so tools
can get such information with better annotation
for jited instructions for performance or other reasons.

The dwarf is too complicated in kernel and for BPF.
Hence, BTF is designed to be the debug format for BPF ([2]).
Right now, pahole supports BTF for types, which
are generated based on dwarf sections in the ELF file.

In order to annotate performance metrics for jited bpf insns,
it is necessary to pass debug line info to the kernel.
Furthermore, we want to pass the actual code to the
kernel because of the following reasons:

. bpf program typically is small so storage overhead
should be small.
. in bpf land, it is totally possible that
an application loads the bpf program into the
kernel and then that application quits, so
holding debug info by the user space application
is not practical.
. having source codes directly kept by kernel
would ease deployment since the original source
code does not need ship on every hosts and
kernel-devel package does not need to be
deployed even if kernel headers are used.

The only reliable time to get the source code is
during compilation time. This will result in both more
accurate information and easier deployment as
stated in the above.

Another consideration is for JIT. The project like bcc
use MCJIT to compile a C program into bpf insns and
load them to the kernel ([3]). The generated BTF sections
will be readily available for such cases as well.

This patch implemented generation of BTF info in llvm
compiler. The BTF related sections will be generated
when both -target bpf and -g are specified. Two sections
are generated:
.BTF contains all the type and string information, and
.BTF.ext contains the func_info and line_info.

The separation is related to how two sections are used
differently in bpf loader, e.g., linux libbpf ([4]).
The .BTF section can be loaded into the kernel directly
while .BTF.ext needs loader manipulation before loading
to the kernel. The format of the each section is roughly
defined in llvm:include/llvm/MC/MCBTFContext.h and
from the implementation in llvm:lib/MC/MCBTFContext.cpp.
A later example also shows the contents in each section.

The type and func_info are gathered during CodeGen/AsmPrinter
by traversing dwarf debug_info. The line_info is
gathered in MCObjectStreamer before writing to
the object file. After all the information is gathered,
the two sections are emitted in MCObjectStreamer::finishImpl.

With cmake CMAKE_BUILD_TYPE=Debug, the compiler can
dump out all the tables except insn offset, which
will be resolved later as relocation records.
The debug type "btf" is used for BTFContext dump.

Dwarf tests the debug info generation with
llvm-dwarfdump to decode the binary sections and
check whether the result is expected. Currently
we do not have such a tool yet. We will implement
btf dump functionality in bpftool ([5]) as the bpftool is
considered the recommended tool for bpf introspection.
The implementation for type and func_info is tested
with linux kernel test cases. The line_info is visually
checked with dump from linux kernel libbpf ([4]) and
checked with readelf dumping section raw data.

Note that the .BTF and .BTF.ext information will not
be emitted to assembly code and there is no assembler
support for BTF either.

In the below, with a clang/llvm built with CMAKE_BUILD_TYPE=Debug,
Each table contents are shown for a simple C program.

-bash-4.2$ cat -n test.c
1 struct A {
2 int a;
3 char b;
4 };
5
6 int test(struct A *t) {
7 return t->a;
8 }
-bash-4.2$ clang -O2 -target bpf -g -mllvm -debug-only=btf -c test.c
Type Table:
[1] FUNC name_off=1 info=0x0c000001 size/type=2
param_type=3
[2] INT name_off=12 info=0x01000000 size/type=4
desc=0x01000020
[3] PTR name_off=0 info=0x02000000 size/type=4
[4] STRUCT name_off=16 info=0x04000002 size/type=8
name_off=18 type=2 bit_offset=0
name_off=20 type=5 bit_offset=32
[5] INT name_off=22 info=0x01000000 size/type=1
desc=0x02000008

String Table:
0 :
1 : test
6 : .text
12 : int
16 : A
18 : a
20 : b
22 : char
27 : test.c
34 : int test(struct A *t) {
58 : return t->a;

FuncInfo Table:
sec_name_off=6
insn_offset=<Omitted> type_id=1

LineInfo Table:
sec_name_off=6
insn_offset=<Omitted> file_name_off=27 line_off=34 line_num=6 column_num=0
insn_offset=<Omitted> file_name_off=27 line_off=58 line_num=7 column_num=3
-bash-4.2$ readelf -S test.o
......
[12] .BTF PROGBITS 0000000000000000 0000028d
00000000000000c1 0000000000000000 0 0 1
[13] .BTF.ext PROGBITS 0000000000000000 0000034e
0000000000000050 0000000000000000 0 0 1
[14] .rel.BTF.ext REL 0000000000000000 00000648
0000000000000030 0000000000000010 16 13 8
......
-bash-4.2$

The latest linux kernel ([6]) can already support .BTF with type information.
The [7] has the reference implementation in linux kernel side
to support .BTF.ext func_info. The .BTF.ext line_info support is not
implemented yet. If you have difficulty accessing [6], you can
manually do the following to access the code:

git clone https://github.com/yonghong-song/bpf-next-linux.git
cd bpf-next-linux
git checkout btf

The change will push to linux kernel soon once this patch is landed.

References:
[1]. https://www.kernel.org/doc/Documentation/networking/filter.txt
[2]. https://lwn.net/Articles/750695/
[3]. https://github.com/iovisor/bcc
[4]. https://github.com/torvalds/linux/tree/master/tools/lib/bpf
[5]. https://github.com/torvalds/linux/tree/master/tools/bpf/bpftool
[6]. https://github.com/torvalds/linux
[7]. https://github.com/yonghong-song/bpf-next-linux/tree/btf

Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Yonghong Song <yhs@fb.com>
Acked-by: Alexei Starovoitov <ast@kernel.org>

Differential Revision: https://reviews.llvm.org/D52950

llvm-svn: 344366

show more ...

Fix typos. NFC

llvm-svn: 341740

Revert r340904 "[llvm-mc] - Allow to set custom flags for debug sections."

It broke PPC64 BB:
http://lab.llvm.org:8011/builders/clang-ppc64be-linux/builds/23252

llvm-svn: 340906

[llvm-mc] - Allow to set custom flags for debug sections.

I am experimenting with a single split dwarf (.dwo sections in .o files).
I want to make linker to ignore .dwo sections in .o, for that I am

[llvm-mc] - Allow to set custom flags for debug sections.

I am experimenting with a single split dwarf (.dwo sections in .o files).
I want to make linker to ignore .dwo sections in .o, for that I am trying to add
SHF_EXCLUDE flag ("E") for them in my asm sample.

I found that currently, it is impossible to add any flag for debug sections using llvm-mc.

That happens because we have a set of predefined unique sections created early with default flags:
https://github.com/llvm-mirror/llvm/blob/master/lib/MC/MCObjectFileInfo.cpp#L391

This patch allows a user to add any flags he wants.

I had to edit TargetLoweringObjectFileImpl.cpp to set MetaData type for debug sections.
Their kind was Data by default (so they were allocatable) and so after changes introduced by
this patch the SHF_ALLOC flag was applied for them, what does not make sense for debug sections.
One of OrcJITTests tests failed because of that.

Differential revision: https://reviews.llvm.org/D51361

llvm-svn: 340904

show more ...

[codeview] Clean up machinery for deferring .cv_loc emission

Now that we create the label at the point of the directive, we don't
need to set the "current CV location", and then later when we emit t

[codeview] Clean up machinery for deferring .cv_loc emission

Now that we create the label at the point of the directive, we don't
need to set the "current CV location", and then later when we emit the
next instruction, create a label for it and emit it.

DWARF still defers the labels used in .debug_loc until the next
instruction or value, for reasons unknown.

llvm-svn: 340883

show more ...

Use debug-prefix-map for AT_NAME

AT_NAME was being emitted before the directory paths were remapped. This
ensures that all paths are remapped before anything is emitted.

An additional test case has

Use debug-prefix-map for AT_NAME

AT_NAME was being emitted before the directory paths were remapped. This
ensures that all paths are remapped before anything is emitted.

An additional test case has been added.

Note that this only works if the replacement string is an absolute path.
If not, then AT_decl_file believes the new path is a relative path, and
joins that path with the compilation directory. I do not know of a good
way to resolve this.

Patch by: Siddhartha Bagaria (starsid)

Differential revision: https://reviews.llvm.org/D49169

llvm-svn: 336793

show more ...

Support -fdebug-prefix-map in llvm-mc. This is useful to omit the
debug compilation dir when compiling assembly files with -g.
Part of PR38050.

Patch by Siddhartha Bagaria!

Differential Revision:

Support -fdebug-prefix-map in llvm-mc. This is useful to omit the
debug compilation dir when compiling assembly files with -g.
Part of PR38050.

Patch by Siddhartha Bagaria!

Differential Revision: https://reviews.llvm.org/D48988

llvm-svn: 336680

show more ...

[DWARFv5] Allow ".loc 0" to refer to the root file.

DWARF v5 explicitly represents file #0 in the line table. Prior
versions did not, so ".loc 0" is still an error in those cases.

Differential Rev

[DWARFv5] Allow ".loc 0" to refer to the root file.

DWARF v5 explicitly represents file #0 in the line table. Prior
versions did not, so ".loc 0" is still an error in those cases.

Differential Revision: https://reviews.llvm.org/D48452

llvm-svn: 335350

show more ...

[WebAssembly] MC: Add compile-twice test and fix corresponding bug

Differential Revision: https://reviews.llvm.org/D47398

llvm-svn: 333494

[WebAssembly] Create section start symbols automatically for all sections

These symbols only get included in the output symbols table if
they are used in a relocation.

This behaviour matches more c

[WebAssembly] Create section start symbols automatically for all sections

These symbols only get included in the output symbols table if
they are used in a relocation.

This behaviour matches more closely the ELF object writer.

Differential Revision: https://reviews.llvm.org/D46561

llvm-svn: 332005

show more ...

[WebAssembly] MC: Create and use first class section symbols

Differential Revision: https://reviews.llvm.org/D46335

llvm-svn: 331413

[codeview] Ignore .cv_loc directives at the end of a function

If no data or instructions are emitted after a location directive, we
should clear the cv_loc when we change sections, or it will be emi

[codeview] Ignore .cv_loc directives at the end of a function

If no data or instructions are emitted after a location directive, we
should clear the cv_loc when we change sections, or it will be emitted
at the beginning of the next section. This violates our invariant that
all .cv_loc directives belong to the same section. Add clearer
assertions for this.

llvm-svn: 330884

show more ...

[DebugInfo] Support DWARF v5 source code embedding extension

In DWARF v5 the Line Number Program Header is extensible, allowing values with
new content types. In this extension a content type is add

[DebugInfo] Support DWARF v5 source code embedding extension

In DWARF v5 the Line Number Program Header is extensible, allowing values with
new content types. In this extension a content type is added,
DW_LNCT_LLVM_source, which contains the embedded source code of the file.

Add new optional attribute for !DIFile IR metadata called source which contains
source text. Use this to output the source to the DWARF line table of code
objects. Analogously extend METADATA_FILE in Bitcode and .file directive in ASM
to support optional source.

Teach llvm-dwarfdump and llvm-objdump about the new values. Update the output
format of llvm-dwarfdump to make room for the new attribute on file_names
entries, and support embedded sources for the -source option in llvm-objdump.

Differential Revision: https://reviews.llvm.org/D42765

llvm-svn: 325970

show more ...

[DWARFv5] Turn an assert into a diagnostic. Hand-coded assembler files
should not trigger assertions.

Differential Revision: https://reviews.llvm.org/D43152

llvm-svn: 325831

[WebAssemlby] MC: Don't write COMDAT symbols as global imports

This was causing undefined references at link time in lld.

Differential Revision: https://reviews.llvm.org/D41959

llvm-svn: 322309

[DWARFv5] MC support for MD5 file checksums

Extend .file directive syntax to allow specifying an MD5 checksum for
the source file. Emit the checksums in DWARF v5 line tables.

llvm-svn: 322134

[WebAssembly] MC: Fix crash when -g specified.

At this point we don't output any debug sections or thier
relocations.

Differential Revision: https://reviews.llvm.org/D39076

llvm-svn: 316240

[WebAssembly] Remove flags from MCSectionWasm

Looks like these were copied from the ELF sections but
don't apply to Wasm and were not used anywhere.

Also remove unused Wasm methods in MCContext.

D

[WebAssembly] Remove flags from MCSectionWasm

Looks like these were copied from the ELF sections but
don't apply to Wasm and were not used anywhere.

Also remove unused Wasm methods in MCContext.

Differential Revision: https://reviews.llvm.org/D37633

llvm-svn: 313058

show more ...

Move Object format code to lib/BinaryFormat.

This creates a new library called BinaryFormat that has all of
the headers from llvm/Support containing structure and layout
definitions for various type

Move Object format code to lib/BinaryFormat.

This creates a new library called BinaryFormat that has all of
the headers from llvm/Support containing structure and layout
definitions for various types of binary formats like dwarf, coff,
elf, etc as well as the code for identifying a file from its
magic.

Differential Revision: https://reviews.llvm.org/D33843

llvm-svn: 304864

show more ...

Sort the remaining #include lines in include/... and lib/....

I did this a long time ago with a janky python script, but now
clang-format has built-in support for this. I fed clang-format every
line

Sort the remaining #include lines in include/... and lib/....

I did this a long time ago with a janky python script, but now
clang-format has built-in support for this. I fed clang-format every
line with a #include and let it re-sort things according to the precise
LLVM rules for include ordering baked into clang-format these days.

I've reverted a number of files where the results of sorting includes
isn't healthy. Either places where we have legacy code relying on
particular include ordering (where possible, I'll fix these separately)
or where we have particular formatting around #include lines that
I didn't want to disturb in this patch.

This patch is *entirely* mechanical. If you get merge conflicts or
anything, just ignore the changes in this patch and run clang-format
over your #include lines in the files.

Sorry for any noise here, but it is important to keep these things
stable. I was seeing an increasing number of patches with irrelevant
re-ordering of #include lines because clang-format was used. This patch
at least isolates that churn, makes it easy to skip when resolving
conflicts, and gets us to a clean baseline (again).

llvm-svn: 304787

show more ...

Fix asm printing of associated sections.

Make MCSectionELF::AssociatedSection be a link to a symbol, because
that's how it works in the assembly, and use it in the asm printer.

llvm-svn: 297769

Disallow redefinition of section symbols.

Differential Revision: https://reviews.llvm.org/D30235

llvm-svn: 296180

Initialize MCContext::InlineSrcMgr in the constructor.

Found with ASan (and a local source change) on test/CodeGen/XCore/section-name.ll.

llvm-svn: 296179

[WebAssembly] Add skeleton MC support for the Wasm container format

This just adds the basic skeleton for supporting a new object file format.
All of the actual encoding will be implemented in follo

[WebAssembly] Add skeleton MC support for the Wasm container format

This just adds the basic skeleton for supporting a new object file format.
All of the actual encoding will be implemented in followup patches.

Differential Revision: https://reviews.llvm.org/D26722

llvm-svn: 295803

show more ...