[lldb/crashlog] Surface error using SBCommandReturnObject argument

This patch allows the crashlog script to surface its errors to lldb by
using the provided SBCommandReturnObject argument.

rdar://9

[lldb/crashlog] Surface error using SBCommandReturnObject argument

This patch allows the crashlog script to surface its errors to lldb by
using the provided SBCommandReturnObject argument.

rdar://95048193

Differential Revision: https://reviews.llvm.org/D129614

Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>

show more ...

[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute.

Resubmission of https://reviews.llvm.org/D130309 with the 2 patches that fixed the linux buildb

[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute.

Resubmission of https://reviews.llvm.org/D130309 with the 2 patches that fixed the linux buildbot, and new windows fixes.

The FileSpec APIs allow users to modify instance variables directly by getting a non const reference to the directory and filename instance variables. This makes it impossible to control all of the times the FileSpec object is modified so we can clear cached member variables like m_resolved and with an upcoming patch caching if the file is relative or absolute. This patch modifies the APIs of FileSpec so no one can modify the directory or filename instance variables directly by adding set accessors and by removing the get accessors that are non const.

Many clients were using FileSpec::GetCString(...) which returned a unique C string from a ConstString'ified version of the result of GetPath() which returned a std::string. This caused many locations to use this convenient function incorrectly and could cause many strings to be added to the constant string pool that didn't need to. Most clients were converted to using FileSpec::GetPath().c_str() when possible. Other clients were modified to use the newly renamed version of this function which returns an actualy ConstString:

ConstString FileSpec::GetPathAsConstString(bool denormalize = true) const;

This avoids the issue where people were getting an already uniqued "const char *" that came from a ConstString only to put the "const char *" back into a "ConstString" object. By returning the ConstString instead of a "const char *" clients can be more efficient with the result.

The patch:
- Removes the non const GetDirectory() and GetFilename() get accessors
- Adds set accessors to replace the above functions: SetDirectory() and SetFilename().
- Adds ClearDirectory() and ClearFilename() to replace usage of the FileSpec::GetDirectory().Clear()/FileSpec::GetFilename().Clear() call sites
- Fixed all incorrect usage of FileSpec::GetCString() to use FileSpec::GetPath().c_str() where appropriate, and updated other call sites that wanted a ConstString to use the newly returned ConstString appropriately and efficiently.

Differential Revision: https://reviews.llvm.org/D130549

show more ...

Revert "[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute." and follow-ups

This reverts commit 9429b67b8e300e638d7828bbcb95585f85c4df4d.

It broke

Revert "[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute." and follow-ups

This reverts commit 9429b67b8e300e638d7828bbcb95585f85c4df4d.

It broke the build on Windows, see comments on https://reviews.llvm.org/D130309

It also reverts these follow-ups:

Revert "Fix buildbot breakage after https://reviews.llvm.org/D130309."
This reverts commit f959d815f4637890ebbacca379f1c38ab47e4e14.

Revert "Fix buildbot breakage after https://reviews.llvm.org/D130309."
This reverts commit 0bbce7a4c2d2bff622bdadd4323f93f5d90e6d24.

Revert "Cache the value for absolute path in FileSpec."
This reverts commit dabe877248b85b34878e75d5510339325ee087d0.

show more ...

Fix buildbot breakage after https://reviews.llvm.org/D130309.

[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute.

The FileSpect APIs allow users to modify instance variables directly by getting a non const ref

[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute.

The FileSpect APIs allow users to modify instance variables directly by getting a non const reference to the directory and filename instance variables. This makes it impossibly to control all of the times the FileSpec object is modified so we can clear the cache. This patch modifies the APIs of FileSpec so no one can modify the directory or filename directly by adding set accessors and by removing the get accessors that are non const.

Many clients were using FileSpec::GetCString(...) which returned a unique C string from a ConstString'ified version of the result of GetPath() which returned a std::string. This caused many locations to use this convenient function incorrectly and could cause many strings to be added to the constant string pool that didn't need to. Most clients were converted to using FileSpec::GetPath().c_str() when possible. Other clients were modified to use the newly renamed version of this function which returns an actualy ConstString:
ConstString FileSpec::GetPathAsConstString(bool denormalize = true) const;

This avoids the issue where people were getting an already uniqued "const char *" that came from a ConstString only to put the "const char *" back into a "ConstString" object. By returning the ConstString instead of a "const char *" clients can be more efficient with the result.

The patch:
- Removes the non const GetDirectory() and GetFilename() get accessors
- Adds set accessors to replace the above functions: SetDirectory() and SetFilename().
- Adds ClearDirectory() and ClearFilename() to replace usage of the FileSpec::GetDirectory().Clear()/FileSpec::GetFilename().Clear() call sites
- Fixed all incorrect usage of FileSpec::GetCString() to use FileSpec::GetPath().c_str() where appropriate, and updated other call sites that wanted a ConstString to use the newly returned ConstString appropriately and efficiently.

Differential Revision: https://reviews.llvm.org/D130309

show more ...

When the module path for `command script import` is invalid, echo the path.

We were just emitting "invalid module" w/o saying which module. That's
not particularly helpful.

Differential Revision:

When the module path for `command script import` is invalid, echo the path.

We were just emitting "invalid module" w/o saying which module. That's
not particularly helpful.

Differential Revision: https://reviews.llvm.org/D129338

show more ...

Remove Python 2 support from the ScriptInterpreter plugin

We dropped downstream support for Python 2 in the previous release. Now
that we have branched for the next release the window where this kin

Remove Python 2 support from the ScriptInterpreter plugin

We dropped downstream support for Python 2 in the previous release. Now
that we have branched for the next release the window where this kind of
change could introduce conflicts is closing too. Start by getting rid of
Python 2 support in the Script Interpreter plugin.

Differential revision: https://reviews.llvm.org/D124429

show more ...

[lldb] Remove FileSystem::Initialize from FileCollector

This patch removes the ability to instantiate the LLDB FileSystem class
with a FileCollector. It keeps the ability to collect files, but uses

[lldb] Remove FileSystem::Initialize from FileCollector

This patch removes the ability to instantiate the LLDB FileSystem class
with a FileCollector. It keeps the ability to collect files, but uses
the FileCollectorFileSystem to do that transparently.

Because the two are intertwined, this patch also removes the
finalization logic which copied the files over out of process.

show more ...

[lldb] Rename Logging.h to LLDBLog.h and clean up includes

Most of our code was including Log.h even though that is not where the
"lldb" log channel is defined (Log.h defines the generic logging
inf

[lldb] Rename Logging.h to LLDBLog.h and clean up includes

Most of our code was including Log.h even though that is not where the
"lldb" log channel is defined (Log.h defines the generic logging
infrastructure). This worked because Log.h included Logging.h, even
though it should.

After the recent refactor, it became impossible the two files include
each other in this direction (the opposite inclusion is needed), so this
patch removes the workaround that was put in place and cleans up all
files to include the right thing. It also renames the file to LLDBLog to
better reflect its purpose.

show more ...

[lldb] Convert "LLDB" log channel to the new API

[lldb] Decouple instrumentation from the reproducers

Remove the last remaining references to the reproducers from the
instrumentation. This patch renames the relevant files and macros.

Differential

[lldb] Decouple instrumentation from the reproducers

Remove the last remaining references to the reproducers from the
instrumentation. This patch renames the relevant files and macros.

Differential revision: https://reviews.llvm.org/D117712

show more ...

[lldb] Initialize Python exactly once

We got a few crash reports that showed LLDB initializing Python on two
separate threads. Make sure Python is initialized exactly once.

rdar://87287005

Differe

[lldb] Initialize Python exactly once

We got a few crash reports that showed LLDB initializing Python on two
separate threads. Make sure Python is initialized exactly once.

rdar://87287005

Differential revision: https://reviews.llvm.org/D117601

show more ...

[lldb/python] Use PythonObject in LLDBSwigPython functions

Return our PythonObject wrappers instead of raw PyObjects (obfuscated as
void *). This ensures that ownership (reference counts) of python

[lldb/python] Use PythonObject in LLDBSwigPython functions

Return our PythonObject wrappers instead of raw PyObjects (obfuscated as
void *). This ensures that ownership (reference counts) of python
objects is automatically tracked.

Differential Revision: https://reviews.llvm.org/D117462

show more ...

[LLDB] Fix Python GIL-not-held issues

The GIL must be held when calling any Python C API functions. In multithreaded applications that use callbacks this requirement can easily be violated by accide

[LLDB] Fix Python GIL-not-held issues

The GIL must be held when calling any Python C API functions. In multithreaded applications that use callbacks this requirement can easily be violated by accident. A general tool to ensure GIL health is not available, but patching Python Py_INCREF to add an assert provides a basic health check:

```
+int PyGILState_Check(void); /* Include/internal/pystate.h */
+
#define Py_INCREF(op) ( \
+ assert(PyGILState_Check()), \
_Py_INC_REFTOTAL _Py_REF_DEBUG_COMMA \
((PyObject *)(op))->ob_refcnt++)

#define Py_DECREF(op) \
do { \
+ assert(PyGILState_Check()); \
PyObject *_py_decref_tmp = (PyObject *)(op); \
if (_Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA \
--(_py_decref_tmp)->ob_refcnt != 0) \
```

Adding this assertion causes around 50 test failures in LLDB. Adjusting the scope of things guarded by `py_lock` fixes them.

More background: https://docs.python.org/3/glossary.html#term-global-interpreter-lock

Patch by Ralf Grosse-Kunstleve

Differential Revision: https://reviews.llvm.org/D114722

show more ...

[lldb] Fix that the embedded Python REPL crashes if it receives SIGINT

When LLDB receives a SIGINT while running the embedded Python REPL it
currently just crashes in ScriptInterpreterPythonImpl::In

[lldb] Fix that the embedded Python REPL crashes if it receives SIGINT

When LLDB receives a SIGINT while running the embedded Python REPL it
currently just crashes in ScriptInterpreterPythonImpl::Interrupt with an
error such as the one below:

Fatal Python error: PyThreadState_Get: the function must be called
with the GIL held, but the GIL is released (the current Python thread
state is NULL)

The faulty code that causes this error is this part of
ScriptInterpreterPythonImpl::Interrupt:

PyThreadState *state = PyThreadState_GET();
if (!state)
state = GetThreadState();
if (state) {
long tid = state->thread_id;
PyThreadState_Swap(state);
int num_threads = PyThreadState_SetAsyncExc(tid, PyExc_KeyboardInterrupt);

The obvious fix I tried is to just acquire the GIL before this code is
running which fixes the crash but the KeyboardInterrupt we want to raise
immediately is actually just queued and would only be raised once the
next line of input has been parsed (which e.g. won't interrupt Python
code that is currently waiting on a timer or IO from what I can see).
Also none of the functions we call here is marked as safe to be called
from a signal handler from what I can see, so we might still end up
crashing here with some bad timing.

Python 3.2 introduced PyErr_SetInterrupt to solve this and the function
takes care of all the details and avoids doing anything that isn't safe
to do inside a signal handler. The only thing we need to do is to
manually setup our own fake SIGINT handler that behaves the same way as
the standalone Python REPL signal handler (which raises a
KeyboardInterrupt).

From what I understand the old code used to work with Python 2 so I kept
the old code around until we officially drop support for Python 2.

There is a small gap here with Python 3.0->3.1 where we might still be
crashing, but those versions have reached their EOL more than a decade
ago so I think we don't need to bother about them.

Differential revision: https://reviews.llvm.org/D104886

show more ...

[lldb] Remove reproducer instrumentation

This patch removes most of the reproducer instrumentation. It keeps
around the LLDB_RECORD_* macros for logging. See [1] for more details.

[1] https://lists

[lldb] Remove reproducer instrumentation

This patch removes most of the reproducer instrumentation. It keeps
around the LLDB_RECORD_* macros for logging. See [1] for more details.

[1] https://lists.llvm.org/pipermail/lldb-dev/2021-September/017045.html

Differential revision: https://reviews.llvm.org/D116847

show more ...

[lldb/python] Fix (some) dangling pointers in our glue code

This starts to fix the other half of the lifetime problems in this code
-- dangling references. SB objects created on the stack will go aw

[lldb/python] Fix (some) dangling pointers in our glue code

This starts to fix the other half of the lifetime problems in this code
-- dangling references. SB objects created on the stack will go away
when the function returns, which is a problem if the python code they
were meant for stashes a reference to them somewhere. Most of the time
this goes by unnoticed, as the code rarely has a reason to store these,
but in case it does, we shouldn't respond by crashing.

This patch fixes the management for a couple of SB objects (Debugger,
Frame, Thread). The SB objects are now created on the heap, and
their ownership is immediately passed on to SWIG, which will ensure they
are destroyed when the last python reference goes away. I will handle
the other objects in separate patches.

I include one test which demonstrates the lifetime issue for SBDebugger.
Strictly speaking, one should create a test case for each of these
objects and each of the contexts they are being used. That would require
figuring out how to persist (and later access) each of these objects.
Some of those may involve a lot of hoop-jumping (we can run python code
from within a frame-format string). I don't think that is
necessary/worth it since the new wrapper functions make it very hard to
get this wrong.

Differential Revision: https://reviews.llvm.org/D115925

show more ...

[lldb] Clarify StructuredDataImpl ownership

StructuredDataImpl ownership semantics is unclear at best. Various
structures were holding a non-owning pointer to it, with a comment that
the object is o

[lldb] Clarify StructuredDataImpl ownership

StructuredDataImpl ownership semantics is unclear at best. Various
structures were holding a non-owning pointer to it, with a comment that
the object is owned somewhere else. From what I was able to gather that
"somewhere else" was the SBStructuredData object, but I am not sure that
all created object eventually made its way there. (It wouldn't matter
even if they did, as we are leaking most of our SBStructuredData
objects.)

Since StructuredDataImpl is just a collection of two (shared) pointers,
there's really no point in elaborate lifetime management, so this patch
replaces all StructuredDataImpl pointers with actual objects or
unique_ptrs to it. This makes it much easier to resolve SBStructuredData
leaks in a follow-up patch.

Differential Revision: https://reviews.llvm.org/D114791

show more ...

[lldb] Remove 'extern "C"' from the lldb-swig-python interface

The LLDBSWIGPython functions had (at least) two problems:
- There wasn't a single source of truth (a header file) for the
prototypes

[lldb] Remove 'extern "C"' from the lldb-swig-python interface

The LLDBSWIGPython functions had (at least) two problems:
- There wasn't a single source of truth (a header file) for the
prototypes of these functions. This meant that subtle differences
in copies of function declarations could go by undetected. And
not-so-subtle differences would result in strange runtime failures.
- All of the declarations had to have an extern "C" interface, because
the function definitions were being placed inside and extert "C" block
generated by swig.

This patch fixes both problems by moving the function definitions to the
%header block of the swig files. This block is not surrounded by extern
"C", and seems more appropriate anyway, as swig docs say it is meant for
"user-defined support code" (whereas the previous %wrapper code was for
automatically-generated wrappers).

It also puts the declarations into the SWIGPythonBridge header file
(which seems to have been created for this purpose), and ensures it is
included by all code wishing to define or use these functions. This
means that any differences in the declaration become a compiler error
instead of a runtime failure.

Differential Revision: https://reviews.llvm.org/D114369

show more ...

[lldb] Fix [some] leaks in python bindings

Using an lldb_private object in the bindings involves three steps
- wrapping the object in it's lldb::SB variant
- using swig to convert/wrap that to a PyO

[lldb] Fix [some] leaks in python bindings

Using an lldb_private object in the bindings involves three steps
- wrapping the object in it's lldb::SB variant
- using swig to convert/wrap that to a PyObject
- wrapping *that* in a lldb_private::python::PythonObject

Our SBTypeToSWIGWrapper was only handling the middle part. This doesn't
just result in increased boilerplate in the callers, but is also a
functionality problem, as it's very hard to get the lifetime of of all
of these objects right. Most of the callers are creating the SB object
(step 1) on the stack, which means that we end up with dangling python
objects after the function terminates. Most of the time this isn't a
problem, because the python code does not need to persist the objects.
However, there are legitimate cases where they can do it (and even if
the use case is not completely legitimate, crashing is not the best
response to that).

For this reason, some of our code creates the SB object on the heap, but
it has another problem -- it never gets cleaned up.

This patch begins to add a new function (ToSWIGWrapper), which does all
of the three steps, while properly taking care of ownership. In the
first step, I have converted most of the leaky code (except for
SBStructuredData, which needs a bit more work).

Differential Revision: https://reviews.llvm.org/D114259

show more ...

[lldb] fix -print-script-interpreter-info on windows

Apparently "{sys.prefix}/bin/python3" isn't where you find the
python interpreter on windows, so the test I wrote for
-print-script-interpreter-i

[lldb] fix -print-script-interpreter-info on windows

Apparently "{sys.prefix}/bin/python3" isn't where you find the
python interpreter on windows, so the test I wrote for
-print-script-interpreter-info is failing.

We can't rely on sys.executable at runtime, because that will point
to lldb.exe not python.exe.

We can't just record sys.executable from build time, because python
could have been moved to a different location.

But it should be OK to apply relative path from sys.prefix to sys.executable
from build-time to the sys.prefix at runtime.

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D113650

show more ...

Revert "[lldb] Fix that the embedded Python REPL crashes if it receives SIGINT"

This reverts commit cef1e07cc6d00b5b429d77133201e1f404a8023c.

It broke the windows bot.

[lldb] Fix that the embedded Python REPL crashes if it receives SIGINT

When LLDB receives a SIGINT while running the embedded Python REPL it currently
just crashes in `ScriptInterpreterPythonImpl::I

[lldb] Fix that the embedded Python REPL crashes if it receives SIGINT

When LLDB receives a SIGINT while running the embedded Python REPL it currently
just crashes in `ScriptInterpreterPythonImpl::Interrupt` with an error such as
the one below:

```

Fatal Python error: PyThreadState_Get: the function must be called with the GIL
held, but the GIL is released (the current Python thread state is NULL)

```

The faulty code that causes this error is this part of `ScriptInterpreterPythonImpl::Interrupt`:
```
PyThreadState *state = PyThreadState_GET();
if (!state)
state = GetThreadState();
if (state) {
long tid = state->thread_id;
PyThreadState_Swap(state);
int num_threads = PyThreadState_SetAsyncExc(tid, PyExc_KeyboardInterrupt);
```

The obvious fix I tried is to just acquire the GIL before this code is running
which fixes the crash but the `KeyboardInterrupt` we want to raise immediately
is actually just queued and would only be raised once the next line of input has
been parsed (which e.g. won't interrupt Python code that is currently waiting on
a timer or IO from what I can see). Also none of the functions we call here is
marked as safe to be called from a signal handler from what I can see, so we
might still end up crashing here with some bad timing.

Python 3.2 introduced `PyErr_SetInterrupt` to solve this and the function takes
care of all the details and avoids doing anything that isn't safe to do inside a
signal handler. The only thing we need to do is to manually setup our own fake
SIGINT handler that behaves the same way as the standalone Python REPL signal
handler (which raises a KeyboardInterrupt).

From what I understand the old code used to work with Python 2 so I kept the old
code around until we officially drop support for Python 2.

There is a small gap here with Python 3.0->3.1 where we might still be crashing,
but those versions have reached their EOL more than a decade ago so I think we
don't need to bother about them.

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D104886

show more ...

[lldb] make it easier to find LLDB's python

It is surprisingly difficult to write a simple python script that
can reliably `import lldb` without failing, or crashing. I'm
currently resorting to co

[lldb] make it easier to find LLDB's python

It is surprisingly difficult to write a simple python script that
can reliably `import lldb` without failing, or crashing. I'm
currently resorting to convolutions like this:

def find_lldb(may_reexec=False):
if prefix := os.environ.get('LLDB_PYTHON_PREFIX'):
if os.path.realpath(prefix) != os.path.realpath(sys.prefix):
raise Exception("cannot import lldb.\n"
f" sys.prefix should be: {prefix}\n"
f" but it is: {sys.prefix}")
else:
line1, line2 = subprocess.run(
['lldb', '-x', '-b', '-o', 'script print(sys.prefix)'],
encoding='utf8', stdout=subprocess.PIPE,
check=True).stdout.strip().splitlines()
assert line1.strip() == '(lldb) script print(sys.prefix)'
prefix = line2.strip()
os.environ['LLDB_PYTHON_PREFIX'] = prefix

if sys.prefix != prefix:
if not may_reexec:
raise Exception(
"cannot import lldb.\n" +
f" This python, at {sys.prefix}\n"
f" does not math LLDB's python at {prefix}")
os.environ['LLDB_PYTHON_PREFIX'] = prefix
python_exe = os.path.join(prefix, 'bin', 'python3')
os.execl(python_exe, python_exe, *sys.argv)

lldb_path = subprocess.run(['lldb', '-P'],
check=True, stdout=subprocess.PIPE,
encoding='utf8').stdout.strip()

sys.path = [lldb_path] + sys.path

This patch aims to replace all that with:

#!/usr/bin/env lldb-python
import lldb
...

... by adding the following features:

* new command line option: --print-script-interpreter-info. This
prints language-specific information about the script interpreter
in JSON format.

* new tool (unix only): lldb-python which finds python and exec's it.

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D112973

show more ...

[lldb] Remove ConstString from Process, ScriptInterpreter and StructuredData plugin names