How cppyy relates to CppInterOp and how it works

- Reducing cppyy Dependencies
- Code Example
- cppyy components
- cppyy repos (enhanced forks+upstream)
- Credits to Devs
- Further Reading Links

New commit:
- Added some changes after David's Review (see comment responses)

Author: QuillPusher

Diff for: docs/UsingCppInterOp.rst

@@ -65,35 +65,243 @@ robust (simple function calls, no inheritance, etc.). The goal is to make it as
 close to the compiler API as possible, and each routine should do just one thing.
 that it was designed for.
 
-Further Enhancing the Dynamic/Automatic bindings in CPPYY
-=========================================================
-The main use case for CppInterOp is the CPPYY service. CPPYY is an
-automatic run-time bindings generator for Python and C++, and supports a wide
-range of C++ features (e.g., template instantiation). It operates on demand and
+How cppyy relates to CppInterOp and how it works
+================================================
+
+CppInterOp ties into several `enhancements in cppyy`_, following are a few:
+
+1. **CppInterOp is a cppyy-inspired Library** that enables interoperability
+   with C++ code, bringing the speed and efficiency of C++ to simpler, more
+   interactive languages like Python.
+
+2. **Reducing dependencies** of cppyy to minimize code bloat and make it
+   faster.
+
+3. **LLVM Integration**: CppInterOp's integration with LLVM's Clang-REPL
+   component (that can then be used as a runtime compiler for CPPYY), will
+   further enhance CPPYY's performance.
+
+**CppInterOp is a cppyy-inspired Library**
+
+A major use case for CppInterOp is the CPPYY service. CPPYY is an automatic
+run-time bindings generator for Python and C++, and supports a wide range of
+C++ features (e.g., template instantiation). It operates on demand and
 generates only what is necessary. It requires a compiler (Cling or Clang-REPL).
 that can be available during programme runtime.
 
+**Reducing Dependencies**
+
+Recent work done on cppyy has been focused on removing unnecessary
+dependencies on domain-specific infrastructure (e.g., the ROOT
+framework). The idea was to convert the cppyy-backend to use Cling
+directly (instead of `ROOT meta`_), and then use it in cppyy.
+
+Only a small set of APIs are needed to connect to the interpreter (Cling),
+since other APIs are already available in the standard compiler. This is what
+led to the creation of CppInterOp (a library of helper functions), that helped
+extract out things that were unnecessary for cppyy, etc.
+
+The cppyy API surface is now incomparably smaller and simpler than what it used
+to be.
+
+**LLVM Integration**
+
 Once CppInterOp is integrated with LLVM's Clang-REPL component (that can then
 be used as a runtime compiler for CPPYY), it will further enhance CPPYY's
 performance in the following ways:
 
 
-**Simpler codebase:** The removal of string parsing logic will lead to a simpler
-code base.
+- *Simpler codebase:* The removal of string parsing logic will lead to a
+  simpler code base.
+
+- *Built into the LLVM toolchain:* The CppInterOp interfaces will be part of
+  the LLVM toolchain (as part of Clang-REPL).
+
+- *Better C++ Support:* C++ features such as Partial Template Specialisation
+  will be available through CppInterOp.
+
+- *Fewer Lines of Code:* A lot of dependencies and workarounds will be
+  removed, reducing the lines of code required to execute CPPYY.
+
+- *Well tested interoperability Layer:* The CppInterOp interfaces have full
+  unit test coverage.
+
+**Making C++ More Social**
+
+cppyy is the first use case demonstrating how CppInterOp can enable C++ to be
+more easily interoperable with other languages. This helps many data scientists
+that are working with legacy C++ code and would like to use simpler, more
+interactive languages.
+
+The goal of these enhancements is to eventually land these interoperability
+tools (including CppInterOp) to greater communities like LLVM and Clang, to
+enable C++ to interact with other languages besides Python.
+
+Example: Template Instantiation
+-------------------------------
+
+The developmental Cppyy version can run basic examples such as the one
+here. Features such as standalone functions and basic classes are also
+supported.
+
+C++ code (Tmpl.h)
+
+::
+
+   template <typename T>
+   struct Tmpl {
+     T m_num;
+     T add (T n) {
+       return m_num + n;
+   }
+   };
+
+Python Interpreter
+
+::
+
+   >>> import cppyy
+   >>> import cppyy.gbl as Cpp
+   >>> cppyy.include("Tmpl.h")
+   >>> tmpl = Tmpl[int]()
+   >>> tmpl.m_num = 4
+   >>> print(tmpl.add(5))
+   9
+   >>> tmpl = Tmpl[float]()
+   >>> tmpl.m_num = 3.0
+   >>> print(tmpl.add(4.0))
+   7.0
+
+Where does the cppyy code reside?
+---------------------------------
+
+Following are the main components where cppyy logic (with Compiler Research
+Organization’s customizations started by `sudo-panda`_) resides:
+
+-  `cppyy <https://github.com/compiler-research/cppyy>`_
+-  `cppyy-backend <https://github.com/compiler-research/cppyy-backend>`_
+-  `CPyCppyy <https://github.com/compiler-research/CPyCppyy>`_
+
+..
+
+   Note: These are forks of the `upstream cppyy`_ repos created by `wlav`_.
+
+CppInterOp is a separate library that helps these packages communicate with C++
+code.
+
+-  `CppInterOp <https://github.com/compiler-research/CppInterOp/tree/main>`_
+
+How cppyy components interact with each other
+---------------------------------------------
 
-**LLVM Integration:** The CppInterOp interfaces will be part of the LLVM 
-toolchain (as part of Clang-REPL).
+cppyy is made up of the following packages: 
 
-**Better C++ Support:** C++ features such as Partial Template Specialisation will
-be available through CppInterOp.
+- A frontend: cppyy, 
 
-**Fewer Lines of Code:** A lot of dependencies and workarounds will be removed,
-reducing the lines of code required to execute CPPYY.
+- A backend: cppyy-backend, and 
 
-**Well tested interoperability Layer:** The CppInterOp interfaces have full
-unit test coverage.
+- An extension: CPyCppyy.
 
+Besides these, the ``CppInterOp`` library serves as an additional layer on top
+of Cling/Clang-REPL that helps these packages in communicating with C++ code.
+
+**1. cppyy-backend**
+
+The `cppyy-backend`_ pacakge forms a layer over ``cppyy``, for example,
+modifying some functionality to provide the functions required for
+``CPyCppyy``. 
+
+  `CPyCppyy`_ is a CPython extension module built on top of the same backend
+  API as PyPy/_cppyy. It thus requires the installation of the cppyy-backend
+  for use, which will pull in Cling. 
+
+``cppyy-backend`` also adds some `utilities`_ to help with repackaging and
+redistribution.
+
+For example, it initializes the interpreter (using the
+``clingwrapper::ApplicationStarter`` function), adds the required
+``include`` paths, and adds the headers required for cppyy to work. It
+also adds some checks and combines two or more functions to help
+CPyCppyy work.
+
+These changes help ensure that any change in ``cppyy`` doesn’t directly
+affect ``CPyCppyy``, and the API for ``CPyCppyy`` remains unchanged.
+
+**2. CPyCppyy**
+
+The ``CPyCppyy`` package uses the functionality provided by ``cppyy-backend``
+and provides Python objects for C++ entities. ``CPyCppyy`` uses separate proxy
+classes for each type of object. It also includes helper classes, for example,
+``Converters.cxx`` helps convert Python type objects to C++ type objects, while
+``Executors.cxx`` is used to execute a function and convert its return value to
+a Python object, so that it can be used inside Python.
+
+**3. cppyy**
+
+The cppyy package provides the front-end for Python. It is `included in code`_
+(using ``import cppyy``) to import cppyy in Python. It initializes things on
+the backend side, provides helper functions (e.g., ``cppdef()``, ``cppexec()``,
+etc.) that the user can utilize, and it calls the relevant backend functions
+required to initialize cppyy.
+
+
+Further Reading
+---------------
+
+-  `High-performance Python-C++ bindings with PyPy and
+   Cling <http://cern.ch/wlav/Cppyy_LavrijsenDutta_PyHPC16.pdf>`_
+
+-  `Efficient and Accurate Automatic Python Bindings with cppyy &
+   Cling <https://arxiv.org/abs/2304.02712>`_
+
+-  cppyy documentation:
+   `cppyy.readthedocs.io <http://cppyy.readthedocs.io/>`_.
+
+-  Notebook-based tutorial: `Cppyy
+   Tutorial <https://github.com/wlav/cppyy/blob/master/doc/tutorial/CppyyTutorial.ipynb>`_.
+
+-  `C++ Language Interoperability
+   Layer <https://compiler-research.org/libinterop/>`_
+
+**Credits:**
+
+-  `Wim Lavrijsen <https://github.com/wlav>`_ (Lawrence Berkeley National Lab.)
+   for his original work in cppyy and mentorship towards student contributors.
+
+-  `Vassil Vasilev <https://github.com/vgvassilev>`_ (Princeton University)
+   for his mentorship towards Compiler Research Org's student contributors.
+
+-  `Baidyanath Kundu <https://github.com/sudo-panda>`_ (Princeton University)
+   for his research work on cppyy and Numba with `Compiler Research Organization`_ 
+   (as discussed in this document).
+   
+- `Aaron Jomy <https://github.com/maximusron>`_ (Princeton University) for
+  continuing this research work with `Compiler Research Organization`_.
 
 In case you haven't already installed CppInterop, please do so before proceeding
 with the Installation And Usage Guide.
-:doc:`Installation and usage <InstallationAndUsage>`
+:doc:`Installation and usage <InstallationAndUsage>`
+
+.. _Compiler Research Organization: https://compiler-research.org/
+
+.. _upstream cppyy: https://github.com/wlav/cppyy
+
+.. _wlav: https://github.com/wlav
+
+.. _utilities: https://cppyy.readthedocs.io/en/latest/utilities.html
+
+.. _included in code: https://cppyy.readthedocs.io/en/latest/starting.html
+
+.. _sudo-panda: https://github.com/sudo-panda
+
+.. _cppyy: https://cppyy.readthedocs.io/en/latest/index.html
+
+.. _CppInterOp: https://github.com/compiler-research/CppInterOp
+
+.. _ROOT meta: https://github.com/root-project/root/tree/master/core/meta
+
+.. _enhancements in cppyy: https://arxiv.org/abs/2304.02712
+
+.. _CPyCppyy: https://github.com/wlav/CPyCppyy
+
+.. _cppyy-backend: https://github.com/wlav/cppyy-backend