docs: Updated README with custom ROBOCORP_HOME + mermaid graph

CHANGELOG.md

@@ -2,6 +2,12 @@
 ## [Unreleased]
 
 
+<a name="v0.2.1"></a>
+## [v0.2.1] - 2024-11-21
+### Docs
+- Updated README with custom ROBOCORP_HOME + mermaid graph
+
+
 <a name="v0.2.0"></a>
 ## [v0.2.0] - 2024-11-20
 ### Fix
@@ -22,5 +28,6 @@
 - self-signed and signed certificates work
 
 
-[Unreleased]: https://github.com/elabit/rccremote-docker/compare/v0.2.0...HEAD
+[Unreleased]: https://github.com/elabit/rccremote-docker/compare/v0.2.1...HEAD
+[v0.2.1]: https://github.com/elabit/rccremote-docker/compare/v0.2.0...v0.2.1
 [v0.2.0]: https://github.com/elabit/rccremote-docker/compare/v0.1.0...v0.2.0

README.md

@@ -4,22 +4,45 @@ Docker-compose setup with SSL for [rccremote](https://sema4.ai/docs/automation/r
 
 ```mermaid
 graph LR
-    rcc -->|TLS| nginx
-    nginx --> rccremote
+    classDef dotted stroke-dasharray: 5 5;
+    rcc -->|HTTPS| nginx
+    nginx -->|HTTP| rccremote
+    ZIP -. import .-> rccremote
+    YAML -. build .-> rccremote
+    rccremote -- fetch --> rccremote2[rccremote]:::dotted
+    rccremote -- fetch --> rccremote-docker:::dotted
+    subgraph import sources
+    ZIP
+    YAML
+    end
+    style ZIP fill:#baa773,stroke-width:0px,color:black
+    style YAML fill:#baa773,stroke-width:0px,color:black
+    style nginx fill:#6589a5,stroke-width:0px,color:black
+    style rccremote2 stroke-line:dotted
 ```
 
 ## Background
 
+### Purpose of this project
+
 In order to built RCC environments with **rcc**, the host must be connected to the Internet in order to download the installation sources for Python/Node.js/etc.  
 However, for security reasons, test clients are often completely isolated from the Internet.
 
-**RCCRemote** solves this problem by serving the blueprints of these environments (aka "_hololib_") for **RCC** clients which can fetch the blueprints from there.  
+**RCCRemote** solves this problem by serving the blueprints of these environments (aka "_Catalogs_") for **RCC** clients which can fetch the blueprints from there.  
 This centralized approach does not only save network traffic and computing resources, but also is a significant performance gain, because when the clients ask for environments, rccremote only relays the missing files, not the whole environment.
 
 By default, **rccremote** operates unencrypted, meaning **rcc** cannot verify the connection, nor is the data transmission encrypted.  
 
 This setup provides a way to run RCCRemote behind a reverse proxy (nginx) which TLS encryption and server authentication. 
 
+### Terminlogy
+
+RCC-internal concepts:
+
+- **Hololib**: A **Collection** of currently available **catalogs**.
+  - **Catalog**: A blueprint of an environment which can be used to create an arbitrary number of instances of this environment.
+- **Holotree**: A **Collection** of currently available **spaces**.
+  - **Space**: An instance of an environment. There is a one-to-many relation between catalog and space.
 
 ## Quick Start 
 
@@ -33,11 +56,31 @@ Copy your certificate files into `/certs`:
 - Root certificate: 
   - `rootCA.pem` (if the server certificate is signed by a CA - recommended) - Note: the server certificate must contain the SAN (X509v3 Subject Alternative Name) attribute.
 
-### Option 2: Self-signed certificate
+### Option 2: Auto-created certificates
+
+There are two way which both are only meant for testing/demonstration purposes: 
+
+- auto-created self-signed
+- auto-created signed (with rootCA)
+
+#### auto-created self-signed
 
 If you want the **nginx** container to create a self-signed certificate, leave the `certs` folder empty.  
 Nginx will then create a self-signed certificate on the very first start. 
 
+#### auto-created signed
+
+If you want to use a signed certificate, you have to execute a generator script: 
+
+```
+cd scripts
+./create-signed-cert.sh
+SERVER_NAME is set to rccremote.local in .env file.
+Openssl configuration created successfully with server name rccremote.local (/tmp/tmp.JIFD8POnxX).
+Ready to generate the certificates now. This will empty the cert dir ../certs.
+=> Are you sure? (y/n)
+```
+
 ### Setting the server name
 
 Set the **server name** used in the certificate to the env variable `SERVER_NAME`: 
@@ -57,22 +100,67 @@ This will spin up 2 containers:
 - **rccremote** 
 - **nginx**
 
+By default, the **rccremote** container does not serve any Catalogs. See next section how to add environments.
+
 ## Usage
 
-### Startup phase of rccremote
+There are basically two modes to add Catalogs by placing files into these folders: 
+
+- A) `./data/robots` - Containing Robot directories with `robot.yaml`/`conda.yaml`: **build and import** on startup
+- B) `./data/hololib_zip` - Containing ZIP files of exported Catalogs: **import** on startup
+- C) using another **rccremote** server (cascaded setup, not yet implemented)
+
+Both modes can be used simultanously. In the following, they are explained. 
 
-While starting, the **rccremote** container locates all `robot.yaml` files in `/robots` and builds the hololibs for each `robot.yaml` which has a `conda.yaml` on its side.  
-All created hololibs in `/opt/robocorp` are saved persistently in a mounted volume.  
-After all hololibs have been built, the **rccremote** process gets started. 
+### Mode A: Add Robot directories
 
-### Adding more environment definitions
+The **rccremote** container first uses **rcc** to build the Catalogs for each directory where it locates `robot.yaml`/`conda.yaml` files.  
 
-`/robots` in the project directory is a host mounted folder in **rccremote** container.  
-Copy the robot folder which contains `robot.yaml` and `conda.yaml` into `/robots` and restart the container:
+As this happens inside of the Docker container, the created Catalogs are **for Linux systems only**.  
 
-`docker compose restart`
+By default, all builds compile for the Holotree path `/opt/robocorp`. If the **rcc** clients operate on systems where this path is useable, this is fine. The environment would be created below of this path then. 
+
+To change the holotree path, the Robot folders can contain a `.env` file: 
+
+```
+ROBOCORP_HOME=/robotmk/rcc_home/current_user
+```
 
-**rccremote** will then build the new hololib (existing hololibs won't be rebuilt).
+(For examples, see `./data/robots-examples/`).
+
+Before each build, **rcc** sources this file so that the Catalog is built against a custom `ROBOCORP_HOME`.
+After each environment creation, the environment gets exported into a ZIP file (volume: hololib_zip_internal).
+
+Finally, all ZIP files are imported into the shared holotree, before the **rccremote** server process gets started. 
+
+### Mode B: Add Hololib ZIP files
+
+This mode is divided into 2 sub steps: 
+
+- **Create the environments** on a build machine (Linux/Windows) which has internet access
+- **Add the ZIP files** into `./data/hololib_zip`
+
+#### Step 1: Create the environments
+
+On the machine where you can build environments with internet access, set `ROBOCORP_HOME` to a path where the Catalogs should be built against. 
+
+**Example**: 
+
+You want to create a Hololib ZIP which can be used on a Windows test client (Windows). The system executing the Robot expects all Catalogs and Spaces for user `alice` in `C:\robotmk\rcc_home\alice`.
+
+```
+set ROBOCORP_HOME=C:\robotmk\rcc_home\alice
+cd myrobot
+# create the catalog (+space), using the custom ROBOCORP_HOME
+rcc holotree vars
+# export the ZIP file
+rcc holotree export -r robot.yaml -z rf_playwright_17_18.zip
+```
+
+#### Step 2: Add the ZIP files
+
+Copy all exported catalog files into the folder `./data/hololib_zip`.  
+When the **rccremote** container gets started, it imports all ZIP files into the shared holotree. 
 
 ## rcc: usage with rccremote
 
@@ -123,17 +211,9 @@ root@0ca74438d77f:/robots/rf7# echo $RCC_REMOTE_ORIGIN
 https://rccremote.local:443
 ```
 
-Execute `rcc holotree vars`. **rcc** should be able to download the hololib from the server: verify that
-
-- **rcc** prints _"Fill hololib from RCC_REMOTE_ORIGIN"_ in step 03
-- the log of **rccremote** (`docker logs rccremote`) shows lines like these: 
+Execute `rcc holotree vars`. **rcc** should be able to download the hololib from the server: 
 
-```
-08.155430.331 [D] Query of catalog "a466d176c7dc6696v12.linux_amd64" took 0.000s
-08.155430.345 [D] query handler: "a466d176c7dc6696v12.linux_amd64" -> true
-08.155430.347 [D] Using existing cache file "/tmp/rccremote/a3d9fb43c5e0589e6b57b1e5608d60e23a3ce2acc256a41d0f2090d62230ae47_parts.zip" [size: 65.3M]
-08.155430.401 [D] Delta of catalog "a466d176c7dc6696v12.linux_amd64" took 0.056s
-```
+![alt text](./img/rcclog.png)
 
 ## Debugging 
 
@@ -152,7 +232,7 @@ Test without the root certificate:
 
 Test with Root certificate: 
 
-    openssl s_client -connect rccremote:443 -CAfile /etc/certs/rootCA.crt
+    openssl s_client -connect rccremote.local:443 -CAfile /etc/certs/rootCA.crt
     Verify return code: 0 (ok)     #  <------------------------------
     Extended master secret: no
     Max Early Data: 0